<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Hello,</p>
<p>this doesn't sound like a leak, but more like the side effect of
defining a lot of variables, which use pkg memory.</p>
<p>In order to speed up (but also avoid some situations the could
end up in leaks when parsing the pv spec allocates memory),
kamailio is caching the definitions of the variables used in the
scripts (kamailio.cfg, but also the scripts for embedded
interpreters).</p>
<p>So, if you use:</p>
<p>$var(x) = "abc";</p>
<p>$sht(t=>$var(x)) = 1;</p>
<p>$var(x) = "xyz";</p>
<p>$sht(t=>$var(x)) = 1;</p>
<p>there are two definitions, respectively for: $var(x) and
$sht(t=>$var(x)).</p>
<p>On the other hand, if you use:</p>
<p>$var(x) = "abc";</p>
<p>$sht(t=>$var(x)) = 1;</p>
<p>$sht(t=>xyz) = 1;</p>
<p>there are three definitions, respectively for: $var(x),
$sht(t=>$var(x)) and $sht(t=>xyz). As you use different
values after "t=>...", there will be different definitions.<br>
</p>
<p>To avoid defining a lot of variable specs, you can try to do:</p>
<p> local bbbb = string.match(variable_a, "^%+?.-([%d|#]+)$") or
"anonymous"<br>
sr.pv.sets("$var(vname)", bbbb)<br>
<b>local aaaa = sr.pv.get("$sht(table=>$var(vname))")</b><br>
if not aaaa then<br>
return<br>
end<br>
sr.pv.sets("$avp(s:cccc)", aaaa)</p>
<div class="moz-cite-prefix">I will try to think of a solution to
avoid defining unlimited pv specs, but for the moment this should
be a solution.<br>
<br>
Cheers,<br>
Daniel<br>
<br>
On 20/02/2017 18:42, david wrote:<br>
</div>
<blockquote cite="mid:1487612532.2999.53.camel@bts.io" type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="GENERATOR" content="GtkHTML/4.8.4">
hello Daniel<br>
<br>
We have been trying to isolate the issue as much as possible.<br>
At first we though that since the issue was affecting only some
UDP receivers, it could be related with some customers traffic
profile, but since the decreased seemed abnormally "constant", and
wondering it might be related to a recent change we did on the
configuration, i think we finally found something that could be
the cause (¿?)<br>
<br>
when we query a hash table from a lua file, seems somehow private
memory is involved and not properly released. <br>
We just loaded a kamailio config file doing only this thing on
request route<br>
request_route {<br>
<br>
xlog("L_INFO","Call-ID $ci: doing lua\n");<br>
lua_run("table","$fU");<br>
xlog("L_INFO","Call-ID $ci: done lua\n");<br>
drop();<br>
}<br>
<br>
the remarked lua command:<br>
function table(variable_a)<br>
local bbbb = string.match(variable_a, "^%+?.-([%d|#]+)$") or
"anonymous"<br>
<b>local aaaa = sr.pv.get("$sht(table=>" .. bbbb .. ")")</b><br>
if not aaaa then<br>
return<br>
end<br>
sr.pv.sets("$avp(s:cccc)", aaaa)<br>
end<br>
<br>
<br>
and launched some sip request to exceute the function, and this
seems to be causing the issue and doing the free pkg mem to
decrease. see file attached<br>
(variable_a is passed to lua function as parameter from an $avp
kam variable too in some other tests and the result is also the
same).<br>
<br>
this is the log in debug mode when we make only a query on the
hash to assign it to a lua variable<br>
<br>
1(12776) INFO: Test Call-ID <a moz-do-not-send="true"
href="mailto:1-15610@79.170.68.171">1-15610@1.1.1.1</a>: doing
lua<br>
1(12776) WARNING: <core> [mem/f_malloc.c:439]: fm_malloc():
fm_malloc(0x7f9b2be0a010, 472) called from core:
parser/parse_from.c: parse_from_header(63)<br>
1(12776) WARNING: <core> [mem/f_malloc.c:514]: fm_malloc():
fm_malloc(0x7f9b2be0a010, 472) returns address 0x7f9b2be61990<br>
1(12776) WARNING: <core> [mem/f_malloc.c:439]: fm_malloc():
fm_malloc(0x7f9b2be0a010, 48) called from core:
parser/parse_addr_spec.c: parse_to_param(281)<br>
1(12776) WARNING: <core> [mem/f_malloc.c:514]: fm_malloc():
fm_malloc(0x7f9b2be0a010, 48) returns address 0x7f9b2be5f0c8<br>
1(12776) DEBUG: <core> [parser/parse_addr_spec.c:172]:
parse_to_param(): DEBUG: add_param: tag=15610SIPpTag001<br>
1(12776) DEBUG: <core> [parser/parse_addr_spec.c:894]:
parse_addr_spec(): end of header reached, state=29<br>
1(12776) DEBUG: app_lua [app_lua_api.c:643]: app_lua_run(): reload
deactivated<br>
1(12776) DEBUG: app_lua [app_lua_api.c:644]: app_lua_run():
executing Lua function: [[table]]<br>
1(12776) DEBUG: app_lua [app_lua_api.c:645]: app_lua_run(): lua
top index is: 4<br>
1(12776) DEBUG: app_lua [app_lua_sr.c:892]: lua_sr_pv_get(): pv
get: $sht(table=>2424627806)<br>
1(12776) WARNING: <core> [mem/f_malloc.c:439]: fm_malloc():
fm_malloc(0x7f9b2be0a010, 166) called from core: pvapi.c:
pv_cache_add(272)<br>
1(12776) WARNING: <core> [mem/f_malloc.c:514]: fm_malloc():
fm_malloc(0x7f9b2be0a010, 168) returns address 0x7f9b2be4a878<br>
1(12776) WARNING: <core> [mem/f_malloc.c:439]: fm_malloc():
fm_malloc(0x7f9b2be0a010, 32) called from htable: ht_var.c:
pv_parse_ht_name(135)<br>
1(12776) WARNING: <core> [mem/f_malloc.c:514]: fm_malloc():
fm_malloc(0x7f9b2be0a010, 32) returns address 0x7f9b2be4a800<br>
1(12776) DEBUG: htable [ht_var.c:172]: pv_parse_ht_name(): htable
[table] - key [2424627806]<br>
1(12776) WARNING: <core> [mem/f_malloc.c:439]: fm_malloc():
fm_malloc(0x7f9b2be0a010, 32) called from core: pvapi.c:
pv_parse_format(1075)<br>
1(12776) WARNING: <core> [mem/f_malloc.c:514]: fm_malloc():
fm_malloc(0x7f9b2be0a010, 32) returns address 0x7f9b2be5efb8<br>
1(12776) DEBUG: htable [ht_api.c:246]: ht_get_table(): htable
found [table]<br>
1(12776) DEBUG: <core> [pvapi.c:293]: pv_cache_add(): pvar
[$sht(table=>2424627806)] added in cache<br>
1(12776) WARNING: <core> [mem/f_malloc.c:439]: fm_malloc():
fm_malloc(0x7f9b2be0a010, 95) called from htable: ht_api.c:
ht_cell_pkg_copy(825)<br>
1(12776) WARNING: <core> [mem/f_malloc.c:514]: fm_malloc():
fm_malloc(0x7f9b2be0a010, 96) returns address 0x7f9b2be5ef10<br>
1(12776) WARNING: <core> [mem/f_malloc.c:575]: fm_free():
fm_free(0x7f9b2be0a010, 0x7f9b2be4acf8), called from htable:
ht_api.c: ht_cell_pkg_free(222)<br>
1(12776) WARNING: <core> [mem/f_malloc.c:594]: fm_free():
fm_free: freeing block alloc'ed from htable: ht_api.c:
ht_cell_pkg_copy(825)<br>
1(12776) DEBUG: app_lua [app_lua_sr.c:1014]: lua_sr_pv_sets(): pv
set: $avp(s:cccc)<br>
1(12776) DEBUG: <core> [pvapi.c:321]: pv_cache_lookup():
pvar [$avp(s:cccc)] found in cache<br>
1(12776) WARNING: <core> [mem/f_malloc.c:439]: fm_malloc():
fm_malloc(0x7f9b222a1000, 75) called from core: usr_avp.c:
create_avp(175)<br>
1(12776) WARNING: <core> [mem/f_malloc.c:514]: fm_malloc():
fm_malloc(0x7f9b222a1000, 80) returns address 0x7f9b22ae7f88<br>
1(12776) INFO: Test Call-ID <a moz-do-not-send="true"
href="mailto:1-15610@79.170.68.171">1-15610@1.1.1.1</a>: done
lua<br>
<br>
In this case the query found an entry on the hash table, but we
could duplicate that the free pkg memory is used and not properly
freeded when the match is not done in the hash table key either.<br>
we don't understand well why this happens, since seems we are
using shared memory variables all the time (except the internal
lua variables).<br>
Besides, we do usually do many sr.pv.sets and sr.pv.get functions
to several $avp (shared) and $var (private), but we never found
this issue or at least was not realized. The only thing i think we
didnt do so much is to read hash tables from lua like in this
case.<br>
<br>
By doing the query on the kamailio conf script file<br>
<b>$avp(s:cccc) = $sht(table=>$variable);</b><br>
seems we are not suffering this leak, or at least is much much
more slow<br>
<br>
<br>
we have tested this with kamailio 4.4.1, 4.4.5 and 5.0.0-rc1
versions, and seems the same.<br>
<br>
do you think this has any sense or could be explained?<br>
<br>
<br>
thanks a lot and regards<br>
david<br>
<br>
<br>
<br>
El vie, 17-02-2017 a las 15:49 +0100, david escribió:<br>
<blockquote type="CITE"> hello Daniel<br>
<br>
i forgot to specify that in the dump attached, the first lines
are for the process UDP receiver carrying the traffic on an
affected IP, yesterday during the test we started the kamailio
only with 1 children per IP, so all the messages would have been
processed by that process<br>
<br>
i see that all the information of allocated mem for the root
process is the same in both moments after and before the traffic
load, but in the PID affected we see more hashes used after the
load, i dont know how to deal with that information, anything
you need from us please let us know and we will pleased to
provide if possible<br>
<br>
best regards<br>
david<br>
<br>
<br>
<br>
El vie, 17-02-2017 a las 14:17 +0100, david escribió:<br>
<blockquote type="CITE"> hello DAniel<br>
<br>
we have 4.4.1 sorry<br>
we are only handling SIP calls without REGISTER messages<br>
<br>
best regards<br>
david<br>
<br>
El vie, 17-02-2017 a las 13:53 +0100, Daniel-Constantin Mierla
escribió:<br>
<blockquote type="CITE"> Hello,<br>
<br>
what kamailio version?<br>
<br>
Are registrations handled by kamailio, or is just used for
calls?<br>
<br>
Cheers,<br>
Daniel<br>
<br>
<br>
<br>
On 17/02/2017 13:31, david wrote:<br>
<br>
<blockquote type="CITE"> Hello all<br>
<br>
We are observing an issue in one kamailio proxy we use.<br>
The consecuence of an unknow cause is that the private
memory of some UDP receiver processes in where we listen
from a particular IP, seems to be draining along the time
while processing traffic. Quite "slowly" but maybe aroung
200K in 3-4hours, depending on the moment of the day.<br>
<br>
This is quite weird for us, because it seemed like it was
caused by some particular traffic profile, since we have
several IP listening (8 children each) and we only see the
issue in the UDP receiver processes of some of those IPs.
Besides we have moved the kamailio service listening on
the same IP to several different physical servers, and the
behaviour is the same.<br>
We look at it ussing pkg.stats kamcmd command and for
instance if we track one of the process affected<br>
pid: 23784<br>
rank: 53<br>
used: 2838616<br>
<b>free: 13005408</b><br>
real_used: 3771808<br>
<br>
pid: 23784<br>
rank: 53<br>
used: 2842584<br>
<b>free: 12999280</b><br>
real_used: 3777936<br>
<br>
This 2 outputs were taken with 10min diff between each
other.<br>
<br>
But we dont know where or how to look at it.<br>
We have been these last days trying to search for patterns
to allocate some particular traffic source causing
something weird, but no luck.<br>
We have checked all the SIP INVITE messages looking for
anything weird at any field but no luck, besides we
sometimes see some bad headers with blank spaces in the
uri or thing like that but those cases we have an error
and we dont think it can cause a memory leak.<br>
Shared memory seems to be stable.<br>
<br>
We also have checked the config we are using and we have
been discarding possible issues since we dont think the
$var variables we are using could cause that issue (but
you never know....)<br>
<br>
We tried to debug the kamailio on one affected IP
compiling it with DBG_SYS_MALLOC , and doing the dump of
the pkg mem status when the kamailio was restarted and no
traffic was processed, and also when it had traffic
processed. in this last case, we removed traffic, waited
for 20min and made the dump<br>
<br>
i attach you both files, we have the debug logs, but they
are very big to be passed here, and we dont know what to
look for :)<br>
<br>
could you please give us some guidance or clue to at least
start to searching for something that can cause this
issue?<br>
<br>
thanks alot and regards<br>
david <br>
<br>
<pre>_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
<a moz-do-not-send="true" href="mailto:sr-users@lists.sip-router.org">sr-users@lists.sip-router.org</a>
<a moz-do-not-send="true" href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<br>
<pre>_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
<a moz-do-not-send="true" href="mailto:sr-users@lists.sip-router.org">sr-users@lists.sip-router.org</a>
<a moz-do-not-send="true" href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<br>
</blockquote>
</blockquote>
<br>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda">www.linkedin.com/in/miconda</a>
Kamailio Advanced Training - Mar 6-8 (Europe) and Mar 20-22 (USA) - <a class="moz-txt-link-abbreviated" href="http://www.asipto.com">www.asipto.com</a>
Kamailio World Conference - May 8-10, 2017 - <a class="moz-txt-link-abbreviated" href="http://www.kamailioworld.com">www.kamailioworld.com</a></pre>
</body>
</html>