<div dir="ltr"><div><div>Hi Daniel,<br><br></div>Kamailio had only been running for about 24 hours with very little traffic on that server. Should we wait for longer and throw some calls at it?<br><br></div>Thanks again.<br>
<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On 2 October 2013 18:39, Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Hello,<br>
<br>
on a second check, those counter chunks for counters/statistics
seems to be from startup, thus not part of a leak.<br>
<br>
How long have you been running kamailio? Looking at the pkg stats
from previous email, the free part is still a lot, with only about
200k used. It could mean the leak was not making any effect yet.<br>
<br>
Cheers,<br>
Daniel<div><div class="h5"><br>
<br>
<div>On 10/2/13 10:19 AM, Daniel-Constantin
Mierla wrote:<br>
</div>
<blockquote type="cite">
Hello,<br>
<br>
the leak is due to keep defining counters or statistics. That
should happen only at startup, but if you execute from perl script
some function from a kamailio module, it may trigger this issue.
Can you share what functions from kamailio are you execute inside
the perl script?<br>
<br>
Cheers,<br>
Daniel<br>
<br>
<div>On 10/2/13 9:26 AM, David Cunningham
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>Hi Daniel,<br>
<br>
</div>
Thank you - I used the restart method and got a whole load
of memory information. 10 megabytes of it! I've compressed
that and attached it to this email.<br>
<br>
</div>
Thanks for your help.<br>
<br>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On 2 October 2013 17:17,
Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Hello,<br>
<br>
ahh, right, forgot that it was disabled because it was
not safe to use at runtime for worker processes. You
have to use:<br>
<br>
kamcmd cfg.set_now_int core mem_dump_pkg <pid><br>
<br>
then wait for a bit so that process is receiving a SIP
message.<br>
<br>
Or, if you restart, the stats are printed in syslog for
all processes.<br>
<br>
Another alternative is to call pkg_status() or
pkg_summary() from cfgutils module in config file, like:<br>
<br>
if(src_ip==127.0.0.1 && is_method("OPTIONS")
&& $rU=="pkgstats") {<br>
send_reply("200", "OK");<br>
pkg_summary();<br>
exit;<br>
}<br>
<br>
You can use sipsak to send an options request to <a>"sip:pkgstats@..."</a><br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
<div>On 10/2/13 2:30 AM, David Cunningham wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Daniel,<br>
<br>
When I send SIGUSR1 to a worker process nothing
is logged. Any ideas there? The following is the
output of "sercmd pkg.stats".<br>
<br>
{<br>
entry: 0<br>
pid: 14570<br>
rank: 0<br>
used: 191100<br>
free: 3731504<br>
real_used: 462800<br>
}<br>
{<br>
entry: 1<br>
pid: 14571<br>
rank: 1<br>
used: 202656<br>
free: 3714236<br>
real_used: 480068<br>
}<br>
{<br>
entry: 2<br>
pid: 14572<br>
rank: 2<br>
used: 202720<br>
free: 3714220<br>
real_used: 480084<br>
}<br>
{<br>
entry: 3<br>
pid: 14573<br>
rank: 3<br>
used: 202664<br>
free: 3714324<br>
real_used: 479980<br>
}<br>
{<br>
entry: 4<br>
pid: 14574<br>
rank: 4<br>
used: 202744<br>
free: 3714100<br>
real_used: 480204<br>
}<br>
{<br>
entry: 5<br>
pid: 14575<br>
rank: 5<br>
used: 201908<br>
free: 3715176<br>
real_used: 479128<br>
}<br>
{<br>
entry: 6<br>
pid: 14576<br>
rank: 6<br>
used: 202572<br>
free: 3714368<br>
real_used: 479936<br>
}<br>
{<br>
entry: 7<br>
pid: 14577<br>
rank: 7<br>
used: 202740<br>
free: 3714488<br>
real_used: 479816<br>
}<br>
{<br>
entry: 8<br>
pid: 14578<br>
rank: 8<br>
used: 199760<br>
free: 3717084<br>
real_used: 477220<br>
}<br>
{<br>
entry: 9<br>
pid: 14579<br>
rank: 9<br>
used: 199440<br>
free: 3722828<br>
real_used: 471476<br>
}<br>
{<br>
entry: 10<br>
pid: 14580<br>
rank: 10<br>
used: 199440<br>
free: 3722828<br>
real_used: 471476<br>
}<br>
{<br>
entry: 11<br>
pid: 14581<br>
rank: 11<br>
used: 199440<br>
free: 3722828<br>
real_used: 471476<br>
}<br>
{<br>
entry: 12<br>
pid: 14582<br>
rank: 12<br>
used: 199440<br>
free: 3722828<br>
real_used: 471476<br>
}<br>
{<br>
entry: 13<br>
pid: 14583<br>
rank: 13<br>
used: 199440<br>
free: 3722828<br>
real_used: 471476<br>
}<br>
{<br>
entry: 14<br>
pid: 14584<br>
rank: 14<br>
used: 199440<br>
free: 3722828<br>
real_used: 471476<br>
}<br>
{<br>
entry: 15<br>
pid: 14585<br>
rank: 15<br>
used: 199440<br>
free: 3722828<br>
real_used: 471476<br>
}<br>
{<br>
entry: 16<br>
pid: 14586<br>
rank: 16<br>
used: 199440<br>
free: 3722828<br>
real_used: 471476<br>
}<br>
{<br>
entry: 17<br>
pid: 14587<br>
rank: -1<br>
used: 199400<br>
free: 3722868<br>
real_used: 471436<br>
}<br>
{<br>
entry: 18<br>
pid: 14588<br>
rank: -1<br>
used: 2105844<br>
free: 3722868<br>
real_used: 2337944<br>
}<br>
{<br>
entry: 19<br>
pid: 14589<br>
rank: -2<br>
used: 198552<br>
free: 3723236<br>
real_used: 471068<br>
}<br>
{<br>
entry: 20<br>
pid: 0<br>
rank: 0<br>
used: 209756<br>
free: 3712368<br>
real_used: 481936<br>
}<br>
{<br>
entry: 21<br>
pid: 14591<br>
rank: -4<br>
used: 253660<br>
free: 3668704<br>
real_used: 525600<br>
}<br>
<br>
<br>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On 1 October 2013
22:45, Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Hello,<br>
<br>
send SIGUSR1 to one of the worker
processes (e.g., UDP listener). The main
process of kamailio doesn't process SIP
messages, so it is not exposed to leaks.<br>
<br>
You can used 'kamcmd pkg.stats" to see
statistics about usage of pkg per each
process (older versions have sercmd). <br>
<br>
'kamctl ps' gives the list of processes.<br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
<div>On 10/1/13 2:35 PM, David
Cunningham wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>Hi Daniel,<br>
<br>
</div>
What I copied and pasted is all
that's listed when I do a kill
-SIGUSR1 on the Kamailio master
process. How do I get fuller
logs?<br>
<br>
</div>
Thanks for your help.<br>
<br>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On 1
October 2013 21:57,
Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br>
<br>
you gave only the first part
of memory log output. The
chunks listed there are from
startup, thus not part of a
leak, because they are not
allocated again at runtime.<br>
<br>
Can you give the full logs
with messages from qm_status?
The leaks are typically listed
towards the end of those
messages.<br>
<br>
Yesterday someone else
reported a potential leak when
using perl module, so it might
be something else there. I
found one, which is related to
an error case, but perhaps you
don't get that, otherwise you
will notice the error message.<br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
On 10/1/13 12:03 PM, David
Cunningham wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello,<br>
<br>
Following up on this
after a long break. It
seems we are still
having a memory leak.
We've added the memory
debugging options as per
the documentation with
the following output.<br>
I'm not sure how to read
it though - can anyone
help? Thank you.<br>
<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: <core>
[main.c:754]: Memory
status (pkg):<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
(0x4019b008):<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status: heap
size= 4194304<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status: used=
191100,
used+overhead=462800,
free=3731504<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status: max
used (+overhead)= 480224<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
dumping all alloc'ed.
fragments:<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
0. N
address=0x401cf414
frag=0x401cf3fc size=512
used=1<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
alloc'd from
<core>:
str_hash.h:
str_hash_alloc(69)<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
start check=f0f0f0f0,
end check= c0c0c0c0,
abcdefed<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
1. N
address=0x401cf644
frag=0x401cf62c size=128
used=1<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
alloc'd from
<core>:
str_hash.h:
str_hash_alloc(69)<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
start check=f0f0f0f0,
end check= c0c0c0c0,
abcdefed<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
2. N
address=0x401cf6f4
frag=0x401cf6dc size=256
used=1<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
alloc'd from
<core>:
counters.c:
cnt_hash_add(331)<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
start check=f0f0f0f0,
end check= c0c0c0c0,
abcdefed<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
3. N
address=0x401cf824
frag=0x401cf80c size=64
used=1<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
alloc'd from
<core>:
counters.c:
init_counters(121)<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
start check=f0f0f0f0,
end check= c0c0c0c0,
abcdefed<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
4. N
address=0x401cf894
frag=0x401cf87c size=8
used=1<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
alloc'd from
<core>: route.c:
init_rlist(180)<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
start check=f0f0f0f0,
end check= c0c0c0c0,
abcdefed<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
5. N
address=0x401cf8cc
frag=0x401cf8b4 size=64
used=1<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
alloc'd from
<core>:
str_hash.h:
str_hash_alloc(69)<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
start check=f0f0f0f0,
end check= c0c0c0c0,
abcdefed<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
6. N
address=0x401cf93c
frag=0x401cf924 size=24
used=1<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
alloc'd from
<core>: route.c:
route_add(159)<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
start check=f0f0f0f0,
end check= c0c0c0c0,
abcdefed<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
7. N
address=0x401cf984
frag=0x401cf96c size=8
used=1<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
alloc'd from
<core>: route.c:
init_rlist(180)<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
start check=f0f0f0f0,
end check= c0c0c0c0,
abcdefed<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
8. N
address=0x401cf9bc
frag=0x401cf9a4 size=64
used=1<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
alloc'd from
<core>:
str_hash.h:
str_hash_alloc(69)<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
start check=f0f0f0f0,
end check= c0c0c0c0,
abcdefed<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
9. N
address=0x401cfa2c
frag=0x401cfa14 size=24
used=1<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
alloc'd from
<core>: route.c:
route_add(159)<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
start check=f0f0f0f0,
end check= c0c0c0c0,
abcdefed<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
10. N
address=0x401cfa74
frag=0x401cfa5c size=8
used=1<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
alloc'd from
<core>: db_id.c:
dupl_string(50)<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
start check=f0f0f0f0,
end check= c0c0c0c0,
abcdefed<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
11. N
address=0x401cfaac
frag=0x401cfa94 size=64
used=1<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
alloc'd from
<core>:
str_hash.h:
str_hash_alloc(69)<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
start check=f0f0f0f0,
end check= c0c0c0c0,
abcdefed<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
12. N
address=0x401cfb1c
frag=0x401cfb04 size=24
used=1<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
alloc'd from
<core>: route.c:
route_add(159)<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
start check=f0f0f0f0,
end check= c0c0c0c0,
abcdefed<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
13. N
address=0x401cfb64
frag=0x401cfb4c size=8
used=1<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
alloc'd from
<core>: route.c:
init_rlist(180)<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
start check=f0f0f0f0,
end check= c0c0c0c0,
abcdefed<br>
Oct 1 05:58:48
sip0-test
/sbin/kamailio[14570]:
NOTICE: qm_status:
14. N
address=0x401cfb9c
frag=0x401cfb84 size=64
used=1<br>
<br>
<br>
</blockquote>
<br>
</div>
</div>
<div> -- <br>
Daniel-Constantin Mierla - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a><br>
<a href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a>
- <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a><br>
</div>
Kamailio Advanced Trainings -
Berlin, Nov 25-28; Miami, Nov
18-20, 2013<br>
- more details about
Kamailio trainings at <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
-<br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
David Cunningham, Voisonics<br>
<a href="http://voisonics.com/" target="_blank">http://voisonics.com/</a><br>
USA: <a href="tel:%2B1%20213%20221%201092" value="+12132211092" target="_blank">+1 213 221 1092</a><br>
UK: <a href="tel:%2B44%20%280%29%2020%203298%201642" value="+442032981642" target="_blank">+44 (0) 20 3298
1642</a><br>
Australia: <a href="tel:%2B61%20%280%29%202%208063%209019" value="+61280639019" target="_blank">+61 (0) 2 8063
9019</a><br>
</div>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a> -
</pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
David Cunningham, Voisonics<br>
<a href="http://voisonics.com/" target="_blank">http://voisonics.com/</a><br>
USA: <a href="tel:%2B1%20213%20221%201092" value="+12132211092" target="_blank">+1 213
221 1092</a><br>
UK: <a href="tel:%2B44%20%280%29%2020%203298%201642" value="+442032981642" target="_blank">+44 (0)
20 3298 1642</a><br>
Australia: <a href="tel:%2B61%20%280%29%202%208063%209019" value="+61280639019" target="_blank">+61 (0) 2
8063 9019</a><br>
</div>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a> -
</pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
David Cunningham, Voisonics<br>
<a href="http://voisonics.com/" target="_blank">http://voisonics.com/</a><br>
USA: <a href="tel:%2B1%20213%20221%201092" value="+12132211092" target="_blank">+1 213 221 1092</a><br>
UK: <a href="tel:%2B44%20%280%29%2020%203298%201642" value="+442032981642" target="_blank">+44 (0) 20 3298 1642</a><br>
Australia: <a href="tel:%2B61%20%280%29%202%208063%209019" value="+61280639019" target="_blank">+61 (0) 2 8063 9019</a><br>
</div>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a> -
</pre>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a href="http://twitter.com/#!/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a> -
</pre>
</div></div></div>
</blockquote></div><br><br clear="all"><br>-- <br>David Cunningham, Voisonics<br><a href="http://voisonics.com/" target="_blank">http://voisonics.com/</a><br>USA: +1 213 221 1092<br>UK: +44 (0) 20 3298 1642<br>Australia: +61 (0) 2 8063 9019<br>
</div>