<div dir="ltr">Hi Daniel,<br><br>Configuration sent to you privately. Thanks.<br><br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On 16 October 2013 20:28, Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Hello,<br>
<br>
indeed, it looks like a system memory leak, for what so ever reason
the discussion was misled to pkg.<br>
<br>
Can you send me the list of modules you load (loadmodule lines) as
well as the functions from your perl script that are related to
kamailio internal functions or variables? You can send directly to
me, without mailing list if there is something you want to protect
from public eyes.<br>
<br>
Cheers,<br>
Daniel<div><div class="h5"><br>
<br>
<div>On 10/15/13 5:12 AM, David Cunningham
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>Hi Daniel,<br>
<br>
</div>
Thanks for the reply again. Looking at the email history,
I'm not sure how we decided it was definitely a pkg memory
problem. What we see is the output of "ps aux" as follows:<br>
<br>
root@sip0-test:~# ps aux | egrep -i "kamailio|mem"<br>
USER PID %CPU %MEM VSZ RSS TTY STAT START
TIME COMMAND<br>
root 6794 0.0 0.0 22480 1868 ? Ss Oct02
0:12 /opt/testuser/current/sbin/testuser_safe_kamailio<br>
testuser 6822 0.0 0.2 556528 4580 ? S Oct02
0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid<br>
testuser 6824 0.3 8.7 825552 180244 ? S Oct02
56:40 /sbin/kamailio -m 512 -P
/var/run/testuser/kamailio.pid<br>
testuser 6825 0.3 8.7 825536 180776 ? S Oct02
56:20 /sbin/kamailio -m 512 -P
/var/run/testuser/kamailio.pid<br>
testuser 6826 0.3 8.7 825912 180296 ? S Oct02
55:54 /sbin/kamailio -m 512 -P
/var/run/testuser/kamailio.pid<br>
testuser 6827 0.3 8.7 825744 180580 ? S Oct02
56:19 /sbin/kamailio -m 512 -P
/var/run/testuser/kamailio.pid<br>
testuser 6828 0.3 8.7 825536 180092 ? S Oct02
56:25 /sbin/kamailio -m 512 -P
/var/run/testuser/kamailio.pid<br>
testuser 6829 0.3 8.7 825536 180632 ? S Oct02
56:21 /sbin/kamailio -m 512 -P
/var/run/testuser/kamailio.pid<br>
testuser 6830 0.3 8.7 825472 180968 ? S Oct02
56:37 /sbin/kamailio -m 512 -P
/var/run/testuser/kamailio.pid<br>
testuser 6831 0.3 8.7 825276 180272 ? S Oct02
56:41 /sbin/kamailio -m 512 -P
/var/run/testuser/kamailio.pid<br>
testuser 6832 0.0 0.0 556528 1324 ? S Oct02
0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid<br>
testuser 6833 0.0 0.0 556528 1324 ? S Oct02
0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid<br>
testuser 6834 0.0 0.0 556528 1324 ? S Oct02
0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid<br>
testuser 6835 0.0 0.0 556528 1324 ? S Oct02
0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid<br>
testuser 6836 0.0 0.0 556528 1324 ? S Oct02
0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid<br>
testuser 6837 0.0 0.0 556528 1324 ? S Oct02
0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid<br>
testuser 6838 0.0 0.0 556528 1324 ? S Oct02
0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid<br>
testuser 6839 0.0 0.0 556528 1324 ? S Oct02
0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid<br>
testuser 6840 0.0 0.0 556528 1776 ? S Oct02
0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid<br>
testuser 6841 0.0 0.0 556528 1780 ? S Oct02
0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid<br>
testuser 6842 0.0 0.0 556528 1780 ? S Oct02
0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid<br>
testuser 6843 0.0 0.0 556528 1328 ? S Oct02
0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid<br>
testuser 6844 0.0 0.0 556528 1780 ? S Oct02
0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid<br>
testuser 6845 0.0 0.0 556528 1328 ? S Oct02
0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid<br>
testuser 6846 0.0 0.0 556528 1328 ? S Oct02
0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid<br>
testuser 6847 0.0 0.0 556528 1328 ? S Oct02
0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid<br>
testuser 6848 0.0 0.0 556528 1676 ? S Oct02
0:02 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid<br>
testuser 6849 0.0 0.1 556528 3568 ? S Oct02
5:28 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid<br>
testuser 6850 0.0 0.0 556612 1600 ? S Oct02
0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid<br>
testuser 6851 0.0 0.0 556532 1188 ? S Oct02
0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid<br>
testuser 6852 0.0 0.0 556528 1360 ? S Oct02
0:02 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid<br>
<br>
</div>
You'll see for example that process 6824 is using 8.7% of
memory, which is much more than it was using 5 days ago. Yet
if I run the same sercmd again I get (exactly!) the same
numbers:<br>
<br>
root@sip0-test:~# sercmd pkg.stats pid 6824 <br>
{<br>
entry: 1<br>
pid: 6824<br>
rank: 1<br>
used: 209836<br>
free: 3704080<br>
real_used: 490224<br>
}<br>
<br>
</div>
Any ideas?<br>
<br>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On 12 October 2013 00:23,
Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi David,
<div><br>
<br>
On 10/10/13 11:36 PM, David Cunningham wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Daniel,<br>
<br>
<div>
Thanks for the reply. Perhaps what we're seeing is
normal, and the memory use is meant to increase as time
progresses. Would you expect to see an ongoing memory
use increase, or when should it stop increasing?<br>
<br>
<br>
</div>
</blockquote>
private memory (pkg) should stay rather constant. It
increases when there is a sip message processed, but once is
sent out, it should come back around the average.<br>
<br>
There are couple of functions that can fill the private
memory and keep it up, such as doing an sql_query() that
returns a big data and the result is not freed
(sql_result_free()). It is not actually a leak as the next
sql_query() will free previous result, but in case you have
such query for some corner case that is not executed
frequently, then the memory can stay filled in. Another
example will be storing very large value in a $var(...)
(e.g., $var(x) ).<br>
<br>
This is private memory, per process, which is meant for
temporary operations. Shared memory (shm) can increase over
the time, being the place where the dynamic data required at
runtime is stored (e.g., location records, hash tables,
transactions) - so as you get more traffic or more phones
using kamailio, more shm is used. But your problem was
reported for pkg.<br>
<br>
Anyhow, keep an eye on the pkg.stats and if you see constant
increase which is substantial, then get a mem log dump.<br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
-- <br>
Daniel-Constantin Mierla - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a><br>
<a href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a>
- <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a><br>
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami,
Nov 18-20, 2013<br>
- more details about Kamailio trainings at <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a> -<br>
<br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
David Cunningham, Voisonics<br>
<a href="http://voisonics.com/" target="_blank">http://voisonics.com/</a><br>
USA: <a href="tel:%2B1%20213%20221%201092" value="+12132211092" target="_blank">+1 213 221 1092</a><br>
UK: <a href="tel:%2B44%20%280%29%2020%203298%201642" value="+442032981642" target="_blank">+44 (0) 20 3298 1642</a><br>
Australia: <a href="tel:%2B61%20%280%29%202%208063%209019" value="+61280639019" target="_blank">+61 (0) 2 8063 9019</a><br>
</div>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a href="http://twitter.com/#!/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a> -
</pre>
</div></div></div>
</blockquote></div><br><br clear="all"><br>-- <br>David Cunningham, Voisonics<br><a href="http://voisonics.com/" target="_blank">http://voisonics.com/</a><br>USA: +1 213 221 1092<br>UK: +44 (0) 20 3298 1642<br>Australia: +61 (0) 2 8063 9019<br>
</div>