<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hello,<br>
<br>
I would say that perl_exec() is the one with the highest chances to
be the reason for the leak. Next is line would be db_mysql module,
if liked with some custom mysql client library, although even in
this case will be unlikely.<br>
<br>
Back to perl, the module itself does not call any malloc, so it
might be the embedding Perl API that is not used properly in the
module.<br>
<br>
Can you use some testbed, set children=1 and run kamailio under
valgrind, then do some calls and see if it detects the source of the
leak?<br>
<br>
I'm not using the perl module, I will try to check it whenever I get
a chance in the next days. What version of perl do you have
installed?<br>
<br>
Cheers,<br>
Daniel<br>
<br>
<div class="moz-cite-prefix">On 7/24/13 10:31 AM, David Cunningham
wrote:<br>
</div>
<blockquote
cite="mid:CAHGbP--dttJQszX7Ut7Q60T4in8F1hdAN4f-NqgGBDRKJiPqMA@mail.gmail.com"
type="cite">Hello,<br>
<br>
We don't do any kamctl commands at all. We do have various modules
loaded, as follows. The primary functions we use Kamailio for are
phone registrations through usrloc, and routing calls to Asterisk
through logic contained in Perl via perl_exec().<br>
Thanks for all your advice so far!<br>
<br>
loadmodule "tm.so"<br>
loadmodule "tmx.so"<br>
loadmodule "usrloc.so"<br>
loadmodule "auth.so"<br>
loadmodule "auth_db.so"<br>
loadmodule "ctl.so"<br>
loadmodule "db_mysql.so"<br>
loadmodule "kex.so"<br>
loadmodule "maxfwd.so"<br>
loadmodule "mi_fifo.so"<br>
loadmodule "mi_rpc.so"<br>
loadmodule "nathelper.so"<br>
loadmodule "perl.so"<br>
loadmodule "pv.so"<br>
loadmodule "registrar.so"<br>
loadmodule "rr.so"<br>
loadmodule "sanity.so"<br>
loadmodule "siputils.so"<br>
loadmodule "sl.so"<br>
loadmodule "textops.so"<br>
loadmodule "xlog.so"<br>
<br>
<br>
<div class="gmail_quote">On 24 July 2013 16:33, Daniel-Constantin
Mierla <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> Hello,
<div class="im"><br>
<br>
<div>On 7/24/13 4:24 AM, David Cunningham wrote:<br>
</div>
</div>
<blockquote type="cite">Hello,<br>
<br>
<div class="im"> Thank you very much for the email. In
reply:<br>
<br>
1. The system ran out of memory. Linux's oom-killer
killed Kamailio.<br>
</div>
</blockquote>
then all the instructions I gave are useless, they are for
debugging kamailio's internal memory manager, which handles
pkg and shm mallocs.<br>
<br>
The chances to be from kamailio itself are very low now. Do
you do lot of mi commands (e.g., kamctl ...)? The mi api
uses system malloc, but the rest of code should use internal
memory manager which does not go beyond the limits set with
-m and -M, thus not causing an OS memory exhaustion.<br>
<br>
Can you list what modules are you loading? At some point it
was a leak in libssl, in case you use tls a lot. But could
be another external library...<br>
<br>
Cheers,<br>
Daniel
<div>
<div class="h5"><br>
<br>
<blockquote type="cite"><br>
2. You're right, DEBUG_MEMORY is a local configuration
setting. If defined it sets memdbg to -2, and memlog
to -2. The debug setting is -1. <br>
<br>
3. We'll try setting mem_summary=12, thanks.<br>
<br>
4. We'll try setting asynchronous syslog, thanks.<br>
<br>
5. Our configuration totals 338 lines, or approx
8.5kb. Is that a lot? <br>
<br>
6. We'll try setting mem_join=1, thanks.<br>
<br>
<br>
<br>
<div class="gmail_quote">On 23 July 2013 16:53,
Daniel-Constantin Mierla <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello,<br>
<br>
first, to clarify, is the system memory or
kamailio's pkg/shm memory running out? If the
operating system runs out of memory, then should
be a leak in a library, because kamailio modules
uses only from a pre-allocated chunk, not going
over it.
<div> <br>
<br>
On 7/23/13 7:33 AM, David Cunningham wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex"> Hello,<br>
<br>
We're running a Kamailio 3.3.4 system, and
Kamailio is slowly using more and more memory.
Over a couple of weeks it will run out of
system memory.<br>
<br>
We tried to enable memory debugging doing the
following, but it resulted in Kamailio not
responding to any SIP packets. Would anyone
have advice please on how to debug the
situation?<br>
<br>
1. In Makefile.defs set MEMDBG to 1 and
recompile Kamailio.<br>
2. In kamailio.cfg add the line:<br>
#!define DEBUG_MEMORY 1<br>
</blockquote>
</div>
do you set something special in config when
DEBUG_MEMORY is 1? It is not by default there, so
I assume you added some rules based on this
pre-processor directive.<br>
<br>
For memory troubleshooting, set memlog to a value
lower than debug parameter in config file and try
with mem_summary=12 for a more compact output. See
more about these parameters in the wiki:<br>
<br>
- <a moz-do-not-send="true"
href="http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog"
target="_blank">http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog</a><br>
<br>
Run kamailio for a while in normal conditions,
then restart it to get the memory usage summaries.
There should be indication if there is some leak,
by seeing memory chunks allocated many times from
a function used at runtime. You can send the
memory summary for a process here, we can look at
it.
<div> <br>
<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex"> <br>
While this was running and Kamailio didn't
respond to packets, it logged lots of lines
like this:<br>
</blockquote>
<br>
</div>
Do you have syslog to be configured in
asynchronous mode? See the notes from:<br>
<br>
- <a moz-do-not-send="true"
href="http://www.kamailio.org/wiki/tutorials/3.2.x/syslog"
target="_blank">http://www.kamailio.org/wiki/tutorials/3.2.x/syslog</a><br>
<br>
The memdbg is less than debug value, that means
printing few log messages for each memory
operation. You can make memdbg higher and rely on
memlog for memory summaries, otherwise will be lot
of log messages related to memory.
<div> <br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex"> <br>
Jul 22 21:32:22 hostname kamailio: :
<core> [mem/q_malloc.c:369]:
qm_malloc(0x4000e008, 128) called from
<core>: cfg.lex: addstr(1438)<br>
Jul 22 21:32:22 hostname kamailio: :
<core> [mem/q_malloc.c:413]:
qm_malloc(0x4000e008, 128) returns address
0x40048918 frag. 0x40048900 (size=128) on 1
-th hit<br>
Jul 22 21:32:22 hostname kamailio: :
<core> [mem/q_malloc.c:369]:
qm_malloc(0x4000e008, 128) called from
<core>: cfg.lex: addstr(1438)<br>
Jul 22 21:32:22 hostname kamailio: :
<core> [mem/q_malloc.c:413]:
qm_malloc(0x4000e008, 128) returns address
0x400489c8 frag. 0x400489b0 (size=128) on 1
-th hit<br>
</blockquote>
</div>
addstr() is a function used only for parsing
configuration file, as long as you can still see
them, the configuration file parsing was not
finish. addstr() is not a source of leaks because
it is not used at runtime.<br>
<br>
If you have large config file, then you can get
close to the limits of the private memory, which
is set to 4MB. You can increase its value using -M
parameter (e.g., start kamailio with -M 8 to set
it to use 8MB of memory).<br>
<br>
Over the time, the private memory can get used due
to fragmentation, you can set the mem_join
parameter in config file to avoid it (works when
compiled with MEMDBG=1).<br>
<br>
To monitor usage of internal pkg memory, then you
can use sercmd with pkg.stats command:<br>
<br>
<a moz-do-not-send="true"
href="http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640"
target="_blank">http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640</a><br>
<br>
Shared memory stats are printed by 'kamctl fifo
get_statistics shmem:'<br>
<br>
When you see significant increase of the memory
usage, then you can restart to get the summaries.<br>
<br>
You should run these commands after start, just to
see the initial usage of memory.<br>
<br>
Cheers,<br>
Daniel<span><font color="#888888"><br>
<br>
-- <br>
Daniel-Constantin Mierla - <a
moz-do-not-send="true"
href="http://www.asipto.com" target="_blank">http://www.asipto.com</a><br>
<a moz-do-not-send="true"
href="http://twitter.com/#%21/miconda"
target="_blank">http://twitter.com/#!/miconda</a>
- <a moz-do-not-send="true"
href="http://www.linkedin.com/in/miconda"
target="_blank">http://www.linkedin.com/in/miconda</a><br>
<br>
<br>
_______________________________________________<br>
SIP Express Router (SER) and Kamailio
(OpenSER) - sr-users mailing list<br>
<a moz-do-not-send="true"
href="mailto:sr-users@lists.sip-router.org"
target="_blank">sr-users@lists.sip-router.org</a><br>
<a moz-do-not-send="true"
href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users"
target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><br>
</font></span></blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
David Cunningham, Voisonics<br>
<a moz-do-not-send="true" href="http://voisonics.com/"
target="_blank">http://voisonics.com/</a><br>
USA: <a moz-do-not-send="true"
href="tel:%2B1%20213%20221%201092"
value="+12132211092" target="_blank">+1 213 221 1092</a><br>
UK: <a moz-do-not-send="true"
href="tel:%2B44%20%280%29%2020%203298%201642"
value="+442032981642" target="_blank">+44 (0) 20
3298 1642</a><br>
Australia: <a moz-do-not-send="true"
href="tel:%2B61%20%280%29%202%208063%209019"
value="+61280639019" target="_blank">+61 (0) 2 8063
9019</a><br>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a moz-do-not-send="true" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a moz-do-not-send="true" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
</pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
David Cunningham, Voisonics<br>
<a moz-do-not-send="true" href="http://voisonics.com/"
target="_blank">http://voisonics.com/</a><br>
USA: +1 213 221 1092<br>
UK: +44 (0) 20 3298 1642<br>
Australia: +61 (0) 2 8063 9019<br>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla - <a class="moz-txt-link-freetext" href="http://www.asipto.com">http://www.asipto.com</a>
<a class="moz-txt-link-freetext" href="http://twitter.com/#!/miconda">http://twitter.com/#!/miconda</a> - <a class="moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda">http://www.linkedin.com/in/miconda</a>
</pre>
</body>
</html>