<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hello,<br>
<br>
<div class="moz-cite-prefix">On 7/24/13 4:24 AM, David Cunningham
wrote:<br>
</div>
<blockquote
cite="mid:CAHGbP-9wioJuQ7YM1fJFXfTXPT4_zJsA+jTT=5aJYVzWO=BO7g@mail.gmail.com"
type="cite">Hello,<br>
<br>
Thank you very much for the email. In reply:<br>
<br>
1. The system ran out of memory. Linux's oom-killer killed
Kamailio.<br>
</blockquote>
then all the instructions I gave are useless, they are for debugging
kamailio's internal memory manager, which handles pkg and shm
mallocs.<br>
<br>
The chances to be from kamailio itself are very low now. Do you do
lot of mi commands (e.g., kamctl ...)? The mi api uses system
malloc, but the rest of code should use internal memory manager
which does not go beyond the limits set with -m and -M, thus not
causing an OS memory exhaustion.<br>
<br>
Can you list what modules are you loading? At some point it was a
leak in libssl, in case you use tls a lot. But could be another
external library...<br>
<br>
Cheers,<br>
Daniel<br>
<br>
<blockquote
cite="mid:CAHGbP-9wioJuQ7YM1fJFXfTXPT4_zJsA+jTT=5aJYVzWO=BO7g@mail.gmail.com"
type="cite"><br>
2. You're right, DEBUG_MEMORY is a local configuration setting. If
defined it sets memdbg to -2, and memlog to -2. The debug setting
is -1. <br>
<br>
3. We'll try setting mem_summary=12, thanks.<br>
<br>
4. We'll try setting asynchronous syslog, thanks.<br>
<br>
5. Our configuration totals 338 lines, or approx 8.5kb. Is that a
lot? <br>
<br>
6. We'll try setting mem_join=1, thanks.<br>
<br>
<br>
<br>
<div class="gmail_quote">On 23 July 2013 16:53, Daniel-Constantin
Mierla <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello,<br>
<br>
first, to clarify, is the system memory or kamailio's pkg/shm
memory running out? If the operating system runs out of
memory, then should be a leak in a library, because kamailio
modules uses only from a pre-allocated chunk, not going over
it.
<div class="im">
<br>
<br>
On 7/23/13 7:33 AM, David Cunningham wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello,<br>
<br>
We're running a Kamailio 3.3.4 system, and Kamailio is
slowly using more and more memory. Over a couple of weeks
it will run out of system memory.<br>
<br>
We tried to enable memory debugging doing the following,
but it resulted in Kamailio not responding to any SIP
packets. Would anyone have advice please on how to debug
the situation?<br>
<br>
1. In Makefile.defs set MEMDBG to 1 and recompile
Kamailio.<br>
2. In kamailio.cfg add the line:<br>
#!define DEBUG_MEMORY 1<br>
</blockquote>
</div>
do you set something special in config when DEBUG_MEMORY is 1?
It is not by default there, so I assume you added some rules
based on this pre-processor directive.<br>
<br>
For memory troubleshooting, set memlog to a value lower than
debug parameter in config file and try with mem_summary=12 for
a more compact output. See more about these parameters in the
wiki:<br>
<br>
- <a moz-do-not-send="true"
href="http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog"
target="_blank">http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog</a><br>
<br>
Run kamailio for a while in normal conditions, then restart it
to get the memory usage summaries. There should be indication
if there is some leak, by seeing memory chunks allocated many
times from a function used at runtime. You can send the memory
summary for a process here, we can look at it.
<div class="im">
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
While this was running and Kamailio didn't respond to
packets, it logged lots of lines like this:<br>
</blockquote>
<br>
</div>
Do you have syslog to be configured in asynchronous mode? See
the notes from:<br>
<br>
- <a moz-do-not-send="true"
href="http://www.kamailio.org/wiki/tutorials/3.2.x/syslog"
target="_blank">http://www.kamailio.org/wiki/tutorials/3.2.x/syslog</a><br>
<br>
The memdbg is less than debug value, that means printing few
log messages for each memory operation. You can make memdbg
higher and rely on memlog for memory summaries, otherwise will
be lot of log messages related to memory.
<div class="im">
<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:369]: qm_malloc(0x4000e008, 128) called
from <core>: cfg.lex: addstr(1438)<br>
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:413]: qm_malloc(0x4000e008, 128) returns
address 0x40048918 frag. 0x40048900 (size=128) on 1 -th
hit<br>
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:369]: qm_malloc(0x4000e008, 128) called
from <core>: cfg.lex: addstr(1438)<br>
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:413]: qm_malloc(0x4000e008, 128) returns
address 0x400489c8 frag. 0x400489b0 (size=128) on 1 -th
hit<br>
</blockquote>
</div>
addstr() is a function used only for parsing configuration
file, as long as you can still see them, the configuration
file parsing was not finish. addstr() is not a source of leaks
because it is not used at runtime.<br>
<br>
If you have large config file, then you can get close to the
limits of the private memory, which is set to 4MB. You can
increase its value using -M parameter (e.g., start kamailio
with -M 8 to set it to use 8MB of memory).<br>
<br>
Over the time, the private memory can get used due to
fragmentation, you can set the mem_join parameter in config
file to avoid it (works when compiled with MEMDBG=1).<br>
<br>
To monitor usage of internal pkg memory, then you can use
sercmd with pkg.stats command:<br>
<br>
<a moz-do-not-send="true"
href="http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640"
target="_blank">http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640</a><br>
<br>
Shared memory stats are printed by 'kamctl fifo get_statistics
shmem:'<br>
<br>
When you see significant increase of the memory usage, then
you can restart to get the summaries.<br>
<br>
You should run these commands after start, just to see the
initial usage of memory.<br>
<br>
Cheers,<br>
Daniel<span class="HOEnZb"><font color="#888888"><br>
<br>
-- <br>
Daniel-Constantin Mierla - <a moz-do-not-send="true"
href="http://www.asipto.com" target="_blank">http://www.asipto.com</a><br>
<a moz-do-not-send="true"
href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a>
- <a moz-do-not-send="true"
href="http://www.linkedin.com/in/miconda"
target="_blank">http://www.linkedin.com/in/miconda</a><br>
<br>
<br>
_______________________________________________<br>
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
mailing list<br>
<a moz-do-not-send="true"
href="mailto:sr-users@lists.sip-router.org"
target="_blank">sr-users@lists.sip-router.org</a><br>
<a moz-do-not-send="true"
href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users"
target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><br>
</font></span></blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
David Cunningham, Voisonics<br>
<a moz-do-not-send="true" href="http://voisonics.com/"
target="_blank">http://voisonics.com/</a><br>
USA: +1 213 221 1092<br>
UK: +44 (0) 20 3298 1642<br>
Australia: +61 (0) 2 8063 9019<br>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla - <a class="moz-txt-link-freetext" href="http://www.asipto.com">http://www.asipto.com</a>
<a class="moz-txt-link-freetext" href="http://twitter.com/#!/miconda">http://twitter.com/#!/miconda</a> - <a class="moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda">http://www.linkedin.com/in/miconda</a>
</pre>
</body>
</html>