<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hello,<br>
<br>
can you try the attached patch? It's the same patch, just for two
versions, one is for 3.3.x and the other for devel version<br>
<br>
It initializes the SIP message variable that is passed to perl after
creating the temporary environment, so it is actually destroyed by
the perl embedded interpreter.<br>
<br>
Cheers,<br>
Daniel<br>
<br>
<div class="moz-cite-prefix">On 7/25/13 1:29 AM, David Cunningham
wrote:<br>
</div>
<blockquote
cite="mid:CAHGbP-_ZjQ4JZdzFQC+WT_RBgF7XLWjeLYVmuqnSmoL=sceCBQ@mail.gmail.com"
type="cite">Hi Daniel,<br>
<br>
The system is running Perl 5.8.8 on Red Hat Enterprise Linux
Server release 5.4. If I remember right programs running under
Valgrind can have issues, so I'm not sure if the customer will
want to do that. Ideally we'd do it on a test system, but I'm not
sure if we have any RHEL available.<br>
I'll see what we can do. Thanks again.<br>
<br>
<br>
<div class="gmail_quote">On 25 July 2013 04:55, Daniel-Constantin
Mierla <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> Hello,<br>
<br>
I would say that perl_exec() is the one with the highest
chances to be the reason for the leak. Next is line would be
db_mysql module, if liked with some custom mysql client
library, although even in this case will be unlikely.<br>
<br>
Back to perl, the module itself does not call any malloc, so
it might be the embedding Perl API that is not used properly
in the module.<br>
<br>
Can you use some testbed, set children=1 and run kamailio
under valgrind, then do some calls and see if it detects the
source of the leak?<br>
<br>
I'm not using the perl module, I will try to check it
whenever I get a chance in the next days. What version of
perl do you have installed?<br>
<br>
Cheers,<br>
Daniel
<div>
<div class="h5"><br>
<br>
<div>On 7/24/13 10:31 AM, David Cunningham wrote:<br>
</div>
<blockquote type="cite">Hello,<br>
<br>
We don't do any kamctl commands at all. We do have
various modules loaded, as follows. The primary
functions we use Kamailio for are phone registrations
through usrloc, and routing calls to Asterisk through
logic contained in Perl via perl_exec().<br>
Thanks for all your advice so far!<br>
<br>
loadmodule "tm.so"<br>
loadmodule "tmx.so"<br>
loadmodule "usrloc.so"<br>
loadmodule "auth.so"<br>
loadmodule "auth_db.so"<br>
loadmodule "ctl.so"<br>
loadmodule "db_mysql.so"<br>
loadmodule "kex.so"<br>
loadmodule "maxfwd.so"<br>
loadmodule "mi_fifo.so"<br>
loadmodule "mi_rpc.so"<br>
loadmodule "nathelper.so"<br>
loadmodule "perl.so"<br>
loadmodule "pv.so"<br>
loadmodule "registrar.so"<br>
loadmodule "rr.so"<br>
loadmodule "sanity.so"<br>
loadmodule "siputils.so"<br>
loadmodule "sl.so"<br>
loadmodule "textops.so"<br>
loadmodule "xlog.so"<br>
<br>
<br>
<div class="gmail_quote">On 24 July 2013 16:33,
Daniel-Constantin Mierla <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> Hello,
<div><br>
<br>
<div>On 7/24/13 4:24 AM, David Cunningham
wrote:<br>
</div>
</div>
<blockquote type="cite">Hello,<br>
<br>
<div> Thank you very much for the email. In
reply:<br>
<br>
1. The system ran out of memory. Linux's
oom-killer killed Kamailio.<br>
</div>
</blockquote>
then all the instructions I gave are useless,
they are for debugging kamailio's internal
memory manager, which handles pkg and shm
mallocs.<br>
<br>
The chances to be from kamailio itself are very
low now. Do you do lot of mi commands (e.g.,
kamctl ...)? The mi api uses system malloc, but
the rest of code should use internal memory
manager which does not go beyond the limits set
with -m and -M, thus not causing an OS memory
exhaustion.<br>
<br>
Can you list what modules are you loading? At
some point it was a leak in libssl, in case you
use tls a lot. But could be another external
library...<br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
<blockquote type="cite"><br>
2. You're right, DEBUG_MEMORY is a local
configuration setting. If defined it sets
memdbg to -2, and memlog to -2. The debug
setting is -1. <br>
<br>
3. We'll try setting mem_summary=12,
thanks.<br>
<br>
4. We'll try setting asynchronous syslog,
thanks.<br>
<br>
5. Our configuration totals 338 lines, or
approx 8.5kb. Is that a lot? <br>
<br>
6. We'll try setting mem_join=1, thanks.<br>
<br>
<br>
<br>
<div class="gmail_quote">On 23 July 2013
16:53, Daniel-Constantin Mierla <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:miconda@gmail.com"
target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex"> Hello,<br>
<br>
first, to clarify, is the system
memory or kamailio's pkg/shm memory
running out? If the operating system
runs out of memory, then should be a
leak in a library, because kamailio
modules uses only from a pre-allocated
chunk, not going over it.
<div> <br>
<br>
On 7/23/13 7:33 AM, David Cunningham
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex"> Hello,<br>
<br>
We're running a Kamailio 3.3.4
system, and Kamailio is slowly
using more and more memory. Over a
couple of weeks it will run out of
system memory.<br>
<br>
We tried to enable memory
debugging doing the following, but
it resulted in Kamailio not
responding to any SIP packets.
Would anyone have advice please on
how to debug the situation?<br>
<br>
1. In Makefile.defs set MEMDBG to
1 and recompile Kamailio.<br>
2. In kamailio.cfg add the line:<br>
#!define DEBUG_MEMORY 1<br>
</blockquote>
</div>
do you set something special in config
when DEBUG_MEMORY is 1? It is not by
default there, so I assume you added
some rules based on this pre-processor
directive.<br>
<br>
For memory troubleshooting, set memlog
to a value lower than debug parameter
in config file and try with
mem_summary=12 for a more compact
output. See more about these
parameters in the wiki:<br>
<br>
- <a moz-do-not-send="true"
href="http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog"
target="_blank">http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog</a><br>
<br>
Run kamailio for a while in normal
conditions, then restart it to get the
memory usage summaries. There should
be indication if there is some leak,
by seeing memory chunks allocated many
times from a function used at runtime.
You can send the memory summary for a
process here, we can look at it.
<div> <br>
<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex"> <br>
While this was running and
Kamailio didn't respond to
packets, it logged lots of lines
like this:<br>
</blockquote>
<br>
</div>
Do you have syslog to be configured in
asynchronous mode? See the notes from:<br>
<br>
- <a moz-do-not-send="true"
href="http://www.kamailio.org/wiki/tutorials/3.2.x/syslog"
target="_blank">http://www.kamailio.org/wiki/tutorials/3.2.x/syslog</a><br>
<br>
The memdbg is less than debug value,
that means printing few log messages
for each memory operation. You can
make memdbg higher and rely on memlog
for memory summaries, otherwise will
be lot of log messages related to
memory.
<div> <br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex"> <br>
Jul 22 21:32:22 hostname kamailio:
: <core>
[mem/q_malloc.c:369]:
qm_malloc(0x4000e008, 128) called
from <core>: cfg.lex:
addstr(1438)<br>
Jul 22 21:32:22 hostname kamailio:
: <core>
[mem/q_malloc.c:413]:
qm_malloc(0x4000e008, 128) returns
address 0x40048918 frag.
0x40048900 (size=128) on 1 -th hit<br>
Jul 22 21:32:22 hostname kamailio:
: <core>
[mem/q_malloc.c:369]:
qm_malloc(0x4000e008, 128) called
from <core>: cfg.lex:
addstr(1438)<br>
Jul 22 21:32:22 hostname kamailio:
: <core>
[mem/q_malloc.c:413]:
qm_malloc(0x4000e008, 128) returns
address 0x400489c8 frag.
0x400489b0 (size=128) on 1 -th hit<br>
</blockquote>
</div>
addstr() is a function used only for
parsing configuration file, as long as
you can still see them, the
configuration file parsing was not
finish. addstr() is not a source of
leaks because it is not used at
runtime.<br>
<br>
If you have large config file, then
you can get close to the limits of the
private memory, which is set to 4MB.
You can increase its value using -M
parameter (e.g., start kamailio with
-M 8 to set it to use 8MB of memory).<br>
<br>
Over the time, the private memory can
get used due to fragmentation, you can
set the mem_join parameter in config
file to avoid it (works when compiled
with MEMDBG=1).<br>
<br>
To monitor usage of internal pkg
memory, then you can use sercmd with
pkg.stats command:<br>
<br>
<a moz-do-not-send="true"
href="http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640"
target="_blank">http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640</a><br>
<br>
Shared memory stats are printed by
'kamctl fifo get_statistics shmem:'<br>
<br>
When you see significant increase of
the memory usage, then you can restart
to get the summaries.<br>
<br>
You should run these commands after
start, just to see the initial usage
of memory.<br>
<br>
Cheers,<br>
Daniel<span><font color="#888888"><br>
<br>
-- <br>
Daniel-Constantin Mierla - <a
moz-do-not-send="true"
href="http://www.asipto.com"
target="_blank">http://www.asipto.com</a><br>
<a moz-do-not-send="true"
href="http://twitter.com/#%21/miconda"
target="_blank">http://twitter.com/#!/miconda</a>
- <a moz-do-not-send="true"
href="http://www.linkedin.com/in/miconda"
target="_blank">http://www.linkedin.com/in/miconda</a><br>
<br>
<br>
_______________________________________________<br>
SIP Express Router (SER) and
Kamailio (OpenSER) - sr-users
mailing list<br>
<a moz-do-not-send="true"
href="mailto:sr-users@lists.sip-router.org"
target="_blank">sr-users@lists.sip-router.org</a><br>
<a moz-do-not-send="true"
href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users"
target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><br>
</font></span></blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
David Cunningham, Voisonics<br>
<a moz-do-not-send="true"
href="http://voisonics.com/"
target="_blank">http://voisonics.com/</a><br>
USA: <a moz-do-not-send="true"
href="tel:%2B1%20213%20221%201092"
value="+12132211092" target="_blank">+1
213 221 1092</a><br>
UK: <a moz-do-not-send="true"
href="tel:%2B44%20%280%29%2020%203298%201642"
value="+442032981642" target="_blank">+44
(0) 20 3298 1642</a><br>
Australia: <a moz-do-not-send="true"
href="tel:%2B61%20%280%29%202%208063%209019"
value="+61280639019" target="_blank">+61
(0) 2 8063 9019</a><br>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a moz-do-not-send="true" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a moz-do-not-send="true" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
</pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
David Cunningham, Voisonics<br>
<a moz-do-not-send="true" href="http://voisonics.com/"
target="_blank">http://voisonics.com/</a><br>
USA: <a moz-do-not-send="true"
href="tel:%2B1%20213%20221%201092"
value="+12132211092" target="_blank">+1 213 221 1092</a><br>
UK: <a moz-do-not-send="true"
href="tel:%2B44%20%280%29%2020%203298%201642"
value="+442032981642" target="_blank">+44 (0) 20
3298 1642</a><br>
Australia: <a moz-do-not-send="true"
href="tel:%2B61%20%280%29%202%208063%209019"
value="+61280639019" target="_blank">+61 (0) 2 8063
9019</a><br>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a moz-do-not-send="true" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a moz-do-not-send="true" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
</pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
David Cunningham, Voisonics<br>
<a moz-do-not-send="true" href="http://voisonics.com/"
target="_blank">http://voisonics.com/</a><br>
USA: +1 213 221 1092<br>
UK: +44 (0) 20 3298 1642<br>
Australia: +61 (0) 2 8063 9019<br>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla - <a class="moz-txt-link-freetext" href="http://www.asipto.com">http://www.asipto.com</a>
<a class="moz-txt-link-freetext" href="http://twitter.com/#!/miconda">http://twitter.com/#!/miconda</a> - <a class="moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda">http://www.linkedin.com/in/miconda</a>
</pre>
</body>
</html>