<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hello,<br>
<br>
revising the patch I noticed I was moving the initialization of the
variable after pushing it to perl environment (from the perl docs,
the variable should have been initialized after initializing the
environment -- what I tried to do with previous patch).<br>
<br>
There is a new smaller patch to be added:<br>
<br>
<a class="moz-txt-link-freetext" href="http://git.sip-router.org/cgi-bin/gitweb.cgi?p=sip-router;a=commitdiff_plain;h=3935fedf23f3bf2b6675182193cef6af3bbd903a">http://git.sip-router.org/cgi-bin/gitweb.cgi?p=sip-router;a=commitdiff_plain;h=3935fedf23f3bf2b6675182193cef6af3bbd903a</a><br>
<br>
Practically, the line
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
XPUSHs(m); has to be moved after the line with
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
SvREADONLY_on(SvRV(m));<br>
<br>
Cheers,<br>
Daniel<br>
<br>
<div class="moz-cite-prefix">On 7/31/13 8:26 AM, David Cunningham
wrote:<br>
</div>
<blockquote
cite="mid:CAHGbP-_o4dL5N4Lg7M_1dx6q=YhOeQoyUGR1GAGv5HfYxFK=Cw@mail.gmail.com"
type="cite">Hi Daniel,<br>
<br>
We tried that patch, but Kamailio logged lots of errors like the
following. The undefined value in question is $m, which should be
the SIP message. Would you have any advice? Thanks.<br>
<br>
Jul 31 02:13:57 hostname /sbin/kamailio[21087]: ERROR: perl
[openserxs.xs:1022]: perl error: Can't call method "pseudoVar" on
an undefined value at Foo.pm line 247.#012<br>
<br>
<br>
<div class="gmail_quote">On 25 July 2013 17:11, David Cunningham <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:dcunningham@voisonics.com" target="_blank">dcunningham@voisonics.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Daniel,<br>
<br>
I'll suggest that to the customer. Thank you!
<div>
<div><br>
<br>
<br>
<div class="gmail_quote">On 25 July 2013 15:45,
Daniel-Constantin Mierla <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> Hello,<br>
<br>
can you try the attached patch? It's the same patch,
just for two versions, one is for 3.3.x and the
other for devel version<br>
<br>
It initializes the SIP message variable that is
passed to perl after creating the temporary
environment, so it is actually destroyed by the perl
embedded interpreter.<br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
<div>On 7/25/13 1:29 AM, David Cunningham wrote:<br>
</div>
<blockquote type="cite">Hi Daniel,<br>
<br>
The system is running Perl 5.8.8 on Red Hat
Enterprise Linux Server release 5.4. If I
remember right programs running under Valgrind
can have issues, so I'm not sure if the
customer will want to do that. Ideally we'd do
it on a test system, but I'm not sure if we
have any RHEL available.<br>
I'll see what we can do. Thanks again.<br>
<br>
<br>
<div class="gmail_quote">On 25 July 2013
04:55, Daniel-Constantin Mierla <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:miconda@gmail.com"
target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
Hello,<br>
<br>
I would say that perl_exec() is the one
with the highest chances to be the
reason for the leak. Next is line would
be db_mysql module, if liked with some
custom mysql client library, although
even in this case will be unlikely.<br>
<br>
Back to perl, the module itself does not
call any malloc, so it might be the
embedding Perl API that is not used
properly in the module.<br>
<br>
Can you use some testbed, set children=1
and run kamailio under valgrind, then do
some calls and see if it detects the
source of the leak?<br>
<br>
I'm not using the perl module, I will
try to check it whenever I get a chance
in the next days. What version of perl
do you have installed?<br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
<div>On 7/24/13 10:31 AM, David
Cunningham wrote:<br>
</div>
<blockquote type="cite">Hello,<br>
<br>
We don't do any kamctl commands at
all. We do have various modules
loaded, as follows. The primary
functions we use Kamailio for are
phone registrations through
usrloc, and routing calls to
Asterisk through logic contained
in Perl via perl_exec().<br>
Thanks for all your advice so far!<br>
<br>
loadmodule "tm.so"<br>
loadmodule "tmx.so"<br>
loadmodule "usrloc.so"<br>
loadmodule "auth.so"<br>
loadmodule "auth_db.so"<br>
loadmodule "ctl.so"<br>
loadmodule "db_mysql.so"<br>
loadmodule "kex.so"<br>
loadmodule "maxfwd.so"<br>
loadmodule "mi_fifo.so"<br>
loadmodule "mi_rpc.so"<br>
loadmodule "nathelper.so"<br>
loadmodule "perl.so"<br>
loadmodule "pv.so"<br>
loadmodule "registrar.so"<br>
loadmodule "rr.so"<br>
loadmodule "sanity.so"<br>
loadmodule "siputils.so"<br>
loadmodule "sl.so"<br>
loadmodule "textops.so"<br>
loadmodule "xlog.so"<br>
<br>
<br>
<div class="gmail_quote">On 24
July 2013 16:33,
Daniel-Constantin Mierla <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:miconda@gmail.com"
target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div text="#000000"
bgcolor="#FFFFFF"> Hello,
<div><br>
<br>
<div>On 7/24/13 4:24 AM,
David Cunningham wrote:<br>
</div>
</div>
<blockquote type="cite">Hello,<br>
<br>
<div> Thank you very much
for the email. In reply:<br>
<br>
1. The system ran out of
memory. Linux's
oom-killer killed
Kamailio.<br>
</div>
</blockquote>
then all the instructions I
gave are useless, they are
for debugging kamailio's
internal memory manager,
which handles pkg and shm
mallocs.<br>
<br>
The chances to be from
kamailio itself are very low
now. Do you do lot of mi
commands (e.g., kamctl ...)?
The mi api uses system
malloc, but the rest of code
should use internal memory
manager which does not go
beyond the limits set with
-m and -M, thus not causing
an OS memory exhaustion.<br>
<br>
Can you list what modules
are you loading? At some
point it was a leak in
libssl, in case you use tls
a lot. But could be another
external library...<br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
<blockquote type="cite"><br>
2. You're right,
DEBUG_MEMORY is a
local configuration
setting. If defined it
sets memdbg to -2, and
memlog to -2. The
debug setting is -1. <br>
<br>
3. We'll try setting
mem_summary=12,
thanks.<br>
<br>
4. We'll try setting
asynchronous syslog,
thanks.<br>
<br>
5. Our configuration
totals 338 lines, or
approx 8.5kb. Is that
a lot? <br>
<br>
6. We'll try setting
mem_join=1, thanks.<br>
<br>
<br>
<br>
<div
class="gmail_quote">On
23 July 2013 16:53,
Daniel-Constantin
Mierla <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0
0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
Hello,<br>
<br>
first, to clarify,
is the system
memory or
kamailio's pkg/shm
memory running
out? If the
operating system
runs out of
memory, then
should be a leak
in a library,
because kamailio
modules uses only
from a
pre-allocated
chunk, not going
over it.
<div> <br>
<br>
On 7/23/13 7:33
AM, David
Cunningham
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
Hello,<br>
<br>
We're running
a Kamailio
3.3.4 system,
and Kamailio
is slowly
using more and
more memory.
Over a couple
of weeks it
will run out
of system
memory.<br>
<br>
We tried to
enable memory
debugging
doing the
following, but
it resulted in
Kamailio not
responding to
any SIP
packets. Would
anyone have
advice please
on how to
debug the
situation?<br>
<br>
1. In
Makefile.defs
set MEMDBG to
1 and
recompile
Kamailio.<br>
2. In
kamailio.cfg
add the line:<br>
#!define
DEBUG_MEMORY 1<br>
</blockquote>
</div>
do you set
something special
in config when
DEBUG_MEMORY is 1?
It is not by
default there, so
I assume you added
some rules based
on this
pre-processor
directive.<br>
<br>
For memory
troubleshooting,
set memlog to a
value lower than
debug parameter in
config file and
try with
mem_summary=12 for
a more compact
output. See more
about these
parameters in the
wiki:<br>
<br>
- <a
moz-do-not-send="true"
href="http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog"
target="_blank">http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog</a><br>
<br>
Run kamailio for a
while in normal
conditions, then
restart it to get
the memory usage
summaries. There
should be
indication if
there is some
leak, by seeing
memory chunks
allocated many
times from a
function used at
runtime. You can
send the memory
summary for a
process here, we
can look at it.
<div> <br>
<br>
<blockquote
class="gmail_quote"
style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<br>
While this was
running and
Kamailio
didn't respond
to packets, it
logged lots of
lines like
this:<br>
</blockquote>
<br>
</div>
Do you have syslog
to be configured
in asynchronous
mode? See the
notes from:<br>
<br>
- <a
moz-do-not-send="true"
href="http://www.kamailio.org/wiki/tutorials/3.2.x/syslog"
target="_blank">http://www.kamailio.org/wiki/tutorials/3.2.x/syslog</a><br>
<br>
The memdbg is less
than debug value,
that means
printing few log
messages for each
memory operation.
You can make
memdbg higher and
rely on memlog for
memory summaries,
otherwise will be
lot of log
messages related
to memory.
<div> <br>
<blockquote
class="gmail_quote"
style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<br>
Jul 22
21:32:22
hostname
kamailio: :
<core>
[mem/q_malloc.c:369]:
qm_malloc(0x4000e008,
128) called
from
<core>:
cfg.lex:
addstr(1438)<br>
Jul 22
21:32:22
hostname
kamailio: :
<core>
[mem/q_malloc.c:413]:
qm_malloc(0x4000e008,
128) returns
address
0x40048918
frag.
0x40048900
(size=128) on
1 -th hit<br>
Jul 22
21:32:22
hostname
kamailio: :
<core>
[mem/q_malloc.c:369]:
qm_malloc(0x4000e008,
128) called
from
<core>:
cfg.lex:
addstr(1438)<br>
Jul 22
21:32:22
hostname
kamailio: :
<core>
[mem/q_malloc.c:413]:
qm_malloc(0x4000e008,
128) returns
address
0x400489c8
frag.
0x400489b0
(size=128) on
1 -th hit<br>
</blockquote>
</div>
addstr() is a
function used only
for parsing
configuration
file, as long as
you can still see
them, the
configuration file
parsing was not
finish. addstr()
is not a source of
leaks because it
is not used at
runtime.<br>
<br>
If you have large
config file, then
you can get close
to the limits of
the private
memory, which is
set to 4MB. You
can increase its
value using -M
parameter (e.g.,
start kamailio
with -M 8 to set
it to use 8MB of
memory).<br>
<br>
Over the time, the
private memory can
get used due to
fragmentation, you
can set the
mem_join parameter
in config file to
avoid it (works
when compiled with
MEMDBG=1).<br>
<br>
To monitor usage
of internal pkg
memory, then you
can use sercmd
with pkg.stats
command:<br>
<br>
<a
moz-do-not-send="true"
href="http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640"
target="_blank">http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640</a><br>
<br>
Shared memory
stats are printed
by 'kamctl fifo
get_statistics
shmem:'<br>
<br>
When you see
significant
increase of the
memory usage, then
you can restart to
get the summaries.<br>
<br>
You should run
these commands
after start, just
to see the initial
usage of memory.<br>
<br>
Cheers,<br>
Daniel<span><font
color="#888888"><br>
<br>
-- <br>
Daniel-Constantin
Mierla - <a
moz-do-not-send="true"
href="http://www.asipto.com" target="_blank">http://www.asipto.com</a><br>
<a
moz-do-not-send="true"
href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a>
- <a
moz-do-not-send="true"
href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a><br>
<br>
<br>
_______________________________________________<br>
SIP Express
Router (SER)
and Kamailio
(OpenSER) -
sr-users
mailing list<br>
<a
moz-do-not-send="true"
href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a><br>
<a
moz-do-not-send="true"
href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users"
target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><br>
</font></span></blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
David Cunningham,
Voisonics<br>
<a
moz-do-not-send="true"
href="http://voisonics.com/" target="_blank">http://voisonics.com/</a><br>
USA: <a
moz-do-not-send="true"
href="tel:%2B1%20213%20221%201092" value="+12132211092" target="_blank">+1
213 221 1092</a><br>
UK: <a
moz-do-not-send="true"
href="tel:%2B44%20%280%29%2020%203298%201642" value="+442032981642"
target="_blank">+44
(0) 20 3298 1642</a><br>
Australia: <a
moz-do-not-send="true"
href="tel:%2B61%20%280%29%202%208063%209019" value="+61280639019"
target="_blank">+61
(0) 2 8063 9019</a><br>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a moz-do-not-send="true" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a moz-do-not-send="true" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
</pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
David Cunningham, Voisonics<br>
<a moz-do-not-send="true"
href="http://voisonics.com/"
target="_blank">http://voisonics.com/</a><br>
USA: <a moz-do-not-send="true"
href="tel:%2B1%20213%20221%201092"
value="+12132211092"
target="_blank">+1 213 221 1092</a><br>
UK: <a moz-do-not-send="true"
href="tel:%2B44%20%280%29%2020%203298%201642"
value="+442032981642"
target="_blank">+44 (0) 20 3298
1642</a><br>
Australia: <a
moz-do-not-send="true"
href="tel:%2B61%20%280%29%202%208063%209019"
value="+61280639019"
target="_blank">+61 (0) 2 8063
9019</a><br>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a moz-do-not-send="true" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a moz-do-not-send="true" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
</pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
David Cunningham, Voisonics<br>
<a moz-do-not-send="true"
href="http://voisonics.com/" target="_blank">http://voisonics.com/</a><br>
USA: <a moz-do-not-send="true"
href="tel:%2B1%20213%20221%201092"
value="+12132211092" target="_blank">+1 213
221 1092</a><br>
UK: <a moz-do-not-send="true"
href="tel:%2B44%20%280%29%2020%203298%201642"
value="+442032981642" target="_blank">+44
(0) 20 3298 1642</a><br>
Australia: <a moz-do-not-send="true"
href="tel:%2B61%20%280%29%202%208063%209019"
value="+61280639019" target="_blank">+61 (0)
2 8063 9019</a><br>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a moz-do-not-send="true" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a moz-do-not-send="true" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
</pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
David Cunningham, Voisonics<br>
<a moz-do-not-send="true" href="http://voisonics.com/"
target="_blank">http://voisonics.com/</a><br>
USA: <a moz-do-not-send="true"
href="tel:%2B1%20213%20221%201092" value="+12132211092"
target="_blank">+1 213 221 1092</a><br>
UK: <a moz-do-not-send="true"
href="tel:%2B44%20%280%29%2020%203298%201642"
value="+442032981642" target="_blank">+44 (0) 20 3298
1642</a><br>
Australia: <a moz-do-not-send="true"
href="tel:%2B61%20%280%29%202%208063%209019"
value="+61280639019" target="_blank">+61 (0) 2 8063 9019</a><br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
David Cunningham, Voisonics<br>
<a moz-do-not-send="true" href="http://voisonics.com/"
target="_blank">http://voisonics.com/</a><br>
USA: <a moz-do-not-send="true" href="tel:%2B1%20213%20221%201092"
value="+12132211092" target="_blank">+1 213 221 1092</a><br>
UK: <a moz-do-not-send="true"
href="tel:%2B44%20%280%29%2020%203298%201642"
value="+442032981642" target="_blank">+44 (0) 20 3298 1642</a><br>
Australia: <a moz-do-not-send="true"
href="tel:%2B61%20%280%29%202%208063%209019"
value="+61280639019" target="_blank">+61 (0) 2 8063 9019</a><br>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla - <a class="moz-txt-link-freetext" href="http://www.asipto.com">http://www.asipto.com</a>
<a class="moz-txt-link-freetext" href="http://twitter.com/#!/miconda">http://twitter.com/#!/miconda</a> - <a class="moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda">http://www.linkedin.com/in/miconda</a>
</pre>
</body>
</html>