<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Hello,</p>
<p>that's the way it was done for older versions of kamailio.</p>
<p>In master and 4.4 the memory debugging is turned on and it is
reflected by the presence of DBG_SR_MEMORY in the output of
'kamailio -v'.</p>
<p>Anyhow, what you reported is not a leak inside kamailio memory
manager, but a leak of using system memory, so it is not affected
by DBG_SR_MEMORY and cannot be troubleshooted using the mechanisms
for pkg and shm managers.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<br>
<div class="moz-cite-prefix">On 07/10/16 15:45, Jurijs Ivolga wrote:<br>
</div>
<blockquote
cite="mid:CAOrA2Uas+BiRH4ha=RK75Yqi4NhkOzA+iPx7a=e=_oOkEOtV3A@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>Hi Daniel,<br>
<br>
</div>
I tried to compile with memory manager in debugging mode like
this:<br>
<br>
MEMDBG=1 make cfg 'include_modules= textopsx db_mysql jansson
json snmpstats tls utils uuid'<br>
make all<br>
/etc/init.d/kamailioa stop<br>
make install<br>
/etc/init.d/kamailioa start<br>
<br>
</div>
But still I'm getting nothing on:<br>
<br>
kamailio -I | grep DBG_QM_MALLOC<br>
<br>
<div>Any hints?<br>
</div>
<div><br>
</div>
<div>With kind regards,<br>
</div>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div class="gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">Jurijs<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On Fri, Oct 7, 2016 at 10:39 AM,
Daniel-Constantin Mierla <span dir="ltr"><<a
moz-do-not-send="true" href="mailto:miconda@gmail.com"
target="_blank">miconda@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Hello,</p>
<p>There are some results when searching for memory leak
on openssl 1.0.1, but might not be valid for this case.</p>
<p>Would you be able to run kamailio through valgrind? It
may slow down a bit the processing, but may be the
fastest way to catch the system memory leak. Maybe you
have an instance will less traffic and you can run
through valgrind for a while, you don't have to wait
until you get the OOM.<br>
</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>
<div class="h5"> <br>
<div class="m_-3322893149780162383moz-cite-prefix">On
07/10/16 09:27, Jurijs Ivolga wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hi Daniel,<br>
<br>
openssl.x86_64 <wbr> <wbr>
1.0.1e-48.el6_8.1<br>
CentOS release 6.8<br>
<br>
</div>
With kind regards,<br>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div
class="m_-3322893149780162383gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">Jurijs<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On Fri, Oct 7, 2016 at
10:20 AM, Daniel-Constantin Mierla <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:miconda@gmail.com"
target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Hello,</p>
<p>the tls module in kamailio is using shm
memory, but can be something internal for
libssl. What operating system do you use
and what is the libssl version?</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>
<div class="m_-3322893149780162383h5"> <br>
<div
class="m_-3322893149780162383m_668784092729894905moz-cite-prefix">On
06/10/16 17:43, Jurijs Ivolga wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>Hi Daniel,<br>
<br>
</div>
I do not use puke.top rpc command.
Maybe this issue related to TLS?
Servers what has this problem are
utilizing TLS heavily, servers
which do not has this problem use
UDP.<br>
<br>
</div>
With kind regards,<br>
</div>
<div class="gmail_extra"><br
clear="all">
<div>
<div
class="m_-3322893149780162383m_668784092729894905gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">Jurijs<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On Thu, Oct
6, 2016 at 6:05 PM,
Daniel-Constantin Mierla <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:miconda@gmail.com"
target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF"
text="#000000">
<p>Hello,</p>
<p>are you using pike.top rpc
command? I noticed in the
code that it uses system
malloc, but I haven't
investigated further yet,
first to see if this would
be a possibility ...</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>
<div
class="m_-3322893149780162383m_668784092729894905h5">
<br>
<div
class="m_-3322893149780162383m_668784092729894905m_1561746082170517228moz-cite-prefix">On
06/10/16 16:33, Jurijs
Ivolga wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hi Daniel,<br>
<br>
</div>
<div>We do not do any
external operations.<br>
<br>
</div>
<div>We are using
janson 2.7
everywhere. I will
try to update to
latest janson
version tomorrow.<br>
</div>
<div>All json
operation is pretty
much same, we are
using only
jansson_get.<br>
<br>
</div>
<div>In attachment you
can see memory
consumption. On the
right 2 servers
which are faced
internet on the left
which don't face
internet. As you can
see memory
consumption is
pretty dramatic.<br>
</div>
<div><br>
</div>
<div>Thank you for
your help!<br>
</div>
<div><br>
</div>
<div>With kind
regards,<br>
</div>
</div>
<div class="gmail_extra"><br
clear="all">
<div>
<div
class="m_-3322893149780162383m_668784092729894905m_1561746082170517228gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">Jurijs<br>
</div>
</div>
</div>
<br>
<div
class="gmail_quote">On
Thu, Oct 6, 2016 at
5:17 PM,
Daniel-Constantin
Mierla <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0
0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
<p>Hello,</p>
<p>are you doing
different
external
operations
than on the
other
instances,
like mi/rpc
commands.</p>
<p>From the list
of the modules
you exposed, I
think jansson
has the higher
probability to
work with
system memory.
Are you doing
different json
operations in
config that in
the other
instances of
kamailio? Are
you using same
version of
libjansson
everywhere?</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>
<div
class="m_-3322893149780162383m_668784092729894905m_1561746082170517228h5">
<div
class="m_-3322893149780162383m_668784092729894905m_1561746082170517228m_-4545043238606932376moz-cite-prefix">On
06/10/16
13:46, Jurijs
Ivolga wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div>
<div>
<div>Hi
Daniel,<br>
<br>
</div>
This modules
what we are
using:<br>
<br>
loadmodule
"mi_fifo.so"<br>
loadmodule
"kex.so"<br>
loadmodule
"corex.so"<br>
loadmodule
"tm.so"<br>
loadmodule
"tmx.so"<br>
loadmodule
"sl.so"<br>
loadmodule
"rr.so"<br>
loadmodule
"pv.so"<br>
loadmodule
"maxfwd.so"<br>
loadmodule
"textops.so"<br>
loadmodule
"siputils.so"<br>
loadmodule
"xlog.so"<br>
loadmodule
"sanity.so"<br>
loadmodule
"ctl.so"<br>
loadmodule
"cfg_rpc.so"<br>
loadmodule
"mi_rpc.so"<br>
loadmodule
"dispatcher.so"<br>
loadmodule
"utils.so"<br>
loadmodule
"path.so"<br>
loadmodule
"ipops.so"<br>
loadmodule
"jansson.so"<br>
loadmodule
"auth.so"<br>
loadmodule
"nathelper.so"<br>
loadmodule
"tls.so"<br>
loadmodule
"htable.so"<br>
loadmodule
"pike.so"<br>
<br>
</div>
We have
several other
Kamailio
instances but
they are not
faced to
internet and
they do not
have such
memory issue.
That other
Kamailio
instances have
same modules,
except modules
listed below.
So if you
think that
issue is
inside
external
library,
probably we
need to check
first modules
from list
below.<br>
<br>
<span
class="m_-3322893149780162383m_668784092729894905m_1561746082170517228m_-4545043238606932376gmail-diff-chunkm_1561746082170517228m_-4545043238606932376gmail-diff-chunk-equal">loadmodule
"ipops.so"<br>
loadmodule
"auth.so"<br>
loadmodule
"nathelper.so"<br>
loadmodule
"pike.so"</span><span
class="m_-3322893149780162383m_668784092729894905m_1561746082170517228m_-4545043238606932376gmail-diff-chunk
m_-4545043238606932376gmail-diff-chunk-removed"></span><br>
<br>
</div>
<div>But maybe
this other
Kamailio
instances do
not have this
memory issue,
just because
they did not
face to
internet and
did not have
same load as
instances with
memory issue.<br>
<br>
kamailio -v<br>
version:
kamailio 4.4.3
(x86_64/linux)
5a2195<br>
flags: STATS:
Off, USE_TCP,
USE_TLS,
USE_SCTP,
TLS_HOOKS,
USE_RAW_SOCKS,
DISABLE_NAGLE,
USE_MCAST,
DNS_IP_HACK,
SHM_MEM,
SHM_MMAP,
PKG_MALLOC,
Q_MALLOC,
F_MALLOC,
TLSF_MALLOC,
DBG_SR_MEMORY,
USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT,
USE_DNS_CACHE,
USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES<br>
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE
1024, BUF_SIZE
65535, DEFAULT
PKG_SIZE 8MB<br>
poll method
support: poll,
epoll_lt,
epoll_et,
sigio_rt,
select.<br>
id: 5a2195<br>
compiled on
08:30:51 Sep
15 2016 with
gcc 4.4.7<br>
<br>
<br>
</div>
<div>With kind
regards,<br>
</div>
<div><br>
</div>
</div>
<div
class="gmail_extra"><br
clear="all">
<div>
<div
class="m_-3322893149780162383m_668784092729894905m_1561746082170517228m_-4545043238606932376gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">Jurijs<br>
</div>
</div>
</div>
<br>
<div
class="gmail_quote">On
Thu, Oct 6,
2016 at 12:52
PM,
Daniel-Constantin
Mierla <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
<p>Hello,</p>
<p>it looks
like a leak
from the
system memory,
not from
kamailio's pkg
or shm memory.
This can be
due to an
improper use
of an external
library (e.g.,
libxml2) by a
kamailio
module or
because of a
problem in the
library.</p>
<p>Can you
list the
modules used
in your config
(the
loadmodule
lines)? I will
try to guess
from the list
which one
relying on
external libs
with higher
risk of leak
issues.</p>
<p>Also,
provide the
version of
kamailio you
are using
(kamailio -v).</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>
<div
class="m_-3322893149780162383m_668784092729894905m_1561746082170517228m_-4545043238606932376h5">
<br>
<div
class="m_-3322893149780162383m_668784092729894905m_1561746082170517228m_-4545043238606932376m_2624872147299544990moz-cite-prefix">On
04/10/16
15:42, Jurijs
Ivolga wrote:<br>
</div>
</div>
</div>
<blockquote
type="cite">
<div>
<div
class="m_-3322893149780162383m_668784092729894905m_1561746082170517228m_-4545043238606932376h5">
<div dir="ltr">
<div>Hi,<br>
<br>
</div>
Our Kamailio
server is
crashing once
per week, with
following
error:<br>
<br>
Oct 1
06:25:06
kamailio
kernel:
[26982632.803789]
Out of memory
in UB 210: OOM
killed process
12261
(kamailio)
score 0
vm:1614768kB,
rss:280200kB,
swap:131408kB<br>
<br>
<div>
<div>
<div>Core dump
was never
created,
probably it is
because of my
environment,
but I will try
to get it.<br>
<br>
</div>
<div>Server
constantly
eats memory,
maybe some
kind of memory
leak?<br>
<br>
</div>
<div>Any help
is highly
appreciated!<br>
</div>
<div><br
clear="all">
<div>
<div
class="m_-3322893149780162383m_668784092729894905m_1561746082170517228m_-4545043238606932376m_2624872147299544990m_-8889579837191443002m_-9076721087679106191gmail_signature">
<div dir="ltr">Jurijs<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset
class="m_-3322893149780162383m_668784092729894905m_1561746082170517228m_-4545043238606932376m_2624872147299544990mimeAttachmentHeader"></fieldset>
<br>
</div>
</div>
<pre>______________________________<wbr>_________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
<a moz-do-not-send="true" class="m_-3322893149780162383m_668784092729894905m_1561746082170517228m_-4545043238606932376m_2624872147299544990moz-txt-link-abbreviated" href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a>
<a moz-do-not-send="true" class="m_-3322893149780162383m_668784092729894905m_1561746082170517228m_-4545043238606932376m_2624872147299544990moz-txt-link-freetext" href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cg<wbr>i-bin/mailman/listinfo/sr-user<wbr>s</a><span class="m_-3322893149780162383m_668784092729894905m_1561746082170517228m_-4545043238606932376HOEnZb"><font color="#888888">
</font></span></pre><span class="m_-3322893149780162383m_668784092729894905m_1561746082170517228m_-4545043238606932376HOEnZb"><font color="#888888">
</font></span></blockquote><span class="m_-3322893149780162383m_668784092729894905m_1561746082170517228m_-4545043238606932376HOEnZb"><font color="#888888">
<pre class="m_-3322893149780162383m_668784092729894905m_1561746082170517228m_-4545043238606932376m_2624872147299544990moz-signature" cols="72">--
Daniel-Constantin Mierla
<a moz-do-not-send="true" class="m_-3322893149780162383m_668784092729894905m_1561746082170517228m_-4545043238606932376m_2624872147299544990moz-txt-link-freetext" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" class="m_-3322893149780162383m_668784092729894905m_1561746082170517228m_-4545043238606932376m_2624872147299544990moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/mic<wbr>onda</a>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - <a moz-do-not-send="true" class="m_-3322893149780162383m_668784092729894905m_1561746082170517228m_-4545043238606932376m_2624872147299544990moz-txt-link-freetext" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a></pre>
</font></span></div>
______________________________<wbr>_________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
<a moz-do-not-send="true" href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a>
<a moz-do-not-send="true" href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">http://lists.sip-router.org/cg<wbr>i-bin/mailman/listinfo/sr-user<wbr>s</a>
</blockquote></div>
</div>
</blockquote>
<pre class="m_-3322893149780162383m_668784092729894905m_1561746082170517228m_-4545043238606932376moz-signature" cols="72">--
Daniel-Constantin Mierla
<a moz-do-not-send="true" class="m_-3322893149780162383m_668784092729894905m_1561746082170517228m_-4545043238606932376moz-txt-link-freetext" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" class="m_-3322893149780162383m_668784092729894905m_1561746082170517228m_-4545043238606932376moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/mic<wbr>onda</a>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - <a moz-do-not-send="true" class="m_-3322893149780162383m_668784092729894905m_1561746082170517228m_-4545043238606932376moz-txt-link-freetext" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a></pre></div></div></div></blockquote></div>
</div>
</blockquote>
<pre class="m_-3322893149780162383m_668784092729894905m_1561746082170517228moz-signature" cols="72">--
Daniel-Constantin Mierla
<a moz-do-not-send="true" class="m_-3322893149780162383m_668784092729894905m_1561746082170517228moz-txt-link-freetext" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" class="m_-3322893149780162383m_668784092729894905m_1561746082170517228moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/mic<wbr>onda</a>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - <a moz-do-not-send="true" class="m_-3322893149780162383m_668784092729894905m_1561746082170517228moz-txt-link-freetext" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a></pre></div></div></div></blockquote></div>
</div>
</blockquote>
<pre class="m_-3322893149780162383m_668784092729894905moz-signature" cols="72">--
Daniel-Constantin Mierla
<a moz-do-not-send="true" class="m_-3322893149780162383m_668784092729894905moz-txt-link-freetext" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" class="m_-3322893149780162383m_668784092729894905moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/mic<wbr>onda</a>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - <a moz-do-not-send="true" class="m_-3322893149780162383m_668784092729894905moz-txt-link-freetext" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a></pre></div></div></div></blockquote></div>
</div>
</blockquote>
<pre class="m_-3322893149780162383moz-signature" cols="72">--
Daniel-Constantin Mierla
<a moz-do-not-send="true" class="m_-3322893149780162383moz-txt-link-freetext" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" class="m_-3322893149780162383moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/<wbr>miconda</a>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - <a moz-do-not-send="true" class="m_-3322893149780162383moz-txt-link-freetext" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a></pre></div></div></div></blockquote></div>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla
<a class="moz-txt-link-freetext" href="http://twitter.com/#!/miconda">http://twitter.com/#!/miconda</a> - <a class="moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - <a class="moz-txt-link-freetext" href="http://www.asipto.com">http://www.asipto.com</a></pre></body></html>