<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Hello,</p>
<p>There are some results when searching for memory leak on openssl
1.0.1, but might not be valid for this case.</p>
<p>Would you be able to run kamailio through valgrind? It may slow
down a bit the processing, but may be the fastest way to catch the
system memory leak. Maybe you have an instance will less traffic
and you can run through valgrind for a while, you don't have to
wait until you get the OOM.<br>
</p>
<p>Cheers,<br>
Daniel<br>
</p>
<br>
<div class="moz-cite-prefix">On 07/10/16 09:27, Jurijs Ivolga wrote:<br>
</div>
<blockquote
cite="mid:CAOrA2UY+Qb2jauBZN5o01j6MRT8tLNfufpVWAPgKrt02ZkM6cQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>Hi Daniel,<br>
<br>
openssl.x86_64
1.0.1e-48.el6_8.1<br>
CentOS release 6.8<br>
<br>
</div>
With kind regards,<br>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div class="gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">Jurijs<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On Fri, Oct 7, 2016 at 10:20 AM,
Daniel-Constantin Mierla <span dir="ltr"><<a
moz-do-not-send="true" href="mailto:miconda@gmail.com"
target="_blank">miconda@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Hello,</p>
<p>the tls module in kamailio is using shm memory, but can
be something internal for libssl. What operating system
do you use and what is the libssl version?</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>
<div class="h5"> <br>
<div class="m_668784092729894905moz-cite-prefix">On
06/10/16 17:43, Jurijs Ivolga wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>Hi Daniel,<br>
<br>
</div>
I do not use puke.top rpc command. Maybe this
issue related to TLS? Servers what has this
problem are utilizing TLS heavily, servers which
do not has this problem use UDP.<br>
<br>
</div>
With kind regards,<br>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div class="m_668784092729894905gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">Jurijs<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On Thu, Oct 6, 2016 at
6:05 PM, Daniel-Constantin Mierla <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:miconda@gmail.com"
target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Hello,</p>
<p>are you using pike.top rpc command? I
noticed in the code that it uses system
malloc, but I haven't investigated further
yet, first to see if this would be a
possibility ...</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>
<div class="m_668784092729894905h5"> <br>
<div
class="m_668784092729894905m_1561746082170517228moz-cite-prefix">On
06/10/16 16:33, Jurijs Ivolga wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hi Daniel,<br>
<br>
</div>
<div>We do not do any external
operations.<br>
<br>
</div>
<div>We are using janson 2.7
everywhere. I will try to update
to latest janson version tomorrow.<br>
</div>
<div>All json operation is pretty
much same, we are using only
jansson_get.<br>
<br>
</div>
<div>In attachment you can see
memory consumption. On the right 2
servers which are faced internet
on the left which don't face
internet. As you can see memory
consumption is pretty dramatic.<br>
</div>
<div><br>
</div>
<div>Thank you for your help!<br>
</div>
<div><br>
</div>
<div>With kind regards,<br>
</div>
</div>
<div class="gmail_extra"><br
clear="all">
<div>
<div
class="m_668784092729894905m_1561746082170517228gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">Jurijs<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On Thu, Oct
6, 2016 at 5:17 PM,
Daniel-Constantin Mierla <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:miconda@gmail.com"
target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF"
text="#000000">
<p>Hello,</p>
<p>are you doing different
external operations than on
the other instances, like
mi/rpc commands.</p>
<p>From the list of the
modules you exposed, I think
jansson has the higher
probability to work with
system memory. Are you doing
different json operations in
config that in the other
instances of kamailio? Are
you using same version of
libjansson everywhere?</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>
<div
class="m_668784092729894905m_1561746082170517228h5">
<div
class="m_668784092729894905m_1561746082170517228m_-4545043238606932376moz-cite-prefix">On
06/10/16 13:46, Jurijs
Ivolga wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>Hi Daniel,<br>
<br>
</div>
This modules what
we are using:<br>
<br>
loadmodule
"mi_fifo.so"<br>
loadmodule
"kex.so"<br>
loadmodule
"corex.so"<br>
loadmodule "tm.so"<br>
loadmodule
"tmx.so"<br>
loadmodule "sl.so"<br>
loadmodule "rr.so"<br>
loadmodule "pv.so"<br>
loadmodule
"maxfwd.so"<br>
loadmodule
"textops.so"<br>
loadmodule
"siputils.so"<br>
loadmodule
"xlog.so"<br>
loadmodule
"sanity.so"<br>
loadmodule
"ctl.so"<br>
loadmodule
"cfg_rpc.so"<br>
loadmodule
"mi_rpc.so"<br>
loadmodule
"dispatcher.so"<br>
loadmodule
"utils.so"<br>
loadmodule
"path.so"<br>
loadmodule
"ipops.so"<br>
loadmodule
"jansson.so"<br>
loadmodule
"auth.so"<br>
loadmodule
"nathelper.so"<br>
loadmodule
"tls.so"<br>
loadmodule
"htable.so"<br>
loadmodule
"pike.so"<br>
<br>
</div>
We have several
other Kamailio
instances but they
are not faced to
internet and they do
not have such memory
issue. That other
Kamailio instances
have same modules,
except modules
listed below. So if
you think that issue
is inside external
library, probably we
need to check first
modules from list
below.<br>
<br>
<span
class="m_668784092729894905m_1561746082170517228m_-4545043238606932376gmail-diff-chunkm_1561746082170517228m_-4545043238606932376gmail-diff-chunk-equal">loadmodule
"ipops.so"<br>
loadmodule
"auth.so"<br>
loadmodule
"nathelper.so"<br>
loadmodule
"pike.so"</span><span
class="m_668784092729894905m_1561746082170517228m_-4545043238606932376gmail-diff-chunk
m_-4545043238606932376gmail-diff-chunk-removed"></span><br>
<br>
</div>
<div>But maybe this
other Kamailio
instances do not
have this memory
issue, just because
they did not face to
internet and did not
have same load as
instances with
memory issue.<br>
<br>
kamailio -v<br>
version: kamailio
4.4.3 (x86_64/linux)
5a2195<br>
flags: STATS: Off,
USE_TCP, USE_TLS,
USE_SCTP, TLS_HOOKS,
USE_RAW_SOCKS,
DISABLE_NAGLE,
USE_MCAST,
DNS_IP_HACK,
SHM_MEM, SHM_MMAP,
PKG_MALLOC,
Q_MALLOC, F_MALLOC,
TLSF_MALLOC,
DBG_SR_MEMORY,
USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT,
USE_DNS_CACHE,
USE_DNS_FAILOVER,
USE_NAPTR,
USE_DST_BLACKLIST,
HAVE_RESOLV_RES<br>
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024,
BUF_SIZE 65535,
DEFAULT PKG_SIZE 8MB<br>
poll method support:
poll, epoll_lt,
epoll_et, sigio_rt,
select.<br>
id: 5a2195<br>
compiled on 08:30:51
Sep 15 2016 with gcc
4.4.7<br>
<br>
<br>
</div>
<div>With kind
regards,<br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br
clear="all">
<div>
<div
class="m_668784092729894905m_1561746082170517228m_-4545043238606932376gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">Jurijs<br>
</div>
</div>
</div>
<br>
<div
class="gmail_quote">On
Thu, Oct 6, 2016 at
12:52 PM,
Daniel-Constantin
Mierla <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0
0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
<p>Hello,</p>
<p>it looks like
a leak from
the system
memory, not
from
kamailio's pkg
or shm memory.
This can be
due to an
improper use
of an external
library (e.g.,
libxml2) by a
kamailio
module or
because of a
problem in the
library.</p>
<p>Can you list
the modules
used in your
config (the
loadmodule
lines)? I will
try to guess
from the list
which one
relying on
external libs
with higher
risk of leak
issues.</p>
<p>Also, provide
the version of
kamailio you
are using
(kamailio -v).</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>
<div
class="m_668784092729894905m_1561746082170517228m_-4545043238606932376h5">
<br>
<div
class="m_668784092729894905m_1561746082170517228m_-4545043238606932376m_2624872147299544990moz-cite-prefix">On
04/10/16
15:42, Jurijs
Ivolga wrote:<br>
</div>
</div>
</div>
<blockquote
type="cite">
<div>
<div
class="m_668784092729894905m_1561746082170517228m_-4545043238606932376h5">
<div dir="ltr">
<div>Hi,<br>
<br>
</div>
Our Kamailio
server is
crashing once
per week, with
following
error:<br>
<br>
Oct 1
06:25:06
kamailio
kernel:
[26982632.803789]
Out of memory
in UB 210: OOM
killed process
12261
(kamailio)
score 0
vm:1614768kB,
rss:280200kB,
swap:131408kB<br>
<br>
<div>
<div>
<div>Core dump
was never
created,
probably it is
because of my
environment,
but I will try
to get it.<br>
<br>
</div>
<div>Server
constantly
eats memory,
maybe some
kind of memory
leak?<br>
<br>
</div>
<div>Any help
is highly
appreciated!<br>
</div>
<div><br
clear="all">
<div>
<div
class="m_668784092729894905m_1561746082170517228m_-4545043238606932376m_2624872147299544990m_-8889579837191443002m_-9076721087679106191gmail_signature">
<div dir="ltr">Jurijs<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset
class="m_668784092729894905m_1561746082170517228m_-4545043238606932376m_2624872147299544990mimeAttachmentHeader"></fieldset>
<br>
</div>
</div>
<pre>______________________________<wbr>_________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
<a moz-do-not-send="true" class="m_668784092729894905m_1561746082170517228m_-4545043238606932376m_2624872147299544990moz-txt-link-abbreviated" href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a>
<a moz-do-not-send="true" class="m_668784092729894905m_1561746082170517228m_-4545043238606932376m_2624872147299544990moz-txt-link-freetext" href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cg<wbr>i-bin/mailman/listinfo/sr-user<wbr>s</a><span class="m_668784092729894905m_1561746082170517228m_-4545043238606932376HOEnZb"><font color="#888888">
</font></span></pre><span class="m_668784092729894905m_1561746082170517228m_-4545043238606932376HOEnZb"><font color="#888888">
</font></span></blockquote><span class="m_668784092729894905m_1561746082170517228m_-4545043238606932376HOEnZb"><font color="#888888">
<pre class="m_668784092729894905m_1561746082170517228m_-4545043238606932376m_2624872147299544990moz-signature" cols="72">--
Daniel-Constantin Mierla
<a moz-do-not-send="true" class="m_668784092729894905m_1561746082170517228m_-4545043238606932376m_2624872147299544990moz-txt-link-freetext" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" class="m_668784092729894905m_1561746082170517228m_-4545043238606932376m_2624872147299544990moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/mic<wbr>onda</a>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - <a moz-do-not-send="true" class="m_668784092729894905m_1561746082170517228m_-4545043238606932376m_2624872147299544990moz-txt-link-freetext" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a></pre>
</font></span></div>
______________________________<wbr>_________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
<a moz-do-not-send="true" href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a>
<a moz-do-not-send="true" href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">http://lists.sip-router.org/cg<wbr>i-bin/mailman/listinfo/sr-user<wbr>s</a>
</blockquote></div>
</div>
</blockquote>
<pre class="m_668784092729894905m_1561746082170517228m_-4545043238606932376moz-signature" cols="72">--
Daniel-Constantin Mierla
<a moz-do-not-send="true" class="m_668784092729894905m_1561746082170517228m_-4545043238606932376moz-txt-link-freetext" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" class="m_668784092729894905m_1561746082170517228m_-4545043238606932376moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/mic<wbr>onda</a>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - <a moz-do-not-send="true" class="m_668784092729894905m_1561746082170517228m_-4545043238606932376moz-txt-link-freetext" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a></pre></div></div></div></blockquote></div>
</div>
</blockquote>
<pre class="m_668784092729894905m_1561746082170517228moz-signature" cols="72">--
Daniel-Constantin Mierla
<a moz-do-not-send="true" class="m_668784092729894905m_1561746082170517228moz-txt-link-freetext" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" class="m_668784092729894905m_1561746082170517228moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/mic<wbr>onda</a>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - <a moz-do-not-send="true" class="m_668784092729894905m_1561746082170517228moz-txt-link-freetext" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a></pre></div></div></div></blockquote></div>
</div>
</blockquote>
<pre class="m_668784092729894905moz-signature" cols="72">--
Daniel-Constantin Mierla
<a moz-do-not-send="true" class="m_668784092729894905moz-txt-link-freetext" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" class="m_668784092729894905moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/<wbr>miconda</a>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - <a moz-do-not-send="true" class="m_668784092729894905moz-txt-link-freetext" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a></pre></div></div></div></blockquote></div>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla
<a class="moz-txt-link-freetext" href="http://twitter.com/#!/miconda">http://twitter.com/#!/miconda</a> - <a class="moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - <a class="moz-txt-link-freetext" href="http://www.asipto.com">http://www.asipto.com</a></pre></body></html>