<div dir="ltr"><div><div>Hi Daniel,<br><br></div>I do not use puke.top rpc command. Maybe this issue related to TLS? Servers what has this problem are utilizing TLS heavily, servers which do not has this problem use UDP.<br><br></div>With kind regards,<br></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Jurijs<br></div></div></div>
<br><div class="gmail_quote">On Thu, Oct 6, 2016 at 6:05 PM, Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Hello,</p>
<p>are you using pike.top rpc command? I noticed in the code that it
uses system malloc, but I haven't investigated further yet, first
to see if this would be a possibility ...</p>
<p>Cheers,<br>
Daniel<br>
</p><div><div class="h5">
<br>
<div class="m_1561746082170517228moz-cite-prefix">On 06/10/16 16:33, Jurijs Ivolga wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hi Daniel,<br>
<br>
</div>
<div>We do not do any external operations.<br>
<br>
</div>
<div>We are using janson 2.7 everywhere. I will try to update to
latest janson version tomorrow.<br>
</div>
<div>All json operation is pretty much same, we are using only
jansson_get.<br>
<br>
</div>
<div>In attachment you can see memory consumption. On the right
2 servers which are faced internet on the left which don't
face internet. As you can see memory consumption is pretty
dramatic.<br>
</div>
<div><br>
</div>
<div>Thank you for your help!<br>
</div>
<div><br>
</div>
<div>With kind regards,<br>
</div>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div class="m_1561746082170517228gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">Jurijs<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On Thu, Oct 6, 2016 at 5:17 PM,
Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Hello,</p>
<p>are you doing different external operations than on the
other instances, like mi/rpc commands.</p>
<p>From the list of the modules you exposed, I think
jansson has the higher probability to work with system
memory. Are you doing different json operations in
config that in the other instances of kamailio? Are you
using same version of libjansson everywhere?</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>
<div class="m_1561746082170517228h5">
<div class="m_1561746082170517228m_-4545043238606932376moz-cite-prefix">On
06/10/16 13:46, Jurijs Ivolga wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>Hi Daniel,<br>
<br>
</div>
This modules what we are using:<br>
<br>
loadmodule "mi_fifo.so"<br>
loadmodule "kex.so"<br>
loadmodule "corex.so"<br>
loadmodule "tm.so"<br>
loadmodule "tmx.so"<br>
loadmodule "sl.so"<br>
loadmodule "rr.so"<br>
loadmodule "pv.so"<br>
loadmodule "maxfwd.so"<br>
loadmodule "textops.so"<br>
loadmodule "siputils.so"<br>
loadmodule "xlog.so"<br>
loadmodule "sanity.so"<br>
loadmodule "ctl.so"<br>
loadmodule "cfg_rpc.so"<br>
loadmodule "mi_rpc.so"<br>
loadmodule "dispatcher.so"<br>
loadmodule "utils.so"<br>
loadmodule "path.so"<br>
loadmodule "ipops.so"<br>
loadmodule "jansson.so"<br>
loadmodule "auth.so"<br>
loadmodule "nathelper.so"<br>
loadmodule "tls.so"<br>
loadmodule "htable.so"<br>
loadmodule "pike.so"<br>
<br>
</div>
We have several other Kamailio instances but
they are not faced to internet and they do not
have such memory issue. That other Kamailio
instances have same modules, except modules
listed below. So if you think that issue is
inside external library, probably we need to
check first modules from list below.<br>
<br>
<span class="m_1561746082170517228m_-4545043238606932376gmail-diff-chunk m_1561746082170517228m_-4545043238606932376gmail-diff-chunk-equal">loadmodule
"ipops.so"<br>
loadmodule "auth.so"<br>
loadmodule "nathelper.so"<br>
loadmodule "pike.so"</span><span class="m_1561746082170517228m_-4545043238606932376gmail-diff-chunk
m_-4545043238606932376gmail-diff-chunk-removed"></span><br>
<br>
</div>
<div>But maybe this other Kamailio instances do
not have this memory issue, just because they
did not face to internet and did not have same
load as instances with memory issue.<br>
<br>
kamailio -v<br>
version: kamailio 4.4.3 (x86_64/linux) 5a2195<br>
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP,
TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE,
USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP,
PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
DBG_SR_MEMORY, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST,
HAVE_RESOLV_RES<br>
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE
262144, MAX_LISTEN 16, MAX_URI_SIZE 1024,
BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB<br>
poll method support: poll, epoll_lt, epoll_et,
sigio_rt, select.<br>
id: 5a2195<br>
compiled on 08:30:51 Sep 15 2016 with gcc 4.4.7<br>
<br>
<br>
</div>
<div>With kind regards,<br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div class="m_1561746082170517228m_-4545043238606932376gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">Jurijs<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On Thu, Oct 6, 2016 at
12:52 PM, Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Hello,</p>
<p>it looks like a leak from the system
memory, not from kamailio's pkg or shm
memory. This can be due to an improper use
of an external library (e.g., libxml2) by
a kamailio module or because of a problem
in the library.</p>
<p>Can you list the modules used in your
config (the loadmodule lines)? I will try
to guess from the list which one relying
on external libs with higher risk of leak
issues.</p>
<p>Also, provide the version of kamailio you
are using (kamailio -v).</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>
<div class="m_1561746082170517228m_-4545043238606932376h5"> <br>
<div class="m_1561746082170517228m_-4545043238606932376m_2624872147299544990moz-cite-prefix">On
04/10/16 15:42, Jurijs Ivolga wrote:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div class="m_1561746082170517228m_-4545043238606932376h5">
<div dir="ltr">
<div>Hi,<br>
<br>
</div>
Our Kamailio server is crashing once
per week, with following error:<br>
<br>
Oct 1 06:25:06 kamailio kernel:
[26982632.803789] Out of memory in
UB 210: OOM killed process 12261
(kamailio) score 0 vm:1614768kB,
rss:280200kB, swap:131408kB<br>
<br>
<div>
<div>
<div>Core dump was never
created, probably it is
because of my environment, but
I will try to get it.<br>
<br>
</div>
<div>Server constantly eats
memory, maybe some kind of
memory leak?<br>
<br>
</div>
<div>Any help is highly
appreciated!<br>
</div>
<div><br clear="all">
<div>
<div class="m_1561746082170517228m_-4545043238606932376m_2624872147299544990m_-8889579837191443002m_-9076721087679106191gmail_signature">
<div dir="ltr">Jurijs<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="m_1561746082170517228m_-4545043238606932376m_2624872147299544990mimeAttachmentHeader"></fieldset>
<br>
</div>
</div>
<pre>______________________________<wbr>_________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
<a class="m_1561746082170517228m_-4545043238606932376m_2624872147299544990moz-txt-link-abbreviated" href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a>
<a class="m_1561746082170517228m_-4545043238606932376m_2624872147299544990moz-txt-link-freetext" href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cg<wbr>i-bin/mailman/listinfo/sr-user<wbr>s</a><span class="m_1561746082170517228m_-4545043238606932376HOEnZb"><font color="#888888">
</font></span></pre><span class="m_1561746082170517228m_-4545043238606932376HOEnZb"><font color="#888888">
</font></span></blockquote><span class="m_1561746082170517228m_-4545043238606932376HOEnZb"><font color="#888888">
<pre class="m_1561746082170517228m_-4545043238606932376m_2624872147299544990moz-signature" cols="72">--
Daniel-Constantin Mierla
<a class="m_1561746082170517228m_-4545043238606932376m_2624872147299544990moz-txt-link-freetext" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a class="m_1561746082170517228m_-4545043238606932376m_2624872147299544990moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/mic<wbr>onda</a>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - <a class="m_1561746082170517228m_-4545043238606932376m_2624872147299544990moz-txt-link-freetext" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a></pre>
</font></span></div>
______________________________<wbr>_________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
<a href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a>
<a href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">http://lists.sip-router.org/cg<wbr>i-bin/mailman/listinfo/sr-user<wbr>s</a>
</blockquote></div>
</div>
</blockquote>
<pre class="m_1561746082170517228m_-4545043238606932376moz-signature" cols="72">--
Daniel-Constantin Mierla
<a class="m_1561746082170517228m_-4545043238606932376moz-txt-link-freetext" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a class="m_1561746082170517228m_-4545043238606932376moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/mic<wbr>onda</a>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - <a class="m_1561746082170517228m_-4545043238606932376moz-txt-link-freetext" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a></pre></div></div></div></blockquote></div>
</div>
</blockquote>
<pre class="m_1561746082170517228moz-signature" cols="72">--
Daniel-Constantin Mierla
<a class="m_1561746082170517228moz-txt-link-freetext" href="http://twitter.com/#!/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a class="m_1561746082170517228moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/<wbr>miconda</a>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - <a class="m_1561746082170517228moz-txt-link-freetext" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a></pre></div></div></div></blockquote></div><br></div>