<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
The logs suggest some leaks in dispatcher module -- it allocates
shared memory only for caching routing records. Few more questions I
need to get the answers in order to troubleshoot further.<br>
<br>
How many records do you have in dispatcher table? Is it in database
or the text file?<br>
<br>
How ofter do you reload? What is the command you use for reload?<br>
<br>
Cheers,<br>
Daniel<br>
<br>
<div class="moz-cite-prefix">On 26/08/14 22:42, Heenan, Timothy
Steven wrote:<br>
</div>
<blockquote
cite="mid:2DB6BB728D67D3458F08F1E4AC6284941FA338@CWWAPP473.windstream.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<meta name="Generator" content="Microsoft Word 12 (filtered
medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]-->
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Times New Roman \, serif";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
color:black;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";
color:black;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle24
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle25
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle26
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">I have attached
is the output of “kamcmd cfg.set_now_int core mem_dump_shm
1” to this email. I had to restart Kamailio to load ctl and
ctl_rpc, so let me know if I need to do another one after
some time passes and calls are processed. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Also, here is
the output of “kamctl mi get_statistics shmem:”<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">shmem:fragments
= 398<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">shmem:free_size
= 259170720<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">shmem:max_used_size
= 9265184<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">shmem:real_used_size
= 9264736<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">shmem:total_size
= 268435456<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">shmem:used_size
= 8559768<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">-Tim<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">
Daniel-Constantin Mierla [<a class="moz-txt-link-freetext" href="mailto:miconda@gmail.com">mailto:miconda@gmail.com</a>]
<br>
<b>Sent:</b> Tuesday, August 26, 2014 10:46 AM<br>
<b>To:</b> Heenan, Timothy Steven; Kamailio (SER) -
Users Mailing List<br>
<b>Subject:</b> Re: [SR-Users] Kamailio 4.0.4 slow
memory leak<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">It is not
needed to see the memory operations logs. Just to dump the
chunks in memory.<br>
<br>
You would need ctl and ctl_rpc modules (iirc -- they should be
in the default config file), then run:<br>
<br>
kamcmd cfg.set_now_int core mem_dump_shm 1<br>
<br>
Extract the logs of the dump from the shared memory and send
them over.<br>
<br>
Send also the output for:<br>
<br>
kamctl mi get_statistics shmem:<br>
<br>
Cheers,<br>
Daniel<o:p></o:p></p>
<div>
<p class="MsoNormal">On 26/08/14 17:34, Heenan, Timothy Steven
wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="color:#1F497D">Thank you. </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">I have
recompiled with MEMDBG, however I am not observing any
additional output in my log files.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">I think I
have something set incorrectly in kamailio.cfg.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Here are my
log settings from Kamailio.cfg</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">debug=3</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">log_stderror=no</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">log_facility=LOG_LOCAL0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">memdbg=2</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">memlog=2</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">…</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">modparam("debugger",
"cfgtrace", 1)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">modparam("debugger",
"log_level", 2)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">modparam("debugger",
"cfgpkgcheck", 1)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">modparam("debugger",
"mod_level", "core=2")</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">modparam("debugger",
"mod_level", "tm=2")</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">modparam("debugger",
"mod_level", "dispatcher=2")</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">modparam("debugger",
"mod_level", "siptrace=2")</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">
Daniel-Constantin Mierla [<a moz-do-not-send="true"
href="mailto:miconda@gmail.com">mailto:miconda@gmail.com</a>]
<br>
<b>Sent:</b> Tuesday, August 26, 2014 3:54 AM<br>
<b>To:</b> Heenan, Timothy Steven; Kamailio (SER) -
Users Mailing List<br>
<b>Subject:</b> Re: [SR-Users] Kamailio 4.0.4 slow
memory leak</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hello,<br>
<br>
I quickly looked over the modules that use shared memory and
I couldn't spot a commit related to any leak.<br>
<br>
Have you had the chance to compile with MEMDBG?<br>
<br>
Cheers,<br>
Daniel<o:p></o:p></p>
<div>
<p class="MsoNormal">On 19/08/14 19:14, Heenan, Timothy
Steven wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="color:#1F497D">Do you
think any of these modules possibly causing this issue?
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">-Tim</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">
<a moz-do-not-send="true"
href="mailto:sr-users-bounces@lists.sip-router.org">sr-users-bounces@lists.sip-router.org</a>
[<a moz-do-not-send="true"
href="mailto:sr-users-bounces@lists.sip-router.org">mailto:sr-users-bounces@lists.sip-router.org</a>]
<b>On Behalf Of </b>Heenan, Timothy Steven<br>
<b>Sent:</b> Tuesday, August 12, 2014 2:37 PM<br>
<b>To:</b> '<a moz-do-not-send="true"
href="mailto:miconda@gmail.com">miconda@gmail.com</a>';
Kamailio (SER) - Users Mailing List<br>
<b>Subject:</b> Re: [SR-Users] Kamailio 4.0.4 slow
memory leak</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Hi Daniel,
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Here’s a
list of the modules I’m running:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">mi_fifo</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">db_mysql</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">sl</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">kex</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">tm</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">tmx</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">rr</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">xlog</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">maxfwd</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">usrloc</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">registrar</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">textops</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">pv</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">acc</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">permissions</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">siputils</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">auth</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">lcr</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">dispatcher</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">sanity</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">debugger</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">siptrace</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">We’re
setting the memory allocation to 256 ( -m 256 –M 64 ).</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">I’ll see
about recompiling with MEMDBG=1.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Regards, </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">- Tim</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">
<a moz-do-not-send="true"
href="mailto:sr-users-bounces@lists.sip-router.org">sr-users-bounces@lists.sip-router.org</a>
[<a moz-do-not-send="true"
href="mailto:sr-users-bounces@lists.sip-router.org">mailto:sr-users-bounces@lists.sip-router.org</a>]
<b>On Behalf Of </b>Daniel-Constantin Mierla<br>
<b>Sent:</b> Tuesday, August 12, 2014 4:17 AM<br>
<b>To:</b> Kamailio (SER) - Users Mailing List<br>
<b>Subject:</b> Re: [SR-Users] Kamailio 4.0.4 slow
memory leak</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hello,<br>
<br>
can you give the list of loaded modules? How much memory
did you allocate (-m parameter value)?<br>
<br>
It will narrow down searches to see if there was anything
similar fixed since 4.0.4.<br>
<br>
To troubleshoot easier, would be good to recompile with
MEMDBG=1, then the details of chunks in memory can be
dumped and analysed.<br>
<br>
Cheers,<br>
Daniel<o:p></o:p></p>
<div>
<p class="MsoNormal">On 12/08/14 01:40, Heenan, Timothy
Steven wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Greetings, <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I’m running into a slow memory leak
on my kamailio 4.0.4 SIP proxies. I’m observing a steady
increase in the memory consumption until there is no
more left. Kamailio then starts repeating this in the
logs:<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">ERROR: dispatcher [dispatch.c:279]:
add_dest2list(): no more memory.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">What would be the best way to debug
this kind of a memory leak? The proxy does not handle
any registrations but does route a fair amount of calls.
<br>
</p>
</blockquote>
</blockquote>
</blockquote>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla
<a class="moz-txt-link-freetext" href="http://twitter.com/#!/miconda">http://twitter.com/#!/miconda</a> - <a class="moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda">http://www.linkedin.com/in/miconda</a>
Next Kamailio Advanced Trainings 2014 - <a class="moz-txt-link-freetext" href="http://www.asipto.com">http://www.asipto.com</a>
Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA</pre>
</body>
</html>