<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Looking at the logs, there are 1336 URIs, which seems ok given your
numbers, because the module keeps also the previous set of records
(needed because at reload time there can be kamailio worker
processes using the records -- perhaps we can improve a bit here, I
will review the current reload code a bit, since it was a
contribution from another developer quite long time ago).<br>
<br>
Some more questions:<br>
- how many destination sets (groups) do you have? Can you estimate
the minimum and maximum addresses in a set?<br>
- when did you take the memory log? In other words, for how long was
kamailio running? Did you wait enough to notice the steady increase
of memory usage?<br>
<br>
Cheers,<br>
Daniel<br>
<br>
<div class="moz-cite-prefix">On 28/08/14 01:56, Heenan, Timothy
Steven wrote:<br>
</div>
<blockquote
cite="mid:2DB6BB728D67D3458F08F1E4AC6284941FAAA7@CWWAPP473.windstream.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<meta name="Generator" content="Microsoft Word 12 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
color:black;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";
color:black;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle24
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle25
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle26
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle27
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Thank you for
the help. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">For dispatcher,
I’m using a database that contains about 700 records.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">A reload is
performed via cronjob every 5 minutes. The command being
used is:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">kamctl
dispatcher reload<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">-Tim<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">
Daniel-Constantin Mierla [<a class="moz-txt-link-freetext" href="mailto:miconda@gmail.com">mailto:miconda@gmail.com</a>]
<br>
<b>Sent:</b> Wednesday, August 27, 2014 5:22 PM<br>
<b>To:</b> Heenan, Timothy Steven; Kamailio (SER) -
Users Mailing List<br>
<b>Subject:</b> Re: [SR-Users] Kamailio 4.0.4 slow
memory leak<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">The logs
suggest some leaks in dispatcher module -- it allocates shared
memory only for caching routing records. Few more questions I
need to get the answers in order to troubleshoot further.<br>
<br>
How many records do you have in dispatcher table? Is it in
database or the text file?<br>
<br>
How ofter do you reload? What is the command you use for
reload?<br>
<br>
Cheers,<br>
Daniel<o:p></o:p></p>
<div>
<p class="MsoNormal">On 26/08/14 22:42, Heenan, Timothy Steven
wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="color:#1F497D">I have
attached is the output of “kamcmd cfg.set_now_int core
mem_dump_shm 1” to this email. I had to restart Kamailio
to load ctl and ctl_rpc, so let me know if I need to do
another one after some time passes and calls are
processed. </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Also, here is
the output of “kamctl mi get_statistics shmem:”</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">shmem:fragments
= 398</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">shmem:free_size
= 259170720</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">shmem:max_used_size
= 9265184</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">shmem:real_used_size
= 9264736</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">shmem:total_size
= 268435456</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">shmem:used_size
= 8559768</span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks!</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">-Tim</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">
Daniel-Constantin Mierla [<a moz-do-not-send="true"
href="mailto:miconda@gmail.com">mailto:miconda@gmail.com</a>]
<br>
<b>Sent:</b> Tuesday, August 26, 2014 10:46 AM<br>
<b>To:</b> Heenan, Timothy Steven; Kamailio (SER) -
Users Mailing List<br>
<b>Subject:</b> Re: [SR-Users] Kamailio 4.0.4 slow
memory leak</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">It is not
needed to see the memory operations logs. Just to dump the
chunks in memory.<br>
<br>
You would need ctl and ctl_rpc modules (iirc -- they should
be in the default config file), then run:<br>
<br>
kamcmd cfg.set_now_int core mem_dump_shm 1<br>
<br>
Extract the logs of the dump from the shared memory and send
them over.<br>
<br>
Send also the output for:<br>
<br>
kamctl mi get_statistics shmem:<br>
<br>
Cheers,<br>
Daniel<o:p></o:p></p>
<div>
<p class="MsoNormal">On 26/08/14 17:34, Heenan, Timothy
Steven wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="color:#1F497D">Thank you.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">I have
recompiled with MEMDBG, however I am not observing any
additional output in my log files.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">I think I
have something set incorrectly in kamailio.cfg.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Here are my
log settings from Kamailio.cfg</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">debug=3</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">log_stderror=no</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">log_facility=LOG_LOCAL0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">memdbg=2</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">memlog=2</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">…</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">modparam("debugger",
"cfgtrace", 1)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">modparam("debugger",
"log_level", 2)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">modparam("debugger",
"cfgpkgcheck", 1)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">modparam("debugger",
"mod_level", "core=2")</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">modparam("debugger",
"mod_level", "tm=2")</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">modparam("debugger",
"mod_level", "dispatcher=2")</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">modparam("debugger",
"mod_level", "siptrace=2")</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">
Daniel-Constantin Mierla [<a moz-do-not-send="true"
href="mailto:miconda@gmail.com">mailto:miconda@gmail.com</a>]
<br>
<b>Sent:</b> Tuesday, August 26, 2014 3:54 AM<br>
<b>To:</b> Heenan, Timothy Steven; Kamailio (SER) -
Users Mailing List<br>
<b>Subject:</b> Re: [SR-Users] Kamailio 4.0.4 slow
memory leak</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hello,<br>
<br>
I quickly looked over the modules that use shared memory
and I couldn't spot a commit related to any leak.<br>
<br>
Have you had the chance to compile with MEMDBG?<br>
<br>
Cheers,<br>
Daniel<o:p></o:p></p>
<div>
<p class="MsoNormal">On 19/08/14 19:14, Heenan, Timothy
Steven wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="color:#1F497D">Do you
think any of these modules possibly causing this
issue?
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">-Tim</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">
<a moz-do-not-send="true"
href="mailto:sr-users-bounces@lists.sip-router.org">sr-users-bounces@lists.sip-router.org</a>
[<a moz-do-not-send="true"
href="mailto:sr-users-bounces@lists.sip-router.org">mailto:sr-users-bounces@lists.sip-router.org</a>]
<b>On Behalf Of </b>Heenan, Timothy Steven<br>
<b>Sent:</b> Tuesday, August 12, 2014 2:37 PM<br>
<b>To:</b> '<a moz-do-not-send="true"
href="mailto:miconda@gmail.com">miconda@gmail.com</a>';
Kamailio (SER) - Users Mailing List<br>
<b>Subject:</b> Re: [SR-Users] Kamailio 4.0.4 slow
memory leak</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Hi
Daniel, </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Here’s a
list of the modules I’m running:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">mi_fifo</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">db_mysql</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">sl</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">kex</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">tm</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">tmx</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">rr</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">xlog</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">maxfwd</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">usrloc</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">registrar</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">textops</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">pv</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">acc</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">permissions</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">siputils</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">auth</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">lcr</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">dispatcher</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">sanity</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">debugger</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">siptrace</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">We’re
setting the memory allocation to 256 ( -m 256 –M 64 ).</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">I’ll see
about recompiling with MEMDBG=1.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Regards,
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">- Tim</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">
<a moz-do-not-send="true"
href="mailto:sr-users-bounces@lists.sip-router.org">sr-users-bounces@lists.sip-router.org</a>
[<a moz-do-not-send="true"
href="mailto:sr-users-bounces@lists.sip-router.org">mailto:sr-users-bounces@lists.sip-router.org</a>]
<b>On Behalf Of </b>Daniel-Constantin Mierla<br>
<b>Sent:</b> Tuesday, August 12, 2014 4:17 AM<br>
<b>To:</b> Kamailio (SER) - Users Mailing List<br>
<b>Subject:</b> Re: [SR-Users] Kamailio 4.0.4 slow
memory leak</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hello,<br>
<br>
can you give the list of loaded modules? How much memory
did you allocate (-m parameter value)?<br>
<br>
It will narrow down searches to see if there was
anything similar fixed since 4.0.4.<br>
<br>
To troubleshoot easier, would be good to recompile with
MEMDBG=1, then the details of chunks in memory can be
dumped and analysed.<br>
<br>
Cheers,<br>
Daniel<o:p></o:p></p>
<div>
<p class="MsoNormal">On 12/08/14 01:40, Heenan, Timothy
Steven wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Greetings, <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I’m running into a slow memory leak
on my kamailio 4.0.4 SIP proxies. I’m observing a
steady increase in the memory consumption until there
is no more left. Kamailio then starts repeating this
in the logs:<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">ERROR: dispatcher [dispatch.c:279]:
add_dest2list(): no more memory.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">What would be the best way to debug
this kind of a memory leak? The proxy does not handle
any registrations but does route a fair amount of
calls.
<o:p></o:p></p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman","serif""><br>
<br>
<o:p></o:p></span></p>
<pre>-- <o:p></o:p></pre>
<pre>Daniel-Constantin Mierla<o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="http://twitter.com/#%21/miconda">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" href="http://www.linkedin.com/in/miconda">http://www.linkedin.com/in/miconda</a><o:p></o:p></pre>
<pre>Next Kamailio Advanced Trainings 2014 - <a moz-do-not-send="true" href="http://www.asipto.com">http://www.asipto.com</a><o:p></o:p></pre>
<pre>Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA<o:p></o:p></pre>
</div>
<hr>This email message and any attachments are for the sole use of
the intended recipient(s). Any unauthorized review, use,
disclosure or distribution is prohibited. If you are not the
intended recipient, please contact the sender by reply email and
destroy all copies of the original message and any attachments.<br>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla
<a class="moz-txt-link-freetext" href="http://twitter.com/#!/miconda">http://twitter.com/#!/miconda</a> - <a class="moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda">http://www.linkedin.com/in/miconda</a>
Next Kamailio Advanced Trainings 2014 - <a class="moz-txt-link-freetext" href="http://www.asipto.com">http://www.asipto.com</a>
Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA</pre>
</body>
</html>