<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Courier New \,serif";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;
mso-fareast-language:EN-US;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New",serif;
color:black;
mso-fareast-language:EN-GB;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;
mso-fareast-language:EN-GB;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;
mso-fareast-language:EN-US;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle24
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:windowtext">4.3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="color:windowtext;mso-fareast-language:EN-GB">From:</span></b><span lang="EN-US" style="color:windowtext;mso-fareast-language:EN-GB"> Daniel-Constantin Mierla [mailto:miconda@gmail.com]
<br>
<b>Sent:</b> 19 September 2016 17:47<br>
<b>To:</b> Phil Lavin <phil.lavin@cloudcall.com>; Kamailio (SER) - Users Mailing List <sr-users@lists.sip-router.org><br>
<b>Subject:</b> Re: [SR-Users] OOM Errors<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p>The rpc command is in 4.4, what version are you using?<span style="font-size:12.0pt;mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p>Cheers,<br>
Daniel<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 19/09/16 18:11, Phil Lavin wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="color:windowtext">Having adding some graphing, I see this memory issue on another production Kamailio server. A few of the processes are using significantly more pkg memory than others. What I assume is happening is that memory
is leaking and will eventually start hitting OOM errors. See <a href="https://gist.github.com/phil-lavin/27f63b472ace53dc956125a47d210bc9">
https://gist.github.com/phil-lavin/27f63b472ace53dc956125a47d210bc9</a> for pkg memory stats output.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext">When was corex.pkg_summary added? I’m getting not found error:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal">error: 500 - command corex.pkg_summary not found<o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="color:windowtext;mso-fareast-language:EN-GB">From:</span></b><span lang="EN-US" style="color:windowtext;mso-fareast-language:EN-GB"> sr-users [<a href="mailto:sr-users-bounces@lists.sip-router.org">mailto:sr-users-bounces@lists.sip-router.org</a>]
<b>On Behalf Of </b>Phil Lavin<br>
<b>Sent:</b> 19 September 2016 14:36<br>
<b>To:</b> <a href="mailto:miconda@gmail.com">miconda@gmail.com</a>; Kamailio (SER) - Users Mailing List
<a href="mailto:sr-users@lists.sip-router.org"><sr-users@lists.sip-router.org></a><br>
<b>Subject:</b> Re: [SR-Users] OOM Errors</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext">Thanks, Daniel.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext">Kamailio has been restarted now so there’s not much of any use in the pkg memory output.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext">I’m going to add some monitoring/graphing on this and see if there’s an upward trend in usage.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="color:windowtext;mso-fareast-language:EN-GB">From:</span></b><span lang="EN-US" style="color:windowtext;mso-fareast-language:EN-GB"> sr-users [<a href="mailto:sr-users-bounces@lists.sip-router.org">mailto:sr-users-bounces@lists.sip-router.org</a>]
<b>On Behalf Of </b>Daniel-Constantin Mierla<br>
<b>Sent:</b> 19 September 2016 13:41<br>
<b>To:</b> Kamailio (SER) - Users Mailing List <<a href="mailto:sr-users@lists.sip-router.org">sr-users@lists.sip-router.org</a>><br>
<b>Subject:</b> Re: [SR-Users] OOM Errors</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p>Hello,<o:p></o:p></p>
<p>it seems that the pkg is out of free space, not shm. You can check it with:<o:p></o:p></p>
<p>kamcmd pkg.stats<o:p></o:p></p>
<p>and look for the same pid that printed the log messages.<o:p></o:p></p>
<p>To troubleshoot, it is good to print the pkg summary, see:<o:p></o:p></p>
<p> - <a href="https://www.kamailio.org/docs/modules/stable/modules/corex.html#corex.rpc.pkg_summary">
https://www.kamailio.org/docs/modules/stable/modules/corex.html#corex.rpc.pkg_summary</a><o:p></o:p></p>
<p>Then get the log messages from syslog printed by the command and send them over here.<o:p></o:p></p>
<p>Cheers,<br>
Daniel<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<p class="MsoNormal">On 19/09/16 14:02, Phil Lavin wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Hi all,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Kamailio stopped performing some functions earlier today, citing OOM errors. An excerpt from the log is below.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">SHM usage, at the time, looked fine. Also see below for that output.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">A restart of Kamailio resolved the issue. Is this SHM related or OS memory manager related?<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">root@ua-proxy-01:~# kamctl stats shmem</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">shmem:fragments = 910</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">shmem:free_size = 104131224</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">shmem:max_used_size = 51590800</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">shmem:real_used_size = 30086504</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">shmem:total_size = 134217728</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">shmem:used_size = 19558800</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">Sep 19 11:35:56 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:161]: insert_new_lump_before(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">Sep 19 11:40:10 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [msg_translator.c:2390]: build_res_buf_from_sip_req(): out of memory; needs 423</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">Sep 19 11:40:16 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [msg_translator.c:2390]: build_res_buf_from_sip_req(): out of memory; needs 453</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">Sep 19 11:42:46 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:136]: insert_new_lump_after(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">Sep 19 11:42:46 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [ut.c:163]: as_asciiz(): Out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">Sep 19 11:42:57 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [ut.c:163]: as_asciiz(): Out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:361]: anchor_lump(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:361]: anchor_lump(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:306]: del_lump(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:306]: del_lump(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [msg_translator.c:326]: received_builder(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [msg_translator.c:326]: received_builder(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [msg_translator.c:326]: received_builder(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">Sep 19 11:43:16 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:136]: insert_new_lump_after(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">Sep 19 11:43:16 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [ut.c:163]: as_asciiz(): Out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">Sep 19 11:44:18 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:161]: insert_new_lump_before(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">Sep 19 11:45:07 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [ut.c:163]: as_asciiz(): Out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">Sep 19 11:45:30 ua-proxy-01 /sbin/kamailio[20218]: ERROR: acc [acc_extra.c:267]: extra2strar(): extra2strar: out of memory.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">Sep 19 11:45:43 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:186]: insert_subst_lump_after(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif">Sep 19 11:45:52 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:161]: insert_new_lump_before(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New ,serif",serif"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:12.0pt"><br>
<br>
<br>
</span><o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list<o:p></o:p></pre>
<pre><a href="mailto:sr-users@lists.sip-router.org">sr-users@lists.sip-router.org</a><o:p></o:p></pre>
<pre><a href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
<pre>-- <o:p></o:p></pre>
<pre>Daniel-Constantin Mierla<o:p></o:p></pre>
<pre><a href="http://twitter.com/#%21/miconda">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda">http://www.linkedin.com/in/miconda</a><o:p></o:p></pre>
<pre>Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - <a href="http://www.asipto.com">http://www.asipto.com</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif;mso-fareast-language:EN-GB"><br>
<br>
<o:p></o:p></span></p>
<pre>-- <o:p></o:p></pre>
<pre>Daniel-Constantin Mierla<o:p></o:p></pre>
<pre><a href="http://twitter.com/#!/miconda">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda">http://www.linkedin.com/in/miconda</a><o:p></o:p></pre>
<pre>Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - <a href="http://www.asipto.com">http://www.asipto.com</a><o:p></o:p></pre>
</div>
</body>
</html>