<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>The rpc command is in 4.4, what version are you using?</p>
<p>Cheers,<br>
Daniel<br>
</p>
<br>
<div class="moz-cite-prefix">On 19/09/16 18:11, Phil Lavin wrote:<br>
</div>
<blockquote
cite="mid:VI1PR03MB184015043F40E00855FCBDD289F40@VI1PR03MB1840.eurprd03.prod.outlook.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;
mso-fareast-language:EN-US;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New",serif;
color:black;
mso-fareast-language:EN-GB;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;
mso-fareast-language:EN-US;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;
mso-fareast-language:EN-GB;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style="color:windowtext">Having
adding some graphing, I see this memory issue on another
production Kamailio server. A few of the processes are using
significantly more pkg memory than others. What I assume is
happening is that memory is leaking and will eventually
start hitting OOM errors. See <a moz-do-not-send="true"
href="https://gist.github.com/phil-lavin/27f63b472ace53dc956125a47d210bc9">https://gist.github.com/phil-lavin/27f63b472ace53dc956125a47d210bc9</a>
for pkg memory stats output.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">When was
corex.pkg_summary added? I’m getting not found error:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif;color:windowtext">error: 500 - command
corex.pkg_summary not found<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span
style="color:windowtext;mso-fareast-language:EN-GB"
lang="EN-US">From:</span></b><span
style="color:windowtext;mso-fareast-language:EN-GB"
lang="EN-US"> sr-users
[<a class="moz-txt-link-freetext" href="mailto:sr-users-bounces@lists.sip-router.org">mailto:sr-users-bounces@lists.sip-router.org</a>]
<b>On Behalf Of </b>Phil Lavin<br>
<b>Sent:</b> 19 September 2016 14:36<br>
<b>To:</b> <a class="moz-txt-link-abbreviated" href="mailto:miconda@gmail.com">miconda@gmail.com</a>; Kamailio (SER) - Users
Mailing List <a class="moz-txt-link-rfc2396E" href="mailto:sr-users@lists.sip-router.org"><sr-users@lists.sip-router.org></a><br>
<b>Subject:</b> Re: [SR-Users] OOM Errors<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:windowtext">Thanks,
Daniel.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">Kamailio has
been restarted now so there’s not much of any use in the pkg
memory output.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">I’m going to
add some monitoring/graphing on this and see if there’s an
upward trend in usage.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span
style="color:windowtext;mso-fareast-language:EN-GB"
lang="EN-US">From:</span></b><span
style="color:windowtext;mso-fareast-language:EN-GB"
lang="EN-US"> sr-users [<a moz-do-not-send="true"
href="mailto:sr-users-bounces@lists.sip-router.org">mailto:sr-users-bounces@lists.sip-router.org</a>]
<b>On Behalf Of </b>Daniel-Constantin Mierla<br>
<b>Sent:</b> 19 September 2016 13:41<br>
<b>To:</b> Kamailio (SER) - Users Mailing List <<a
moz-do-not-send="true"
href="mailto:sr-users@lists.sip-router.org">sr-users@lists.sip-router.org</a>><br>
<b>Subject:</b> Re: [SR-Users] OOM Errors<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p>Hello,<span
style="font-size:12.0pt;mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p>it seems that the pkg is out of free space, not shm. You can
check it with:<o:p></o:p></p>
<p>kamcmd pkg.stats<o:p></o:p></p>
<p>and look for the same pid that printed the log messages.<o:p></o:p></p>
<p>To troubleshoot, it is good to print the pkg summary, see:<o:p></o:p></p>
<p> - <a moz-do-not-send="true"
href="https://www.kamailio.org/docs/modules/stable/modules/corex.html#corex.rpc.pkg_summary">https://www.kamailio.org/docs/modules/stable/modules/corex.html#corex.rpc.pkg_summary</a><o:p></o:p></p>
<p>Then get the log messages from syslog printed by the command
and send them over here.<o:p></o:p></p>
<p>Cheers,<br>
Daniel<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 19/09/16 14:02, Phil Lavin wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Hi all,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Kamailio stopped performing some
functions earlier today, citing OOM errors. An excerpt from
the log is below.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">SHM usage, at the time, looked fine. Also
see below for that output.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">A restart of Kamailio resolved the issue.
Is this SHM related or OS memory manager related?<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">root@ua-proxy-01:~# kamctl stats shmem</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">shmem:fragments = 910</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">shmem:free_size = 104131224</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">shmem:max_used_size = 51590800</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">shmem:real_used_size = 30086504</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">shmem:total_size = 134217728</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">shmem:used_size = 19558800</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">Sep 19 11:35:56 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[data_lump.c:161]: insert_new_lump_before(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">Sep 19 11:40:10 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[msg_translator.c:2390]: build_res_buf_from_sip_req(): out
of memory; needs 423</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">Sep 19 11:40:16 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[msg_translator.c:2390]: build_res_buf_from_sip_req(): out
of memory; needs 453</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">Sep 19 11:42:46 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[data_lump.c:136]: insert_new_lump_after(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">Sep 19 11:42:46 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core> [ut.c:163]:
as_asciiz(): Out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">Sep 19 11:42:57 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core> [ut.c:163]:
as_asciiz(): Out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">Sep 19 11:43:12 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[data_lump.c:361]: anchor_lump(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">Sep 19 11:43:12 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[data_lump.c:361]: anchor_lump(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">Sep 19 11:43:12 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[data_lump.c:306]: del_lump(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">Sep 19 11:43:12 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[data_lump.c:306]: del_lump(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">Sep 19 11:43:12 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[msg_translator.c:326]: received_builder(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">Sep 19 11:43:12 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[msg_translator.c:326]: received_builder(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">Sep 19 11:43:12 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[msg_translator.c:326]: received_builder(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">Sep 19 11:43:16 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[data_lump.c:136]: insert_new_lump_after(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">Sep 19 11:43:16 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core> [ut.c:163]:
as_asciiz(): Out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">Sep 19 11:44:18 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[data_lump.c:161]: insert_new_lump_before(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">Sep 19 11:45:07 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core> [ut.c:163]:
as_asciiz(): Out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">Sep 19 11:45:30 ua-proxy-01
/sbin/kamailio[20218]: ERROR: acc [acc_extra.c:267]:
extra2strar(): extra2strar: out of memory.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">Sep 19 11:45:43 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[data_lump.c:186]: insert_subst_lump_after(): out of
memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif">Sep 19 11:45:52 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[data_lump.c:161]: insert_new_lump_before(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif;mso-fareast-language:EN-GB"><br>
<br>
<o:p></o:p></span></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list<o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="mailto:sr-users@lists.sip-router.org">sr-users@lists.sip-router.org</a><o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif;mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<pre>-- <o:p></o:p></pre>
<pre>Daniel-Constantin Mierla<o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="http://twitter.com/#%21/miconda">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" href="http://www.linkedin.com/in/miconda">http://www.linkedin.com/in/miconda</a><o:p></o:p></pre>
<pre>Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - <a moz-do-not-send="true" href="http://www.asipto.com">http://www.asipto.com</a><o:p></o:p></pre>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla
<a class="moz-txt-link-freetext" href="http://twitter.com/#!/miconda">http://twitter.com/#!/miconda</a> - <a class="moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - <a class="moz-txt-link-freetext" href="http://www.asipto.com">http://www.asipto.com</a></pre>
</body>
</html>