<div dir="ltr">Hey Daniel,<div><br></div><div>Please check this commit. I think you are running before this was added?</div><div><br></div><div><div>commit 1d6b1d8765d7a6d21c291d2076a3a80a1710c11c</div><div>Author: Jason Penton <<a href="mailto:jason.penton@gmail.com">jason.penton@gmail.com</a>></div>
<div>Date: Fri Feb 14 09:57:45 2014 +0200</div><div><br></div><div> lib/ims:modules/ims_charging,ims_icscf,ims_qos,ims_registrar_scscf: fixed pkg_mem leak</div><div> - fixed leak when parsing msg headers on a shm msg</div>
<div> - instead now pass in flag to say if parsing is on shm_msg or not (is_shm)</div><div> - assumes all common headers are already parsed before pkg_mem msg is cloned</div></div><div><br></div><div><br></div>
<div>Cheers</div><div>Jason</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Mar 3, 2014 at 11:22 PM, Jason Penton <span dir="ltr"><<a href="mailto:jason.penton@smilecoms.com" target="_blank">jason.penton@smilecoms.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks Daniel,<div><br></div><div>How much pkg and shm memory is assigned to your Kamailio instance? Also, how many users are you registering concurrently? Please also recreate and dump shm memory when you get a chance. If you are not sure how to dump shm memory - see here - <a href="http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory" target="_blank">http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory</a></div>
<div><br></div><div>Cheers</div><span class="HOEnZb"><font color="#888888"><div>Jason</div></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Mar 3, 2014 at 11:07 PM, Daniel Ciprus <span dir="ltr"><<a href="mailto:daniel.ciprus@acision.com" target="_blank">daniel.ciprus@acision.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Jason, <br>
<br>
HW configuration : this is a virtual machine running 4 cores with 24GB ram (VM hosting).
<br>
<br>
processor : 3<br>
vendor_id : GenuineIntel<br>
cpu family : 6<br>
model : 26<br>
model name : Intel(R) Xeon(R) CPU E5-2667 0 @ 2.90GHz<br>
stepping : 4<br>
cpu MHz : 2900.000<br>
cache size : 15360 KB<br>
fpu : yes<br>
fpu_exception : yes<br>
cpuid level : 11<br>
wp : yes<br>
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni ssse3 cx16 sse4_1
sse4_2 popcnt hypervisor lahf_lm ida arat epb pln pts dts<br>
bogomips : 5800.00<br>
clflush size : 64<br>
cache_alignment : 64<br>
address sizes : 40 bits physical, 48 bits virtual<br>
power management:<br>
<br>
As far as scenario: it's a mixture of REGISTER/MESSAGE/SUBSCRIBE/NOTIFY. I will send logs (configs + /var/log/messages) privately .. but generally speaking config is almost identical with configs from samples directory for p-cscf.
<br>
<br>
Dan<div><div><br>
<br>
<br>
<div>On 03/03/2014 04:02 PM, Jason Penton wrote:<br>
</div>
</div></div><blockquote style="border-left:2px solid #009900!important;border-right:2px solid #009900!important;padding:0px 15px 0px 15px;margin:8px 2px" type="cite"><div><div>
<div dir="ltr">Hi Daniel,
<div><br>
</div>
<div>no we have not seen that... Can you provide details around the scenario - registration, calling, mixture, etc? Load? We have done extensive testing especially w.r.t. memory usage so I am very interested in this. Could you do a memory dump and send us the
log?</div>
<div><br>
</div>
<div>Also, will you share your cfg file? you can send the cfg file directly to me if you prefer...</div>
<div><br>
</div>
<div>Cheers</div>
<div>Jason</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Mon, Mar 3, 2014 at 10:58 PM, Daniel Ciprus <span dir="ltr">
<<a href="mailto:daniel.ciprus@acision.com" target="_blank">daniel.ciprus@acision.com</a>></span> wrote:<br>
<blockquote class="gmail_quote">
<div>Hi, <br>
<br>
Has anybody seen this : <br>
<br>
Mar 2 17:25:16 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2011]: generate_res_buf_from_sip_res():
<b>out of mem</b><br>
Mar 2 17:25:16 kamailio kam-pcscf[20741]: ERROR: tm [t_reply.c:1943]: relay_reply(): ERROR: relay_reply: no mem for outbound reply buffer<br>
<br>
followed by: <br>
<br>
Mar 2 17:27:03 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2011]: generate_res_buf_from_sip_res():
<b>out of mem</b><br>
Mar 2 17:27:03 kamailio kam-pcscf[20741]: ERROR: tm [t_reply.c:1943]: relay_reply(): ERROR: relay_reply: no mem for outbound reply buffer<br>
Mar 2 17:27:03 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2168]: build_res_buf_from_sip_req():
<b>ERROR: build_res_buf_from_sip_req: out of memory ; needs 386</b><br>
<br>
Mar 2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/contact.c:194]: parse_contacts(): parse_contacts():
<b>No memory left</b><br>
Mar 2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/parse_contact.c:59]: contact_parser(): contact_parser(): Error while parsing contacts<br>
Mar 2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/parse_contact.c:88]: parse_contact(): parse_contact(): Error while parsing<br>
<br>
<br>
However, seems like we still had enough mem: <br>
<br>
[root@kamailio kamailio]# vmstat 1<br>
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----<br>
r b swpd free buff cache si so bi bo in cs us sy id wa st<br>
3 0 0 9526348 230308 13282472 0 0 0 4 0 1 6 0 94 0 0<br>
1 0 0 9526216 230308 13282524 0 0 0 0 1313 84424 17 8 75 0 0<br>
1 0 0 9526092 230308 13282616 0 0 0 0 1482 86882 16 9 75 0 0<br>
<br>
[root@kamailio monit.d]# kamailio -v<br>
version: kamailio 4.2.0-dev0 (x86_64/linux) <br>
flags: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES<br>
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB<br>
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.<br>
id: unknown <br>
compiled on 17:42:42 Dec 18 2013 with gcc 4.4.7<br>
<br>
Unfortunately I had to reload p-cscf to get service working again. <br>
<br>
<div>-- <br>
<b>Daniel Ciprus</b><br>
Integration engineer<br>
<a href="http://www.acision.com" target="_blank">http://www.acision.com</a><br>
<br>
9954 Mayland Dr<br>
Suite 3100<br>
Richmond, VA 23233<br>
USA<br>
T: <a href="tel:%2B1%20804%20762%205601" value="+18047625601" target="_blank">
+1 804 762 5601</a><br>
E: <a href="mailto:daniel.ciprus@acision.com" target="_blank">
daniel.ciprus@acision.com</a><br>
</div>
<br>
<hr>
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other
party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.<br>
<br>
</div>
<br>
_______________________________________________<br>
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list<br>
<a href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a><br>
<a href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><br>
<br>
</blockquote>
</div>
<br>
<br>
<div><br>
</div>
-- <br>
<div dir="ltr">
<div>
<table width="600">
<tbody>
<tr>
<td><br>
</td>
</tr>
</tbody>
</table>
<table width="320px">
<tbody>
<tr>
<td><strong>Jason Penton</strong></td>
</tr>
<tr>
<td><strong>Senior Manager: Applications and Services</strong></td>
</tr>
<tr>
<td><strong>Smile Communications Pty (Ltd)</strong></td>
</tr>
<tr>
<td>
<table width="100%">
<tbody>
<tr>
<td width="27%"><strong>Mobile:</strong></td>
<td width="73%"><a href="tel:%2B27%20%280%29%2083%20283%207000" value="+27832837000" target="_blank">+27 (0) 83 283 7000</a></td>
</tr>
<tr>
<td><strong>Skype:</strong></td>
<td>jason.barry.penton</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td><a href="mailto:name.surname@smilecoms.com" target="_blank">jason.penton@smilecoms.com</a></td>
</tr>
<tr>
<td><a href="http://www.smilecoms.com/" target="_blank">www.smilecoms.com</a><br>
<img height="144" width="420"><br>
</td>
</tr>
</tbody>
</table>
</div>
<br>
</div>
</div>
<br>
</div></div><span>
<pre>This email is subject to the disclaimer of Smile Communications at <a href="http://www.smilecoms.com/home/email-disclaimer/" target="_blank">http://www.smilecoms.com/home/<u></u>email-disclaimer/</a></pre>
</span></blockquote><div>
<br>
<div>-- <br>
<b>Daniel Ciprus</b><br>
Integration engineer<br>
<a href="http://www.acision.com" target="_blank">http://www.acision.com</a><br>
<br>
9954 Mayland Dr<br>
Suite 3100<br>
Richmond, VA 23233<br>
USA<br>
T: <a href="tel:%2B1%20804%20762%205601" value="+18047625601" target="_blank">+1 804 762 5601</a><br>
E: <a href="mailto:daniel.ciprus@acision.com" target="_blank">daniel.ciprus@acision.com</a><br>
</div>
<br>
<hr>
<font face="Arial" color="Gray" size="1">This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed
to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.<br>
</font>
<br clear="all">
</div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><div><table border="0" cellspacing="0" cellpadding="0" width="600" style="width:450pt"><tbody><tr><td valign="top" style="padding:0cm"><p></p></td>
</tr></tbody></table><table width="320px" border="0" cellpadding="0" cellspacing="0" style="font-family:monospace"><tbody><tr><td style="font-family:Arial,sans-serif;font-size:16px;color:rgb(117,179,67);line-height:18px;padding-left:15px;padding-top:10px">
<strong>Jason Penton</strong></td></tr><tr><td style="font-family:Arial,sans-serif;font-size:16px;color:rgb(117,179,67);line-height:18px;padding-left:15px;padding-top:10px"><strong>Senior Manager: Applications and Services</strong></td>
</tr><tr><td style="font-family:Arial,sans-serif;font-size:16px;color:rgb(134,145,154);line-height:18px;padding-left:15px;padding-top:10px"><strong>Smile Communications Pty (Ltd)</strong></td></tr><tr><td><table cellpadding="0" cellspacing="0" width="100%">
<tbody><tr><td width="27%" style="font-family:Arial,sans-serif;font-size:16px;color:rgb(134,145,154);line-height:18px;padding-left:15px;padding-top:10px"><strong>Mobile:</strong></td><td width="73%" style="font-family:Arial,sans-serif;font-size:16px;color:rgb(134,145,154);line-height:18px;padding-left:10px;padding-top:10px">
<a href="tel:%2B27%20%280%29%2083%20283%207000" value="+27832837000" target="_blank">+27 (0) 83 283 7000</a></td></tr><tr><td style="font-family:Arial,sans-serif;font-size:16px;color:rgb(134,145,154);line-height:18px;padding-left:15px;padding-top:10px">
<strong>Skype:</strong></td><td style="font-family:Arial,sans-serif;font-size:16px;color:rgb(134,145,154);line-height:18px;padding-left:10px;padding-top:10px">
jason.barry.penton</td></tr></tbody></table></td></tr><tr><td style="font-family:Arial,sans-serif;font-size:16px;color:rgb(117,179,67);line-height:18px;padding-left:15px;padding-top:10px"><a href="mailto:name.surname@smilecoms.com" style="font-family:Arial;color:rgb(117,179,67);text-decoration:none" target="_blank">jason.penton@smilecoms.com</a></td>
</tr><tr><td style="font-family:Arial,sans-serif;font-size:16px;color:rgb(117,179,67);line-height:18px;padding-left:15px;padding-top:10px"><a href="http://www.smilecoms.com/" style="font-family:Arial;color:rgb(117,179,67)" target="_blank">www.smilecoms.com</a><br>
<img src="http://196.33.227.129/~smlcoms/sigs/pty/images/smile_signature_07_09.jpg" width="420" height="144"><br></td></tr></tbody></table></div><br></div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><div><table border="0" cellspacing="0" cellpadding="0" width="600" style="width:450pt"><tbody><tr><td valign="top" style="padding:0cm">
<p></p></td></tr></tbody></table><table width="320px" border="0" cellpadding="0" cellspacing="0" style="font-family:monospace"><tbody><tr><td style="font-family:Arial,sans-serif;font-size:16px;color:rgb(117,179,67);line-height:18px;padding-left:15px;padding-top:10px">
<strong>Jason Penton</strong></td></tr><tr><td style="font-family:Arial,sans-serif;font-size:16px;color:rgb(117,179,67);line-height:18px;padding-left:15px;padding-top:10px"><strong>Senior Manager: Applications and Services</strong></td>
</tr><tr><td style="font-family:Arial,sans-serif;font-size:16px;color:rgb(134,145,154);line-height:18px;padding-left:15px;padding-top:10px"><strong>Smile Communications Pty (Ltd)</strong></td></tr><tr><td><table cellpadding="0" cellspacing="0" width="100%">
<tbody><tr><td width="27%" style="font-family:Arial,sans-serif;font-size:16px;color:rgb(134,145,154);line-height:18px;padding-left:15px;padding-top:10px"><strong>Mobile:</strong></td><td width="73%" style="font-family:Arial,sans-serif;font-size:16px;color:rgb(134,145,154);line-height:18px;padding-left:10px;padding-top:10px">
+27 (0) 83 283 7000</td></tr><tr><td style="font-family:Arial,sans-serif;font-size:16px;color:rgb(134,145,154);line-height:18px;padding-left:15px;padding-top:10px"><strong>Skype:</strong></td><td style="font-family:Arial,sans-serif;font-size:16px;color:rgb(134,145,154);line-height:18px;padding-left:10px;padding-top:10px">
jason.barry.penton</td></tr></tbody></table></td></tr><tr><td style="font-family:Arial,sans-serif;font-size:16px;color:rgb(117,179,67);line-height:18px;padding-left:15px;padding-top:10px"><a href="mailto:name.surname@smilecoms.com" style="font-family:Arial;color:rgb(117,179,67);text-decoration:none" target="_blank">jason.penton@smilecoms.com</a></td>
</tr><tr><td style="font-family:Arial,sans-serif;font-size:16px;color:rgb(117,179,67);line-height:18px;padding-left:15px;padding-top:10px"><a href="http://www.smilecoms.com/" style="font-family:Arial;color:rgb(117,179,67)" target="_blank">www.smilecoms.com</a><br>
<img src="http://196.33.227.129/~smlcoms/sigs/pty/images/smile_signature_07_09.jpg" width="420" height="144"><br></td></tr></tbody></table></div><br></div>
</div>
<br>
<span><font color="#888888"><pre>This email is subject to the disclaimer of Smile Communications at <a href="http://www.smilecoms.com/home/email-disclaimer/" target="_blank">http://www.smilecoms.com/home/<WBR>email-disclaimer/</a><a href="http://www.smilecoms.com/disclaimer" target="_blank"></a></pre></font></span>