<div dir="ltr">It seems located in tcp connection:<div><br></div><div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: summarizing all alloc'ed. fragments:</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 800 bytes from <core>: cfg/cfg_struct.c: cfg_clone_global(626)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 246 size= 9320 bytes from usrloc: urecord.c: new_urecord(65)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 250 size= 58000 bytes from usrloc: ucontact.c: new_ucontact(98)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1279 size= 51288 bytes from usrloc: ../../ut.h: shm_str_dup(691)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 8 size= 2048 bytes from tmx: tmx_pretran.c: tmx_check_pretran(271)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 8 size= 1088 bytes from tmx: tmx_pretran.c: tmx_check_pretran(250)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 246 size= 15856 bytes from usrloc: urecord.c: new_urecord(58)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 2555 size= 43782480 bytes from <core>: tcp_main.c: tcpconn_new(957)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 12800 bytes from <core>: counters.c: counters_prefork_init(207)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 3840 bytes from sl: sl_stats.c: init_sl_stats_child(125)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 256 bytes from tmx: tmx_pretran.c: tmx_init_pretran_table(90)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 5120 bytes from tm: t_stats.c: init_tm_stats_child(60)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 960 bytes from kex: pkg_stats.c: pkg_proc_stats_init(79)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 2 size= 64 bytes from <core>: cfg/cfg_struct.c: cfg_clone_str(130)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 3 size= 64 bytes from usrloc: udomain.c: build_stat_name(51)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 40960 bytes from usrloc: udomain.c: new_udomain(93)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 48 bytes from usrloc: udomain.c: new_udomain(86)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from usrloc: dlist.c: new_dlist(556)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 32 bytes from usrloc: dlist.c: new_dlist(548)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: pt.c: init_pt(110)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: pt.c: init_pt(105)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 2800 bytes from <core>: pt.c: init_pt(104)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 24 bytes from app_lua: ../../lock_alloc.h: lock_set_alloc(70)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from app_lua: app_lua_api.c: lua_sr_alloc_script_ver(90)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from app_lua: app_lua_api.c: lua_sr_alloc_script_ver(83)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 72 bytes from <core>: timer.c: register_timer(1011)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 32768 bytes from htable: ht_api.c: ht_init_tables(336)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 65536 bytes from tsilo: ts_hash.c: init_ts_table(105)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8208 bytes from tsilo: ../../lock_alloc.h: lock_set_alloc(70)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 32 bytes from tsilo: ts_hash.c: init_ts_table(75)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(920)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(914)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(906)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 96 bytes from rtpproxy: rtpproxy.c: insert_rtpp_node(563)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 48 bytes from rtpproxy: rtpproxy.c: get_rtpp_set(535)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from rtpproxy: rtpproxy.c: mod_init(911)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from nathelper: nathelper.c: mod_init(662)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 40 bytes from tm: t_hooks.c: insert_tmcb(137)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 32 bytes from usrloc: ul_callback.c: register_ulcb(94)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from usrloc: ul_callback.c: init_ulcb_list(45)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 4112 bytes from usrloc: ../../lock_alloc.h: lock_set_alloc(70)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from sl: sl_funcs.c: sl_startup(83)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from sl: sl_stats.c: init_sl_stats(110)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(74)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(72)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 2097152 bytes from tm: h_table.c: init_hash_table(450)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 2 size= 64 bytes from <core>: cfg/cfg_ctx.c: cfg_register_ctx(47)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8192 bytes from <core>: tcp_main.c: init_tcp(4635)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 32768 bytes from <core>: tcp_main.c: init_tcp(4629)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4621)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4614)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4607)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4601)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4589)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: usr_avp.c: init_avps(90)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: usr_avp.c: init_avps(89)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16384 bytes from <core>: dst_blacklist.c: init_dst_blacklist(437)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: dst_blacklist.c: init_dst_blacklist(430)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 2 size= 96 bytes from <core>: timer.c: timer_alloc(514)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: dns_cache.c: init_dns_cache(366)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16384 bytes from <core>: dns_cache.c: init_dns_cache(358)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from <core>: dns_cache.c: init_dns_cache(351)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: dns_cache.c: init_dns_cache(345)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(283)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16384 bytes from <core>: timer.c: init_timer(282)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(281)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(280)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(269)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(237)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 278544 bytes from <core>: timer.c: init_timer(220)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(219)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(206)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 64 bytes from <core>: cfg/cfg_struct.c: cfg_child_cb_new(830)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(361)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(354)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(347)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(335)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(323)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 72 bytes from mqueue: mqueue_api.c: mq_head_add(168)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 96 bytes from htable: ht_api.c: ht_add_table(275)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: mem/shm_mem.c: shm_mem_init_mallocs(179)</div><div>Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: -----------------------------</div></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">2015-11-20 12:49 GMT+01:00 Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
If you get the shm used size increasing, run following commands:<br>
<br>
kamcmd cfg.set_now_int core memlog 1<br>
kamcmd corex.shm_summary<br>
<br>
Do it after some time of running, when you see that the increase has
happened.<br>
<br>
Then take the logs from syslog related to shm summary and send them
over here.<br>
<br>
On the other hand, I saw recently some article about latency spikes
for tcp:<br>
<br>
- <a href="https://http2.cloudflare.com/the-story-of-one-latency-spike/" target="_blank">https://http2.cloudflare.com/the-story-of-one-latency-spike/</a><br>
<br>
It is about http, but actually relates to tcp setting of os. Maybe
you can verify your values in OS.<br>
<br>
Cheers,<br>
Daniel<div><div class="h5"><br>
<br>
<div>On 20/11/15 12:40, Thibault Gueslin
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><span style="font-size:13px">I am adding the info
in a mqueue.</span>
<div style="font-size:13px">In a single rtimer, I am consuming
the mqueue and calling a lua script.</div>
<div style="font-size:13px"><br>
</div>
<div style="font-size:13px">(I have followed the example of
twitter).</div>
<div style="font-size:13px"><br>
</div>
<div style="font-size:13px">I have increase the number of
tcp_children and I got issue for memory allocation on
allocations new tcp connection after a few hours.</div>
<div style="font-size:13px">Restart with SHMEM increased.</div>
<div style="font-size:13px">(but <font face="Menlo"><span style="font-size:11px">shame:used_size seems to increase)</span></font></div>
<div style="font-size:13px"><br>
</div>
<div style="font-size:13px">Regards</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2015-11-19 11:55 GMT+01:00
Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank"></a><a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Are you doing the
http requests for all sip requests? How many rtimer
processes do you have consuming from mqueue?<br>
<br>
What do you mean by "all tcp workers were busy"? They used
lot of CPU?<br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
<div>On 19/11/15 01:30, Thibault Gueslin wrote:<br>
</div>
<blockquote type="cite">
<div>It seems to appear randomly.</div>
<div><br>
</div>
<div>I don't have database access (default config
without db)</div>
<div>I have a mqueue with rtimer to do asynchronous
http request.</div>
<div><br>
</div>
<div>I have seen also sometimes that all tcp worker
were busy even on a test server with 4 clients.</div>
<div><br>
</div>
<div>Regards,</div>
<div><br>
</div>
<div>Thibault</div>
<div><br>
<br>
</div>
<div><br>
<div><br>
<br>
<br>
</div>
Le 18 nov. 2015 à 13:28, Daniel-Constantin Mierla
<<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>>
a écrit :<br>
<br>
</div>
<blockquote type="cite">
<div> Is it something repeating at constant
interval of times, or are they random? Are you
doing database queries in your config? Any of
them via rtimer module?<br>
<br>
Cheers,<br>
Daniel<br>
<br>
<br>
<div>On 18/11/15 12:00, Thibault Gueslin wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">I am facing an issue with
kamailio 4.3.3 that is very similar to
<div><a href="http://lists.sip-router.org/pipermail/sr-users/2015-May/088195.html" target="_blank">http://lists.sip-router.org/pipermail/sr-users/2015-May/088195.html</a><br>
</div>
<div><br>
</div>
<div>Had someone found the solution or faced
similar issue ?</div>
<div><br>
</div>
<div>Sometimes kamailio is receiving the
message (check with wireshark, logs in
tcp) but the message is not handled nor
relayed.</div>
<div>After a certain amount of time
everything works fine.</div>
<div><br>
</div>
<div>I will do what as has been advised :
enabling pike & check with gdb when it
appears.</div>
<div><br>
</div>
<div>thibault</div>
<div><br>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
<a href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a>
<a href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla
<a href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Book: SIP Routing With Kamailio - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
Kamailio Advanced Training, Nov 30-Dec 2, Berlin - <a href="http://asipto.com/kat" target="_blank">http://asipto.com/kat</a></pre>
</div>
</blockquote>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla
<a href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Book: SIP Routing With Kamailio - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
Kamailio Advanced Training, Nov 30-Dec 2, Berlin - <a href="http://asipto.com/kat" target="_blank">http://asipto.com/kat</a></pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla
<a href="http://twitter.com/#!/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Book: SIP Routing With Kamailio - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
Kamailio Advanced Training, Nov 30-Dec 2, Berlin - <a href="http://asipto.com/kat" target="_blank">http://asipto.com/kat</a></pre>
</div></div></div>
</blockquote></div><br></div>