<h4>Use Case Description:</h4>
<p>I'm trying to log "user last register status" using function <code>jsonrpc_notification</code> from module <code>jsonrpc-c</code></p>
<h5>invoke in kamailio.cfg:</h5>
<p><code>jsonrpc_notification("last_reg", "{'user': '$au', 'data':['$Ts', '$au', '$ct', '$ua', '$sel(received.proto_ip_port)']}");</code></p>
<p><strong>according to <a href="http://kamailio.org/docs/modules/4.3.x/modules/jsonrpc-c.html#idp2463944">doc</a>:</strong><br>
jsonrpc_notification is <em>asynchronous delayed job</em> and <em>do not receive response</em></p>
<blockquote>
<p>notifications do not receive a response. Script processing continues in the usual fashion as soon as the notification has been sent.</p>
</blockquote>
<p>The rpc server on localhost is implemented with Node.js which never return responses and never close connections ( it only support jsonrpc notification )</p>
<h4>Observation and Test:</h4>
<h5>Crash occurs</h5>
<p>I found kamailio crashed periodically about 2-3 days. ( back trace attached at the end)</p>
<p><strong>settings:</strong><br>
1. the amount of total registered users are about 15k ~ 50k; register expiration is 1200 Secs.<br>
2. kamailio start with parameter pkg 1G and shm 5G.<br>
3. I had patched jsonrpc_io.c from <a href="https://github.com/kamailio/kamailio/pull/268" class="issue-link" title="jsonrpc-c: fix crash at reconnect_cb if server was never connected">#268</a> </p>
<p><strong>json crash log:</strong></p>
<pre><code>Aug 18 22:12:04 SIP-4 kernel: [50538943.565484] kamailio[84670]: segfault at 0 ip 00007f424a3ea3bf sp 00007fff8b3eab60 error 4 in libjson.so.0.1.0[7f424a3e6000+9000]
Aug 18 22:12:23 SIP-4 /home/pkg/kamailio/sbin/kamailio[84620]: ALERT: <core> [main.c:728]: handle_sigs(): child process 84670 exited by a signal 11
Aug 18 22:12:23 SIP-4 /home/pkg/kamailio/sbin/kamailio[84620]: ALERT: <core> [main.c:731]: handle_sigs(): core was generated
</code></pre>
<p>sometimes it would not crash, but result in allocation failure. no new connections can be accept.</p>
<pre><code>Aug 20 12:26:04 SIP-5 /home/pkg/kamailio/sbin/kamailio[181104]: ERROR: <core> [tcp_main.c:959]: tcpconn_new(): mem. allocation failure
Aug 20 12:26:04 SIP-5 /home/pkg/kamailio/sbin/kamailio[181104]: ERROR: <core> [tcp_main.c:3977]: handle_new_connect(): tcpconn_new failed, closing socket
</code></pre>
<h5>Test</h5>
<p>use <code>sip-p</code> to start a stress test with RPS 2000 register and observing results both from <code>kamcmd core.shmmem</code> and <code>top</code> command</p>
<p>Kamailio died quickly along with depleting shm memory.<br>
<code>top</code> shows kamailio SHR consumed 4.8G<br>
<code>kamcmd core.shmmem</code> shows <strong>free</strong> is less than 1MB </p>
<h5>the more <code>jsonrpc_notification</code> call we've made, the quicker kamailio out of memory.</h5>
<h4>MEMDBG</h4>
<p>switch on memdbg and shutdown kamailio to get mem status.<br>
see Memory still-in-use summary (shm):<br>
<code>shm_strdup</code> and <code>jsonrpc_notification</code> from jsonrpc_request.c</p>
<pre><code>Aug 24 14:39:41 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: -----------------------------
Aug 24 14:39:41 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : <core> [main.c:569]: cleanup(): Memory still-in-use summary (shm):
Aug 24 14:39:41 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: summarizing all alloc'ed. fragments:
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 96 size= 4608 bytes from dialog: dlg_var.c: new_dlg_var(97)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 96 size= 2016 bytes from dialog: dlg_var.c: new_dlg_var(106)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 96 size= 1152 bytes from dialog: dlg_var.c: new_dlg_var(116)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 40 size= 10568 bytes from tmx: tmx_pretran.c: tmx_check_pretran(264)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 40 size= 5520 bytes from tmx: tmx_pretran.c: tmx_check_pretran(243)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 993 size= 7960 bytes from dialog: dlg_handlers.c: dlg_get_iuid_shm_clone(279)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count=997818 size= 113547928 bytes from jsonrpc-c: jsonrpc_request.c: shm_strdup(158)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1 size= 1024 bytes from tmx: tmx_pretran.c: tmx_init_pretran_table(90)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 3 size= 64 bytes from usrloc: udomain.c: build_stat_name(51)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1 size= 8 bytes from <core>: pt.c: init_pt(110)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1 size= 8 bytes from <core>: pt.c: init_pt(105)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count=498909 size= 40377480 bytes from jsonrpc-c: jsonrpc_request.c: jsonrpc_notification(135)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1 size= 8 bytes from dialog: dlg_hash.c: init_dlg_table(269)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1 size= 8 bytes from dialog: dlg_hash.c: init_dlg_table(262)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1 size= 8 bytes from dialog: dlg_hash.c: init_dlg_table(257)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 3 size= 216 bytes from <core>: timer.c: register_timer(1011)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1 size= 8 bytes from <core>: usr_avp.c: init_avps(90)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1 size= 8 bytes from <core>: usr_avp.c: init_avps(89)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1 size= 8 bytes from <core>: mem/shm_mem.c: shm_mem_init_mallocs(179)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: -----------------------------
</code></pre>
<h4>crash on jsonrpc back trace</h4>
<pre><code>#0 0x00007f424a3ea3bf in json_tokener_parse_ex () from /lib/x86_64-linux-gnu/libjson.so.0
No symbol table info available.
#1 0x00007f424a3ebda6 in json_tokener_parse_verbose () from /lib/x86_64-linux-gnu/libjson.so.0
No symbol table info available.
#2 0x00007f424a3ebdfe in json_tokener_parse () from /lib/x86_64-linux-gnu/libjson.so.0
No symbol table info available.
#3 0x00007f424a5f4e1a in cmd_pipe_cb (fd=6, event=2, arg=0x7fff8b3eae60) at jsonrpc_io.c:152
cmd = 0x7f4304a93cf8
__FUNCTION__ = "cmd_pipe_cb"
params = 0x1bc0aa0
payload = 0x3985b4d6
req = 0x304a9c5
json = 0x7fff8b3ead00 "0\f\274\001"
ns = 0x7f4391ebdc60 "298:{ \"jsonrpc\": \"2.0\", \"method\": \"last_reg\", \"params\": { \"user\": \"597438910\", \"data\": [ \"1439907102\", \"597438910\", \"<sip:597438910@211.246.212.119:55430;transport=tcp>;+sip.instance=\\\"<urn:uuid:f1016"...
bytes = 140735529528624
g = 0x7f424a1ac2f8
sent = 0
#4 0x00007f424a1af254 in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
No symbol table info available.
#5 0x00007f424a5f45ef in jsonrpc_io_child_process (cmd_pipe=6, _servers=0x7f4391aba578 "10.1.3.9:5013,3 10.1.3.10:5013,2 10.1.3.11:5013,1")
at jsonrpc_io.c:92
__FUNCTION__ = "jsonrpc_io_child_process"
pipe_ev = {ev_active_next = {tqe_next = 0x0, tqe_prev = 0x1bb9dc0}, ev_next = {tqe_next = 0x7f4391ebda20, tqe_prev = 0x1bc0c00},
ev_timeout_pos = {ev_next_with_common_timeout = {tqe_next = 0xffffffff, tqe_prev = 0x0}, min_heap_idx = -1}, ev_fd = 6,
ev_base = 0x1bc0aa0, _ev = {ev_io = {ev_io_next = {tqe_next = 0x0, tqe_prev = 0x1bc0fd0}, ev_timeout = {tv_sec = 0, tv_usec = 0}},
ev_signal = {ev_signal_next = {tqe_next = 0x0, tqe_prev = 0x1bc0fd0}, ev_ncalls = 0, ev_pncalls = 0x0}}, ev_events = 18,
ev_res = 2, ev_flags = 130, ev_pri = 0 '\000', ev_closure = 2 '\002', ev_timeout = {tv_sec = 140733193388081, tv_usec = 6631108320},
ev_callback = 0x7f424a5f4aa7 <cmd_pipe_cb>, ev_arg = 0x7fff8b3eae60}
#6 0x00007f424a5fd50b in child_init (rank=0) at jsonrpc_mod.c:135
pid = 0
#7 0x0000000000596a41 in init_mod_child (m=0x7f4391ab0db8, rank=0) at sr_module.c:898
__FUNCTION__ = "init_mod_child"
#8 0x0000000000596779 in init_mod_child (m=0x7f4391ab1110, rank=0) at sr_module.c:895
__FUNCTION__ = "init_mod_child"
#9 0x0000000000596d55 in init_child (rank=0) at sr_module.c:924
No locals.
#10 0x00000000004aafd3 in main_loop () at main.c:1650
i = 24
pid = 84647
si = 0x0
si_desc = "udp receiver child=23 sock=38.106.58.4:5063\000n\000\000\000~/q\000\001\000\000\000\260\344\213OB\177\000\000\020\261>\213\377\177\000\000\032\264N\000\000\000\000\000`\261>\213\377\177\000\000\000\224\351\221C\177\000\000 \333\330OB\177\000\000\260\344\213OB\177\000\000$!\314\a\000\000\000\000\260\210A\000\001\000\000"
nrprocs = 24
__FUNCTION__ = "main_loop"
#11 0x00000000004b0b5b in main (argc=13, argv=0x7fff8b3eb3d8) at main.c:2533
cfg_stream = 0x1b1a010
c = -1
r = 0
tmp = 0x7fff8b3ecd59 ""
tmp_len = 32579
port = -767281296
proto = 0
options = 0x707db8 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 189487301
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x76
p = 0x7fff8b3eb2de ""
__FUNCTION__ = "main"
quit
</code></pre>
<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br>Reply to this email directly or <a href="https://github.com/kamailio/kamailio/issues/296">view it on GitHub</a>.<img alt="" height="1" src="https://github.com/notifications/beacon/AF36ZSID6Ttf6ZxtiZVRKDKdhjQfuF5wks5oqslFgaJpZM4Fw0dn.gif" width="1" /></p>
<div itemscope itemtype="http://schema.org/EmailMessage">
<div itemprop="action" itemscope itemtype="http://schema.org/ViewAction">
<link itemprop="url" href="https://github.com/kamailio/kamailio/issues/296"></link>
<meta itemprop="name" content="View Issue"></meta>
</div>
<meta itemprop="description" content="View this Issue on GitHub"></meta>
</div>