[sr-dev] [kamailio] DMQ under high load kills Kamailio (#399)
Sebastian Damm
notifications at github.com
Tue Nov 17 15:10:32 CET 2015
We started a Kamailio handling about 30k registrations with dmq_usrloc module enabled (only sending the traffic to one other host). It crashes almost right away.
That's what the log says for the instance. You see, there are not only about 45 seconds from starting up to crashing.
```
root at pleisse:~# grep '\[35833\]' /var/log/kamailio/proxy.log
Nov 17 14:46:40 pleisse /usr/sbin/kamailio[35833]: WARNING: tm [t_funcs.c:384]: init_avp_params(): using AVP for TM fr_timer is deprecated, use t_set_fr(...) instead
Nov 17 14:46:40 pleisse /usr/sbin/kamailio[35833]: WARNING: tm [t_funcs.c:416]: init_avp_params(): using AVP for TM fr_inv_timer is deprecated, use t_set_fr(...) instead
Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: ALERT: <core> [main.c:728]: handle_sigs(): child process 35911 exited by a signal 6
Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: ALERT: <core> [main.c:731]: handle_sigs(): core was generated
Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: ERROR: tm [uac.c:296]: t_uac_prepare(): t_uac: short of cell shmem
Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: ERROR: dmq [dmq_funcs.c:243]: dmq_send_message(): error in tmb.t_request_within
Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: : <core> [mem/q_malloc.c:453]: qm_free(): BUG: qm_free: freeing already freed pointer (0x7f3c896c11c0), called from <core>: parser/parse_param.c: do_free_params(623), first free <core>: parser/parse_param.c: do_free_params(623) - aborting
```
We first thought, it is a problem with shm, so we doubled it (from 512 to 1024M), but the problem stays.
Those are the last lines from the log.
```
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35911]: ERROR: tm [uac.c:296]: t_uac_prepare(): t_uac: short of cell shmem
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35911]: ERROR: dmq [dmq_funcs.c:243]: dmq_send_message(): error in tmb.t_request_within
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35838]: ERROR: <core> [sip_msg_clone.c:497]: sip_msg_shm_clone(): cannot allocate memory
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35838]: ERROR: dmq [worker.c:164]: add_dmq_job(): error cloning sip message
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35912]: ERROR: tm [uac.c:296]: t_uac_prepare(): t_uac: short of cell shmem
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35838]: ERROR: dmq [message.c:67]: dmq_handle_message(): failed to add dmq job
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35912]: ERROR: dmq [dmq_funcs.c:243]: dmq_send_message(): error in tmb.t_request_within
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35871]: ERROR: <core> [sip_msg_clone.c:497]: sip_msg_shm_clone(): cannot allocate memory
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35871]: ERROR: dmq [worker.c:164]: add_dmq_job(): error cloning sip message
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35911]: ERROR: tm [uac.c:296]: t_uac_prepare(): t_uac: short of cell shmem
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35871]: ERROR: dmq [message.c:67]: dmq_handle_message(): failed to add dmq job
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35911]: ERROR: dmq [dmq_funcs.c:243]: dmq_send_message(): error in tmb.t_request_within
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35909]: ERROR: tm [uac.c:296]: t_uac_prepare(): t_uac: short of cell shmem
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35910]: ERROR: tm [uac.c:296]: t_uac_prepare(): t_uac: short of cell shmem
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35911]: : <core> [mem/q_malloc.c:453]: qm_free(): BUG: qm_free: freeing already freed pointer (0x7f3c896c11c0), called from <core>: parser
/parse_param.c: do_free_params(623), first free <core>: parser/parse_param.c: do_free_params(623) - aborting
Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: ALERT: <core> [main.c:728]: handle_sigs(): child process 35911 exited by a signal 6
Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: ALERT: <core> [main.c:731]: handle_sigs(): core was generated
Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: ERROR: tm [uac.c:296]: t_uac_prepare(): t_uac: short of cell shmem
Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: ERROR: dmq [dmq_funcs.c:243]: dmq_send_message(): error in tmb.t_request_within
Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: : <core> [mem/q_malloc.c:453]: qm_free(): BUG: qm_free: freeing already freed pointer (0x7f3c896c11c0), called from <core>: parser/parse_param.c: do_free_params(623), first free <core>: parser/parse_param.c: do_free_params(623) - aborting
```
This is the core dump of process 35833:
```
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/kamailio...Reading symbols from /usr/lib/debug/.build-id/c3/a4fc0cef8da17048bce2dc080e543509e39d37.debug...done.
done.
[New LWP 35833]
warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio_sip_proxy.cfg -P /var/run/kamailio'.
Program terminated with signal 6, Aborted.
#0 0x00007f3ccc4a6165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
Thread 1 (Thread 0x7f3ccce2e700 (LWP 35833)):
#0 0x00007f3ccc4a6165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f3ccc4a93e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00000000005f29ae in qm_free (qm=<optimized out>, p=<optimized out>, file=<optimized out>, func=<optimized out>, line=623) at mem/q_malloc.c:441
#3 0x0000000000624dd8 in do_free_params (_shm=1, _p=0x0) at parser/parse_param.c:623
#4 shm_free_params (_p=<optimized out>) at parser/parse_param.c:643
#5 0x00007f3cc3af95c9 in destroy_dmq_node (node=0x7f3cc3484c98, shm=<optimized out>) at dmqnode.c:252
#6 destroy_dmq_node (node=0x7f3cc3484c98, shm=<optimized out>) at dmqnode.c:248
#7 0x00007f3cc3af3a6a in dmq_send_message (peer=peer at entry=0x7f3c837c9658, body=body at entry=0x7f3cca9992f0, node=node at entry=0x7f3c89c16880,
resp_cback=resp_cback at entry=0x7f3cc3d11650, max_forwards=max_forwards at entry=1, content_type=content_type at entry=0x7f3cc3d11660) at dmq_funcs.c:258
#8 0x00007f3cc3af49b5 in bcast_dmq_message (peer=0x7f3c837c9658, body=body at entry=0x7f3cca9992f0, except=except at entry=0x0, resp_cback=resp_cback at entry=0x7f3cc3d11650,
max_forwards=max_forwards at entry=1, content_type=content_type at entry=0x7f3cc3d11660) at dmq_funcs.c:167
#9 0x00007f3cc3b044a2 in request_nodelist (node=<optimized out>, forward=forward at entry=1) at notification_peer.c:562
#10 0x00007f3cc3aee88f in destroy () at dmq.c:331
#11 0x000000000056e872 in destroy_modules () at sr_module.c:805
#12 0x00000000004a410d in cleanup (show_status=show_status at entry=1) at main.c:513
#13 0x00000000004a4d98 in shutdown_children (show_status=1, sig=15) at main.c:655
#14 0x00000000004a66e8 in handle_sigs () at main.c:746
#15 0x00000000004aa976 in main_loop () at main.c:1701
#16 0x0000000000427292 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2547
#0 0x00007f3ccc4a6165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1 0x00007f3ccc4a93e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#2 0x00000000005f29ae in qm_free (qm=<optimized out>, p=<optimized out>, file=<optimized out>, func=<optimized out>, line=623) at mem/q_malloc.c:441
f = <optimized out>
size = <optimized out>
next = <optimized out>
prev = <optimized out>
__FUNCTION__ = "qm_free"
#3 0x0000000000624dd8 in do_free_params (_shm=1, _p=0x0) at parser/parse_param.c:623
ptr = <optimized out>
#4 shm_free_params (_p=<optimized out>) at parser/parse_param.c:643
No locals.
#5 0x00007f3cc3af95c9 in destroy_dmq_node (node=0x7f3cc3484c98, shm=<optimized out>) at dmqnode.c:252
No locals.
#6 destroy_dmq_node (node=0x7f3cc3484c98, shm=<optimized out>) at dmqnode.c:248
No locals.
#7 0x00007f3cc3af3a6a in dmq_send_message (peer=peer at entry=0x7f3c837c9658, body=body at entry=0x7f3cca9992f0, node=node at entry=0x7f3c89c16880,
resp_cback=resp_cback at entry=0x7f3cc3d11650, max_forwards=max_forwards at entry=1, content_type=content_type at entry=0x7f3cc3d11660) at dmq_funcs.c:258
uac_r = {method = 0x7f3cc3d11340, headers = 0x7ffd03cd77d0, body = 0x7f3cca9992f0, ssock = 0x7f3cc3d116e0, dialog = 0x0, cb_flags = 1024,
cb = 0x7f3cc3af1e70 <dmq_tm_callback>, cbp = 0x7f3cc34646c8, callid = 0x0}
str_hdr = {s = 0x7f3cca711348 "Max-Forwards: 1\r\nContent-Type: text/plain\r\n", len = 43}
from = {s = 0x7f3cca999368 "sip:notification_peer at 172.20.40.5:5060ain\r\n", len = 38}
to = {s = 0x7f3cca999408 "sip:notification_peer at 217.116.118.19:5060", len = 41}
cb_param = <optimized out>
result = <optimized out>
len = <optimized out>
__FUNCTION__ = "dmq_send_message"
#8 0x00007f3cc3af49b5 in bcast_dmq_message (peer=0x7f3c837c9658, body=body at entry=0x7f3cca9992f0, except=except at entry=0x0, resp_cback=resp_cback at entry=0x7f3cc3d11650,
max_forwards=max_forwards at entry=1, content_type=content_type at entry=0x7f3cc3d11660) at dmq_funcs.c:167
node = 0x7f3c89c16880
__FUNCTION__ = "bcast_dmq_message"
#9 0x00007f3cc3b044a2 in request_nodelist (node=<optimized out>, forward=forward at entry=1) at notification_peer.c:562
body = 0x7f3cca9992f0
ret = <optimized out>
__FUNCTION__ = "request_nodelist"
#10 0x00007f3cc3aee88f in destroy () at dmq.c:331
__FUNCTION__ = "destroy"
#11 0x000000000056e872 in destroy_modules () at sr_module.c:805
t = <optimized out>
foo = 0x7f3cca4d5f20
__FUNCTION__ = "destroy_modules"
#12 0x00000000004a410d in cleanup (show_status=show_status at entry=1) at main.c:513
memlog = <optimized out>
__FUNCTION__ = "cleanup"
#13 0x00000000004a4d98 in shutdown_children (show_status=1, sig=15) at main.c:655
No locals.
#14 0x00000000004a66e8 in handle_sigs () at main.c:746
chld = 0
chld_status = 134
memlog = <optimized out>
__FUNCTION__ = "handle_sigs"
#15 0x00000000004aa976 in main_loop () at main.c:1701
i = <optimized out>
pid = <optimized out>
si = <optimized out>
si_desc = "udp receiver child=31 sock=172.20.40.5:5060\000\000\000\000\000\030\216\231\312<\177\000\000\016\000\000\000\000\000\000\000\003\000\000\000\000\000\000\000(\245K\203<\177\000\000\001\000\000\000\000\000\000\000\000Bp\367\353\301<~ \000\000\000\000\000\000\000\002\000\000\000\000\000\000\000\310\222\241\000\000\000\000\000\300\222\241\000\000\000\000"
nrprocs = <optimized out>
__FUNCTION__ = "main_loop"
#16 0x0000000000427292 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2547
cfg_stream = <optimized out>
c = <optimized out>
r = <optimized out>
tmp = 0x7ffd03cd9f15 ""
tmp_len = 0
port = 1
proto = 1
options = 0x6baf50 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 3232125741
rfd = 4
debug_save = <optimized out>
debug_flag = <optimized out>
dont_fork_cnt = <optimized out>
n_lst = 0x21
p = <optimized out>
st = {st_dev = 14, st_ino = 23606, st_nlink = 2, st_mode = 16877, st_uid = 110, st_gid = 301, __pad0 = 0, st_rdev = 0, st_size = 200, st_blksize = 4096, st_blocks = 0,
st_atim = {tv_sec = 1447757500, tv_nsec = 873437536}, st_mtim = {tv_sec = 1447767996, tv_nsec = 445431681}, st_ctim = {tv_sec = 1447768000, tv_nsec = 253521630},
__unused = {0, 0, 0}}
__FUNCTION__ = "main"
```
There is also a core dump of PID 35911:
```
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/kamailio...Reading symbols from /usr/lib/debug/.build-id/c3/a4fc0cef8da17048bce2dc080e543509e39d37.debug...done.
done.
[New LWP 35911]
warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio_sip_proxy.cfg -P /var/run/kamailio'.
Program terminated with signal 6, Aborted.
#0 0x00007f3ccc4a6165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
Thread 1 (Thread 0x7f3ccce2e700 (LWP 35911)):
#0 0x00007f3ccc4a6165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f3ccc4a93e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00000000005f29ae in qm_free (qm=<optimized out>, p=<optimized out>, file=<optimized out>, func=<optimized out>, line=623) at mem/q_malloc.c:441
#3 0x0000000000624dd8 in do_free_params (_shm=1, _p=0x0) at parser/parse_param.c:623
#4 shm_free_params (_p=<optimized out>) at parser/parse_param.c:643
#5 0x00007f3cc3af95c9 in destroy_dmq_node (node=0x7f3cc3484218, shm=<optimized out>) at dmqnode.c:252
#6 destroy_dmq_node (node=0x7f3cc3484218, shm=<optimized out>) at dmqnode.c:248
#7 0x00007f3cc3af3a6a in dmq_send_message (peer=<optimized out>, body=0x7ffd03cd7540, node=<optimized out>, resp_cback=<optimized out>, max_forwards=<optimized out>,
content_type=<optimized out>) at dmq_funcs.c:258
#8 0x00007f3cc38e00b2 in usrloc_dmq_send (body=body at entry=0x7ffd03cd7540, node=node at entry=0x7f3c89c16880) at usrloc_sync.c:261
#9 0x00007f3cc38e13fe in usrloc_dmq_send_contact (ptr=<optimized out>, aor=..., action=action at entry=1, node=node at entry=0x7f3c89c16880) at usrloc_sync.c:505
#10 0x00007f3cc38e1bfc in usrloc_get_all_ucontact (node=node at entry=0x7f3c89c16880) at usrloc_sync.c:211
#11 0x00007f3cc38e2e96 in usrloc_dmq_handle_msg (msg=<optimized out>, resp=0x7ffd03cd7880, node=0x7f3c89c16880) at usrloc_sync.c:398
#12 0x00007f3cc3b0710b in worker_loop (id=id at entry=2) at worker.c:105
#13 0x00007f3cc3aee1e8 in child_init (rank=<optimized out>) at dmq.c:293
#14 child_init (rank=<optimized out>) at dmq.c:279
#15 0x000000000056d4f8 in init_mod_child (m=0x7f3cca4d6b38, rank=rank at entry=0) at sr_module.c:915
#16 0x000000000056d442 in init_mod_child (m=0x7f3cca4d7030, rank=rank at entry=0) at sr_module.c:912
#17 0x000000000056d442 in init_mod_child (m=0x7f3cca4d7308, rank=rank at entry=0) at sr_module.c:912
#18 0x000000000056d442 in init_mod_child (m=0x7f3cca4d7930, rank=rank at entry=0) at sr_module.c:912
#19 0x000000000056e91b in init_child (rank=rank at entry=0) at sr_module.c:941
#20 0x00000000004aa78c in main_loop () at main.c:1650
#21 0x0000000000427292 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2547
#0 0x00007f3ccc4a6165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1 0x00007f3ccc4a93e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#2 0x00000000005f29ae in qm_free (qm=<optimized out>, p=<optimized out>, file=<optimized out>, func=<optimized out>, line=623) at mem/q_malloc.c:441
f = <optimized out>
size = <optimized out>
next = <optimized out>
prev = <optimized out>
__FUNCTION__ = "qm_free"
#3 0x0000000000624dd8 in do_free_params (_shm=1, _p=0x0) at parser/parse_param.c:623
ptr = <optimized out>
#4 shm_free_params (_p=<optimized out>) at parser/parse_param.c:643
No locals.
#5 0x00007f3cc3af95c9 in destroy_dmq_node (node=0x7f3cc3484218, shm=<optimized out>) at dmqnode.c:252
No locals.
#6 destroy_dmq_node (node=0x7f3cc3484218, shm=<optimized out>) at dmqnode.c:248
No locals.
#7 0x00007f3cc3af3a6a in dmq_send_message (peer=<optimized out>, body=0x7ffd03cd7540, node=<optimized out>, resp_cback=<optimized out>, max_forwards=<optimized out>,
content_type=<optimized out>) at dmq_funcs.c:258
uac_r = {method = 0x7f3cc3d11340, headers = 0x7ffd03cd73c0, body = 0x7ffd03cd7540, ssock = 0x7f3cc3d116e0, dialog = 0x0, cb_flags = 1024,
cb = 0x7f3cc3af1e70 <dmq_tm_callback>, cbp = 0x7f3cc34844c8, callid = 0x0}
str_hdr = {s = 0x7f3cca713a20 "Max-Forwards: 1\r\nContent-Type: application/json\r\n", len = 49}
from = {s = 0x7f3cca713918 "sip:usrloc at 172.20.40.5:50600/UDP\300\300\300\300", len = 27}
to = {s = 0x7f3cca713998 "sip:usrloc at 217.116.118.19:506047698f4.9e\300\300\300\300", len = 30}
cb_param = <optimized out>
result = <optimized out>
len = <optimized out>
__FUNCTION__ = "dmq_send_message"
#8 0x00007f3cc38e00b2 in usrloc_dmq_send (body=body at entry=0x7ffd03cd7540, node=node at entry=0x7f3c89c16880) at usrloc_sync.c:261
__FUNCTION__ = "usrloc_dmq_send"
#9 0x00007f3cc38e13fe in usrloc_dmq_send_contact (ptr=<optimized out>, aor=..., action=action at entry=1, node=node at entry=0x7f3c89c16880) at usrloc_sync.c:505
jdoc = {root = 0xd71160, flags = 0, buf = {
s = 0xd70070 "{\"action\":1,\"aor\":\"2241474e0 at sipgate.de\",\"ruid\":\"uloc-564b246e-7416-dec\",\"c\":\"sip:2241474 at 192.168.0.140:5060\",\"received\":\"sip:87.79.157.233:1027\",\"path\":\"<sip:217.10.79.9;lr;received=sip:87.79.157.233"..., len = 414}, malloc_fn = 0x7f3ccc4eea00 <malloc>, free_fn = 0x7f3ccc4ee920 <free>}
flags = <optimized out>
__FUNCTION__ = "usrloc_dmq_send_contact"
#10 0x00007f3cc38e1bfc in usrloc_get_all_ucontact (node=node at entry=0x7f3c89c16880) at usrloc_sync.c:211
rval = <optimized out>
len = <optimized out>
buf = 0x7f3cca9c8080
cp = 0x7f3ccab62a49
ruid = {s = 0x7f3ccab62a2f "uloc-564b246e-7416-dec\337\321z\205\036", len = 22}
aorhash = <optimized out>
send_sock = <optimized out>
flags = <optimized out>
aor = {s = 0x7f3c8608a528 "2241474e0 at sipgate.de", len = 20}
r = 0x7f3c8608a488
_d = 0x7f3c837257a0
ptr = 0x7f3c8608a5a0
res = <optimized out>
__FUNCTION__ = "usrloc_get_all_ucontact"
#11 0x00007f3cc38e2e96 in usrloc_dmq_handle_msg (msg=<optimized out>, resp=0x7ffd03cd7880, node=0x7f3c89c16880) at usrloc_sync.c:398
content_length = <optimized out>
body = <optimized out>
jdoc = {root = 0xd69490, flags = 0, buf = {s = 0x7f3c8eb7f378 "{\"action\":3}", len = 12}, malloc_fn = 0x7f3ccc4eea00 <malloc>, free_fn = 0x7f3ccc4ee920 <free>}
it = <optimized out>
ci = {ruid = {s = 0x0, len = 0}, c = 0x7ffd03cd7750, received = {s = 0x0, len = 0}, path = 0x7ffd03cd7760, expires = 0, q = 0, callid = 0x7ffd03cd7770, cseq = 0,
flags = 2, cflags = 0, user_agent = 0x7ffd03cd7780, sock = 0x0, methods = 0, instance = {s = 0x0, len = 0}, reg_id = 0, server_id = 0, tcpconn_id = -1, keepalive = 0,
xavp = 0x0, last_modified = 0}
action = <optimized out>
expires = <optimized out>
cseq = <optimized out>
flags = <optimized out>
cflags = <optimized out>
q = <optimized out>
last_modified = <optimized out>
methods = <optimized out>
reg_id = <optimized out>
aor = {s = <optimized out>, len = 0}
ruid = <optimized out>
c = {s = 0x0, len = 0}
received = <optimized out>
path = {s = 0x0, len = 0}
callid = {s = 0x0, len = 0}
user_agent = {s = 0x0, len = 0}
instance = <optimized out>
__FUNCTION__ = "usrloc_dmq_handle_msg"
#12 0x00007f3cc3b0710b in worker_loop (id=id at entry=2) at worker.c:105
worker = 0x7f3c837c9560
current_job = 0x7f3c8eb7bdd8
peer_response = {resp_code = 0, content_type = {s = 0x0, len = 0}, reason = {s = 0x0, len = 0}, body = {s = 0x0, len = 0}}
ret_value = <optimized out>
not_parsed = 1
dmq_node = 0x7f3c89c16880
__FUNCTION__ = "worker_loop"
#13 0x00007f3cc3aee1e8 in child_init (rank=<optimized out>) at dmq.c:293
i = <optimized out>
newpid = <optimized out>
#14 child_init (rank=<optimized out>) at dmq.c:279
No locals.
#15 0x000000000056d4f8 in init_mod_child (m=0x7f3cca4d6b38, rank=rank at entry=0) at sr_module.c:915
__FUNCTION__ = "init_mod_child"
#16 0x000000000056d442 in init_mod_child (m=0x7f3cca4d7030, rank=rank at entry=0) at sr_module.c:912
__FUNCTION__ = "init_mod_child"
#17 0x000000000056d442 in init_mod_child (m=0x7f3cca4d7308, rank=rank at entry=0) at sr_module.c:912
__FUNCTION__ = "init_mod_child"
#18 0x000000000056d442 in init_mod_child (m=0x7f3cca4d7930, rank=rank at entry=0) at sr_module.c:912
__FUNCTION__ = "init_mod_child"
#19 0x000000000056e91b in init_child (rank=rank at entry=0) at sr_module.c:941
No locals.
#20 0x00000000004aa78c in main_loop () at main.c:1650
i = <optimized out>
pid = <optimized out>
si = <optimized out>
si_desc = "udp receiver child=31 sock=172.20.40.5:5060\000\000\000\000\000\030\216\231\312<\177\000\000\016\000\000\000\000\000\000\000\003\000\000\000\000\000\000\000(\245K\203<\177\000\000\001\000\000\000\000\000\000\000\000Bp\367\353\301<~ \000\000\000\000\000\000\000\002\000\000\000\000\000\000\000\310\222\241\000\000\000\000\000\300\222\241\000\000\000\000"
nrprocs = <optimized out>
__FUNCTION__ = "main_loop"
#21 0x0000000000427292 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2547
cfg_stream = <optimized out>
c = <optimized out>
r = <optimized out>
tmp = 0x7ffd03cd9f15 ""
tmp_len = 0
port = 1
proto = 1
options = 0x6baf50 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 3232125741
rfd = 4
debug_save = <optimized out>
debug_flag = <optimized out>
dont_fork_cnt = <optimized out>
n_lst = 0x21
p = <optimized out>
st = {st_dev = 14, st_ino = 23606, st_nlink = 2, st_mode = 16877, st_uid = 110, st_gid = 301, __pad0 = 0, st_rdev = 0, st_size = 200, st_blksize = 4096, st_blocks = 0,
st_atim = {tv_sec = 1447757500, tv_nsec = 873437536}, st_mtim = {tv_sec = 1447767996, tv_nsec = 445431681}, st_ctim = {tv_sec = 1447768000, tv_nsec = 253521630},
__unused = {0, 0, 0}}
__FUNCTION__ = "main"
```
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/399
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-dev/attachments/20151117/91c60121/attachment-0001.html>
More information about the sr-dev
mailing list