<div dir="ltr"><div><div>Hello,<br><br></div>Here is the bt of all cores. This time, the scenario is different, when kamailio receives an ACK and try to forward the message<br><br>Log file:<br><b>Feb 13 11:09:19 kamailio23 /usr/local/sbin/kamailio[21086]: : <core> [pass_fd.c:293]: receive_fd(): ERROR: receive_fd: EOF on 31 </b><br>
Feb 13 11:09:19 kamailio23 /usr/local/sbin/kamailio[21038]: ALERT: <core> [main.c:788]: handle_sigs(): child process 21063 exited by a signal 11<br>Feb 13 11:09:19 kamailio23 /usr/local/sbin/kamailio[21038]: ALERT: <core> [main.c:791]: handle_sigs(): core was generated<br>
Feb 13 11:09:19 kamailio23 /usr/local/sbin/kamailio[21038]: INFO: <core> [main.c:803]: handle_sigs(): INFO: terminating due to SIGCHLD<br><br><b>root@kamailio23:/var/log/kamailio# gdb /usr/local/sbin/kamailio cores/core.kamailio.21063</b><br>
(gdb) bt<br>#0 0x000000000052971e in timer_list_expire (t=275552247, h=0x7f5ea9f30aa0, slow_l=0x7f5ea9f31a50, slow_mark=59628) at timer.c:883<br>#1 0x0000000000529a80 in timer_handler () at timer.c:959<br>#2 0x0000000000529d0f in timer_main () at timer.c:998<br>
#3 0x0000000000469eda in main_loop () at main.c:1709<br>#4 0x000000000046ca85 in main (argc=5, argv=0x7fffcd0f5828) at main.c:2566<br>(gdb) bt full<br>#0 0x000000000052971e in timer_list_expire (t=275552247, h=0x7f5ea9f30aa0, slow_l=0x7f5ea9f31a50, slow_mark=59628) at timer.c:883<br>
tl = 0x7f5eaa1bf5f0<br> ret = 32767<br>#1 0x0000000000529a80 in timer_handler () at timer.c:959<br> saved_ticks = 275552247<br> run_slow_timer = 0<br> i = 236<br> __FUNCTION__ = "timer_handler"<br>
#2 0x0000000000529d0f in timer_main () at timer.c:998<br>No locals.<br>#3 0x0000000000469eda in main_loop () at main.c:1709<br> i = 8<br> pid = 0<br> si = 0x0<br> si_desc = "udp receiver child=7 sock=<a href="http://91.213.79.31:5060">91.213.79.31:5060</a>\000^\177", '\000' <repeats 18 times>, "0V\017\315\377\177\000\000u\234\021O\000\000\000\000\240mA\000\000\000\000\000 X\017\315\377\177", '\000' <repeats 18 times>, "`V\017\315\377\177\000\000\030>K\000\000\000\000"<br>
nrprocs = 8<br> __FUNCTION__ = "main_loop"<br>#4 0x000000000046ca85 in main (argc=5, argv=0x7fffcd0f5828) at main.c:2566<br> cfg_stream = 0x1bd2010<br> c = -1<br> r = 0<br> tmp = 0x7fffcd0f5858 "\233n\017\315\377\177"<br>
tmp_len = 0<br> port = 5<br> proto = 0<br> options = 0x5c7c18 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"<br> ret = -1<br> seed = 1955773403<br> rfd = 4<br>
debug_save = 0<br> debug_flag = 0<br> dont_fork_cnt = 0<br> n_lst = 0xbf<br> p = 0x4162c9 "H\203\304\b\303" <Address 0x4162ce out of bounds><br> __FUNCTION__ = "main"<br>
(gdb)<br><br><br><b>root@kamailio23:/var/log/kamailio# gdb /usr/local/sbin/kamailio cores/core.kamailio.21038</b><br>(gdb) bt full<br>#0 0x000000000053a81d in qm_free (qm=0x7f5ea9ee2000, p=0x424f9a6b00000000) at mem/q_malloc.c:476<br>
f = 0x424f9a6afffffff0<br> size = 4880<br> next = 0x7f5eaa17eff0<br> prev = 0x7f5ea9eec840<br> __FUNCTION__ = "qm_free"<br>#1 0x00007f5eb150d81c in free_cell (dead_cell=0x7f5eaa047710) at h_table.c:162<br>
b = 0x0<br> i = 1<br> rpl = 0x0<br> tt = 0x0<br> foo = 0x0<br> cbs = 0x0<br> cbs_tmp = 0x7f5eaa186950<br> __FUNCTION__ = "free_cell"<br>#2 0x00007f5eb150e783 in free_hash_table () at h_table.c:443<br>
p_cell = 0x7f5eaa047710<br> tmp_cell = 0x7f5eaa047710<br> i = 59334<br>#3 0x00007f5eb152181b in tm_shutdown () at t_funcs.c:122<br> __FUNCTION__ = "tm_shutdown"<br>#4 0x00000000004ee6f0 in destroy_modules () at sr_module.c:790<br>
t = 0x7f5eb2aea2f0<br> foo = 0x7f5eb2ae9cc0<br>#5 0x0000000000463baa in cleanup (show_status=1) at main.c:573<br> memlog = 32606<br> __FUNCTION__ = "cleanup"<br>#6 0x0000000000464c32 in shutdown_children (sig=15, show_status=1) at main.c:715<br>
__FUNCTION__ = "shutdown_children"<br>#7 0x0000000000466223 in handle_sigs () at main.c:806<br> chld = 0<br> chld_status = 139<br> memlog = 0<br> __FUNCTION__ = "handle_sigs"<br>
#8 0x000000000046a459 in main_loop () at main.c:1767<br> i = 8<br> pid = 21086<br> si = 0x0<br> si_desc = "udp receiver child=7 sock=<a href="http://91.213.79.31:5060">91.213.79.31:5060</a>\000^\177", '\000' <repeats 18 times>, "0V\017\315\377\177\000\000u\234\021O\000\000\000\000\240mA\000\000\000\000\000 X\017\315\377\177", '\000' <repeats 18 times>, "`V\017\315\377\177\000\000\030>K\000\000\000\000"<br>
nrprocs = 8<br> __FUNCTION__ = "main_loop"<br>#9 0x000000000046ca85 in main (argc=5, argv=0x7fffcd0f5828) at main.c:2566<br> cfg_stream = 0x1bd2010<br> c = -1<br> r = 0<br> tmp = 0x7fffcd0f5858 "\233n\017\315\377\177"<br>
tmp_len = 0<br> port = 5<br> proto = 0<br> options = 0x5c7c18 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"<br> ret = -1<br> seed = 1955773403<br> rfd = 4<br>
debug_save = 0<br> debug_flag = 0<br> dont_fork_cnt = 0<br> n_lst = 0xbf<br> p = 0x4162c9 "H\203\304\b\303" <Address 0x4162ce out of bounds><br> __FUNCTION__ = "main"<br>
(gdb) <br><br></div>Regards,<br><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Feb 12, 2014 at 12:24 PM, Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
Hello,<br>
<br>
the backtrace you got is not relevant, because it is from the main
process that is handling the singnal of the child process that
crashed. So the corefile from crashed process has been overwritten.<br>
<br>
Can you enable one core file per process and try to reproduce? Then
get the backtrace from each core generated.<br>
<br>
Cheers,<br>
Daniel<div><div class="h5"><br>
<br>
<div>On 11/02/14 12:38, Tuan Viet Nguyen
wrote:<br>
</div>
</div></div><blockquote type="cite"><div><div class="h5">
<div dir="ltr">
<div>
<div>Hello, <br>
<br>
</div>
Upon receipt an 502 Bad gateway, I have the following error
then Kamailio crashed<br>
Feb 11 11:49:57 kamailio23 /usr/local/sbin/kamailio[28895]:
WARNING: tm [t_lookup.c:1564]: t_unref(): WARNING: script
writer didn't release transaction<br>
Feb 11 11:50:03 kamailio23 /usr/local/sbin/kamailio[28926]: :
<core> [pass_fd.c:293]: receive_fd(): ERROR: receive_fd:
EOF on 31<br>
Feb 11 11:50:03 kamailio23 /usr/local/sbin/kamailio[28876]:
ALERT: <core> [main.c:788]: handle_sigs(): child process
28903 exited by a signal 11<br>
Feb 11 11:50:03 kamailio23 /usr/local/sbin/kamailio[28876]:
ALERT: <core> [main.c:791]: handle_sigs(): core was
generated<br>
<br>
<br>
</div>
<div>(gdb) bt<br>
#0 0x000000000053a81d in qm_free (qm=0x7fd6f962a000,
p=0x75d63ff100000000) at mem/q_malloc.c:476<br>
#1 0x00007fd700c5581c in free_cell (dead_cell=0x7fd6f9777bf0)
at h_table.c:162<br>
#2 0x00007fd700c56783 in free_hash_table () at h_table.c:443<br>
#3 0x00007fd700c6981b in tm_shutdown () at t_funcs.c:122<br>
#4 0x00000000004ee6f0 in destroy_modules () at
sr_module.c:790<br>
#5 0x0000000000463baa in cleanup (show_status=1) at
main.c:573<br>
#6 0x0000000000464c32 in shutdown_children (sig=15,
show_status=1) at main.c:715<br>
#7 0x0000000000466223 in handle_sigs () at main.c:806<br>
#8 0x000000000046a459 in main_loop () at main.c:1767<br>
#9 0x000000000046ca85 in main (argc=5, argv=0x7fff2f13ec38)
at main.c:2566<br>
(gdb) bt full<br>
#0 0x000000000053a81d in qm_free (qm=0x7fd6f962a000,
p=0x75d63ff100000000) at mem/q_malloc.c:476<br>
f = 0x75d63ff0fffffff0<br>
size = 4880<br>
next = 0x7fd6f98c5380<br>
prev = 0x7fd6f9634840<br>
__FUNCTION__ = "qm_free"<br>
#1 0x00007fd700c5581c in free_cell (dead_cell=0x7fd6f9777bf0)
at h_table.c:162<br>
b = 0x0<br>
i = 1<br>
rpl = 0x0<br>
tt = 0x0<br>
foo = 0x59e92da4<br>
cbs = 0x0<br>
cbs_tmp = 0x7fd6f98c5370<br>
__FUNCTION__ = "free_cell"<br>
#2 0x00007fd700c56783 in free_hash_table () at h_table.c:443<br>
p_cell = 0x7fd6f9777bf0<br>
tmp_cell = 0x7fd6f98fb9c0<br>
i = 3084<br>
#3 0x00007fd700c6981b in tm_shutdown () at t_funcs.c:122<br>
__FUNCTION__ = "tm_shutdown"<br>
#4 0x00000000004ee6f0 in destroy_modules () at
sr_module.c:790<br>
t = 0x7fd7022322f0<br>
foo = 0x7fd702231cc0<br>
#5 0x0000000000463baa in cleanup (show_status=1) at
main.c:573<br>
memlog = 32727<br>
__FUNCTION__ = "cleanup"<br>
#6 0x0000000000464c32 in shutdown_children (sig=15,
show_status=1) at main.c:715<br>
__FUNCTION__ = "shutdown_children"<br>
#7 0x0000000000466223 in handle_sigs () at main.c:806<br>
chld = 0<br>
chld_status = 139<br>
memlog = 0<br>
__FUNCTION__ = "handle_sigs"<br>
#8 0x000000000046a459 in main_loop () at main.c:1767<br>
i = 8<br>
pid = 28926<br>
si = 0x0<br>
si_desc = "udp receiver child=7 sock=<a href="http://91.213.79.31:5060" target="_blank">91.213.79.31:5060</a>\000\326\177",
'\000' <repeats 18 times>,
"@\352\023/\377\177\000\000\244-\351Y\000\000\000\000\240mA\000\000\000\000\000\060\354\023/\377\177",
'\000' <repeats 18 times>,
"p\352\023/\377\177\000\000\030>K\000\000\000\000"<br>
nrprocs = 8<br>
__FUNCTION__ = "main_loop"<br>
#9 0x000000000046ca85 in main (argc=5, argv=0x7fff2f13ec38)
at main.c:2566<br>
cfg_stream = 0xc36010<br>
c = -1<br>
r = 0<br>
tmp = 0x7fff2f13ec68 "\233\356\023/\377\177"<br>
tmp_len = 0<br>
port = 5<br>
proto = 0<br>
options = 0x5c7c18
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"<br>
ret = -1<br>
seed = 3628627122<br>
rfd = 4<br>
debug_save = 0<br>
debug_flag = 0<br>
dont_fork_cnt = 0<br>
n_lst = 0xbf<br>
p = 0x4162c9 "H\203\304\b\303" <Address 0x4162ce
out of bounds><br>
__FUNCTION__ = "main"<br>
(gdb)<br>
<br>
</div>
<div>Regards,<br>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
</div></div><pre>_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
<a href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a>
<a href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><span class="HOEnZb"><font color="#888888">
</font></span></pre><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a href="http://twitter.com/#!/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a></pre>
</font></span></div>
<br>_______________________________________________<br>
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list<br>
<a href="mailto:sr-users@lists.sip-router.org">sr-users@lists.sip-router.org</a><br>
<a href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><br>
<br></blockquote></div><br></div>