<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
This is for an ACK and slightly different case -- it can happen only
with MEMDBG=1 -- I will look over it as well.<br>
<br>
Can you run another test with:<br>
<br>
mem_safety=1<br>
<br>
?<br>
<br>
Still keeping MEMDBG=1<br>
<br>
Daniel<br>
<br>
<div class="moz-cite-prefix">On 09/12/13 19:22, Kelvin Chua wrote:<br>
</div>
<blockquote
cite="mid:CAMF97-djDEJH=C-=SVjSCEE-YDjvdqE+=h6i9RpVPc2Fnu1fkQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>#0 0x00007fbf3aee9425 in raise () from
/lib/x86_64-linux-gnu/libc.so.6</div>
<div>No symbol table info available.</div>
<div>#1 0x00007fbf3aeecb8b in abort () from
/lib/x86_64-linux-gnu/libc.so.6</div>
<div>
No symbol table info available.</div>
<div>#2 0x000000000054d710 in qm_free (qm=0x7fbf30414000,
p=0x7fbf3088a9a8, file=0x7fbf38b85cb5 "tm: h_table.c", </div>
<div> func=0x7fbf38b85e58 "free_cell", line=157) at
mem/q_malloc.c:470</div>
<div> f = 0x7fbf3088a978</div>
<div> size = 0</div>
<div> next = 0x7fbf30449518</div>
<div> prev = 0x7fbf3088c660</div>
<div> __FUNCTION__ = "qm_free"</div>
<div>#3 0x00007fbf38b19b10 in free_cell
(dead_cell=0x7fbf30887de8) at h_table.c:157</div>
<div> b = 0x7fff7da2d4b0 " ֢}\377\177"</div>
<div> i = 1</div>
<div> rpl = 0x7fbf30887de8</div>
<div> tt = 0x7fff7da2d9c0</div>
<div> foo = 0x22b7da2d3d0</div>
<div> cbs = 0x0</div>
<div> cbs_tmp = 0x0</div>
<div> __FUNCTION__ = "free_cell"</div>
<div>#4 0x00007fbf38b4a486 in t_unref (p_msg=0x7fbf3abbf368) at
t_lookup.c:1574</div>
<div> kr = 12</div>
<div> __FUNCTION__ = "t_unref"</div>
<div>#5 0x00007fbf38b4e60b in w_t_unref (foo=0x7fbf3abbf368,
flags=2147483649, bar=0x0) at tm.c:725</div>
<div>No locals.</div>
<div>#6 0x00000000004db5bb in exec_post_script_cb
(msg=0x7fbf3abbf368, type=REQUEST_CB_TYPE) at script_cb.c:195</div>
<div> cb = 0x7fbf3abf4fa0</div>
<div> flags = 2147483649</div>
<div>#7 0x00000000004a9cd8 in receive_msg (</div>
<div> buf=0x9263c0 "ACK <a class="moz-txt-link-freetext" href="sip:18X88X441X1@2X8.1X1.3X.2X">sip:18X88X441X1@2X8.1X1.3X.2X</a>
SIP/2.0\r\nVia: SIP/2.0/UDP
125.60.156.243:17218;rport;branch=z9hG4bKogunmjtk\r\nMax-Forwards:
16\r\nProxy-Authorization: Digest
username=\"kelvin\",realm=\"2X8.1X1.3X.2X\",non".</div>
<div>
<div>.., len=555, rcv_info=0x7fff7da2d680) at receive.c:227</div>
<div> msg = 0x7fbf3abbf368</div>
<div> ctx = {rec_lev = -1, run_flags = 0, last_retcode
= 984293392, jmp_env = {{__jmpbuf = {140459304019208, </div>
<div> 1204527218, 8881072, 0, 0, 0, 0,
140459312289652}, __mask_was_saved = 5, __saved_mask =
{__val = {</div>
<div> 0, 140459304039824, 1, 140459125283080,
1204527218, 2107823648, 1024, 6402790880, </div>
<div> 140459125283080, 140735301211616,
5493883, 4087102589, 140459125283080, 16963,
140459125283080, </div>
<div> 140735301211772}}}}}</div>
<div> ret = 0</div>
<div> inb = {</div>
<div> s = 0x9263c0 "ACK <a class="moz-txt-link-freetext" href="sip:18X88X441X1@2X8.1X1.3X.2X">sip:18X88X441X1@2X8.1X1.3X.2X</a>
SIP/2.0\r\nVia: SIP/2.0/UDP
125.60.156.243:17218;rport;branch=z9hG4bKogunmjtk\r\nMax-Forwards:
16\r\nProxy-Authorization: Digest
username=\"kelvin\",realm=\"2X8.1X1.3X.2X\",non"..., len =
555}</div>
<div> __FUNCTION__ = "receive_msg"</div>
<div>#8 0x000000000054148b in udp_rcv_loop () at
udp_server.c:557</div>
<div> len = 555</div>
<div> buf = "ACK <a class="moz-txt-link-freetext" href="sip:18X88X441X1@2X8.1X1.3X.2X">sip:18X88X441X1@2X8.1X1.3X.2X</a>
SIP/2.0\r\nVia: SIP/2.0/UDP
125.60.156.243:17218;rport;branch=z9hG4bKogunmjtk\r\nMax-Forwards:
16\r\nProxy-Authorization: Digest
username=\"kelvin\",realm=\"2X8.1X1.3X.2X\",non"...</div>
<div> tmp = 0x7fff7da2d690 "\020ע}\377\177"</div>
<div> from = 0x7fbf3abc4df0</div>
<div> fromlen = 16</div>
<div> ri = {src_ip = {af = 2, len = 4, u = {addrl =
{8382069885, 140735301211920}, addr32 = {4087102589, 1, </div>
<div> 2107823888, 32767}, addr16 = {15485,
62364, 1, 0, 55056, 32162, 32767, 0}, </div>
<div> addr =
"}<\234\363\001\000\000\000\020ע}\377\177\000"}}, dst_ip
= {af = 2, len = 4, u = {addrl = {</div>
<div> 489123792, 0}, addr32 = {489123792, 0, 0,
0}, addr16 = {28624, 7463, 0, 0, 0, 0, 0, 0}, </div>
<div> addr = "\320o'\035", '\000' <repeats 11
times>}}, src_port = 17218, dst_port = 5060, </div>
<div> proto_reserved1 = 0, proto_reserved2 = 0,
src_su = {s = {sa_family = 2, </div>
<div> sa_data =
"CB}<\234\363\000\000\000\000\000\000\000"}, sin =
{sin_family = 2, sin_port = 16963, </div>
<div> sin_addr = {s_addr = 4087102589}, sin_zero
= "\000\000\000\000\000\000\000"}, sin6 = {</div>
<div> sin6_family = 2, sin6_port = 16963,
sin6_flowinfo = 4087102589, sin6_addr = {__in6_u = {</div>
<div> __u6_addr8 = '\000' <repeats 15
times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
__u6_addr32 = {</div>
<div> 0, 0, 0, 0}}}, sin6_scope_id = 0}},
bind_address = 0x7fbf3aaf03d8, proto = 1 '\001'}</div>
</div>
<div>
<div> __FUNCTION__ = "udp_rcv_loop"</div>
<div>#9 0x000000000046ce9c in main_loop () at main.c:1638</div>
<div> i = 0</div>
<div> pid = 0</div>
<div> si = 0x7fbf3aaf03d8</div>
<div> si_desc = "udp receiver child=0
sock=2X8.1X1.3X.2X:5060\000\364\357:\000\\,5\277\177\000\000r\240\313G\000\000\000\000\000DA\000\000\000\000\000\300٢}\377\177",
'\000' <repeats 18 times>"\360,
ע}\377\177\000\000\232\373K\000\000\000\000\000\020آ}\377\177\000\000\240Ó\000\000\000\000"</div>
<div> nrprocs = 4</div>
<div> __FUNCTION__ = "main_loop"</div>
<div>#10 0x000000000046fec0 in main (argc=15,
argv=0x7fff7da2d9c8) at main.c:2566</div>
<div> cfg_stream = 0xfd6010</div>
<div> c = -1</div>
<div> r = 0</div>
<div> tmp = 0x7fff7da2df48 ""</div>
<div> tmp_len = 0</div>
<div> port = 4274515</div>
<div> proto = 0</div>
<div> options = 0x5e5c10
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"</div>
<div> ret = -1</div>
<div> seed = 3558558127</div>
<div> rfd = 4</div>
<div> debug_save = 0</div>
<div> debug_flag = 0</div>
<div> dont_fork_cnt = 0</div>
<div> n_lst = 0x5d0670</div>
<div> p = 0xc2 <Address 0xc2 out of bounds></div>
<div> __FUNCTION__ = "main"</div>
</div>
<div><br>
</div>
<div>...cut</div>
</div>
<div class="gmail_extra"><br clear="all">
<div>Kelvin Chua</div>
<br>
<br>
<div class="gmail_quote">On Tue, Dec 10, 2013 at 1:04 AM,
Daniel-Constantin Mierla <span dir="ltr"><<a
moz-do-not-send="true" href="mailto:miconda@gmail.com"
target="_blank">miconda@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> It is a different
place now, can you give the bt full output?<br>
<br>
Cheers,<br>
Daniel
<div>
<div class="h5"><br>
<br>
<div>On 09/12/13 17:18, Kelvin Chua wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>i didn's set any global parameter, just the
defaults.</div>
<div>yes, i can reproduce it consistently, i just
send an INVITE and CANCEL while kamailio is
trying the destinations. (serial forks to
non-existent IP)</div>
<div>after i send the CANCEL, kamailio will crash.</div>
<div><br>
</div>
<div>below is the MEMDBG=1 syslog:</div>
<div><br>
</div>
<div>Dec 9 11:12:14 kelvin
/usr/local/sbin/kamailio[22191]: WARNING: tm
[t_lookup.c:1564]: t_unref(): WARNING: script
writer didn't release transaction</div>
<div>Dec 9 11:12:14 kelvin
/usr/local/sbin/kamailio[22189]: : <core>
[mem/q_malloc.c:468]: qm_free(): BUG: qm_free:
freeing already freed pointer (0x7f35a60e01e0),
called from tm: h_table.c: free_cell(157), first
free tm: h_table.c: free_cell(157) - aborting</div>
<div>Dec 9 11:12:14 kelvin
/usr/local/sbin/kamailio[22209]: : <core>
[pass_fd.c:293]: receive_fd(): ERROR:
receive_fd: EOF on 13</div>
<div>Dec 9 11:12:14 kelvin
/usr/local/sbin/kamailio[22186]: ALERT:
<core> [main.c:788]: handle_sigs(): child
process 22189 exited by a signal 6</div>
<div>Dec 9 11:12:14 kelvin
/usr/local/sbin/kamailio[22186]: ALERT:
<core> [main.c:791]: handle_sigs(): core
was generated</div>
<div>Dec 9 11:12:14 kelvin
/usr/local/sbin/kamailio[22186]: ERROR: ctl
[ctl.c:379]: mod_destroy(): ERROR: ctl: could
not delete unix socket /tmp/kamailio_ctl:
Operation not permitted (1)</div>
<div>Dec 9 11:12:14 kelvin
/usr/local/sbin/kamailio[22186]: : <core>
[mem/q_malloc.c:468]: qm_free(): BUG: qm_free:
freeing already freed pointer (0x7f35a60e01e0),
called from tm: h_table.c: free_cell(157), first
free tm: h_table.c: free_cell(157)</div>
</div>
<div class="gmail_extra"><br clear="all">
<div>Kelvin Chua</div>
<br>
<br>
<div class="gmail_quote">On Mon, Dec 9, 2013 at
11:54 PM, Daniel-Constantin Mierla <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:miconda@gmail.com"
target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Can you
compile with MEMDBG=1 in Makefile.defs and
try again? Might be a memory overwritten
issues somewhere. Look in the syslog for
memory related message.<br>
<br>
Few more details needed ... do you have
memjoin global parameter set? Is the
situation reproducible, or it happens
sporadically?<br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
<div>On 09/12/13 16:45, Kelvin Chua
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">tried latest branch
4.0, also segfault
<div>similar backtrace</div>
</div>
<div class="gmail_extra"><br
clear="all">
<div>Kelvin Chua</div>
<br>
<br>
<div class="gmail_quote">On Mon, Dec
9, 2013 at 7:13 PM,
Daniel-Constantin Mierla <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:miconda@gmail.com"
target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF"
text="#000000"> Hello,<br>
<br>
can you try latest branch 4.0?
There were some fixes for
similar case. Soon will be a
4.0.5 release.<br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
<div>On 09/12/13 10:35,
Kelvin Chua wrote:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div>
<div dir="ltr">
<div>
<div>following is
backtrace:</div>
<div><br>
</div>
<div>#0
qm_detach_free
(frag=0x7f91a4656170,
qm=<optimized
out>) at
mem/q_malloc.c:269</div>
<div> prev =
0x7f91a4656170</div>
<div> next =
0x0</div>
<div>#1 qm_malloc
(qm=0x7f91a429b000,
size=3840) at
mem/q_malloc.c:386</div>
<div> f =
0x7f91a4656170</div>
<div> hash =
-1536859792</div>
<div>#2
0x000000000050977f
in shm_malloc
(size=3832) at
parser/../mem/shm_mem.h:262</div>
<div> p =
<optimized
out></div>
<div>#3
sip_msg_shm_clone
(org_msg=0x7f91ae9d7630,
sip_msg_len=0x7fff56a2dde0,
clone_lumps=0) at
sip_msg_clone.c:502</div>
<div> len =
3832</div>
<div> hdr =
<optimized
out></div>
<div> new_hdr
= <optimized
out></div>
<div>
last_hdr =
<optimized
out></div>
<div> via =
<optimized
out></div>
<div> prm =
<optimized
out></div>
<div> to_prm
= <optimized
out></div>
<div>
new_to_prm =
<optimized
out></div>
<div> new_msg
= <optimized
out></div>
<div> p =
<optimized
out></div>
<div>
__FUNCTION__ =
"sip_msg_shm_clone"</div>
<div>#4
0x00007f91ac989062
in build_cell
(p_msg=0x7f91ae9d7630)
at h_table.c:372</div>
<div>
new_cell =
0x7f91a4656180</div>
<div>
sip_msg_len =
<optimized
out></div>
<div> old =
<optimized
out></div>
<div> cbs =
<optimized
out></div>
<div> cbs_tmp
= <optimized
out></div>
<div> xold =
<optimized
out></div>
<div>#5
0x00007f91ac9b5fc6
in new_t
(p_msg=0x7f91ae9d7630)
at t_lookup.c:1357</div>
<div>
new_cell =
<optimized
out></div>
<div>#6 t_newtran
(p_msg=0x7f91ae9d7630)
at t_lookup.c:1497</div>
<div> lret =
<optimized
out></div>
<div> my_err
= <optimized
out></div>
<div>
canceled =
<optimized
out></div>
<div>
__FUNCTION__ =
"t_newtran"</div>
<div>#7
0x00007f91ac9a99e0
in
t_forward_cancel
(p_msg=0x7f91ae9d7630,
proxy=0x0,
proto=0,
tran=0x7fff56a2e2c0)
at t_fwd.c:1607</div>
<div>
t_invite =
<optimized
out></div>
<div> t = 0x0</div>
<div> ret =
<optimized
out></div>
<div>
new_tran =
<optimized
out></div>
</div>
<div>
<div> dst =
{send_sock =
0x63371f, to = {s
= {sa_family =
54224, sa_data =
"\003\255\221\177\000\000\020\300\220\256\221\177\000"},
sin = {sin_family
= 54224, </div>
<div>
sin_port = 44291,
sin_addr = {s_addr
= 32657}, sin_zero
=
"\020\300\220\256\221\177\000"},
sin6 =
{sin6_family =
54224, sin6_port =
44291, </div>
<div>
sin6_flowinfo =
32657, sin6_addr =
{__in6_u =
{__u6_addr8 =
"\020\300\220\256\221\177\000\000\070\261W\000\000\000\000",
__u6_addr16 =
{49168, 44688, </div>
<div>
32657, 0,
45368, 87, 0, 0},
__u6_addr32 =
{2928721936,
32657, 5747000,
0}}},
sin6_scope_id =
0}}, id = 0, proto
= 96 '`',
send_flags = {</div>
<div> f =
225 '\341',
blst_imask = 162
'\242'}}</div>
<div> host =
{s = 0x0, len = 0}</div>
<div> port =
<optimized
out></div>
<div>
__FUNCTION__ =
"t_forward_cancel"</div>
<div>#8
0x00007f91ac99ae15
in t_relay_to
(p_msg=0x7f91ae9d7630,
proxy=0x0,
proto=0,
replicate=0) at
t_funcs.c:264</div>
<div> ret = 0</div>
<div>
new_tran =
<optimized
out></div>
<div> t =
<optimized
out></div>
<div> dst =
{send_sock = 0x0,
to = {s =
{sa_family = 0,
sa_data = '\000'
<repeats 13
times>}, sin =
{sin_family = 0,
sin_port = 0,
sin_addr = {s_addr
= 0}, </div>
<div>
sin_zero =
"\000\000\000\000\000\000\000"},
sin6 =
{sin6_family = 0,
sin6_port = 0,
sin6_flowinfo = 0,
sin6_addr =
{__in6_u = {</div>
<div>
__u6_addr8 =
'\000' <repeats
15 times>,
__u6_addr16 = {0,
0, 0, 0, 0, 0, 0,
0}, __u6_addr32 =
{0, 0, 0, 0}}},
sin6_scope_id =
0}}, id = 0, </div>
<div> proto
= 64 '@',
send_flags = {f =
243 '\363',
blst_imask = 162
'\242'}}</div>
<div> port =
<optimized
out></div>
<div> host =
{s =
0x7fff56a2eab8
"0[*\244\221\177",
len = -1398876638}</div>
<div>
__FUNCTION__ =
"t_relay_to"</div>
</div>
<div><br>
</div>
<div><br>
</div>
<br clear="all">
<div>Kelvin Chua</div>
</div>
<br>
<fieldset></fieldset>
<br>
</div>
</div>
<pre>_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
<a moz-do-not-send="true" href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a>
<a moz-do-not-send="true" href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><span><font color="#888888">
</font></span></pre>
<span><font color="#888888">
</font></span></blockquote>
<span><font color="#888888"> <br>
<pre cols="72">--
Daniel-Constantin Mierla - <a moz-do-not-send="true" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a moz-do-not-send="true" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a></pre>
</font></span></div>
<br>
_______________________________________________<br>
SIP Express Router (SER) and
Kamailio (OpenSER) - sr-users
mailing list<br>
<a moz-do-not-send="true"
href="mailto:sr-users@lists.sip-router.org"
target="_blank">sr-users@lists.sip-router.org</a><br>
<a moz-do-not-send="true"
href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users"
target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a moz-do-not-send="true" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a moz-do-not-send="true" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a></pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a moz-do-not-send="true" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a moz-do-not-send="true" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a></pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla - <a class="moz-txt-link-freetext" href="http://www.asipto.com">http://www.asipto.com</a>
<a class="moz-txt-link-freetext" href="http://twitter.com/#!/miconda">http://twitter.com/#!/miconda</a> - <a class="moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda">http://www.linkedin.com/in/miconda</a></pre>
</body>
</html>