<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hello,<br>
<br>
the problem was that a structure in shared memory (the request
cloned in tm) could have been used in parallel by different kamailio
processes.<br>
<br>
If there were two processes at the same time, parsing PAI resulted
in setting the header pointer to a private memory. The other process
could overtake in processing, using the same cloned request, and
this time the pai pointer is set, but to another private memory
zone. I added the locks for calling the callbacks, so the process
that parse the PAI is the one cleaning it.<br>
<br>
Performances should not be impacted that much, the transaction lock
is used and will add sequential processing when there are two
replies at the same time, which is not the common.<br>
<br>
Cheers,<br>
Daniel<br>
<br>
<div class="moz-cite-prefix">On 07/07/14 12:40, Igor Potjevlesch
wrote:<br>
</div>
<blockquote
cite="mid:CAJmJ=jH0Rbx7V7aHrOjEYP_LNq_k0bPpeCUNhCpKGcGq8wA4aA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>Hello,</div>
<div><br>
</div>
<div>Can you explain the modification and the impact on our
plateform?</div>
<div>Is it for the pai problem?</div>
<div><br>
</div>
<div>Do you have explanation for the km_val.c problem wich cause
crash for Kamailio too?</div>
<div><br>
</div>
<div>Regards,</div>
<div><br>
</div>
<div>Igor</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">2014-07-01 16:40 GMT+02:00
Daniel-Constantin Mierla <span dir="ltr"><<a
moz-do-not-send="true" href="mailto:miconda@gmail.com"
target="_blank">miconda@gmail.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> Hello,<br>
<br>
can you give it a try with the patch from next commit?<br>
<br>
-
<a moz-do-not-send="true"
href="http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=da9d56be28e050dd0cb4aed50efcbda043a3e5cf"
target="_blank">http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=da9d56be28e050dd0cb4aed50efcbda043a3e5cf</a><br>
<br>
If all goes fine while testing, I will backport.<br>
<br>
Cheers,<br>
Daniel
<div>
<div class="h5"><br>
<br>
<div>On 26/06/14 12:58, Igor Potjevlesch wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div>Hello,</div>
<div><br>
</div>
<div>Here the result :</div>
<div><br>
</div>
<div>(gdb) frame 6
<div><br>
#6 0x00007f127cb6dde6 in acc_onreply
(t=0x7f1274c157f0, req=0x7f1274c3ac08, <br>
reply=0x7f12804a6d70, code=200) at
acc_logic.c:501<br>
</div>
501
clean_hdr_field(hdr);<br>
(gdb) print hdr<br>
$1 = (hdr_field_t *) 0x7f1274c3c238<br>
(gdb) print *hdr<br>
$2 = {type = HDR_PAI_T, name = {<br>
s = 0x7f1274c3b6cd "P-Asserted-Identity: <a
moz-do-not-send="true"><sip:0123456789@domain;user=phone></a>\r\nP-Sig-Options:
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN
IP4 A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336
RTP/AVP 8 101 13\r\nc=IN IP4 A.B.C"..., len =
19}, body = {<br>
s = 0x7f1274c3b6e2 "<a
moz-do-not-send="true"><sip:0123456789@domain;user=phone></a>\r\nP-Sig-Options:
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN
IP4 A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336
RTP/AVP 8 101 13\r\nc=IN IP4
A.B.C.D\r\na=rtpmap:101 tele"..., len = 44},
len = 67, parsed = 0x0, next = 0x7f1274c3c278}</div>
<div><br>
</div>
<div>(gdb) frame 4
<div><br>
#4 0x000000000056e5e6 in free_pai_ppi_body
(pid_b=0x7f12803cb480)<br>
at parser/parse_ppi_pai.c:102<br>
</div>
102 pkg_free(pid_b);<br>
(gdb) print *pid_b<br>
$3 = {id = 0x0, num_ids = 0, next = 0x1d0}</div>
<div><br>
</div>
<div>This is the bt full :
<div><br>
<br>
#0 0x0000003d6f6328a5 in raise () from
/lib64/libc.so.6<br>
No symbol table info available.<br>
#1 0x0000003d6f634085 in abort () from
/lib64/libc.so.6<br>
No symbol table info available.<br>
#2 0x0000000000546d3c in qm_debug_frag
(qm=0x7f1280275010, f=0x7f12803cb450) at
mem/q_malloc.c:142<br>
</div>
__FUNCTION__ = "qm_debug_frag"
<div><br>
#3 0x0000000000548b26 in qm_free
(qm=0x7f1280275010, p=0x7f12803cb480,
file=0x6276a0 "<core>:
parser/parse_ppi_pai.c", func=0x627a00
"free_pai_ppi_body", line=102) at
mem/q_malloc.c:464<br>
</div>
f = 0x7f12803cb450<br>
size = 139717434027144<br>
next = 0xf00000000<br>
prev = 0x7f127cd79e00<br>
__FUNCTION__ = "qm_free"
<div><br>
#4 0x000000000056e5e6 in free_pai_ppi_body
(pid_b=0x7f12803cb480) at
parser/parse_ppi_pai.c:102<br>
</div>
__FUNCTION__ = "free_pai_ppi_body"
<div><br>
#5 0x000000000054fee0 in clean_hdr_field
(hf=0x7f1274c3c238) at parser/hf.c:126<br>
</div>
h_parsed = 0x7f1274c3c268<br>
__FUNCTION__ = "clean_hdr_field"
<div> <br>
#6 0x00007f127cb6dde6 in acc_onreply
(t=0x7f1274c157f0, req=0x7f1274c3ac08,
reply=0x7f12804a6d70, code=200) at
acc_logic.c:501<br>
</div>
new_uri_bk = {s = 0x7f1274b53cdf "<a
moz-do-not-send="true">sip:0987654321@GW</a>
SIP/2.0\r\nRecord-Route: <a
moz-do-not-send="true"><sip:A.B.C.D;lr=on></a>\r\nVia:
SIP/2.0/UDP
A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0\r\nVia:
SIP/2.0/UDP
A.B.C.D:2057;branch=z9hG4bK-12"..., len = 19}<br>
br = 0<br>
hdr = 0x7f1274c3c238<br>
__FUNCTION__ = "acc_onreply"
<div><br>
#7 0x00007f127cb6e30a in tmcb_func
(t=0x7f1274c157f0, type=512,
ps=0x7fff0b015580) at acc_logic.c:573<br>
</div>
__FUNCTION__ = "tmcb_func"
<div><br>
#8 0x00007f127ed68478 in
run_trans_callbacks_internal
(cb_lst=0x7f1274c15860, type=512,
trans=0x7f1274c157f0, params=0x7fff0b015580)
at t_hooks.c:290<br>
</div>
cbp = 0x7f1274ac0e90<br>
backup_from = 0x934630<br>
backup_to = 0x934638<br>
backup_dom_from = 0x934640<br>
backup_dom_to = 0x934648<br>
backup_uri_from = 0x934620<br>
backup_uri_to = 0x934628<br>
backup_xavps = 0x934760<br>
__FUNCTION__ =
"run_trans_callbacks_internal"
<div><br>
#9 0x00007f127ed6868a in
run_trans_callbacks_with_buf (type=512,
rbuf=0x7f1274c158b0, req=0x7f1274c3ac08,
repl=0x7f12804a6d70, flags=200) at
t_hooks.c:336<br>
</div>
params = {req = 0x7f1274c3ac08, rpl =
0x7f12804a6d70, param = 0x7f1274ac0ea0, code =
200, flags = 200, branch = 0, t_rbuf =
0x7f1274c158b0, dst = 0x7f1274c15900, send_buf
= {<br>
s = 0x7f1274c27620 "SIP/2.0 200
OK\r\nVia: SIP/2.0/UDP
A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID:
<a moz-do-not-send="true"
href="mailto:cb03dc02e909d3118f86009033290024@A.B.C.D%5Cr%5CnFrom"
target="_blank">cb03dc02e909d3118f86009033290024@A.B.C.D\r\nFrom</a>:
<a moz-do-not-send="true"><sip:0123456789@domain;user=phone></a>;epid=00903"...,
len = 1021}}<br>
trans = 0x7f1274c157f0
<div><br>
#10 0x00007f127ed9ac06 in relay_reply
(t=0x7f1274c157f0, p_msg=0x7f12804a6d70,
branch=0, msg_status=200,
cancel_data=0x7fff0b0158e0,
do_put_on_wait=1) at t_reply.c:2001<br>
</div>
relay = 0<br>
save_clone = 0<br>
buf = 0x7f12804a7cc0 "SIP/2.0 200
OK\r\nVia: SIP/2.0/UDP
A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID:
<a moz-do-not-send="true"
href="mailto:cb03dc02e909d3118f86009033290024@A.B.C.D%5Cr%5CnFrom"
target="_blank">cb03dc02e909d3118f86009033290024@A.B.C.D\r\nFrom</a>:
<a moz-do-not-send="true"><sip:0123456789@domain;user=phone></a>;epid=00903"...<br>
res_len = 1021<br>
relayed_code = 200<br>
relayed_msg = 0x7f12804a6d70<br>
reply_bak = 0x7fff0b015730<br>
bm = {to_tag_val = {s = 0x7f1274c16d88
"", len = 5449343}}<br>
totag_retr = 0<br>
reply_status = RPS_COMPLETED<br>
uas_rb = 0x7f1274c158b0<br>
to_tag = 0x0<br>
reason = {s = 0x10b0156e0 <Address
0x10b0156e0 out of bounds>, len = 1}<br>
onsend_params = {req = 0x200924a64,
rpl = 0x7f127edbaf90, param = 0x414cc0, code =
1, flags = 0, branch = 0, t_rbuf =
0x7f126a80c828, dst = 0x7f12804a6f68, send_buf
= {s = 0xb015700 <Address 0xb015700 out of
bounds>, len = 1024}}<br>
__FUNCTION__ = "relay_reply"
<div><br>
#11 0x00007f127ed9d0b7 in reply_received
(p_msg=0x7f12804a6d70) at t_reply.c:2499<br>
</div>
msg_status = 200<br>
last_uac_status = 183<br>
ack = 0x40 <Address 0x40 out of
bounds><br>
ack_len = 0<br>
branch = 0<br>
reply_status = -<a
moz-do-not-send="true" href="tel:2143420688"
target="_blank" value="+12143420688">2143420688</a><br>
onreply_route = 1<br>
cancel_data = {cancel_bitmap = 0,
reason = {cause = 200, u = {text = {s = 0x0,
len = 9586191}, e2e_cancel = 0x0, packed_hdrs
= {s = 0x0, len = 9586191}}}}<br>
uac = 0x7f1274c15958<br>
t = 0x7f1274c157f0<br>
lack_dst = {send_sock =
0x7f12803e4110, to = {s = {sa_family = 20496,
sa_data =
"'\200\022\177\000\000\310\036#\000\000\000\000"},
sin = {sin_family = 20496, sin_port = 32807,
sin_addr = {s_addr = 32530}, sin_zero =
"\310\036#\000\000\000\000"}, sin6 = {<br>
sin6_family = 20496, sin6_port =
32807, sin6_flowinfo = 32530, sin6_addr =
{__in6_u = {__u6_addr8 =
"\310\036#\000\000\000\000\000\360\247=\200\022\177\000",
__u6_addr16 = {7880, 35, 0, 0, 42992, 32829,
32530, 0}, __u6_addr32 = {2301640, 0,
2151524336, <br>
32530}}}, sin6_scope_id =
2150060928}}, id = 32530, proto = 72 'H',
send_flags = {f = 228 '\344', blst_imask = 61
'='}}<br>
backup_user_from = 0x934630<br>
backup_user_to = 0x934638<br>
backup_domain_from = 0x934640<br>
backup_domain_to = 0x934648<br>
backup_uri_from = 0x934620<br>
backup_uri_to = 0x934628<br>
backup_xavps = 0x934760<br>
replies_locked = 1<br>
branch_ret = 0<br>
prev_branch = 184637856<br>
blst_503_timeout = 32767<br>
hf = 0x7f12804a6d90<br>
onsend_params = {req = 0x7fff0b015960,
rpl = 0x550b94, param = 0x231dc8, code = 0,
flags = 3, branch = 0, t_rbuf =
0x7f1280275380, dst = 0x7f12803de418, send_buf
= {s = 0x7fff0b015960 "`G\223", len =
5538037}}<br>
ctx = {rec_lev = 0, run_flags = 0,
last_retcode = 0, jmp_env = {{__jmpbuf =
{139717438500712, 3644308075193502665,
4279488, 140733378027408, 0, 0,
3644308075281583049, -3644194520509117495},
__mask_was_saved = 0, __saved_mask = {__val =
{9586395, <br>
1065161476041, 124554051613,
9586471, 139717437685488, 9587300, 9586197,
361695345073193192, 9586295, 9586274,
2151546560, 139717437685488, 139717437615640,
139717438500712, 4279488, 140733378027408}}}}}<br>
__FUNCTION__ = "reply_received"
<div><br>
#12 0x000000000045d837 in do_forward_reply
(msg=0x7f12804a6d70, mode=0) at
forward.c:777<br>
</div>
new_buf = 0x0<br>
dst = {send_sock = 0x0, to = {s =
{sa_family = 0, sa_data = '\000' <repeats
13 times>}, sin = {sin_family = 0, sin_port
= 0, sin_addr = {s_addr = 0}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 =
{sin6_family = 0, sin6_port = 0, sin6_flowinfo
= 0, <br>
sin6_addr = {__in6_u =
{__u6_addr8 = '\000' <repeats 15 times>,
__u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
__u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id =
0}}, id = 0, proto = 0 '\000', send_flags = {f
= 0 '\000', blst_imask = 0 '\000'}}<br>
new_len = 32530<br>
r = 1<br>
s = 0x464804a6d78 <Address
0x464804a6d78 out of bounds><br>
len = 0<br>
__FUNCTION__ = "do_forward_reply"
<div><br>
#13 0x000000000045e0f8 in forward_reply
(msg=0x7f12804a6d70) at forward.c:860<br>
</div>
No locals.<br>
#14 0x00000000004a58e7 in receive_msg
(buf=0x924600 "SIP/2.0 200 OK\r\nVia:
SIP/2.0/UDP
185.20.8.4;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=185.20.8.4\r\nVia:
SIP/2.0/UDP
10.143.1.2:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID:
cb0"..., <br>
<div> len=1124, rcv_info=0x7fff0b015c60) at
receive.c:273<br>
</div>
msg = 0x7f12804a6d70<br>
ctx = {rec_lev = 8868984, run_flags =
0, last_retcode = 0, jmp_env = {{__jmpbuf =
{0, 0, 0, 263853236176, 1, 0, 169653785368,
9586112}, __mask_was_saved = 184638568,
__saved_mask = {__val = {139717436454816,
12884901899, 139717436454816, 4279488, <br>
140733378027408,
140733378026464, 5477954, 0, 139717072962944,
50195, 169290548608, 9586112, 140733378026592,
140733378026512, 5474789, 4279488}}}}}<br>
ret = 32530<br>
inb = {s = 0x924600 "SIP/2.0 200
OK\r\nVia: SIP/2.0/UDP
A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=A.B.C.D\r\nVia:
SIP/2.0/UDP
A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID:
cb0"..., len = 1124}<br>
__FUNCTION__ = "receive_msg"
<div><br>
#15 0x000000000053c9a8 in udp_rcv_loop () at
udp_server.c:536<br>
</div>
len = 1124<br>
buf = "SIP/2.0 200 OK\r\nVia:
SIP/2.0/UDP
A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=A.B.C.D\r\nVia:
SIP/2.0/UDP
A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID:
cb0"...<br>
tmp = 0x9245c0 "10.143.1.10"<br>
from = 0x7f12803e3f68<br>
fromlen = 16<br>
ri = {src_ip = {af = 2, len = 4, u =
{addrl = {403182777, 139717436454816}, addr32
= {403182777, 0, 2150315936, 32530}, addr16 =
{5305, 6152, 0, 0, 14240, 32811, 32530, 0},
addr =
"\271\024\b\030\000\000\000\000\240\067+\200\022\177\000"}},
dst_ip = {af = 2, <br>
len = 4, u = {addrl = {67638457,
0}, addr32 = {67638457, 0, 0, 0}, addr16 =
{5305, 1032, 0, 0, 0, 0, 0, 0}, addr =
"\271\024\b\004", '\000' <repeats 11
times>}}, src_port = 5060, dst_port = 5060,
proto_reserved1 = 0, proto_reserved2 = 0,
src_su = {s = {<br>
sa_family = 2, sa_data =
"\023Ĺ\024\b\030\000\000\000\000\000\000\000"},
sin = {sin_family = 2, sin_port = 50195,
sin_addr = {s_addr = 403182777}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 =
{sin6_family = 2, sin6_port = 50195, <br>
sin6_flowinfo = 403182777,
sin6_addr = {__in6_u = {__u6_addr8 = '\000'
<repeats 15 times>, __u6_addr16 = {0, 0,
0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0,
0}}}, sin6_scope_id = 0}}, bind_address =
0x7f12802b3638, proto = 1 '\001'}<br>
__FUNCTION__ = "udp_rcv_loop"
<div><br>
#16 0x000000000046d42b in main_loop () at
main.c:1617<br>
</div>
i = 1<br>
pid = 0<br>
si = 0x7f12802b3638<br>
si_desc = "udp receiver child=1
sock=A.B.C.D:5060\000D\200\022\177\000\000\000\206\063\200\022\177\000\000.\205^\000\000\000\000\000\020w^\000\000\000\000\000\275\005r/\000\000\000\000\300LA\000\000\000\000\000\220_\001\v\377\177",
'\000' <repeats 18 times>"\320,
]\001\v\377\177\000\000\364\244K\000\000\000\000"<br>
nrprocs = 15<br>
__FUNCTION__ = "main_loop"
<div><br>
#17 0x0000000000470533 in main (argc=7,
argv=0x7fff0b015f98) at main.c:2545<br>
</div>
cfg_stream = 0xe20010<br>
c = -1<br>
r = 0<br>
tmp = 0x7fff0b017f70 ""<br>
tmp_len = 0<br>
port = 0<br>
proto = 0<br>
options = 0x5e0a68
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"<br>
ret = -1<br>
seed = 1972285608<br>
rfd = 4<br>
debug_save = 0<br>
debug_flag = 0<br>
dont_fork_cnt = 0<br>
n_lst = 0x3d6f60fb88<br>
p = 0x5cab80
"H\211l$\330L\211d$\340H\215-o\244*"<br>
__FUNCTION__ = "main"<br>
</div>
<div><br>
</div>
<div>In a next mail you will find a new bt full
of Kamailio 's crash but about km_val.c :
db_mysql_val2str</div>
<div><br>
</div>
</div>
<div class="gmail_extra"> <br>
<br>
<div class="gmail_quote">2014-06-25 18:26
GMT+02:00 Daniel-Constantin Mierla <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:miconda@gmail.com"
target="_blank">miconda@gmail.com</a>></span>:</div>
</div>
</div>
<div class="gmail_extra"> <br>
<br>
<div class="gmail_quote">2014-06-25 18:26
GMT+02:00 Daniel-Constantin Mierla <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:miconda@gmail.com"
target="_blank">miconda@gmail.com</a>></span>:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
Hello,<br>
<br>
can you give the output of:<br>
<br>
frame 6<br>
print hdr<br>
print *hdr<br>
<br>
frame 4<br>
print *pid_b<br>
<br>
Also, it would be good to have full trace for
other details:<br>
<br>
bt full<br>
<br>
Cheers,<br>
Daniel
<div><br>
<br>
On 25/06/14 14:49, Igor Potjevlesch wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">Hello,<br>
<br>
<div> We updated this morning Kamailio in
4.1.4 with your patch.<br>
</div>
[...]<br>
</blockquote>
<div>
<div> <br>
-- <br>
Daniel-Constantin Mierla - <a
moz-do-not-send="true"
href="http://www.asipto.com"
target="_blank">http://www.asipto.com</a><br>
<a moz-do-not-send="true"
href="http://twitter.com/#%21/miconda"
target="_blank">http://twitter.com/#!/miconda</a>
- <a moz-do-not-send="true"
href="http://www.linkedin.com/in/miconda"
target="_blank">http://www.linkedin.com/in/miconda</a><br>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a moz-do-not-send="true" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a moz-do-not-send="true" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a></pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla - <a class="moz-txt-link-freetext" href="http://www.asipto.com">http://www.asipto.com</a>
<a class="moz-txt-link-freetext" href="http://twitter.com/#!/miconda">http://twitter.com/#!/miconda</a> - <a class="moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda">http://www.linkedin.com/in/miconda</a></pre>
</body>
</html>