<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Hello,</p>
<p>I think you were hit by an issue solved with commit
15fc8b9c59aaf31f005e38f54d363f1e9d0a068e :<br>
</p>
<a class="moz-txt-link-freetext"
href="https://github.com/kamailio/kamailio/commit/15fc8b9c59aaf31f005e38f54d363f1e9d0a068e">https://github.com/kamailio/kamailio/commit/15fc8b9c59aaf31f005e38f54d363f1e9d0a068e</a><br>
<br>
The 4.1.3 was released before, in April 2014:<br>
<br>
- <a class="moz-txt-link-freetext"
href="https://www.kamailio.org/pub/kamailio/4.1.3/README">https://www.kamailio.org/pub/kamailio/4.1.3/README</a><br>
<br>
I am not sure if it was backported to 4.1 branch, but should not be
hard to backport.<br>
<br>
The issues was with many processing handling the same transaction,
which has the sip_msg in shared memory, but then parsing of some
headers created pointers to private memory of the process doing the
parsing. Another process coming shortly after would see the pointer
in sip_msg, but it would be to another process private memory and
accessing it does a seg fault as expected.<br>
<br>
Not all headers are cloned in shared memory, like it's done for
To/From, and results in this kind of processing with parsing again
in private memory.<br>
<br>
Anyhow, I didn't have much time to dig in too much, so it's mu quick
guess based on the fact you run a rather old release.<br>
<br>
Cheers,<br>
Daniel<br>
<br>
<div class="moz-cite-prefix">On 14/01/2017 17:53, Matthew Jordan
wrote:<br>
</div>
<blockquote
cite="mid:CAN2PU+6hjZXgFWNQZz7vu5sKWx+5Dmc86WMTrCQU3Z06PSZf+g@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Jan 13, 2017 at 2:57 PM,
Joshua Colp <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:jcolp@digium.com" target="_blank">jcolp@digium.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">Greetings all,<br>
<br>
I'm currently investigating a crash in a Kamailio 4.1.3
deployment. The<br>
backtrace is as follows:<br>
<br>
(I've commented out phone numbers and IP addresses using X
but have<br>
retained the correct length for each)<br>
<br>
(gdb) bt<br>
#0 0x00007f017f2ac96e in memcpy () from /lib64/libc.so.6<br>
#1 0x00007f0174e1bb50 in rc_avpair_assign () from<br>
/usr/lib64/libradiusclient-ng.<wbr>so.2<br>
#2 0x00007f0174e1bc92 in rc_avpair_new () from<br>
/usr/lib64/libradiusclient-ng.<wbr>so.2<br>
#3 0x00007f0174e1bdd1 in rc_avpair_add () from<br>
/usr/lib64/libradiusclient-ng.<wbr>so.2<br>
#4 0x00007f0175026806 in acc_radius_send_request
(req=0x7f0162b6c1f0,<br>
inf=0x7fff113e9cc0) at acc_radius_mod.c:365<br>
#5 0x00007f017522ff13 in acc_run_engines
(msg=0x7f0162b6c1f0, type=0,<br>
reset=0x0) at acc.c:957<br>
#6 0x00007f0175239b49 in acc_onreply (t=0x7f0161a12f00,<br>
req=0x7f0162b6c1f0, reply=0x7f017ca845c8, code=200) at
acc_logic.c:485<br>
#7 0x00007f017523a07d in tmcb_func (t=0x7f0161a12f00,
type=512,<br>
ps=0x7fff113e9ee0) at acc_logic.c:559<br>
#8 0x00007f0177434478 in run_trans_callbacks_internal<br>
(cb_lst=0x7f0161a12f70, type=512, trans=0x7f0161a12f00,<br>
params=0x7fff113e9ee0) at t_hooks.c:290<br>
#9 0x00007f017743468a in run_trans_callbacks_with_buf
(type=512,<br>
rbuf=0x7f0161a12fc0, req=0x7f0162b6c1f0,
repl=0x7f017ca845c8, flags=200)<br>
at t_hooks.c:336<br>
#10 0x00007f0177466c06 in relay_reply (t=0x7f0161a12f00,<br>
p_msg=0x7f017ca845c8, branch=0, msg_status=200,<br>
cancel_data=0x7fff113ea240, do_put_on_wait=1)<br>
at t_reply.c:2001<br>
#11 0x00007f01774690b7 in reply_received
(p_msg=0x7f017ca845c8) at<br>
t_reply.c:2499<br>
#12 0x000000000045d837 in do_forward_reply
(msg=0x7f017ca845c8, mode=0)<br>
at forward.c:777<br>
#13 0x000000000045e0f8 in forward_reply
(msg=0x7f017ca845c8) at<br>
forward.c:860<br>
#14 0x00000000004a5887 in receive_msg (<br>
buf=0x9245e0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP<br>
X.XX.XX.XX:5060;rport=5060;<wbr>received=X.XX.XX.XX;branch=<wbr>z9hG4bK9981.<wbr>5d2e8535c8eb94d43283f83230b501<wbr>1e.0\r\nVia:<br>
SIP/2.0/UDP X.XX.XX.XX;rport=5060;branch=<wbr>z9hG4bK9981.78d8b40"...,<br>
len=1161, rcv_info=0x7fff113ea5c0) at receive.c:273<br>
#15 0x000000000053c838 in udp_rcv_loop () at
udp_server.c:536<br>
#16 0x000000000046d42b in main_loop () at main.c:1617<br>
#17 0x00000000004704d3 in main (argc=11,
argv=0x7fff113ea8f8) at<br>
main.c:2533<br>
<br>
As you can see it's crashing when interacting with the
radius client.<br>
<br>
Looking at frame 4 we can examine what is being provided
to radius:<br>
<br>
(gdb) frame 4<br>
#4 0x00007f0175026806 in acc_radius_send_request
(req=0x7f0162b6c1f0,<br>
inf=0x7fff113e9cc0) at acc_radius_mod.c:365<br>
365 ADD_RAD_AVPAIR(offset+i,
inf->varr[i].s,<br>
inf->varr[i].len);<br>
(gdb) print inf->varr[i].s<br>
$12 = 0x0<br>
(gdb) print inf->varr[i].len<br>
$13 = 3<br>
(gdb) print inf->tarr[i]<br>
$14 = 2 '\002'<br>
<br>
This is a TYPE_STR with a length of 3, but no actual
string. Needless to<br>
say this upsets radius. Based on the configuration of the
system and the<br>
value of 'i' this is actually for the Remote-Party-ID
header:<br>
<br>
(gdb) print i<br>
$15 = 8<br>
(gdb) print *rad_extra->next->next->next-><wbr>next<br>
$16 = {name = {s = 0x7f017b27fcf4 "Sip-RPid", len = 8},
spec = {type =<br>
PVT_OTHER, getf = 0x7f0176b79883 <pv_get_rpid>, setf
= 0, pvp = {pvn =<br>
{type = 0,<br>
nfree = 0, u = {isname = {type = 0, name = {n = 0,
s = {s = 0x0,<br>
len = 0}, re = 0x0}}, dname = 0x0}}, pvi = {type =
0, u = {ival<br>
= 0, dval = 0x0}}},<br>
trans = 0x0}, next = 0x7f017b3ae4b0}<br>
<br>
Going up a frame and looking at the RPID URI on the
message it matches<br>
the bad TYPE_STR:<br>
<br>
(gdb) frame 5<br>
#5 0x00007f017522ff13 in acc_run_engines
(msg=0x7f0162b6c1f0, type=0,<br>
reset=0x0) at acc.c:957<br>
957 e->acc_req(msg,
&inf);<br>
(gdb) print ((struct to_body*)(msg)->rpid->parsed)-<wbr>>uri<br>
$17 = {s = 0x0, len = 3}<br>
<br>
And digging into the full parsed information shows that
it's not correct<br>
at all:<br>
(gdb) print *((struct to_body*)(msg)->rpid->parsed)<br>
$30 = {error = 3, body = {s = 0x0, len = 11064280}, uri =
{s = 0x0, len<br>
= 3}, display = {s = 0x0, len = 11064313}, tag_value = {s
= 0x0, len =<br>
5},<br>
parsed_uri = {user = {s = 0x0, len = 1484108237}, passwd
= {s = 0x0,<br>
len = 2}, host = {s = 0x0, len = 0}, port = {s = 0x0,
len = 3}, params<br>
= {s = 0x0,<br>
len = 11064395}, sip_params = {s = 0x0, len = 0},
headers = {s =<br>
0x0, len = 8619}, port_no = 0, proto = 0, type =
ERROR_URI_T,<br>
flags = 0, transport = {<br>
s = 0x0, len = 0}, ttl = {s = 0x0, len = 0},
user_param = {s =<br>
0x0, len = 192}, maddr = {s = 0x0, len = 3}, method
= {s = 0x0,<br>
len = 11064439}, lr = {<br>
s = 0x0, len = 3}, r2 = {s = 0x0, len = 11064464},
gr = {s = 0x0,<br>
len = 3}, transport_val = {s = 0x0, len =
2039735278}, ttl_val =<br>
{s = 0x0, len = 3},<br>
user_param_val = {s = 0x0, len = 11064487}, maddr_val
= {s = 0x0,<br>
len = 0}, method_val = {s = 0x0, len = 16367}, lr_val
= {s = 0x0,<br>
len = 5}, r2_val = {<br>
s = 0x0, len = 1484104637}, gr_val = {s = 0x0, len =
3}},<br>
param_lst = 0x0, last_param = 0xa8d4d6}<br>
<br>
Looking at the actual Remote-Party-ID header on the other
hand itself<br>
shows nothing out of the ordinary:<br>
<br>
(gdb) print *msg->rpid<br>
$18 = {type = HDR_RPID_T, name = {<br>
s = 0x7f0162b6cba0 "Remote-Party-ID:<br>
<<a class="moz-txt-link-freetext"
href="sip:XXXXXXXXXX@XXX.XX.XX.XX">sip:XXXXXXXXXX@XXX.XX.XX.XX</a>:<wbr>5060>;privacy=off\r\nP-Charge-<wbr>Info:<br>
<a class="moz-txt-link-freetext"
href="sip:XXXXXXXXXX@XXX.XX.XX.XX">sip:XXXXXXXXXX@XXX.XX.XX.XX</a>:<wbr>5060\r\nSupported:<br>
timer,replaces\r\nSession-<wbr>Expires:
3600\r\nMin-SE:<br>
90\r\nContent-Length: 260\r\nCo"..., len = 15}, body
= {<br>
s = 0x7f0162b6cbb1<br>
"<<a class="moz-txt-link-freetext"
href="sip:XXXXXXXXXX@XXX.XX.XX.XX">sip:XXXXXXXXXX@XXX.XX.XX.XX</a>:<wbr>5060>;privacy=off\r\nP-Charge-<wbr>Info:<br>
<a class="moz-txt-link-freetext"
href="sip:XXXXXXXXXX@XXX.XX.XX.XX">sip:XXXXXXXXXX@XXX.XX.XX.XX</a>:<wbr>5060\r\nSupported:<br>
timer,replaces\r\nSession-<wbr>Expires:
3600\r\nMin-SE:<br>
90\r\nContent-Length: 260\r\nContent-Disposition"...<wbr>,
len = 46},<br>
len = 65, parsed = 0x7f017b33c710, next =
0x7f0162b6d720}<br>
<br>
I should also add that the value before and after RPID
appears to be<br>
correct:<br>
<br>
(gdb) print inf->varr[i-1].s<br>
$24 = 0x7f0161ce8427 "<a class="moz-txt-link-freetext"
href="sip:XXXXXXXXXX@X.XX.XX.XXX">sip:XXXXXXXXXX@X.XX.XX.XXX</a>:<wbr>5060<br>
SIP/2.0\r\nRecord-Route:<br>
<<a class="moz-txt-link-freetext"
href="sip:X.XX.XX.XX;lr=on;ftag=">sip:X.XX.XX.XX;lr=on;ftag=</a><wbr>gK0c7f2820;did=438.50e1>\r\<wbr>nRecord-Route:<br>
<<a class="moz-txt-link-freetext"
href="sip:X.XX.XX.XX;lr=on;did=ee5">sip:X.XX.XX.XX;lr=on;did=ee5</a>.<wbr>4da5ae1>\r\nVia:
SIP/2.0/UDP<br>
X.XX.XX.XX:5060;branc"...<br>
(gdb) print inf->varr[i-1].len<br>
$25 = 31<br>
(gdb) print inf->varr[i+1].s<br>
$26 = 0x7f0176dbe720 "X.XX.XX.XX"<br>
(gdb) print inf->varr[i+1].len<br>
$27 = 10<br>
<br>
Has anyone seen anything like this before or have a
suggestion on where<br>
to look further? I've traced through the code in question
and it all<br>
seems to be correct.<br>
</blockquote>
<div><br>
</div>
<div>We have a theory that we hope to test out later today.
If anyone more familiar with the inner workings of
Kamailio could comment on it, that'd be hugely
appreciated.</div>
<div><br>
</div>
<div>Looking through the Kamailio code base, not many things
manipulate rpid->parsed once it has been set. One place
that it could be manipulated is in sip_msg_shm_clone.
Comparing the handling of the rpid header in that routine
to the from/to headers is instructive - the from/to header
handling directly handles the case of parsed being set,
while rpid does not:</div>
<div><br>
</div>
<div>from/to:</div>
<div><br>
</div>
<div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>case
HDR_TO_T:</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>case
HDR_FROM_T:</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>if
(hdr->type == HDR_TO_T) {</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>if
(!HOOK_SET(to)) new_msg->to = new_hdr;</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>}
else {</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>if
(!HOOK_SET(from)) new_msg->from = new_hdr;</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>}</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>
/* From header might be unparsed */</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>if
(!hdr->parsed) break;</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>new_hdr->parsed
= p;</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>p
+=ROUND4(sizeof(struct to_body));</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>memcpy(new_hdr->parsed,
hdr->parsed, sizeof(struct to_body));</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>((struct
to_body*)new_hdr->parsed)->body.s =</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>translate_pointer(
new_msg->buf , org_msg->buf ,</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>
((struct to_body*)hdr->parsed)->body.s );</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>((struct
to_body*)new_hdr->parsed)->display.s =</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>translate_pointer(
new_msg->buf, org_msg->buf,</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>
((struct to_body*)hdr->parsed)->display.s);</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>((struct
to_body*)new_hdr->parsed)->uri.s =</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>translate_pointer(
new_msg->buf , org_msg->buf ,</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>
((struct to_body*)hdr->parsed)->uri.s );</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>if
( ((struct to_body*)hdr->parsed)->tag_value.s )</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>((struct
to_body*)new_hdr->parsed)->tag_value.s =</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>translate_pointer(
new_msg->buf , org_msg->buf ,</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>
((struct to_body*)hdr->parsed)->tag_value.s );</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>if
( (((struct
to_body*)new_hdr->parsed)->parsed_uri.user.s)</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>||
(((struct
to_body*)new_hdr->parsed)->parsed_uri.host.s) )</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>uri_trans(new_msg->buf,
org_msg->buf,</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>&((struct
to_body*)new_hdr->parsed)->parsed_uri);</div>
</div>
<div><br>
</div>
<div>etc.</div>
<div><br>
</div>
<div>rpid:</div>
<div><br>
</div>
<div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>case
HDR_RPID_T:</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>if
(!HOOK_SET(rpid)) {</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>new_msg->rpid
= new_hdr;</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>}</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>break;</div>
</div>
<div><br>
</div>
<div>Since the rpid->parsed pointer points to a struct
to_body, I would expect it would need the same handling as
the from/to headers.</div>
<div><br>
</div>
<div>It's interesting to note in our backtrace that the
crash occurs in a reply route, and - I think - Kamailio
will clone the SIP message prior to executing the reply
route (again - I think). Since the crash only happens if
you evaluate the rpid pseudo variable, and since the
Radius configuration we're using maps an 'extra field' to
said psuedo variable, that might explain the issue we're
seeing.</div>
<div><br>
</div>
<div>Does this sound plausible, or are we potentially going
down the wrong path here?</div>
</div>
<br clear="all">
<div>-- <br>
</div>
<div class="gmail_signature">Matthew Jordan<br>
Digium, Inc. | CTO<br>
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA<br>
Check us out at: <a moz-do-not-send="true"
href="http://digium.com" target="_blank">http://digium.com</a>
& <a moz-do-not-send="true" href="http://asterisk.org"
target="_blank">http://asterisk.org</a></div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
sr-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:sr-dev@lists.sip-router.org">sr-dev@lists.sip-router.org</a>
<a class="moz-txt-link-freetext" href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda">www.linkedin.com/in/miconda</a>
Kamailio World Conference - May 8-10, 2017 - <a class="moz-txt-link-abbreviated" href="http://www.kamailioworld.com">www.kamailioworld.com</a></pre>
</body>
</html>