<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hello,<br>
<br>
yes, I'll apply it, just that I was mainly offline these days to had
a proper chance to look at it.<br>
<br>
Thanks for troubleshooting and patching,<br>
Daniel<br>
<br>
<div class="moz-cite-prefix">On 9/20/12 2:36 PM, Jijo wrote:<br>
</div>
<blockquote
cite="mid:CAOYmDE-uiSsi+hwj-tEw10iaY6GjtNWQ2hgySA-zPK8Sq75vdA@mail.gmail.com"
type="cite">Hi Daniel,
<div><br>
</div>
<div>This patch needs to be applied to avoid core.</div>
<div><br>
</div>
<div>Thanks</div>
<div>Jijo<br>
<br>
<div class="gmail_quote">On Wed, Sep 19, 2012 at 10:54 AM, Jijo
<span dir="ltr"><<a moz-do-not-send="true"
href="mailto:realjijo@gmail.com" target="_blank">realjijo@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><br>
</div>
Hi All,
<div><br>
</div>
<div>Finally i found the issue, </div>
<div><br>
</div>
<div>
<p class="MsoNormal"><span style="color:#1f497d">Here is
one of the bad trace for
SUBSCRIBE(722bytes) and NOTIFY(1282bytes) which
corrupted the memory. The
messages came in the order NOTIFY and SUBSCRIBE. The
core is generated in a
different place but I believe this could be the reason
for memory corruption. </span></p>
<p class="MsoNormal"><span style="color:#1f497d">Here is
the trace UDP Process </span><span
style="color:#c00000">27294</span><span
style="color:#1f497d"> processing
NOTIFY and Process </span><span style="color:#00b050">27303</span><span
style="font-size:8.0pt;color:#00b050"> </span><span
style="color:#1f497d">processing
SUBSCRIBE .</span></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="color:#1f497d">The
explanation and
implementation is below</span></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:17+01:00
[info]
sipserver: [27303] INFO: <script>: <a
moz-do-not-send="true"
href="mailto:CI=1-3292@10.233.20.152"
target="_blank">CI=1-3292@10.233.20.152</a> -R39 -
Entry
R-URI=1234 FD=10.233.20.152 SI=10.233.20.152</span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:17+01:00
[info]
sipserver: [</span><span
style="font-size:8.0pt;color:#c00000">27294</span><span
style="font-size:8.0pt;color:#1f497d">] INFO:
<script>: <a moz-do-not-send="true"
href="mailto:CI=1-3292@10.233.20.152"
target="_blank">CI=1-3292@10.233.20.152</a> -R1 -
Force
LAN socket: tcp:<a moz-do-not-send="true"
href="http://10.233.20.151:5060" target="_blank">10.233.20.151:5060</a>
<null></span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:17+01:00
[info]
sipserver: [</span><span
style="font-size:8.0pt;color:#00b050">27303</span><span
style="font-size:8.0pt;color:#1f497d">] INFO:
<script>: <a moz-do-not-send="true"
href="mailto:CI=1-3292@10.233.20.152"
target="_blank">CI=1-3292@10.233.20.152</a> -R1 -
Force
LAN socket: tcp:<a moz-do-not-send="true"
href="http://10.233.20.151:5060" target="_blank">10.233.20.151:5060</a>
<null></span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:17+01:00
[err]
sipserver: [27303] ERROR: <core>
[tcp_main.c:2357]:
tcp_conn_send_put : calling wbufq_add</span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:17+01:00
[err]
sipserver: [27303] ERROR: <core>
[tcp_main.c:730]: ERROR:
wbufq_add(722 bytes): buf:</span><span
style="font-size:8.0pt;color:#00b050">SUBSCRIBE
</span><span style="font-size:8.0pt;color:#1f497d"><a
moz-do-not-send="true">sip:1234@10.233.20.141:5063;transport=tcp</a>
SIP/2.0</span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d">Record-Route:
<<a class="moz-txt-link-freetext" href="sip:10.233.20.151;tran">sip:10.233.20.151;tran</a></span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:17+01:00
[err]
sipserver: [27303] ERROR: <core>
[tcp_main.c:747]: ERROR:
wbufq_add(722 bytes): first:b00519f4 last:b00519f4</span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:17+01:00
[err]
sipserver: [27303] ERROR: <core>
[tcp_main.c:774]: ERROR:
wbufq_add(2 last free crt_size:722): first:b00519f4
last:b00519f4</span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:17+01:00
[err]
sipserver: [27294] ERROR: <core>
[tcp_main.c:796]: ERROR:
wbufq_insert(1282 bytes): buf:</span><span
style="font-size:8.0pt;color:#c00000">NOTIFY
</span><span style="font-size:8.0pt;color:#1f497d"><a
moz-do-not-send="true">sip:1234@10.233.20.141:5063;transport=tcp</a>
SIP/2.0</span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d">Record-Route:
<<a class="moz-txt-link-freetext" href="sip:10.233.20.151;transpo">sip:10.233.20.151;transpo</a></span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:17+01:00
[err]
sipserver: [27294] ERROR: <core>
[tcp_main.c:801]: ERROR: wbufq_insert(2
last free ): first:b00519f4 last:b00519f4</span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:17+01:00
[err]
sipserver: [27294] ERROR: <core>
[tcp_main.c:820]: ERROR:
wbufq_insert(22 last free ): first:b00519f4
last:b00519f4</span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:17+01:00
[err]
sipserver: [27359] ERROR: <core>
[tcp_main.c:887]: ERROR:
wbufq_run(3 last free ): first:b00519f4 last:b00519f4</span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:17+01:00
[crit]
sipserver: [27359] : <core>
[mem/q_malloc.c:157]: BUG: qm_*:
fragm. 0xb00519dc (address 0xb00519f4) end
overwritten(0, 0)!</span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:18+01:00
[alert]
sipserver: [27265] ALERT: <core> [main.c:755]:
child process
27359 exited by a signal 11</span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:18+01:00
[alert]
sipserver: [27265] ALERT: <core> [main.c:758]:
core was generated</span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:18+01:00
[info]
sipserver: [27265] INFO: <core> [main.c:770]:
INFO: terminating
due to SIGCHLD</span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="color:#1f497d">Process
27294(NOTIFY) created
the TCP connection structure for destination IP and
just before calling
wbufq_insert(), context switch happened and process
27303(SUBSCRIBE) got the
cpu. Since the connection structure is already
available process 27303 add the
SUBSCRIBE message(722 bytes) to wbufq. Afterwards
process 27294 got the CPU and
invoked wbufq_insert() which basically corrupted the
memory due to an overflow
with the existing implementation.</span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d"> </span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d"> </span></p>
<p class="MsoNormal"><span
style="font-size:9.0pt;color:#1f497d">inline static
int _wbufq_insert(struct tcp_connection* c, const
char* data, </span></p>
<p class="MsoNormal"><span
style="font-size:9.0pt;color:#1f497d">
unsigned
int size)</span></p>
<p class="MsoNormal"><span
style="font-size:9.0pt;color:#1f497d">{</span></p>
<p class="MsoNormal"><span
style="font-size:9.0pt;color:#1f497d">
struct tcp_wbuffer_queue* q;</span></p>
<p class="MsoNormal"><span
style="font-size:9.0pt;color:#1f497d">
struct tcp_wbuffer* wb;</span></p>
<p class="MsoNormal"><span
style="font-size:9.0pt;color:#1f497d">
</span></p>
<p class="MsoNormal"><span
style="font-size:9.0pt;color:#1f497d">
q=&c->wbuf_q;</span></p>
<p class="MsoNormal"><span
style="font-size:9.0pt;color:#1f497d">
if (likely(q->first==0)) /* if empty, use wbufq_add
*/</span></p>
<p class="MsoNormal"><span
style="font-size:9.0pt;color:#1f497d">
return
_wbufq_add(c, data, size);</span></p>
<p class="MsoNormal"><span
style="font-size:9.0pt;color:#1f497d">:</span></p>
<p class="MsoNormal"><span
style="font-size:9.0pt;color:#1f497d">:</span></p>
<p class="MsoNormal"><span
style="font-size:9.0pt;color:#1f497d">:</span></p>
<p class="MsoNormal"><span
style="font-size:9.0pt;color:#1f497d">:</span></p>
<p class="MsoNormal"><span
style="font-size:9.0pt;color:#00b050">
if ((q->first==q->last) &&
((q->last->b_size-q->last_used)>=size)){</span></p>
<p class="MsoNormal"><span
style="font-size:9.0pt;color:#00b050">
/*
one block with enough space in it for size bytes */</span></p>
<p class="MsoNormal"><span
style="font-size:9.0pt;color:#00b050"> </span></p>
<p class="MsoNormal"><span
style="font-size:9.0pt;color:#1f497d">
</span><span style="font-size:9.0pt;color:#c00000">memmove(q->first->buf+size,
q->first->buf,
size);</span></p>
<p class="MsoNormal"><span
style="font-size:9.0pt;color:#1f497d">
memcpy(q->first->buf,
data, size);</span></p>
<p class="MsoNormal"><span
style="font-size:9.0pt;color:#1f497d">
q->last_used+=size;</span></p>
<p class="MsoNormal"><span
style="font-size:9.0pt;color:#00b050">
}</span></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="color:#1f497d">The above
condition shall be
true in this case and memmove was moving the pointer
which was causing the
overflow.</span></p>
<p class="MsoNormal"><span style="color:#1f497d">
memmove(void
*dest, const void *src, size_t n); </span></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="color:#1f497d">As per
the memmove man page, the
src shall be copied with size ‘n’ to a temporary
buffer and then temporary
buffer to dest.</span></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d">dest is
q->first->buf+size: which is basically
(q->first->buf +
NOTIFY MSG SIZE). so the dst will move my 1282 bytes,
so we have remaining
space of only 818 bytes.</span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d"> </span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d">src is
q->first->buf: which is basically copied to temp
buffer with NOTIFY
SIZE(1282bytes).</span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d"> </span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d">Finally we are
moving the buffer from temporary buffer of size 1282
bytes to buffer which we
left with 818 bytes, This basically corrupt the memory
and on wbufq_run we see
the memory corruption</span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#1f497d"> </span></p>
<p class="MsoNormal"><span
style="font-size:8.0pt;color:#c00000">2012-09-19T02:06:17+01:00
[crit]
sipserver: [27359] : <core>
[mem/q_malloc.c:157]: BUG: qm_*:
fragm. 0xb00519dc (address 0xb00519f4) end
overwritten(0, 0)!</span></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="color:#1f497d">I think
we don’t need memove, so
we can change the code as below</span></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="color:#1f497d"> if
((q->first==q->last) &&
((q->last->b_size-q->last_used)>=size)){</span></p>
<p class="MsoNormal"><span style="color:#1f497d">
/*
one block with enough space in it for size bytes */</span></p>
<p class="MsoNormal"><s><span style="color:#c00000">
//memmove(q->first->buf+size,
q->first->buf, size);</span></s></p>
<p class="MsoNormal"><span style="color:#1f497d">
memcpy(q->first->buf+q->last_used,
data, size);</span></p>
<p class="MsoNormal"><span style="color:#1f497d">
q->last_used+=size;</span></p>
<p class="MsoNormal"><span style="color:#1f497d">
}</span></p>
<div><br>
</div>
<font color="#1f497d">OR</font></div>
<div> we need only the else part as it always add the block
to the first.</div>
<div><br>
</div>
<div>Thanks</div>
<span class="HOEnZb"><font color="#888888">
<div>
Jijo</div>
</font></span>
<div class="HOEnZb">
<div class="h5">
<div><br>
<div class="gmail_quote">
On Wed, Jul 18, 2012 at 12:42 PM, Daniel-Constantin
Mierla <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Hello,<br>
<br>
sorry, I just read the last messages in the
thread and I didn't noticed it is about a crash,
but thought it is about an annoying log message.<br>
<br>
The backtrace is no longer matching the latest
sources of the 3.1 branch, but I expect it is
due to a double free issue, so you have to
update to latest 3.1.6, as suggested by other
users here. There were many fixes from 3.1.0 to
3.1.6.<br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
<div>On 7/17/12 6:03 PM, Jijo wrote:<br>
</div>
<blockquote type="cite">Hi,
<div><br>
</div>
<div>This is not happening at shutdown or
status check. Its aborting when the
system is active.</div>
<div><br>
Thanks</div>
<div>Jijo<br>
<div class="gmail_quote">On Tue, Jul 17,
2012 at 3:06 AM, Daniel-Constantin
Mierla <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:miconda@gmail.com"
target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF"
text="#000000"> Hello,<br>
<br>
does it keep being or it is one
time and that's it?<br>
<br>
That is printed at shut down or if
pkg_status() or shm_status() is
executed from some part of code or
in config via cfgutils module
functions.<br>
<br>
You can get rid of them by setting
memdbg and memlog to a value
higher than debug global
parameter.<br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
<div>On 7/16/12 5:28 PM, Jijo
wrote:<br>
</div>
<blockquote type="cite">Thanks.. It
is not easy to upgrade as
it is happening at customer
system.
<div>Is there any
change occurred for this
issue.I looked at it, but
didn't see anything in
q_malloc.c/qm_status()</div>
<div><br>
</div>
<div>
<div> <br>
<div class="gmail_quote">On
Mon, Jul 16, 2012 at
11:12 AM, Jon Bonilla
<span dir="ltr"><<a
moz-do-not-send="true" href="mailto:manwe@aholab.ehu.es" target="_blank">manwe@aholab.ehu.es</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
El Mon, 16 Jul 2012
10:27:42 -0400<br>
Jijo <<a
moz-do-not-send="true"
href="mailto:realjijo@gmail.com" target="_blank">realjijo@gmail.com</a>>
escribió:<br>
<div><br>
> Hi All,<br>
><br>
> I'm observing
a core
intermittently at
"qm_status
(qm=0x786cd000) at<br>
>
mem/q_malloc.c:763"
for kamailio
version 3.1.0<br>
><br>
<br>
</div>
I'd say that you're
using a very old
version. You should
update your branch
to<br>
3.1.6 or upgrade to
a newer branch.<br>
<br>
<br>
_______________________________________________<br>
sr-dev mailing list<br>
<a
moz-do-not-send="true"
href="mailto:sr-dev@lists.sip-router.org" target="_blank">sr-dev@lists.sip-router.org</a><br>
<a
moz-do-not-send="true"
href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev"
target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev</a><br>
</blockquote>
</div>
<br>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
sr-dev mailing list
<a moz-do-not-send="true" href="mailto:sr-dev@lists.sip-router.org" target="_blank">sr-dev@lists.sip-router.org</a>
<a moz-do-not-send="true" href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev</a>
</pre>
</blockquote>
<br>
</div>
</div>
<span><font color="#888888">
<pre cols="72">--
Daniel-Constantin Mierla - <a moz-do-not-send="true" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a moz-do-not-send="true" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - <a moz-do-not-send="true" href="http://asipto.com/u/katu" target="_blank">http://asipto.com/u/katu</a>
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - <a moz-do-not-send="true" href="http://asipto.com/u/kpw" target="_blank">http://asipto.com/u/kpw</a></pre>
</font></span></div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a moz-do-not-send="true" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a moz-do-not-send="true" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - <a moz-do-not-send="true" href="http://asipto.com/u/katu" target="_blank">http://asipto.com/u/katu</a>
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - <a moz-do-not-send="true" href="http://asipto.com/u/kpw" target="_blank">http://asipto.com/u/kpw</a></pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla - <a class="moz-txt-link-freetext" href="http://www.asipto.com">http://www.asipto.com</a>
<a class="moz-txt-link-freetext" href="http://twitter.com/#!/miconda">http://twitter.com/#!/miconda</a> - <a class="moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Training, Berlin, Nov 5-8, 2012 - <a class="moz-txt-link-freetext" href="http://asipto.com/u/kat">http://asipto.com/u/kat</a>
Kamailio Advanced Training, Miami, USA, Nov 12-14, 2012 - <a class="moz-txt-link-freetext" href="http://asipto.com/u/katu">http://asipto.com/u/katu</a></pre>
</body>
</html>