Hi Daniel,<div><br></div><div>This patch needs to be applied to avoid core.</div><div><br></div><div>Thanks</div><div>Jijo<br><br><div class="gmail_quote">On Wed, Sep 19, 2012 at 10:54 AM, Jijo <span dir="ltr"><<a href="mailto:realjijo@gmail.com" target="_blank">realjijo@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><br></div>Hi All,<div><br></div><div>Finally i found the issue, </div><div><br></div><div><p class="MsoNormal"><span style="color:#1f497d">Here is one of the bad trace for
SUBSCRIBE(722bytes) and NOTIFY(1282bytes) which corrupted the memory. The
messages came in the order NOTIFY and SUBSCRIBE. The core is generated in a
different place but I believe this could be the reason for memory corruption. </span></p>
<p class="MsoNormal"><span style="color:#1f497d">Here is the trace UDP Process </span><span style="color:#c00000">27294</span><span style="color:#1f497d"> processing
NOTIFY and Process </span><span style="color:#00b050">27303</span><span style="font-size:8.0pt;color:#00b050"> </span><span style="color:#1f497d">processing
SUBSCRIBE .</span></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="color:#1f497d">The explanation and
implementation is below</span></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:17+01:00
[info] sipserver: [27303] INFO: <script>: <a href="mailto:CI=1-3292@10.233.20.152" target="_blank">CI=1-3292@10.233.20.152</a> -R39 - Entry
R-URI=1234 FD=10.233.20.152 SI=10.233.20.152</span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:17+01:00
[info] sipserver: [</span><span style="font-size:8.0pt;color:#c00000">27294</span><span style="font-size:8.0pt;color:#1f497d">] INFO: <script>: <a href="mailto:CI=1-3292@10.233.20.152" target="_blank">CI=1-3292@10.233.20.152</a> -R1 - Force
LAN socket: tcp:<a href="http://10.233.20.151:5060" target="_blank">10.233.20.151:5060</a> <null></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:17+01:00
[info] sipserver: [</span><span style="font-size:8.0pt;color:#00b050">27303</span><span style="font-size:8.0pt;color:#1f497d">] INFO: <script>: <a href="mailto:CI=1-3292@10.233.20.152" target="_blank">CI=1-3292@10.233.20.152</a> -R1 - Force
LAN socket: tcp:<a href="http://10.233.20.151:5060" target="_blank">10.233.20.151:5060</a> <null></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:17+01:00
[err] sipserver: [27303] ERROR: <core> [tcp_main.c:2357]:
tcp_conn_send_put : calling wbufq_add</span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:17+01:00
[err] sipserver: [27303] ERROR: <core> [tcp_main.c:730]: ERROR:
wbufq_add(722 bytes): buf:</span><span style="font-size:8.0pt;color:#00b050">SUBSCRIBE
</span><span style="font-size:8.0pt;color:#1f497d"><a>sip:1234@10.233.20.141:5063;transport=tcp</a>
SIP/2.0</span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d">Record-Route:
<sip:10.233.20.151;tran</span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:17+01:00
[err] sipserver: [27303] ERROR: <core> [tcp_main.c:747]: ERROR:
wbufq_add(722 bytes): first:b00519f4 last:b00519f4</span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:17+01:00
[err] sipserver: [27303] ERROR: <core> [tcp_main.c:774]: ERROR:
wbufq_add(2 last free crt_size:722): first:b00519f4 last:b00519f4</span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:17+01:00
[err] sipserver: [27294] ERROR: <core> [tcp_main.c:796]: ERROR:
wbufq_insert(1282 bytes): buf:</span><span style="font-size:8.0pt;color:#c00000">NOTIFY
</span><span style="font-size:8.0pt;color:#1f497d"><a>sip:1234@10.233.20.141:5063;transport=tcp</a>
SIP/2.0</span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d">Record-Route:
<sip:10.233.20.151;transpo</span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:17+01:00
[err] sipserver: [27294] ERROR: <core> [tcp_main.c:801]: ERROR: wbufq_insert(2
last free ): first:b00519f4 last:b00519f4</span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:17+01:00
[err] sipserver: [27294] ERROR: <core> [tcp_main.c:820]: ERROR:
wbufq_insert(22 last free ): first:b00519f4 last:b00519f4</span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:17+01:00
[err] sipserver: [27359] ERROR: <core> [tcp_main.c:887]: ERROR:
wbufq_run(3 last free ): first:b00519f4 last:b00519f4</span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:17+01:00
[crit] sipserver: [27359] : <core> [mem/q_malloc.c:157]: BUG: qm_*:
fragm. 0xb00519dc (address 0xb00519f4) end overwritten(0, 0)!</span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:18+01:00
[alert] sipserver: [27265] ALERT: <core> [main.c:755]: child process
27359 exited by a signal 11</span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:18+01:00
[alert] sipserver: [27265] ALERT: <core> [main.c:758]: core was generated</span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d">2012-09-19T02:06:18+01:00
[info] sipserver: [27265] INFO: <core> [main.c:770]: INFO: terminating
due to SIGCHLD</span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="color:#1f497d">Process 27294(NOTIFY) created
the TCP connection structure for destination IP and just before calling
wbufq_insert(), context switch happened and process 27303(SUBSCRIBE) got the
cpu. Since the connection structure is already available process 27303 add the
SUBSCRIBE message(722 bytes) to wbufq. Afterwards process 27294 got the CPU and
invoked wbufq_insert() which basically corrupted the memory due to an overflow
with the existing implementation.</span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1f497d">inline static
int _wbufq_insert(struct tcp_connection* c, const char* data, </span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1f497d">
unsigned int size)</span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1f497d">{</span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1f497d">
struct tcp_wbuffer_queue* q;</span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1f497d">
struct tcp_wbuffer* wb;</span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1f497d">
</span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1f497d">
q=&c->wbuf_q;</span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1f497d">
if (likely(q->first==0)) /* if empty, use wbufq_add */</span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1f497d">
return _wbufq_add(c, data, size);</span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1f497d">:</span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1f497d">:</span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1f497d">:</span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1f497d">:</span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#00b050">
if ((q->first==q->last) &&
((q->last->b_size-q->last_used)>=size)){</span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#00b050">
/* one block with enough space in it for size bytes */</span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#00b050"> </span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1f497d">
</span><span style="font-size:9.0pt;color:#c00000">memmove(q->first->buf+size,
q->first->buf, size);</span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1f497d">
memcpy(q->first->buf, data, size);</span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1f497d">
q->last_used+=size;</span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#00b050">
}</span></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="color:#1f497d">The above condition shall be
true in this case and memmove was moving the pointer which was causing the
overflow.</span></p>
<p class="MsoNormal"><span style="color:#1f497d">
memmove(void
*dest, const void *src, size_t n); </span></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="color:#1f497d">As per the memmove man page, the
src shall be copied with size ‘n’ to a temporary buffer and then temporary
buffer to dest.</span></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d">dest is
q->first->buf+size: which is basically (q->first->buf +
NOTIFY MSG SIZE). so the dst will move my 1282 bytes, so we have remaining
space of only 818 bytes.</span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d">src is
q->first->buf: which is basically copied to temp buffer with NOTIFY
SIZE(1282bytes).</span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d">Finally we are
moving the buffer from temporary buffer of size 1282 bytes to buffer which we
left with 818 bytes, This basically corrupt the memory and on wbufq_run we see
the memory corruption</span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#c00000">2012-09-19T02:06:17+01:00
[crit] sipserver: [27359] : <core> [mem/q_malloc.c:157]: BUG: qm_*:
fragm. 0xb00519dc (address 0xb00519f4) end overwritten(0, 0)!</span></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="color:#1f497d">I think we don’t need memove, so
we can change the code as below</span></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="color:#1f497d"> if
((q->first==q->last) &&
((q->last->b_size-q->last_used)>=size)){</span></p>
<p class="MsoNormal"><span style="color:#1f497d">
/* one block with enough space in it for size bytes */</span></p>
<p class="MsoNormal"><s><span style="color:#c00000">
//memmove(q->first->buf+size, q->first->buf, size);</span></s></p>
<p class="MsoNormal"><span style="color:#1f497d">
memcpy(q->first->buf+q->last_used, data, size);</span></p>
<p class="MsoNormal"><span style="color:#1f497d">
q->last_used+=size;</span></p>
<p class="MsoNormal"><span style="color:#1f497d">
}</span></p><div><br></div><font color="#1f497d">OR</font></div><div> we need only the else part as it always add the block to the first.</div><div><br></div><div>Thanks</div><span class="HOEnZb"><font color="#888888"><div>
Jijo</div></font></span><div class="HOEnZb"><div class="h5"><div><br><div class="gmail_quote">
On Wed, Jul 18, 2012 at 12:42 PM, Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Hello,<br>
<br>
sorry, I just read the last messages in the thread and I didn't
noticed it is about a crash, but thought it is about an annoying log
message.<br>
<br>
The backtrace is no longer matching the latest sources of the 3.1
branch, but I expect it is due to a double free issue, so you have
to update to latest 3.1.6, as suggested by other users here. There
were many fixes from 3.1.0 to 3.1.6.<br>
<br>
Cheers,<br>
Daniel<div><div><br>
<br>
<div>On 7/17/12 6:03 PM, Jijo wrote:<br>
</div>
<blockquote type="cite">Hi,
<div><br>
</div>
<div>This is not happening at shutdown or status check. Its
aborting when the system is active.</div>
<div><br>
Thanks</div>
<div>Jijo<br>
<div class="gmail_quote">On Tue, Jul 17, 2012 at 3:06 AM,
Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Hello,<br>
<br>
does it keep being or it is one time and that's it?<br>
<br>
That is printed at shut down or if pkg_status() or
shm_status() is executed from some part of code or in
config via cfgutils module functions.<br>
<br>
You can get rid of them by setting memdbg and memlog to a
value higher than debug global parameter.<br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
<div>On 7/16/12 5:28 PM, Jijo wrote:<br>
</div>
<blockquote type="cite">Thanks.. It is not easy to
upgrade as it is happening at customer system.
<div>Is there any change occurred for this issue.I
looked at it, but didn't see anything in
q_malloc.c/qm_status()</div>
<div><br>
</div>
<div>
<div> <br>
<div class="gmail_quote">On Mon, Jul 16, 2012 at
11:12 AM, Jon Bonilla <span dir="ltr"><<a href="mailto:manwe@aholab.ehu.es" target="_blank">manwe@aholab.ehu.es</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> El Mon, 16 Jul
2012 10:27:42 -0400<br>
Jijo <<a href="mailto:realjijo@gmail.com" target="_blank">realjijo@gmail.com</a>>
escribió:<br>
<div><br>
> Hi All,<br>
><br>
> I'm observing a core intermittently
at "qm_status (qm=0x786cd000) at<br>
> mem/q_malloc.c:763" for kamailio
version 3.1.0<br>
><br>
<br>
</div>
I'd say that you're using a very old
version. You should update your branch to<br>
3.1.6 or upgrade to a newer branch.<br>
<br>
<br>
_______________________________________________<br>
sr-dev mailing list<br>
<a href="mailto:sr-dev@lists.sip-router.org" target="_blank">sr-dev@lists.sip-router.org</a><br>
<a href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev</a><br>
</blockquote>
</div>
<br>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
sr-dev mailing list
<a href="mailto:sr-dev@lists.sip-router.org" target="_blank">sr-dev@lists.sip-router.org</a>
<a href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev</a>
</pre>
</blockquote>
<br>
</div>
</div>
<span><font color="#888888">
<pre cols="72">--
Daniel-Constantin Mierla - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - <a href="http://asipto.com/u/katu" target="_blank">http://asipto.com/u/katu</a>
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - <a href="http://asipto.com/u/kpw" target="_blank">http://asipto.com/u/kpw</a></pre>
</font></span></div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a href="http://twitter.com/#!/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - <a href="http://asipto.com/u/katu" target="_blank">http://asipto.com/u/katu</a>
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - <a href="http://asipto.com/u/kpw" target="_blank">http://asipto.com/u/kpw</a></pre>
</div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>