<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
Hello,<br>
<br>
On 3/26/10 4:13 PM, Zappasodi Daniele wrote:
<blockquote
cite="mid:14A9059A37EF9A45BC2520DE64C30F60075256@slttex002.seltatel.local"
type="cite">
<meta http-equiv="Content-Type"
content="text/html; charset=ISO-8859-1">
<title>Fast lock loop with arm</title>
<meta content="MSHTML 6.00.6000.16981" name="GENERATOR">
<div>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span style=""
lang="EN-GB">Hello,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span style=""
lang="EN-GB">this is what I get with gdb:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span style=""
lang="EN-GB"> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span style=""
lang="EN-GB">(gdb) bt full<o:p></o:p></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span style=""
lang="EN-GB">#0<span style=""> </span>0x40137e54 in sched_yield ()
from /usr/local/lib2/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span style=""
lang="EN-GB">No symbol table info available.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span style=""
lang="EN-GB"> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span style=""
lang="EN-GB">I don’t think that it can help, but I am not able to load
the symbol table for openser<span class="292020015-26032010"> on the
server</span>.</span></p>
</div>
</blockquote>
hmm, strange. Did you try with many processes? Sam result in the
backtrace?<br>
<br>
Cheers,<br>
Daniel<br>
<br>
<blockquote
cite="mid:14A9059A37EF9A45BC2520DE64C30F60075256@slttex002.seltatel.local"
type="cite">
<div>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span style=""
lang="EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span style=""
lang="EN-GB"><span class="292020015-26032010"></span></span><span
style="" lang="EN-GB"></span> </p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span style=""
lang="EN-GB"><span class="292020015-26032010"><font color="#0000ff"
face="Arial" size="2">thanks,</font></span></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span style=""
lang="EN-GB"><o:p><span class="292020015-26032010"><font
color="#0000ff" face="Arial" size="2">Daniele</font></span></o:p></span></p>
</div>
<blockquote dir="ltr" style="margin-right: 0px;">
<div class="OutlookMessageHeader" dir="ltr" align="left"><font
face="Tahoma" size="2">-----Messaggio originale-----<br>
<b>Da:</b> Daniel-Constantin Mierla [<a class="moz-txt-link-freetext" href="mailto:miconda@gmail.com">mailto:miconda@gmail.com</a>]<br>
<b>Inviato:</b> mercoledì 24 marzo 2010 16.47<br>
<b>A:</b> Zappasodi Daniele<br>
<b>Cc:</b> <a class="moz-txt-link-abbreviated" href="mailto:users@lists.kamailio.org">users@lists.kamailio.org</a>; sr-dev<br>
<b>Oggetto:</b> Re: [Kamailio-Users] Fast lock loop with arm<br>
<br>
</font></div>
Hello,<br>
<br>
what version of kamailio do you use?<br>
<br>
Can you grab a backtrace with gdb once you get this high cpu usage?
Spot one of the processes, take the pid and do:<br>
<br>
gdb /path/to/kamailio _pid_<br>
<br>
Then:<br>
<br>
bt<br>
<br>
You should see the bactrace of executed function getting to deadloc.
Make sure you get at least one SIP worker process backtrace (do kamctl
ps to see the type of kamailio process).<br>
<br>
Cheers,<br>
Daniel<br>
<br>
On 3/24/10 4:29 PM, Zappasodi Daniele wrote:
<blockquote
cite="mid:14A9059A37EF9A45BC2520DE64C30F60075252@slttex002.seltatel.local"
type="cite">
<meta content="MS Exchange Server version 6.5.7654.12"
name="Generator">
<!-- Converted from text/rtf format -->
<p><span lang="en-gb"><font face="Times New Roman">Hi,</font></span>
<br>
<span lang="en-gb"><font face="Times New Roman">I have a big
problem with fast lock mutex on arm CPU: </font></span><br>
<span lang="en-gb"><font face="Times New Roman">sometimes I find
one or more children that go in an infinite loop, in the while loop of
get_lock function. </font></span><br>
<span lang="en-gb"><font face="Times New Roman">They remain in
Run for long time (sometimes hours) and openser keeps 100% CPU.</font></span>
</p>
<p><span lang="en-gb"><font face="Times New Roman">Now I have
changed the functions get_lock and tsl in order to obtain more info and
</font></span><br>
<span lang="en-gb"><font face="Times New Roman">I observe that
(if and) when the process finish the cycle, it have done a big amount
of cycles. </font></span></p>
<p><span lang="en-gb"><font face="Times New Roman">This the log
with my added info:</font></span> <br>
<span lang="en-gb"><font face="Times New Roman">Mar 26 20:29:08
SAM-IP openser[1645]: NOTICE:tm:_lock: ret 1 (cycle)</font></span> <br>
<span lang="en-gb"><font face="Times New Roman">Mar 26 20:29:51
SAM-IP openser[1645]: NOTICE:tm:_lock: ret 1 (cycle)</font></span> <br>
<span lang="en-gb"><font face="Times New Roman">Mar 26 20:29:55
SAM-IP openser[1645]: NOTICE:tm:_lock: ret 1 (cycle)</font></span> </p>
<p><span lang="en-gb"><font face="Times New Roman">Mar 26
20:29:55 SAM-IP openser[1645]: NOTICE:tm:t_retransmit_reply: MYTM lock </font></span><br>
<span lang="en-gb"><font face="Times New Roman">[process in loop]</font></span>
<br>
<span lang="en-gb"><font face="Times New Roman">Mar 26 20:33:46
SAM-IP openser[1645]: NOTICE:tm:_lock: ret 59374917 (cycle)</font></span>
<br>
<span lang="en-gb"><font face="Times New Roman">[after 4 minutes
and 59374917 cycles, this is an example with a "short" loop]</font></span>
<br>
<span lang="en-gb"><font face="Times New Roman">Mar 26 20:33:46
SAM-IP openser[1645]: NOTICE:tm:t_retransmit_reply: MYTM lock done </font></span></p>
<p><span lang="en-gb"><font face="Times New Roman">I’m not able
to recognize the call flow that generates the loop (I know only that it
always happens when t_retransmit_reply calls LOCK_REPLIES), </font></span></p>
<p><span lang="en-gb"><font face="Times New Roman">but I urgently
need a work around to escape from the loop.</font></span> </p>
<p><span lang="en-gb"><font face="Times New Roman">This is the
get_lock function with my counter j:</font></span> </p>
<p><span lang="en-gb"><font face="Times New Roman">inline static
int get_lock(fl_lock_t* lock)</font></span> <br>
<span lang="en-gb"><font face="Times New Roman">{</font></span> <br>
<span lang="en-gb"><font face="Times New Roman">#ifdef
ADAPTIVE_WAIT</font></span> <br>
<span lang="en-gb"> <font face="Times New Roman">int
i=ADAPTIVE_WAIT_LOOPS;</font></span> <br>
<span lang="en-gb"> <font face="Times New Roman">int
j=1; /***** my change ****/</font></span>
<br>
<span lang="en-gb"><font face="Times New Roman">#endif</font></span>
<br>
<span lang="en-gb"> </span><br>
<span lang="en-gb"> <font face="Times New Roman">while(tsl(lock)){</font></span>
<br>
<span lang="en-gb"><font face="Times New Roman">#ifdef BUSY_WAIT</font></span>
<br>
<span lang="en-gb"><font face="Times New Roman">#elif defined
ADAPTIVE_WAIT</font></span> <br>
<span lang="en-gb"> <font face="Times New Roman">j++;
/***** my change ****/</font></span> <br>
<span lang="en-gb"> <font face="Times New Roman">if
(i>0) i--;</font></span> <br>
<span lang="en-gb"> <font face="Times New Roman">else
sched_yield();</font></span> <br>
<span lang="en-gb"><font face="Times New Roman">#else</font></span>
<br>
<span lang="en-gb"> <font face="Times New Roman">sched_yield();</font></span>
<br>
<span lang="en-gb"><font face="Times New Roman">#endif</font></span>
<br>
<span lang="en-gb"> <font face="Times New Roman">}</font></span>
<br>
<span lang="en-gb"> <font face="Times New Roman">return(j);</font></span>
<br>
<span lang="en-gb"><font face="Times New Roman">}</font></span> </p>
<p><span lang="en-gb"><font face="Times New Roman">Can I break
the lock when my counter reaches a big value? </font></span><br>
<span lang="en-gb"><font face="Times New Roman">Should I call the
unlock function after the break?</font></span> <br>
<span lang="en-gb"><font face="Times New Roman">which value can
be considered too big?</font></span> </p>
<p><span lang="en-gb"><font face="Times New Roman">Thanks,</font></span>
<br>
<span lang="en-gb"><font face="Times New Roman">Daniele </font></span></p>
<br>
<br>
<p>**********************************************************************
The information in this message is confidential and may be legally
privileged. It is intended solely for the addressee. Access to this
message by anyone else is unauthorized. If you are not the intended
recipient, any disclosure, copying, or distribution of the message, or
any action or omission taken by you in reliance on it, is prohibited
and may be unlawful. Please immediately contact the sender if you have
received this message in error.
**********************************************************************</p>
<pre wrap=""><fieldset class="mimeAttachmentHeader"></fieldset>
_______________________________________________
Kamailio (OpenSER) - Users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:Users@lists.kamailio.org">Users@lists.kamailio.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://lists.kamailio.org/cgi-bin/mailman/listinfo/users">http://lists.kamailio.org/cgi-bin/mailman/listinfo/users</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://lists.openser-project.org/cgi-bin/mailman/listinfo/users">http://lists.openser-project.org/cgi-bin/mailman/listinfo/users</a></pre>
</blockquote>
<br>
<div class="moz-signature">-- <br>
Daniel-Constantin Mierla * <a moz-do-not-send="true"
class="moz-txt-link-freetext" href="http://www.asipto.com/">http://www.asipto.com/</a>
</div>
</blockquote>
<p>**********************************************************************
The
information in this message is confidential and may be legally
privileged. It is intended solely for the addressee. Access to this
message
by anyone else is unauthorized. If you are not the intended recipient,
any
disclosure, copying, or distribution of the message, or any action or
omission taken by you in reliance on it, is prohibited and may be
unlawful.
Please immediately contact the sender if you have received this message
in error.
**********************************************************************</p>
</blockquote>
<br>
<div class="moz-signature">-- <br>
Daniel-Constantin Mierla
* <a class="moz-txt-link-freetext" href="http://www.asipto.com/">http://www.asipto.com/</a> </div>
</body>
</html>