<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
Hello,<br>
<br>
looks like related to the callbacks for dialog module. Are you
loading other modules that require dialog module?<br>
<br>
Checking the time staps from the acc and the crash log, the BYE for
the dialog was before the crash but the To-tag is not printed from
dlg_hash.c, although it is in the acc for INVITE and BYE. Do you
have parallel forking in front of this SIP server? I mean, is there
another proxy that can do parallel forking then send two or more
branches to this instance?<br>
<br>
I will dig in more to see what went wrong there.<br>
<br>
Thanks,<br>
Daniel<br>
<br>
On 3/2/11 4:34 PM, Anton Roman wrote:
<blockquote
cite="mid:AANLkTimwFgc3=m3jSv-osk51zhdw=AeX85sgHxWmBhgo@mail.gmail.com"
type="cite">Hi all,<br>
<br>
we are running Kamailio 3.1.2 in a production environment, using
the dialog module, and it crashed two hours ago. <br
style="font-family: courier new,monospace;">
<br>
<br>
Here you have the logs we got (addtional log fragments with the
acc records involved in this call are appended at the end of the
mail):<br>
<br>
<span style="font-family: courier new,monospace;">Mar 2 14:43:05
kamailio2 /usr/local/sbin/kamailio[28927]: CRITICAL: dialog
[dlg_hash.c:599]: bogus ref -1 with cnt 1 for dlg 0x7f23f472db30
[2490:1070436595] with clid
'e0a20cb844d211e0acd8001422093865@<CLIENT IP>' and tags
'1577886432-3759264324-335599788-1698171170' ''</span><br
style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Mar 2 14:43:05
kamailio2 /usr/local/sbin/kamailio[28927]: : <core>
[mem/q_malloc.c:446]: BUG: qm_free: freeing already freed
pointer, first free: dialog: dlg_cb.c:
destroy_dlg_callbacks_list(80) - aborting</span><br
style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Mar 2 14:43:05
kamailio2 /usr/local/sbin/kamailio[28896]: ALERT: <core>
[main.c:741]: child process 28927 exited by a signal 6</span><br
style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Mar 2 14:43:05
kamailio2 /usr/local/sbin/kamailio[28896]: ALERT: <core>
[main.c:744]: core was not generated</span><br
style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Mar 2 14:43:05
kamailio2 /usr/local/sbin/kamailio[28896]: INFO: <core>
[main.c:756]: INFO: terminating due to SIGCHLD</span><br
style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Mar 2 14:43:05
kamailio2 /usr/local/sbin/kamailio[28948]: INFO: <core>
[main.c:807]: INFO: signal 15 received</span><br
style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Mar 2 14:43:05
kamailio2 /usr/local/sbin/kamailio[28942]: INFO: <core>
[main.c:807]: INFO: signal 15 received</span><br
style="font-family: courier new,monospace;">
<br>
We get the kamailio code from git last week: <br>
<br>
<span style="font-family: courier new,monospace;">sercmd> <a
moz-do-not-send="true" href="http://core.info/"
target="_blank">core.info</a></span><br style="font-family:
courier new,monospace;">
<span style="font-family: courier new,monospace;">{</span><br
style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> version:
kamailio 3.1.2</span><br style="font-family: courier
new,monospace;">
<span style="font-family: courier new,monospace;"> id: 4ace86 </span><br
style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> compiler:
gcc 4.3.2</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> compiled:
09:12:36 Feb 23 2011</span><br style="font-family: courier
new,monospace;">
<span style="font-family: courier new,monospace;"> flags:
STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS,
USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM,
SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER,
USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES</span><br
style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">}</span><br>
<br>
The problem looks like this other one already fixed: <a
moz-do-not-send="true"
href="http://lists.sip-router.org/pipermail/sr-users/2009-November/027351.html">http://lists.sip-router.org/pipermail/sr-users/2009-November/027351.html</a><br>
<br>
We set the Kamailio to debug level in case it happens again.<br>
<br>
On the other side, I need to know why the core is not been
generated. I have already checked the points mentioned in <a
moz-do-not-send="true"
href="http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles">
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles</a><br>
<br>
<span style="font-family: courier new,monospace;"><span
style="font-family: arial,helvetica,sans-serif;">1.</span> d</span><span
style="font-family: courier new,monospace;">is</span><span
style="font-family: courier new,monospace;">able_</span><span
style="font-family: courier new,monospace;">core</span><span
style="font-family: courier new,monospace;">_dump</span> is not
set in the config file.<br>
<br>
2. From /etc/default/kamailio:<br>
...<br>
<span style="font-family: courier new,monospace;">DUMP_CORE=yes</span><br>
...<br>
<br>
2. From /etc/init.d/kamailio: <br>
...<br>
<span style="font-family: courier new,monospace;">if test
"$DUMP_CORE" = "yes" ; then</span><br style="font-family:
courier new,monospace;">
<span style="font-family: courier new,monospace;"> # set proper
ulimit</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> ulimit -c
unlimited</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> # directory
for the core dump files</span><br style="font-family: courier
new,monospace;">
<span style="font-family: courier new,monospace;">
COREDIR=/home/corefiles</span><br style="font-family: courier
new,monospace;">
<span style="font-family: courier new,monospace;"> [ -d
$COREDIR ] || mkdir $COREDIR</span><br style="font-family:
courier new,monospace;">
<span style="font-family: courier new,monospace;"> chmod 777
$COREDIR</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> echo
"$COREDIR/core.%e.sig%s.%p" > /proc/sys/kernel/core_pattern</span><br
style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">fi</span><br>
...<br>
<br>
4. Writting permissions of $COREDIR<br>
<br>
<span style="font-family: courier new,monospace;">ls -hall /home</span><br
style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">...</span><br
style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">drwxrwxrwx 2
root root 4.0K 2010-12-21 09:15 corefiles</span><br
style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">...<br>
<br>
<span style="font-family: arial,helvetica,sans-serif;">What else
should I check?<br>
<br>
Thanks in advance,<br>
regards<br>
<br>
Antón<br>
<br>
<br>
<b>Acc records related to the dialog whose destruction causes
the problem:</b><br>
<br>
<span style="font-family: courier new,monospace;">Mar 2
14:42:44 kamailio2 /usr/local/sbin/kamailio[28902]: NOTICE:
acc [acc.c:275]: ACC: transaction answered:
timestamp=1299073364;method=INVITE;from_tag=1577886432-3759264324-335599788-1698171170;to_tag=5FFAEA34-6A;call_id=e0a20cb844d211e0acd8001422093865@<client
IP>;code=200;reason=OK;src_user=<caller
number>;src_domain=</span></span></span><span
style="font-family: courier new,monospace;"><span
style="font-family: arial,helvetica,sans-serif;"><span
style="font-family: courier new,monospace;"><client
IP></span></span></span><span style="font-family: courier
new,monospace;"><span style="font-family:
arial,helvetica,sans-serif;"><span style="font-family: courier
new,monospace;">;dst_ouser=<called number>;dst_user=</span></span></span><span
style="font-family: courier new,monospace;"><called
number></span><span style="font-family: courier
new,monospace;"><span style="font-family:
arial,helvetica,sans-serif;"><span style="font-family: courier
new,monospace;">;dst_domain=10.90.1.251;src_ip=<client
IP></span><br>
<br>
...<br>
<br>
<span style="font-family: courier new,monospace;">Mar 2
14:42:44 kamailio2 /usr/local/sbin/kamailio[28920]: NOTICE:
acc [acc.c:275]: ACC: request acknowledged:
timestamp=1299073364;method=ACK;from_tag=1577886432-3759264324-335599788-1698171170;to_tag=5FFAEA34-6A;call_id=e0a20cb844d211e0acd8001422093865@</span></span></span><span
style="font-family: courier new,monospace;"><span
style="font-family: arial,helvetica,sans-serif;"><span
style="font-family: courier new,monospace;"><client
IP></span></span></span><span style="font-family: courier
new,monospace;"><span style="font-family:
arial,helvetica,sans-serif;"><span style="font-family: courier
new,monospace;">;code=200;reason=OK;src_user=<caller
number>;src_domain=<client IP>;dst_ouser=</span></span></span><span
style="font-family: courier new,monospace;"><called
number></span><span style="font-family: courier
new,monospace;"><span style="font-family:
arial,helvetica,sans-serif;"><span style="font-family: courier
new,monospace;">;dst_user=<called
number>;dst_domain=10.90.1.251;src_ip=<client IP></span><br
style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">...</span><br
style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Mar 2
14:43:00 kamailio2 /usr/local/sbin/kamailio[28903]: ERROR:
<script>: ACK WITHOUT MATCHING TRANSACTION in
e0a20cb844d211e0acd8001422093865@<client IP> call...
ignore and discard.</span><br style="font-family: courier
new,monospace;">
<br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">... </span><br>
<br>
</span></span><span style="font-family: courier new,monospace;">Mar
2 14:43:00 kamailio2 /usr/local/sbin/kamailio[28904]: NOTICE:
acc [acc.c:275]: ACC: transaction answered:
timestamp=1299073380;method=BYE;from_tag=1577886432-3759264324-335599788-1698171170;to_tag=5FFAEA34-6A;call_id=e0a20cb844d211e0acd8001422093865@</span><span
style="font-family: courier new,monospace;"><span
style="font-family: arial,helvetica,sans-serif;"><span
style="font-family: courier new,monospace;"><client
IP></span></span></span><span style="font-family: courier
new,monospace;">;code=200;reason=OK;src_user=<caller
number>;src_domain=<client IP>;dst_ouser=</span><span
style="font-family: courier new,monospace;"><called
number></span><span style="font-family: courier
new,monospace;">;dst_user=<called
number>;dst_domain=10.90.1.251;src_ip=<client IP></span><br>
<span style="font-family: courier new,monospace;"><span
style="font-family: arial,helvetica,sans-serif;"><br>
</span></span>
<pre wrap="">
<fieldset class="mimeAttachmentHeader"></fieldset>
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:sr-users@lists.sip-router.org">sr-users@lists.sip-router.org</a>
<a class="moz-txt-link-freetext" href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla
<a class="moz-txt-link-freetext" href="http://www.asipto.com">http://www.asipto.com</a></pre>
</body>
</html>