Hello,<div><br></div><div>thanks for your quick reply, my answer is inline.<br><br><div class="gmail_quote">2011/3/2 Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com">miconda@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div bgcolor="#ffffff" text="#000000">
Hello,<br>
<br>
looks like related to the callbacks for dialog module. Are you
loading other modules that require dialog module?<br></div></blockquote><div> </div><div>we are using some features of dialog module such as ending dialogs after a timeout period, and we are using engage_mediaproxy() function, as well. It's an old configuration we had to put in production with no time enough to test. Do you recommend not to use dialog module if not strictly required?</div>
<meta http-equiv="content-type" content="text/html; charset=utf-8"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div bgcolor="#ffffff" text="#000000">
<br>
Checking the time staps from the acc and the crash log, the BYE for
the dialog was before the crash but the To-tag is not printed from
dlg_hash.c, although it is in the acc for INVITE and BYE. Do you
have parallel forking in front of this SIP server? I mean, is there
another proxy that can do parallel forking then send two or more
branches to this instance?<br>
<br></div></blockquote><div>AFAIK the the client who is sending that calls is not doing parallel forking, they are sending calls over a SIP trunk to our Kamailio. They are calling to PSTN numbers and we are sending that calls to a gateway, so they shouldn't do parallel forking, I'll get some traces to check it. </div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div bgcolor="#ffffff" text="#000000">
I will dig in more to see what went wrong there.<br>
<br></div></blockquote><div>I got the attached debug level logs for the reported problem, hope it helps. I'm still trying to find out why the core is not being generated. </div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div bgcolor="#ffffff" text="#000000">
Thanks,<br>
Daniel</div></blockquote><div>Thanks a lot,</div><div>Anton</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div bgcolor="#ffffff" text="#000000">
<div><div></div><div class="h5"><br>
<br>
On 3/2/11 4:34 PM, Anton Roman wrote:
</div></div><blockquote type="cite"><div><div></div><div class="h5">Hi all,<br>
<br>
we are running Kamailio 3.1.2 in a production environment, using
the dialog module, and it crashed two hours ago. <br style="font-family:courier new,monospace">
<br>
<br>
Here you have the logs we got (addtional log fragments with the
acc records involved in this call are appended at the end of the
mail):<br>
<br>
<span style="font-family:courier new,monospace">Mar 2 14:43:05
kamailio2 /usr/local/sbin/kamailio[28927]: CRITICAL: dialog
[dlg_hash.c:599]: bogus ref -1 with cnt 1 for dlg 0x7f23f472db30
[2490:1070436595] with clid
'e0a20cb844d211e0acd8001422093865@<CLIENT IP>' and tags
'1577886432-3759264324-335599788-1698171170' ''</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">Mar 2 14:43:05
kamailio2 /usr/local/sbin/kamailio[28927]: : <core>
[mem/q_malloc.c:446]: BUG: qm_free: freeing already freed
pointer, first free: dialog: dlg_cb.c:
destroy_dlg_callbacks_list(80) - aborting</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">Mar 2 14:43:05
kamailio2 /usr/local/sbin/kamailio[28896]: ALERT: <core>
[main.c:741]: child process 28927 exited by a signal 6</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">Mar 2 14:43:05
kamailio2 /usr/local/sbin/kamailio[28896]: ALERT: <core>
[main.c:744]: core was not generated</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">Mar 2 14:43:05
kamailio2 /usr/local/sbin/kamailio[28896]: INFO: <core>
[main.c:756]: INFO: terminating due to SIGCHLD</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">Mar 2 14:43:05
kamailio2 /usr/local/sbin/kamailio[28948]: INFO: <core>
[main.c:807]: INFO: signal 15 received</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">Mar 2 14:43:05
kamailio2 /usr/local/sbin/kamailio[28942]: INFO: <core>
[main.c:807]: INFO: signal 15 received</span><br style="font-family:courier new,monospace">
<br>
We get the kamailio code from git last week: <br>
<br>
<span style="font-family:courier new,monospace">sercmd> <a href="http://core.info/" target="_blank">core.info</a></span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">{</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace"> version:
kamailio 3.1.2</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace"> id: 4ace86 </span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace"> compiler:
gcc 4.3.2</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace"> compiled:
09:12:36 Feb 23 2011</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace"> flags:
STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS,
USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM,
SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER,
USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">}</span><br>
<br>
The problem looks like this other one already fixed: <a href="http://lists.sip-router.org/pipermail/sr-users/2009-November/027351.html" target="_blank">http://lists.sip-router.org/pipermail/sr-users/2009-November/027351.html</a><br>
<br>
We set the Kamailio to debug level in case it happens again.<br>
<br>
On the other side, I need to know why the core is not been
generated. I have already checked the points mentioned in <a href="http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles" target="_blank">
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles</a><br>
<br>
<span style="font-family:courier new,monospace"><span style="font-family:arial,helvetica,sans-serif">1.</span> d</span><span style="font-family:courier new,monospace">is</span><span style="font-family:courier new,monospace">able_</span><span style="font-family:courier new,monospace">core</span><span style="font-family:courier new,monospace">_dump</span> is not
set in the config file.<br>
<br>
2. From /etc/default/kamailio:<br>
...<br>
<span style="font-family:courier new,monospace">DUMP_CORE=yes</span><br>
...<br>
<br>
2. From /etc/init.d/kamailio: <br>
...<br>
<span style="font-family:courier new,monospace">if test
"$DUMP_CORE" = "yes" ; then</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace"> # set proper
ulimit</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace"> ulimit -c
unlimited</span><br style="font-family:courier new,monospace">
<br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace"> # directory
for the core dump files</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">
COREDIR=/home/corefiles</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace"> [ -d
$COREDIR ] || mkdir $COREDIR</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace"> chmod 777
$COREDIR</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace"> echo
"$COREDIR/core.%e.sig%s.%p" > /proc/sys/kernel/core_pattern</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">fi</span><br>
...<br>
<br>
4. Writting permissions of $COREDIR<br>
<br>
<span style="font-family:courier new,monospace">ls -hall /home</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">...</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">drwxrwxrwx 2
root root 4.0K 2010-12-21 09:15 corefiles</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">...<br>
<br>
<span style="font-family:arial,helvetica,sans-serif">What else
should I check?<br>
<br>
Thanks in advance,<br>
regards<br>
<br>
Antón<br>
<br>
<br>
<b>Acc records related to the dialog whose destruction causes
the problem:</b><br>
<br>
<span style="font-family:courier new,monospace">Mar 2
14:42:44 kamailio2 /usr/local/sbin/kamailio[28902]: NOTICE:
acc [acc.c:275]: ACC: transaction answered:
timestamp=1299073364;method=INVITE;from_tag=1577886432-3759264324-335599788-1698171170;to_tag=5FFAEA34-6A;call_id=e0a20cb844d211e0acd8001422093865@<client
IP>;code=200;reason=OK;src_user=<caller
number>;src_domain=</span></span></span><span style="font-family:courier new,monospace"><span style="font-family:arial,helvetica,sans-serif"><span style="font-family:courier new,monospace"><client
IP></span></span></span><span style="font-family:courier new,monospace"><span style="font-family:arial,helvetica,sans-serif"><span style="font-family:courier new,monospace">;dst_ouser=<called number>;dst_user=</span></span></span><span style="font-family:courier new,monospace"><called
number></span><span style="font-family:courier new,monospace"><span style="font-family:arial,helvetica,sans-serif"><span style="font-family:courier new,monospace">;dst_domain=10.90.1.251;src_ip=<client
IP></span><br>
<br>
...<br>
<br>
<span style="font-family:courier new,monospace">Mar 2
14:42:44 kamailio2 /usr/local/sbin/kamailio[28920]: NOTICE:
acc [acc.c:275]: ACC: request acknowledged:
timestamp=1299073364;method=ACK;from_tag=1577886432-3759264324-335599788-1698171170;to_tag=5FFAEA34-6A;call_id=e0a20cb844d211e0acd8001422093865@</span></span></span><span style="font-family:courier new,monospace"><span style="font-family:arial,helvetica,sans-serif"><span style="font-family:courier new,monospace"><client
IP></span></span></span><span style="font-family:courier new,monospace"><span style="font-family:arial,helvetica,sans-serif"><span style="font-family:courier new,monospace">;code=200;reason=OK;src_user=<caller
number>;src_domain=<client IP>;dst_ouser=</span></span></span><span style="font-family:courier new,monospace"><called
number></span><span style="font-family:courier new,monospace"><span style="font-family:arial,helvetica,sans-serif"><span style="font-family:courier new,monospace">;dst_user=<called
number>;dst_domain=10.90.1.251;src_ip=<client IP></span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">...</span><br style="font-family:courier new,monospace">
<br style="font-family:courier new,monospace">
<br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">Mar 2
14:43:00 kamailio2 /usr/local/sbin/kamailio[28903]: ERROR:
<script>: ACK WITHOUT MATCHING TRANSACTION in
e0a20cb844d211e0acd8001422093865@<client IP> call...
ignore and discard.</span><br style="font-family:courier new,monospace">
<br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">... </span><br>
<br>
</span></span><span style="font-family:courier new,monospace">Mar
2 14:43:00 kamailio2 /usr/local/sbin/kamailio[28904]: NOTICE:
acc [acc.c:275]: ACC: transaction answered:
timestamp=1299073380;method=BYE;from_tag=1577886432-3759264324-335599788-1698171170;to_tag=5FFAEA34-6A;call_id=e0a20cb844d211e0acd8001422093865@</span><span style="font-family:courier new,monospace"><span style="font-family:arial,helvetica,sans-serif"><span style="font-family:courier new,monospace"><client
IP></span></span></span><span style="font-family:courier new,monospace">;code=200;reason=OK;src_user=<caller
number>;src_domain=<client IP>;dst_ouser=</span><span style="font-family:courier new,monospace"><called
number></span><span style="font-family:courier new,monospace">;dst_user=<called
number>;dst_domain=10.90.1.251;src_ip=<client IP></span><br>
<span style="font-family:courier new,monospace"><span style="font-family:arial,helvetica,sans-serif"><br>
</span></span>
</div></div><pre><fieldset></fieldset>
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
<a href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a>
<a href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla
<a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a></pre>
</div>
</blockquote></div><br></div>