Hi all,<br><br>we are running Kamailio 3.1.2 in a production environment, using the dialog module, and it crashed two hours ago. <br style="font-family: courier new,monospace;"><br><br>Here you have the logs we got (addtional log fragments with the acc records involved in this call are appended at the end of the mail):<br>
<br><span style="font-family: courier new,monospace;">Mar 2 14:43:05 kamailio2 /usr/local/sbin/kamailio[28927]: CRITICAL: dialog [dlg_hash.c:599]: bogus ref -1 with cnt 1 for dlg 0x7f23f472db30 [2490:1070436595] with clid 'e0a20cb844d211e0acd8001422093865@<CLIENT IP>' and tags '1577886432-3759264324-335599788-1698171170' ''</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Mar 2 14:43:05 kamailio2 /usr/local/sbin/kamailio[28927]: : <core> [mem/q_malloc.c:446]: BUG: qm_free: freeing already freed pointer, first free: dialog: dlg_cb.c: destroy_dlg_callbacks_list(80) - aborting</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Mar 2 14:43:05 kamailio2 /usr/local/sbin/kamailio[28896]: ALERT: <core> [main.c:741]: child process 28927 exited by a signal 6</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Mar 2 14:43:05 kamailio2 /usr/local/sbin/kamailio[28896]: ALERT: <core> [main.c:744]: core was not generated</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Mar 2 14:43:05 kamailio2 /usr/local/sbin/kamailio[28896]: INFO: <core> [main.c:756]: INFO: terminating due to SIGCHLD</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Mar 2 14:43:05 kamailio2 /usr/local/sbin/kamailio[28948]: INFO: <core> [main.c:807]: INFO: signal 15 received</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Mar 2 14:43:05 kamailio2 /usr/local/sbin/kamailio[28942]: INFO: <core> [main.c:807]: INFO: signal 15 received</span><br style="font-family: courier new,monospace;">
<br>We get the kamailio code from git last week: <br><br><span style="font-family: courier new,monospace;">sercmd> <a href="http://core.info/" target="_blank">core.info</a></span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">{</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> version: kamailio 3.1.2</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> id: 4ace86 </span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> compiler: gcc 4.3.2</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> compiled: 09:12:36 Feb 23 2011</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">
flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS,
USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP,
PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST,
HAVE_RESOLV_RES</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">}</span><br><br>The problem looks like this other one already fixed: <a href="http://lists.sip-router.org/pipermail/sr-users/2009-November/027351.html">http://lists.sip-router.org/pipermail/sr-users/2009-November/027351.html</a><br>
<br>We set the Kamailio to debug level in case it happens again.<br><br>On the other side, I need to know why the core is not been generated. I have already checked the points mentioned in <a href="http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles"> http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles</a><br>
<br><span style="font-family: courier new,monospace;"><span style="font-family: arial,helvetica,sans-serif;">1.</span> d</span><span style="font-family: courier new,monospace;">is</span><span style="font-family: courier new,monospace;">able_</span><span style="font-family: courier new,monospace;">core</span><span style="font-family: courier new,monospace;">_dump</span> is not set in the config file.<br>
<br>2. From /etc/default/kamailio:<br>...<br><span style="font-family: courier new,monospace;">DUMP_CORE=yes</span><br>...<br><br>2. From /etc/init.d/kamailio: <br>...<br><span style="font-family: courier new,monospace;">if test "$DUMP_CORE" = "yes" ; then</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> # set proper ulimit</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> ulimit -c unlimited</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> # directory for the core dump files</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> COREDIR=/home/corefiles</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> [ -d $COREDIR ] || mkdir $COREDIR</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> chmod 777 $COREDIR</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> echo "$COREDIR/core.%e.sig%s.%p" > /proc/sys/kernel/core_pattern</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">fi</span><br>
...<br><br>4. Writting permissions of $COREDIR<br><br><span style="font-family: courier new,monospace;">ls -hall /home</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">...</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">drwxrwxrwx 2 root root 4.0K 2010-12-21 09:15 corefiles</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">...<br>
<br><span style="font-family: arial,helvetica,sans-serif;">What else should I check?<br><br>Thanks in advance,<br>regards<br><br>Antón<br><br><br><b>Acc records related to the dialog whose destruction causes the problem:</b><br>
<br><span style="font-family: courier new,monospace;">Mar 2 14:42:44 kamailio2 /usr/local/sbin/kamailio[28902]: NOTICE: acc [acc.c:275]: ACC: transaction answered: timestamp=1299073364;method=INVITE;from_tag=1577886432-3759264324-335599788-1698171170;to_tag=5FFAEA34-6A;call_id=e0a20cb844d211e0acd8001422093865@<client IP>;code=200;reason=OK;src_user=<caller number>;src_domain=</span></span></span><span style="font-family: courier new,monospace;"><span style="font-family: arial,helvetica,sans-serif;"><span style="font-family: courier new,monospace;"><client IP></span></span></span><span style="font-family: courier new,monospace;"><span style="font-family: arial,helvetica,sans-serif;"><span style="font-family: courier new,monospace;">;dst_ouser=<called number>;dst_user=</span></span></span><span style="font-family: courier new,monospace;"><called number></span><span style="font-family: courier new,monospace;"><span style="font-family: arial,helvetica,sans-serif;"><span style="font-family: courier new,monospace;">;dst_domain=10.90.1.251;src_ip=<client IP></span><br>
<br>...<br><br><span style="font-family: courier new,monospace;">Mar 2 14:42:44 kamailio2 /usr/local/sbin/kamailio[28920]: NOTICE: acc [acc.c:275]: ACC: request acknowledged: timestamp=1299073364;method=ACK;from_tag=1577886432-3759264324-335599788-1698171170;to_tag=5FFAEA34-6A;call_id=e0a20cb844d211e0acd8001422093865@</span></span></span><span style="font-family: courier new,monospace;"><span style="font-family: arial,helvetica,sans-serif;"><span style="font-family: courier new,monospace;"><client IP></span></span></span><span style="font-family: courier new,monospace;"><span style="font-family: arial,helvetica,sans-serif;"><span style="font-family: courier new,monospace;">;code=200;reason=OK;src_user=<caller number>;src_domain=<client IP>;dst_ouser=</span></span></span><span style="font-family: courier new,monospace;"><called number></span><span style="font-family: courier new,monospace;"><span style="font-family: arial,helvetica,sans-serif;"><span style="font-family: courier new,monospace;">;dst_user=<called number>;dst_domain=10.90.1.251;src_ip=<client IP></span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">...</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">Mar 2 14:43:00 kamailio2 /usr/local/sbin/kamailio[28903]: ERROR: <script>: ACK WITHOUT MATCHING TRANSACTION in e0a20cb844d211e0acd8001422093865@<client IP> call... ignore and discard.</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">... </span><br><br></span></span><span style="font-family: courier new,monospace;">Mar 2 14:43:00
kamailio2 /usr/local/sbin/kamailio[28904]: NOTICE: acc [acc.c:275]: ACC:
transaction answered:
timestamp=1299073380;method=BYE;from_tag=1577886432-3759264324-335599788-1698171170;to_tag=5FFAEA34-6A;call_id=e0a20cb844d211e0acd8001422093865@</span><span style="font-family: courier new,monospace;"><span style="font-family: arial,helvetica,sans-serif;"><span style="font-family: courier new,monospace;"><client IP></span></span></span><span style="font-family: courier new,monospace;">;code=200;reason=OK;src_user=<caller number>;src_domain=<client IP>;dst_ouser=</span><span style="font-family: courier new,monospace;"><called number></span><span style="font-family: courier new,monospace;">;dst_user=<called number>;dst_domain=10.90.1.251;src_ip=<client IP></span><br>
<span style="font-family: courier new,monospace;"><span style="font-family: arial,helvetica,sans-serif;"><br></span></span>