<h3>Description</h3>
<p>Hi,</p>
<p>I got a problem with Kamailio, it crashed 11 times in a row, with no visible reason, only thing I notice is MALLOC always on all the core-dumps.</p>
<h3>Troubleshooting</h3>
<h4>Reproduction</h4>
<p>It was aleatory 11 crashes in a row</p>
<h4>Debugging Data</h4>
<pre><code>Program terminated with signal 11, Segmentation fault.
#0 0x0000000000622796 in qm_status (qm=0x7fcbe89b4000) at mem/q_malloc.c:788
788 f!=&(qm->free_hash[h].head); f=f->u.nxt_free, i++, j++){
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.5-7.el6_0.x86_64 db4-4.7.25-18.el6_4.x86_64 elfutils-libelf-0.152-1.el6.x86_64 glibc-2.12-1.132.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-10.el6_4.6.x86_64 libacl-2.2.49-6.el6.x86_64 libattr-2.4.44-7.el6.x86_64 libcap-2.16-5.5.el6.x86_64 libcom_err-1.41.12-18.el6.x86_64 libgcc-4.4.7-11.el6.x86_64 libselinux-2.0.94-5.3.el6_4.1.x86_64 lm_sensors-libs-3.1.1-17.el6.x86_64 lua-5.1.4-4.1.el6.x86_64 mysql-libs-5.1.73-3.el6_5.x86_64 net-snmp-libs-5.5-50.el6_6.1.x86_64 nspr-4.10.0-1.el6.x86_64 nss-3.15.1-15.el6.x86_64 nss-softokn-freebl-3.14.3-9.el6.x86_64 nss-util-3.15.1-3.el6.x86_64 openssl-1.0.1e-30.el6_6.4.x86_64 pcre-7.8-6.el6.x86_64 perl-libs-5.10.1-136.el6.x86_64 popt-1.13-7.el6.x86_64 rpm-libs-4.8.0-37.el6.x86_64 tcp_wrappers-libs-7.6-57.el6.x86_64 xz-libs-4.999.9-0.3.beta.20091007git.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt full
#0 0x0000000000622796 in qm_status (qm=0x7fcbe89b4000) at mem/q_malloc.c:788
f = 0x69746163696c7070
i = 304
j = 3
h = 10
unused = 0
memlog = 5
mem_summary = 3
__FUNCTION__ = "qm_status"
#1 0x000000000061a758 in qm_debug_frag (qm=0x7fcbe89b4000, f=0x7fcbf83fa268) at mem/q_malloc.c:150
__FUNCTION__ = "qm_debug_frag"
#2 0x000000000061cd90 in qm_free (qm=0x7fcbe89b4000, p=0x7fcbf83fa298, file=0x7fcc0f8d336d "tm: h_table.c", func=0x7fcc0f8d3648 "free_cell", line=186) at mem/q_malloc.c:468
f = 0x7fcbf83fa268
size = 40
next = 0x400
prev = 0x7ffffdc3b7a0
__FUNCTION__ = "qm_free"
#3 0x00007fcc0f814c9d in free_cell (dead_cell=0x7fcbf83ea0e8) at h_table.c:186
b = 0x7fcbf83fa298 "INVITE sip:0034126498563@proxy SIP/2.0\r\nRecord-Route: <sip:192.168.1.5;lr;did=d4d.7fa>\r\nCSeq: 1 INVITE\r\nCall-ID: SD7jt2a01-b81723f229af11e61f26a15065\r\nFrom: <sip:012854697@192.168.1."...
i = 0
rpl = 0x0
tt = 0x7fcbe8ad0f18
foo = 0x7ffffdc3b870
cbs = 0x0
cbs_tmp = 0x7fcbf83f2308
__FUNCTION__ = "free_cell"
#4 0x00007fcc0f859a1c in wait_handler (ti=269273136, wait_tl=0x7fcbf83ea168, data=0x7fcbf83ea0e8) at timer.c:675
p_cell = 0x7fcbf83ea0e8
ret = 1
#5 0x00000000005fd647 in timer_list_expire (t=269273136, h=0x7fcbe8a2d908, slow_l=0x7fcbe8a2fb38, slow_mark=15874) at timer.c:888
tl = 0x7fcbf83ea168
ret = 269273136
#6 0x00000000005fda8f in timer_handler () at timer.c:953
saved_ticks = 269273136
run_slow_timer = 0
i = 514
__FUNCTION__ = "timer_handler"
#7 0x00000000005fdefd in timer_main () at timer.c:992
No locals.
#8 0x00000000004a77e1 in main_loop () at main.c:1700
i = 8
pid = 0
si = 0x0
si_desc = "udp receiver child=7 sock=192.168.1.5:5060\000\177\000\000\200\272\303\375\377\177\000\000\023{N\000\000\000\000\000к\303\375\377\177\000\000\004\000\000\000\000\000\000\000`TA\000\000\000\000\000(\225\236\350\313\177", '\000' <repeats 14 times>, "\001\000\000\000к\303\375\377\177\000\000\266{N\000\000\000\000"
nrprocs = 8
__FUNCTION__ = "main_loop"
#9 0x00000000004acfa6 in main (argc=7, argv=0x7ffffdc3bd48) at main.c:2581
cfg_stream = 0x13b8010
c = -1
r = 0
tmp = 0x7ffffdc3cf70 ""
tmp_len = 32767
port = -37503970
proto = 0
options = 0x703718 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 2884067974
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x40d134
p = 0xc2 <Address 0xc2 out of bounds>
__FUNCTION__ = "main"
(gdb) info locals
f = 0x69746163696c7070
i = 304
j = 3
h = 10
unused = 0
memlog = 5
mem_summary = 3
__FUNCTION__ = "qm_status"
(gdb) list
783 LOG_(DEFAULT_FACILITY, memlog, "qm_status: ",
784 "dumping free list stats :\n");
785 for(h=0,i=0;h<QM_HASH_SIZE;h++){
786 unused=0;
787 for (f=qm->free_hash[h].head.u.nxt_free,j=0;
788 f!=&(qm->free_hash[h].head); f=f->u.nxt_free, i++, j++){
789 if (!FRAG_WAS_USED(f)){
790 unused++;
791 #ifdef DBG_QM_MALLOC
792 LOG_(DEFAULT_FACILITY, memlog, "qm_status: ",
(gdb) quit
</code></pre>
<h4>Log Messages</h4>
<pre><code>Mar 9 10:33:49 kamserv /usr/local/sbin/kamailio[2945]: : <core> [mem/q_malloc.c:140]: qm_debug_frag(): BUG: qm_*: fragm. 0x7f064ea52480 (address 0x7f064ea524b0) beginning overwritten(646e756f622d6575)!
Mar 9 10:33:52 kamserv abrtd: Directory 'ccpp-2017-03-09-10:33:49-2945' creation detected
Mar 9 10:33:52 kamserv abrt[3793]: Saved core dump of pid 2945 (/usr/local/sbin/kamailio) to /var/spool/abrt/ccpp-2017-03-09-10:33:49-2945 (608370688 bytes)
Mar 9 10:33:52 kamserv abrtd: Executable '/usr/local/sbin/kamailio' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Mar 9 10:33:52 kamserv /usr/local/sbin/kamailio[2908]: ALERT: <core> [main.c:784]: handle_sigs(): child process 2945 exited by a signal 6
Mar 9 10:33:52 kamserv /usr/local/sbin/kamailio[2908]: ALERT: <core> [main.c:787]: handle_sigs(): core was generated
Mar 9 10:33:52 kamserv /usr/local/sbin/kamailio[2908]: INFO: <core> [main.c:799]: handle_sigs(): terminating due to SIGCHLD
Mar 9 10:33:52 kamserv /usr/local/sbin/kamailio[2953]: INFO: <core> [main.c:850]: sig_usr(): signal 15 received
Mar 9 10:33:52 kamserv /usr/local/sbin/kamailio[2955]: INFO: <core> [main.c:850]: sig_usr(): signal 15 received
Mar 9 10:33:52 kamserv /usr/local/sbin/kamailio[2947]: INFO: <core> [main.c:850]: sig_usr(): signal 15 received
Mar 9 10:33:52 kamserv /usr/local/sbin/kamailio[2941]: INFO: <core> [main.c:850]: sig_usr(): signal 15 received
Mar 9 10:33:52 kamserv /usr/local/sbin/kamailio[2939]: INFO: <core> [main.c:850]: sig_usr(): signal 15 received
Mar 9 10:33:52 kamserv /usr/local/sbin/kamailio[2935]: INFO: <core> [main.c:850]: sig_usr(): signal 15 received
Mar 9 10:33:52 kamserv /usr/local/sbin/kamailio[2927]: INFO: <core> [main.c:850]: sig_usr(): signal 15 received
Mar 9 10:33:52 kamserv /usr/local/sbin/kamailio[2943]: INFO: <core> [main.c:850]: sig_usr(): signal 15 received
Mar 9 10:33:52 kamserv /usr/local/sbin/kamailio[2929]: INFO: <core> [main.c:850]: sig_usr(): signal 15 received
Mar 9 10:33:52 kamserv /usr/local/sbin/kamailio[2937]: INFO: <core> [main.c:850]: sig_usr(): signal 15 received
Mar 9 10:33:52 kamserv /usr/local/sbin/kamailio[2949]: INFO: <core> [main.c:850]: sig_usr(): signal 15 received
Mar 9 10:33:52 kamserv /usr/local/sbin/kamailio[2931]: INFO: <core> [main.c:850]: sig_usr(): signal 15 received
Mar 9 10:33:52 kamserv /usr/local/sbin/kamailio[2933]: INFO: <core> [main.c:850]: sig_usr(): signal 15 received
Mar 9 10:33:52 kamserv abrtd: 'post-create' on '/var/spool/abrt/ccpp-2017-03-09-10:33:49-2945' exited with 1
Mar 9 10:33:52 kamserv abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2017-03-09-10:33:49-2945'
</code></pre>
<h4>SIP Traffic</h4>
<pre><code>(paste your sip traffic here)
</code></pre>
<h3>Possible Solutions</h3>
<h3>Additional Information</h3>
<ul>
<li><strong>Kamailio Version</strong> - output of <code>kamailio -v</code></li>
</ul>
<pre><code>version: kamailio 4.2.8 (x86_64/linux) 4507b8
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 64MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 4507b8
compiled on 16:19:58 Oct 26 2016 with gcc 4.4.7
</code></pre>
<ul>
<li><strong>Operating System</strong>:</li>
</ul>
<pre><code>Linux kamserv 2.6.32-431.el6.x86_64 #1 SMP Sun Nov 10 22:19:54 EST 2013 x86_64 x86_64 x86_64 GNU/Linux
</code></pre>
<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />You are receiving this because you are subscribed to this thread.<br />Reply to this email directly, <a href="https://github.com/kamailio/kamailio/issues/1026">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/AF36ZYze4RLg3q9nUrq2FTbWZA-f9_qJks5rj9fxgaJpZM4MX6sv">mute the thread</a>.<img alt="" height="1" src="https://github.com/notifications/beacon/AF36ZcKqO71ra6bh61jZYsG3p2OFYusoks5rj9fxgaJpZM4MX6sv.gif" width="1" /></p>
<div itemscope itemtype="http://schema.org/EmailMessage">
<div itemprop="action" itemscope itemtype="http://schema.org/ViewAction">
<link itemprop="url" href="https://github.com/kamailio/kamailio/issues/1026"></link>
<meta itemprop="name" content="View Issue"></meta>
</div>
<meta itemprop="description" content="View this Issue on GitHub"></meta>
</div>
<script type="application/json" data-scope="inboxmarkup">{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/kamailio/kamailio","title":"kamailio/kamailio","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/kamailio/kamailio"}},"updates":{"snippets":[{"icon":"DESCRIPTION","message":"Kamailio Crashes 11 times: Malloc (#1026)"}],"action":{"name":"View Issue","url":"https://github.com/kamailio/kamailio/issues/1026"}}}</script>