<h3>Description</h3>
<p>We have a Kamailio v4.4.5 Git-Rev. e9d576 running on Debian Wheezy 32bit KVM machine. We are experiencing random crashes since morning and so far Kamailio has crashed at least twice.</p>
<h3>Troubleshooting</h3>
<h4>Reproduction</h4>
<p>This issue occurs randomly. We are not able to reproduce it manually in our lab.</p>
<h4>Debugging Data</h4>
<pre><code>GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/adx-webrtc/sbin/kamailio...done.
[New LWP 29499]
warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Core was generated by `/usr/local/adx-webrtc/sbin/kamailio -f /usr/local/adx-webrtc/etc/kamailio/kamai'.
Program terminated with signal 11, Segmentation fault.
#0 0xb5a91e6b in append_time_request_f (msg=0xb6a6fa80, p1=0x0, p2=0x0) at textops.c:1223
1223 textops.c: No such file or directory.
(gdb) bt full
#0 0xb5a91e6b in append_time_request_f (msg=0xb6a6fa80, p1=0x0, p2=0x0) at textops.c:1223
time_str = {s = 0x0, len = 37}
now = 1490005155
bd_time = 0xb775b6a0
hf = 0xb7117590
anchor = 0xb75f0f9c
__FUNCTION__ = "append_time_request_f"
#1 0x0806509e in do_action (h=0xbfdf9b7c, a=0xb668c940, msg=0xb6a6fa80) at action.c:1054
ret = -5
v = 0
dst = {send_sock = 0x38, to = {s = {sa_family = 53644, sa_data = "2\bX\323\062\b9\000\000\000\314\317\062\b"}, sin = {sin_family = 53644, sin_port = 2098, sin_addr = {
s_addr = 137548632}, sin_zero = "9\000\000\000\314\317\062\b"}, sin6 = {sin6_family = 53644, sin6_port = 2098, sin6_flowinfo = 137548632, sin6_addr = {__in6_u = {
__u6_addr8 = "9\000\000\000\314\317\062\b\000\000\000\000\250\002U\377", __u6_addr16 = {57, 0, 53196, 2098, 0, 0, 680, 65365}, __u6_addr32 = {57, 137547724, 0,
4283761320}}}, sin6_scope_id = 1529706238}}, id = -1218478588, proto = -40 '\330', send_flags = {f = 205 '\315', blst_imask = 121 'y'}}
tmp = 0x80e1081 "\213E\344\353\032\220\353\001\220\203", <incomplete sequence \340>
new_uri = 0x832cfcc "core"
end = 0x832d358 "add_lump_rpl2"
crt = 0xbfdf8598 "x\217\337\277\"P\006\b|\233\337\277@\311h\266\200\372\246\266\360\304h\266\b\020_\266\360ꑶt"
cmd = 0xb6619e00
len = -1230570520
user = -1075863780
uri = {user = {s = 0x20014 <Address 0x20014 out of bounds>, len = 1490002293}, passwd = {s = 0xb5c6629c "\340\060\t", len = -1075863780}, host = {
s = 0xb6a6fbe8 "\340\257D\bs\001", len = -1075871080}, port = {s = 0x823f044 "\211E\364\203", <incomplete sequence \364>, len = 275}, params = {
s = 0xb6762891 "uq_saas_domain)", len = 14}, sip_params = {
s = 0x844b09c "exen.vpn:5080;tag=5nwi2d\r\nCall-ID: 76rjac@lifeguard.andrexen.vpn\r\nCSeq: 1 OPTIONS\r\nContact: <sip:checksip@lifeguard.andrexen.vpn:48713>\r\nAccept: application/sdp\r\nContent-Length: 0\r\n\r\n", len = 13}, headers = {s = 0xffffffff <Address 0xffffffff out of bounds>, len = 642}, port_no = 4104, proto = 46687, type = 2133685845,
flags = 196617, transport = {s = 0xbfdf82c8 "\350\257E\b\020", len = 1}, ttl = {s = 0xb65f1008 "\001", len = -1075871068}, user_param = {s = 0x1 <Address 0x1 out of bounds>,
len = 0}, maddr = {s = 0xbfdf9f1c "\020", len = -1230570520}, method = {s = 0xbfdf8398 "\230\205\337\277\273B\030\b\020\216\337\277\004", len = 135790452}, lr = {
s = 0xbfdf9b7c "\003", len = -1234489668}, r2 = {s = 0xb6a6fa80 "\204\256", len = -1245799139}, gr = {s = 0xb6a6fa80 "\204\256", len = -1233770052}, transport_val = {
s = 0x845afe8 "", len = 16}, ttl_val = {s = 0xbfdf9f1c "\020", len = -1230570520}, user_param_val = {s = 0xbfdf83b8 "x\204\337\277\353\004\016\b", len = -1245742364},
maddr_val = {s = 0x113 <Address 0x113 out of bounds>, len = -1233770351}, method_val = {s = 0xe <Address 0xe out of bounds>, len = 138719388}, lr_val = {
s = 0xd <Address 0xd out of bounds>, len = 332}, r2_val = {s = 0xbfdf83c8 "\376", len = -1234654324}, gr_val = {s = 0xb668a73c "\a", len = 0}}
next_hop = {user = {s = 0xbfdf833c "", len = 0}, passwd = {s = 0x0, len = -1230570616}, host = {s = 0xbfdf9f1c "\020", len = -1230570520}, port = {
s = 0xbfdf8398 "\230\205\337\277\273B\030\b\020\216\337\277\004", len = 135806558}, params = {s = 0xbfdf9b7c "\003", len = -1230570880}, sip_params = {s = 0xbfdf8568 "",
len = -1234648760}, headers = {s = 0x0, len = 0}, port_no = 43916, proto = 46696, type = ERROR_URI_T, flags = 262164, transport = {s = 0xb771161f "\201\303\001\255\002",
len = -1245289828}, ttl = {s = 0xb6a6fb88 "\002", len = -1075863780}, user_param = {s = 0xb6a6fbe8 "\340\257D\bs\001", len = -1075871064}, maddr = {
s = 0x8070fc5 "\211E\360\213\225P\377\377\377\213E\b\211P\b\213\205L\377\377\377\200\344\376\211\302\vU\364\213E\b\211P\004\213E\360\311\303U\211\345\201\354", <incomplete sequence \330>, len = -1075871248}, method = {s = 0xb66b2ebc "g\006", len = -1230570880}, lr = {s = 0x60014 <Address 0x60014 out of bounds>, len = 0}, r2 = {s = 0x0, len = 1}, gr = {
s = 0xb6a6fb88 "\002", len = -1075863780}, transport_val = {s = 0xb6a6fbe8 "\340\257D\bs\001", len = -1075871272}, ttl_val = {
s = 0xd1e9220b <Address 0xd1e9220b out of bounds>, len = 1625566052}, user_param_val = {s = 0x0, len = -1075871064}, maddr_val = {
s = 0x8169901 "\211E\344", <incomplete sequence \351>, len = -1075864708}, method_val = {s = 0xb676d94c "\002", len = -1230570880}, lr_val = {
s = 0x60014 <Address 0x60014 out of bounds>, len = -1494096780}, r2_val = {s = 0xffffffff <Address 0xffffffff out of bounds>, len = 628}, gr_val = {
s = 0x39 <Address 0x39 out of bounds>, len = 64}}
u = 0xb6a6fb88
port = 49119
dst_host = 0x7d0
---Type <return> to continue, or q <return> to quit---
i = -1230570520
flags = 0
avp = 0xbfdf9f1c
st = {flags = 2, id = 42812, name = {n = 0, s = {s = 0x0, len = 0}, re = 0x0}, avp = 0x0}
sct = 0x1
sjt = 0x0
rve = 0xb66b6300
mct = 0xb75f3020
rv = 0xb6a6fa80
rv1 = 0x1
c1 = {cache_type = 30, val_type = 3062845320, c = {avp_val = {n = 0, s = {s = 0x0, len = -65536}, re = 0x0}, pval = {rs = {s = 0x0, len = -65536}, ri = -1217329915,
flags = -1230570616}}, i2s = "\210\373\246\266\034\237\337\277\350\373\246\266\230\201\337\277%;\030\b\000"}
s = {s = 0x8290229 "\205\300\017\211^\002", len = 138719371}
srevp = {0xb6a6fbe8, 0xbfdf8198}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0,
attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {
number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0},
data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {
type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
__FUNCTION__ = "do_action"
#2 0x08070970 in run_actions (h=0xbfdf9b7c, a=0xb668c940, msg=0xb6a6fa80) at action.c:1549
t = 0xb668c940
ret = -1
mod = 0x25
ms = 0
__FUNCTION__ = "run_actions"
#3 0x08065022 in do_action (h=0xbfdf9b7c, a=0xb668c9dc, msg=0xb6a6fa80) at action.c:1045
ret = 1
v = 1
dst = {send_sock = 0x20, to = {s = {sa_family = 42387, sa_data = ":\bC\304:\b\316\t\000\000\370\207:\b"}, sin = {sin_family = 42387, sin_port = 2106, sin_addr = {
s_addr = 138069059}, sin_zero = "\316\t\000\000\370\207:\b"}, sin6 = {sin6_family = 42387, sin6_port = 2106, sin6_flowinfo = 138069059, sin6_addr = {__in6_u = {
__u6_addr8 = "\316\t\000\000\370\207:\b\034\216\337\277\b\020_\266", __u6_addr16 = {2510, 0, 34808, 2106, 36380, 49119, 4104, 46687}, __u6_addr32 = {2510, 138053624,
3219099164, 3059683336}}}, sin6_scope_id = 8}}, id = -1217179589, proto = 1 '\001', send_flags = {f = 0 '\000', blst_imask = 0 '\000'}}
tmp = 0x1e <Address 0x1e out of bounds>
new_uri = 0xb68f8c10 "\001"
end = 0xbfdf9486 "D\b\244J\217\266\234bƵ\034\237\337\277\350\373\246\266\b\227\337\277\032\267&\bD]9\b\004"
crt = 0x83475ac "(): memory allocation failure\n"
cmd = 0x5
len = 4
user = 0
uri = {user = {s = 0x436bf57 <Address 0x436bf57 out of bounds>, len = -1075868328}, passwd = {s = 0x80c5317 "\211E\364\213E\364\351\373\001", len = 0}, host = {
s = 0xb7710505 "\201Û\265\002", len = -1241058888}, port = {s = 0x8253671 "\213E\364\213P\004\213E\f\211\020\213E\f\307@\024\004", len = -1075868284}, params = {s = 0x0,
len = 28}, sip_params = {s = 0x0, len = 70696791}, headers = {s = 0x8 <Address 0x8 out of bounds>, len = -1075868120}, port_no = 4104, proto = 46687, type = URN_URI_T,
flags = 0, transport = {s = 0xbfdf8da8 "؍\337\277\027S\f\b", len = -1241075329}, ttl = {s = 0xb65f1008 "\001", len = -1075868284}, user_param = {s = 0x8398904 "core",
len = 0}, maddr = {s = 0xb773417d "(fwd->size & 0x4) == 0", len = 16777216}, method = {s = 0x364 <Address 0x364 out of bounds>, len = 8110980}, lr = {
---Type <return> to continue, or q <return> to quit---
s = 0xfffc9c <Address 0xfffc9c out of bounds>, len = 16777216}, r2 = {s = 0x4 <Address 0x4 out of bounds>, len = 13}, gr = {s = 0x0, len = -1075863780}, transport_val = {
s = 0xbfdf8dd8 "\250\216\337\277\203\361$\b\005", len = 135025431}, ttl_val = {s = 0x0, len = -1230570520}, user_param_val = {
s = 0xb76624e3 "\213U\344\003u\360\211B\024\353\257f\220\213E\020)\370\203\304\034[^_]Ív", len = 157513914}, maddr_val = {s = 0x838c064 "); Free fragment not found!\n",
len = 256}, method_val = {s = 0x8 <Address 0x8 out of bounds>, len = 0}, lr_val = {s = 0xbfdf9f1c "\020", len = 0}, r2_val = {
s = 0xbfdf8ea8 "\230\225\337\277\356\331+\b.\260D\b\005", len = 136638851}, gr_val = {s = 0x5 <Address 0x5 out of bounds>, len = 0}}
next_hop = {user = {s = 0x0, len = 0}, passwd = {s = 0xbfdf92a8 "", len = 0}, host = {s = 0x0, len = 248}, port = {s = 0x2 <Address 0x2 out of bounds>, len = 136638062},
params = {s = 0x0, len = -1}, sip_params = {s = 0x14 <Address 0x14 out of bounds>, len = -1253735563}, headers = {s = 0x0, len = -1253660952}, port_no = 65535, proto = 65535,
type = ERROR_URI_T, flags = 2772135936, transport = {s = 0x35 <Address 0x35 out of bounds>, len = -1514431728}, ttl = {s = 0xb6a6fb88 "\002", len = -1075863780}, user_param = {
s = 0xb7710505 "\201Û\265\002", len = -1241058888}, maddr = {s = 0x8253671 "\213E\364\213P\004\213E\f\211\020\213E\f\307@\024\004", len = -1075868412}, method = {s = 0x0,
len = 28}, lr = {s = 0x1 <Address 0x1 out of bounds>, len = 0}, r2 = {s = 0x0, len = -1}, gr = {s = 0xb65f1008 "\001", len = -1075863780}, transport_val = {
s = 0xb6a6fbe8 "\340\257D\bs\001", len = -1075868376}, ttl_val = {s = 0xb606b17f "\213\223,\b", len = -1235283960}, user_param_val = {s = 0xbfdf8d04 "", len = 0},
maddr_val = {s = 0xb65c5ea8 "U\211\345WVS\203\354|\350%\327\377\377\201\303ަ\002", len = -1235450529}, method_val = {s = 0x1000000 <Address 0x1000000 out of bounds>,
len = 72}, lr_val = {s = 0x7bc6a0 <Address 0x7bc6a0 out of bounds>, len = 16777144}, r2_val = {s = 0x1000000 <Address 0x1000000 out of bounds>, len = 4}, gr_val = {
s = 0x1 <Address 0x1 out of bounds>, len = -1252044316}}
u = 0x1
port = 46963
dst_host = 0x83475ca
i = -1
flags = 10
avp = 0xffffffff
st = {flags = 3219098728, id = 35944, name = {n = -1249644216, s = {s = 0xb583f148 "\f \001", len = 0}, re = 0xb583f148}, avp = 0xbfdf90bf}
sct = 0xb7757ff4
sjt = 0x1b
rve = 0xb668c4f0
mct = 0x963771c
rv = 0xb583d328
rv1 = 0xb62d0564
c1 = {cache_type = 3077930996, val_type = 3077936064, c = {avp_val = {n = 157485576, s = {s = 0x9630a08 "71.234", len = -1075868616}, re = 0x9630a08}, pval = {rs = {
s = 0x9630a08 "71.234", len = -1075868616}, ri = -1, flags = 0}}, i2s = '\000' <repeats 20 times>, " \n"}
s = {s = 0xb581a8d8 "\205\300u\f\213D$\034\203\304([Ív", len = -1217031232}
srevp = {0xbfdfa3ac, 0x21}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0,
attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {
number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0},
data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {
type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
__FUNCTION__ = "do_action"
#4 0x08070970 in run_actions (h=0xbfdf9b7c, a=0xb6687218, msg=0xb6a6fa80) at action.c:1549
t = 0xb668c9dc
ret = 1
mod = 0xbfdf9618
ms = 0
__FUNCTION__ = "run_actions"
#5 0x08061d7e in do_action (h=0xbfdf9b7c, a=0xb66340c4, msg=0xb6a6fa80) at action.c:678
ret = -5
---Type <return> to continue, or q <return> to quit---
v = 0
dst = {send_sock = 0x11a, to = {s = {sa_family = 0, sa_data = "\000\000z\001\000\000\000\000\000\000\n\001\000"}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 378},
sin_zero = "\000\000\000\000\n\001\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 378, sin6_addr = {__in6_u = {
__u6_addr8 = "\000\000\000\000\n\001\000\000\000\000\000\000\005\000\000", __u6_addr16 = {0, 0, 266, 0, 0, 0, 5, 0}, __u6_addr32 = {0, 266, 0, 5}}},
sin6_scope_id = 0}}, id = 346, proto = 0 '\000', send_flags = {f = 0 '\000', blst_imask = 0 '\000'}}
tmp = 0x844b006 "Via: SIP/2.0/UDP lifeguard.andrexen.vpn:48713;branch=z9hG4bKhjhs8ass877\r\nMax-Forwards: 69\r\nTo: sip:777@85.118.45.91\r\nFrom: sip:checksip@lifeguard.andrexen.vpn:5080;tag=5nwi2d\r\nCall-ID: 76rjac@lifeguar"...
new_uri = 0xb7710505 "\201Û\265\002"
end = 0xbfdf9aa4 ""
crt = 0xb68f5120 " "
cmd = 0xbfdf9f1c
len = 0
user = 8
uri = {user = {s = 0xb6a6fbe8 "\340\257D\bs\001", len = -1075864952}, passwd = {s = 0x8270416 "\203\370\002\017\216.\002", len = 137977156}, host = {
s = 0x4 <Address 0x4 out of bounds>, len = -1223592560}, port = {s = 0x83988e7 "parse_headers", len = 318}, params = {s = 0x8395d44 "core", len = -1075865496}, sip_params = {
s = 0x10 <Address 0x10 out of bounds>, len = 70696791}, headers = {s = 0xbfdf99dd "\261\006\266\b\020_\266\364\231\337\277h", len = -1075865496}, port_no = 20964,
proto = 46431, type = SIPS_URI_T, flags = 0, transport = {s = 0x4001fa <Address 0x4001fa out of bounds>, len = 0}, ttl = {s = 0x0, len = 138807552}, user_param = {
s = 0x16 <Address 0x16 out of bounds>, len = -1252044316}, maddr = {s = 0x436bf57 <Address 0x436bf57 out of bounds>, len = -1075865175}, method = {
s = 0xbfdf9a38 " Q\217\266\006\260D\b\034\237\337\277\001", len = -1252118708}, lr = {s = 0xbfdf9934 "\350\373\246\266\210\232\337\277/\352(\b\376\257D\b\365\337\071\b\006",
len = 0}, r2 = {s = 0xb <Address 0xb out of bounds>, len = -1075865288}, gr = {s = 0xbfdf9940 "\376\257D\b\365\337\071\b\006", len = -1218461980}, transport_val = {
s = 0xd696910 <Address 0xd696910 out of bounds>, len = -1075865404}, ttl_val = {s = 0xb778cd76 "\213M\340\205\300u3\211\360\213]\364\213u\370\213}\374\211\354]Ít&",
len = -1218418128}, user_param_val = {s = 0x4011fa <Address 0x4011fa out of bounds>, len = 0}, maddr_val = {s = 0x8054fde "libc.so.6", len = -1218498528}, method_val = {
s = 0xbfdf0002 "", len = -1}, lr_val = {s = 0x8054fde "libc.so.6", len = -1216865232}, r2_val = {s = 0x102 <Address 0x102 out of bounds>, len = 0}, gr_val = {
s = 0x1fa <Address 0x1fa out of bounds>, len = 0}}
next_hop = {user = {s = 0xffffffff <Address 0xffffffff out of bounds>, len = -1}, passwd = {s = 0x20 <Address 0x20 out of bounds>, len = 137978236}, host = {
s = 0x83988e7 "parse_headers", len = 318}, port = {s = 0x8395d44 "core", len = 106}, params = {s = 0xb622d9a4 "p\330\002", len = 59}, sip_params = {
s = 0x8401940 "{1 1 OPTIONS 76rjac@lifeguard.andrexen.vpn} ", len = 137565758}, headers = {s = 0xb65f1008 "\001", len = 0}, port_no = 47, proto = 0, type = 59,
flags = 138418496, transport = {s = 0x833163e "ERROR", len = -1232122012}, ttl = {s = 0x44b1a6 <Address 0x44b1a6 out of bounds>, len = 88}, user_param = {s = 0x0, len = 0},
maddr = {s = 0x0, len = 0}, method = {s = 0x0, len = 0}, lr = {s = 0x0, len = 0}, r2 = {s = 0x0, len = 1}, gr = {s = 0xb55ddf58 "\201Ìr\001", len = 0}, transport_val = {
s = 0x0, len = -1}, ttl_val = {s = 0x0, len = -1514551080}, user_param_val = {s = 0x21 <Address 0x21 out of bounds>, len = 3}, maddr_val = {s = 0xb68f8c10 "\001",
len = -1235283960}, method_val = {s = 0x8 <Address 0x8 out of bounds>, len = -1223592596}, lr_val = {
s = 0x844b04f "Max-Forwards: 69\r\nTo: sip:777@85.118.45.91\r\nFrom: sip:checksip@lifeguard.andrexen.vpn:5080;tag=5nwi2d\r\nCall-ID: 76rjac@lifeguard.andrexen.vpn\r\nCSeq: 1 OPTIONS\r\nContact: <sip:checksip@lifeguard.andrexe"..., len = -1075865292}, r2_val = {s = 0x1c8 <Address 0x1c8 out of bounds>, len = 0}, gr_val = {s = 0x0, len = -1075863780}}
u = 0x0
port = 0
dst_host = 0xfffc88
i = 4
flags = 136638851
avp = 0x0
st = {flags = 8, id = 0, name = {n = -1075863780, s = {s = 0xbfdf9f1c "\020", len = 0}, re = 0xbfdf9f1c}, avp = 0xbfdf9808}
sct = 0xffffffff
sjt = 0x30
rve = 0x2
mct = 0xa
rv = 0x8395d44
rv1 = 0x0
---Type <return> to continue, or q <return> to quit---
c1 = {cache_type = 16777216, val_type = RV_ACTION_ST, c = {avp_val = {n = -1245289828, s = {s = 0xb5c6629c "\340\060\t", len = -1075863780}, re = 0xb5c6629c}, pval = {rs = {
s = 0xb5c6629c "\340\060\t", len = -1075863780}, ri = -1230570520, flags = -1075865208}}, i2s = "\324\365&\b\b\020_\266\270K\217\266|a9\b\347\210\071\bN\001"}
s = {s = 0xb65f1008 "\001", len = 16776200}
srevp = {0x8, 0xb68f4b94}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0,
attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {
number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0},
data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {
type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
__FUNCTION__ = "do_action"
#6 0x08070970 in run_actions (h=0xbfdf9b7c, a=0xb66340c4, msg=0xb6a6fa80) at action.c:1549
t = 0xb66340c4
ret = -1
mod = 0x0
ms = 0
__FUNCTION__ = "run_actions"
#7 0x0807105d in run_top_route (a=0xb66340c4, msg=0xb6a6fa80, c=0x0) at action.c:1635
ctx = {rec_lev = 3, run_flags = 0, last_retcode = -1, jmp_env = {{__jmpbuf = {-1230570616, -1075863780, -1230570520, -1075864744, -774102517, 1625566052}, __mask_was_saved = 0,
__saved_mask = {__val = {4, 3219102888, 3064396680, 3071374700, 3064396776, 3053849578, 3053849578, 3053908408, 3219103516, 3219102696, 3053849925, 3219062798, 48, 0,
3053908408, 3219103516, 3064396776, 3219102728, 3053850835, 138719229, 3053894780, 7, 1, 3219103516, 1, 3219102776, 135912396, 3064396416, 1073741825, 0, 134977546,
3064396416}}}}}
p = 0xbfdf9b7c
ret = 48
sfbk = 0
#8 0x081479fc in receive_msg (
buf=0x844afe0 "OPTIONS sip:777@85.118.45.91 SIP/2.0\r\nVia: SIP/2.0/UDP lifeguard.andrexen.vpn:48713;branch=z9hG4bKhjhs8ass877\r\nMax-Forwards: 69\r\nTo: sip:777@85.118.45.91\r\nFrom: sip:checksip@lifeguard.andrexen.vpn:508"..., len=371, rcv_info=0xbfdf9ebc) at receive.c:240
msg = 0xb6a6fa80
ctx = {rec_lev = 5, run_flags = -1218441964, last_retcode = -1216865128, jmp_env = {{__jmpbuf = {-1216734968, -1, -1216737292, 134567086, 5, -1075864192},
__mask_was_saved = -1216799770, __saved_mask = {__val = {3078232768, 3076468768, 1, 5, 0, 5, 134565684, 138392384, 0, 3076525332, 961078559, 0, 3219104684, 3219103256,
3078191504, 3219103428, 3077456128, 33200, 3077641759, 961078559, 136537889, 18878, 3062860384, 4, 136537807, 2772170560, 4294967295, 3078230004, 134566635, 5,
3219103232, 3078167526}}}}}
ret = 0
inb = {
s = 0x844afe0 "OPTIONS sip:777@85.118.45.91 SIP/2.0\r\nVia: SIP/2.0/UDP lifeguard.andrexen.vpn:48713;branch=z9hG4bKhjhs8ass877\r\nMax-Forwards: 69\r\nTo: sip:777@85.118.45.91\r\nFrom: sip:checksip@lifeguard.andrexen.vpn:508"..., len = 371}
netinfo = {data = {s = 0xbfdf9db4 "\377\377\377\377\364\017z\267\353R\005\b\005", len = 134537572}, dst = 0xbfdf9da8, rcv = 0xb77a1a64}
__FUNCTION__ = "receive_msg"
#9 0x0823da26 in udp_rcv_loop () at udp_server.c:495
len = 371
buf = "OPTIONS sip:777@85.118.45.91 SIP/2.0\r\nVia: SIP/2.0/UDP lifeguard.andrexen.vpn:48713;branch=z9hG4bKhjhs8ass877\r\nMax-Forwards: 69\r\nTo: sip:777@85.118.45.91\r\nFrom: sip:checksip@lifeguard.andrexen.vpn:508"...
tmp = 0x8329042 "counters_prefork_init"
from = 0xb68f8a5c
fromlen = 16
---Type <return> to continue, or q <return> to quit---
ri = {src_ip = {af = 2, len = 4, u = {addrl = {1378711125, 4, 961078559, 1}, addr32 = {1378711125, 4, 961078559, 1}, addr16 = {30293, 21037, 4, 0, 58655, 14664, 1, 0},
addr = "Uv-R\004\000\000\000\037\345H9\001\000\000"}}, dst_ip = {af = 2, len = 4, u = {addrl = {1529706069, 0, 0, 0}, addr32 = {1529706069, 0, 0, 0}, addr16 = {30293,
23341, 0, 0, 0, 0, 0, 0}, addr = "Uv-[", '\000' <repeats 11 times>}}, src_port = 48713, dst_port = 5080, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {
sa_family = 2, sa_data = "\276IUv-R\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 18878, sin_addr = {s_addr = 1378711125},
sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 18878, sin6_flowinfo = 1378711125, sin6_addr = {__in6_u = {
__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0xb6611b94,
proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"
#10 0x080eaf7b in main_loop () at main.c:1614
i = 2
pid = 0
si = 0xb6611b94
si_desc = "udp receiver child=2 sock=85.118.45.91:5080\000̛\206\245̛\206\245\000\000\000\000 \v\353\265\364\317\344\265\037\345H9\001\000\000\000\037\345H9\370\240\337\277\303I%\bHF\222\266\004\000\000\000\071x4\b\204\211\064\bn\000\000\000\064x4\b\024\000\000\000\037\345H9", '\000' <repeats 11 times>
nrprocs = 4
woneinit = 1
__FUNCTION__ = "main_loop"
#11 0x080f23d4 in main (argc=13, argv=0xbfdfa524) at main.c:2631
cfg_stream = 0x90c2008
c = -1
r = 0
tmp = 0xbfdfbd8f ""
tmp_len = 1
port = 0
proto = 0
options = 0x833deb4 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:"
ret = -1
seed = 2912300657
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0xbfdfa450
p = 0x805c9c8 "[\201\303H\347\071"
st = {st_dev = 14, __pad1 = 0, st_ino = 6001, st_mode = 16832, st_nlink = 2, st_uid = 0, st_gid = 0, st_rdev = 0, __pad2 = 0, st_size = 60, st_blksize = 4096, st_blocks = 0,
st_atim = {tv_sec = 1487716504, tv_nsec = 24863882}, st_mtim = {tv_sec = 1490002293, tv_nsec = 141472099}, st_ctim = {tv_sec = 1490002293, tv_nsec = 141472099}, __unused4 = 0,
__unused5 = 0}
__FUNCTION__ = "main"
</code></pre>
<h4>Log Messages</h4>
<p>Here are Kamailio logs in time DESC order.</p>
<pre><code>Mar 20 12:02:15 webrtc-as kernel: [2291290.426195] kamailio[2963]: segfault at 23 ip b5a0be6b sp bfca2950 error 6 in textops.so[b59fb000+40000]
Mar 20 12:02:15 webrtc-as adx-webrtc[2963]: {1 1 OPTIONS osj8es@lifeguard.andrexen.vpn} ERROR: <core> [mem/f_malloc.c:498]: fm_malloc(): fm_malloc(0xb656b008, 64) called from textops: textops.c: append_time_request_f(1216), module: textops; Free fragment not found!
Mar 20 12:02:15 webrtc-as adx-webrtc[2963]: {1 1 OPTIONS osj8es@lifeguard.andrexen.vpn} ERROR: <core> [mem/f_malloc.c:415]: fm_search_defrag(): fm_search_defrag(0xb656b008, 64); Free fragment not found!
Mar 20 12:02:15 webrtc-as adx-webrtc[2963]: {1 1 OPTIONS osj8es@lifeguard.andrexen.vpn} ERROR: <core> [mem/f_malloc.c:498]: fm_malloc(): fm_malloc(0xb656b008, 520) called from core: rvalue.c: rval_new_empty(224), module: core; Free fragment not found!
Mar 20 12:02:15 webrtc-as adx-webrtc[2963]: {1 1 OPTIONS osj8es@lifeguard.andrexen.vpn} ERROR: <core> [mem/f_malloc.c:415]: fm_search_defrag(): fm_search_defrag(0xb656b008, 520); Free fragment not found!
Mar 20 12:02:15 webrtc-as adx-webrtc[2963]: {1 1 OPTIONS osj8es@lifeguard.andrexen.vpn} ERROR: <core> [mem/f_malloc.c:498]: fm_malloc(): fm_malloc(0xb656b008, 520) called from core: rvalue.c: rval_new_empty(224), module: core; Free fragment not found!
Mar 20 12:02:15 webrtc-as adx-webrtc[2963]: {1 1 OPTIONS osj8es@lifeguard.andrexen.vpn} ERROR: <core> [mem/f_malloc.c:415]: fm_search_defrag(): fm_search_defrag(0xb656b008, 520); Free fragment not found!
...
Mar 20 11:43:53 webrtc-as adx-webrtc[2963]: {1 102 ACK 62e26bfc56736b99020d5c1a3eaa34c1@172.16.171.16} ERROR: siptrace [siptrace.c:1603]: trace_send_duplicate(): bad host name in uri
Mar 20 11:43:53 webrtc-as adx-webrtc[2963]: {1 102 ACK 62e26bfc56736b99020d5c1a3eaa34c1@172.16.171.16} CRITICAL: <core> [proxy.c:179]: hostent_cpy(): memory allocation failure
Mar 20 11:43:53 webrtc-as adx-webrtc[2963]: {1 102 ACK 62e26bfc56736b99020d5c1a3eaa34c1@172.16.171.16} ERROR: <core> [mem/f_malloc.c:498]: fm_malloc(): fm_malloc(0xb656b008, 16) called from core: proxy.c: hostent_cpy(177), module: core; Free fragment not found!
Mar 20 11:43:53 webrtc-as adx-webrtc[2963]: {1 102 ACK 62e26bfc56736b99020d5c1a3eaa34c1@172.16.171.16} ERROR: <core> [mem/f_malloc.c:415]: fm_search_defrag(): fm_search_defrag(0xb656b008, 16); Free fragment not found!
...
Mar 20 11:43:37 webrtc-as adx-webrtc[2963]: {1 102 ACK 2dcb43b73bc08adf29c5c7ed086a1b47@172.16.171.16} ERROR: siptrace [siptrace.c:1614]: trace_send_duplicate(): bad host name in uri
Mar 20 11:43:37 webrtc-as adx-webrtc[2963]: {1 102 ACK 2dcb43b73bc08adf29c5c7ed086a1b47@172.16.171.16} CRITICAL: <core> [proxy.c:179]: hostent_cpy(): memory allocation failure
Mar 20 11:43:37 webrtc-as adx-webrtc[2963]: {1 102 ACK 2dcb43b73bc08adf29c5c7ed086a1b47@172.16.171.16} ERROR: <core> [mem/f_malloc.c:498]: fm_malloc(): fm_malloc(0xb656b008, 16) called from core: proxy.c: hostent_cpy(177), module: core; Free fragment not found!
Mar 20 11:43:37 webrtc-as adx-webrtc[2963]: {1 102 ACK 2dcb43b73bc08adf29c5c7ed086a1b47@172.16.171.16} ERROR: <core> [mem/f_malloc.c:415]: fm_search_defrag(): fm_search_defrag(0xb656b008, 16); Free fragment not found!
</code></pre>
<h4>SIP Traffic</h4>
<pre><code>No Data Available
</code></pre>
<h3>Possible Solutions</h3>
<p>Currently no idea. However one strange thing is that although siptrace module is load by kamailio but none of its functions are called or flags are set, i wonder why it shows up in kamailio log in the first place.</p>
<h3>Additional Information</h3>
<ul>
<li><strong>Kamailio Version</strong> - output of <code>kamailio -v</code></li>
</ul>
<pre><code>version: kamailio 4.4.5 (i386/linux) e9d576
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, select.
id: e9d576 -dirty
compiled on 06:32:04 Jan 25 2017 with gcc 4.7.2
</code></pre>
<ul>
<li><strong>Operating System</strong>:</li>
</ul>
<pre><code>Linux webrtc-as 3.10-0.bpo.2-686-pae #1 SMP Debian 3.10.5-1~bpo70+1 (2013-08-11) i686 GNU/Linux
</code></pre>
<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />You are receiving this because you are subscribed to this thread.<br />Reply to this email directly, <a href="https://github.com/kamailio/kamailio/issues/1040">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/AF36ZYBFeT9srNybY6YDcdxG0ttfREvGks5rnocHgaJpZM4MieDF">mute the thread</a>.<img alt="" height="1" src="https://github.com/notifications/beacon/AF36ZeNLFd4SnTB6HE5k0FN1raL1PM0pks5rnocHgaJpZM4MieDF.gif" width="1" /></p>
<div itemscope itemtype="http://schema.org/EmailMessage">
<div itemprop="action" itemscope itemtype="http://schema.org/ViewAction">
<link itemprop="url" href="https://github.com/kamailio/kamailio/issues/1040"></link>
<meta itemprop="name" content="View Issue"></meta>
</div>
<meta itemprop="description" content="View this Issue on GitHub"></meta>
</div>
<script type="application/json" data-scope="inboxmarkup">{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/kamailio/kamailio","title":"kamailio/kamailio","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/kamailio/kamailio"}},"updates":{"snippets":[{"icon":"DESCRIPTION","message":"Segmentation Fault in Kamailio v4.4.5 Git-Rev. e9d576 (#1040)"}],"action":{"name":"View Issue","url":"https://github.com/kamailio/kamailio/issues/1040"}}}</script>