Hello,<br><br>my answer is inline:<br><br><div class="gmail_quote">2011/5/12 Timo Reimann <span dir="ltr"><<a href="mailto:timo.reimann@1und1.de">timo.reimann@1und1.de</a>></span><br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Hey,<br>
<div class="im"><br>
<br>
On 12.05.2011 12:37, Anton Roman wrote:<br>
> we got a core in dialog module. We are using kamailio 3.1.2. Below you<br>
> can find a full backtrace from the dump and the Kamailio compilation<br>
> options. Please, if you need further information don't hesitate to ask<br>
> me for it. I can't precise the situation when it is generated because<br>
> we have a quite high load in this server.<br>
<br>
</div>The call path seems to be like this:<br>
<br>
transaction timer fires -> tm module walking through callback list finds<br>
unref_dlg() -> tm module calls unref_dlg() -> boom.<br>
<br>
I wonder why unref_dlg() was registered as a tm callback in the first<br>
place -- the dialog module shouldn't do that. Are you using any custom<br>
modules that would possibly do such registrations?<br></blockquote><div><br>No, we aren't. We got the code of all the modules directly from the git repository. <br><br></div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<br>
As to the reason of the segfault, the dialog structure or hash table may<br>
already be gone when unref_dlg() is called. Can you go to stack #0 and<br>
tell us what the value of each of the following data structures is (use<br>
"p <data structure> in gdb):<br>
<br>
*dlg<br>
d_table<br>
d_table->entries<br></blockquote><div><br>Here you have:<br><br></div>(gdb) p *dlg<br>$1 = {ref = 793790803, next = 0xa0d4b4f20303032, prev = 0x504953203a616956, h_id = 808333871, h_entry = 1346655535, state = 775174432, <br>
lifetime = 841888562, start_ts = 892219952, dflags = 808794678, sflags = 1648046134, toroute = 1668178290, toroute_name = {<br> s = 0x62344768397a3d68 <Address 0x62344768397a3d68 out of bounds>, len = 946221643}, from_rr_nb = 1886534457, tl = {<br>
next = 0x72460a0d30363035, prev = 0x6f6e4122203a6d6f, timeout = 1869445486}, callid = {<br> s = 0x6f6e613a7069733c <Address 0x6f6e613a7069733c out of bounds>, len = 1869445486}, from_uri = {<br> s = 0x3230322e33322e34 <Address 0x3230322e33322e34 out of bounds>, len = 1043739950}, to_uri = {<br>
s = 0x396637643173613d <Address 0x396637643173613d out of bounds>, len = 221656933}, req_uri = {<br> s = 0x34333a7069733c20 <Address 0x34333a7069733c20 out of bounds>, len = 925972025}, tag = {{<br> s = 0x33322e3539314030 <Address 0x33322e3539314030 out of bounds>, len = 942747189}, {<br>
s = 0x743b3e303630353a <Address 0x743b3e303630353a out of bounds>, len = 1178429281}}, cseq = {{<br> s = 0x364134322d344434 <Address 0x364134322d344434 out of bounds>, len = 1631848973}, {<br> s = 0x203932202c697246 <Address 0x203932202c697246 out of bounds>, len = 544236883}}, route_set = {{<br>
s = 0x343a30313a333020 <Address 0x343a30313a333020 out of bounds>, len = 1296506937}, {<br> s = 0x203a44492d6c6c61 <Address 0x203a44492d6c6c61 out of bounds>, len = 1630549808}}, contact = {{<br> s = 0x6639633663313634 <Address 0x6639633663313634 out of bounds>, len = 858808881}, {<br>
s = 0x6464363632663631 <Address 0x6464363632663631 out of bounds>, len = 775174464}}, bind_addr = {0x530a0d36352e3230, <br> 0x43203a7265767265}, cbs = {first = 0x5049532d6f637369, types = 1702125895}, profile_links = 0x782e32312d534f49}<br>
(gdb) p d_table<br>$2 = (struct dlg_table *) 0x7f08a9e38ee0<br>(gdb) p d_table->entries<br>$3 = (struct dlg_entry *) 0x7f08a9e38f00<br>(gdb) info f<br>Stack level 0, frame at 0x7fff49059980:<br> rip = 0x7f08cc18fa39 in unref_dlg (dlg_hash.c:598); saved rip 0x7f08ce92fa02<br>
called by frame at 0x7fff49059a10<br> source language c.<br> Arglist at 0x7fff490598a8, args: dlg=0x7f08a9f67da8, cnt=1<br> Locals at 0x7fff490598a8, Previous frame's sp is 0x7fff49059980<br> Saved registers:<br> rbx at 0x7fff49059948, rbp at 0x7fff49059950, r12 at 0x7fff49059958, r13 at 0x7fff49059960, r14 at 0x7fff49059968, r15 at 0x7fff49059970,<br>
rip at 0x7fff49059978<br>(gdb) <br><br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<br>
<br>
Cheers,<br>
<font color="#888888"><br>
--Timo<br>
</font><div><div></div><div class="h5"><br></div></div></blockquote><div> <br></div><div>Thank you very much,<br><br>Regards,<br>Antón<br> </div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div><div class="h5">
<br>
<br>
> (gdb) bt full<br>
> #0 unref_dlg (dlg=0x7f08a9f67da8, cnt=1) at dlg_hash.c:598<br>
> d_entry = (struct dlg_entry *) 0x7f10304b8b68<br>
> #1 0x00007f08ce92fa02 in run_trans_callbacks_internal<br>
> (cb_lst=0x7f08aa203e98, type=32768, trans=0x7f08aa203e28,<br>
> params=0x7fff49059a10)<br>
> at t_hooks.c:290<br>
> cbp = (struct tm_callback *) 0x7f08a9f6e7e0<br>
> backup_from = (avp_list_t *) 0x8b3330<br>
> backup_to = (avp_list_t *) 0x8b3338<br>
> backup_dom_from = (avp_list_t *) 0x8b3340<br>
> backup_dom_to = (avp_list_t *) 0x8b3348<br>
> backup_uri_from = (avp_list_t *) 0x8b3320<br>
> backup_uri_to = (avp_list_t *) 0x8b3328<br>
> #2 0x00007f08ce92fc56 in run_trans_callbacks (type=32768, trans=<value<br>
> optimized out>, req=0x1, rpl=0x7f10304b8b68, code=-868566200)<br>
> at t_hooks.c:317<br>
> params = {req = 0x0, rpl = 0x0, param = 0x7f08a9f6e7f0, code = 0,<br>
> flags = 0, branch = 0, t_rbuf = 0x0, dst = 0x0, send_buf = {<br>
> s = 0x0, len = 0}}<br>
> #3 0x00007f08ce915b36 in free_cell (dead_cell=0x7f08aa203e28) at<br>
> h_table.c:136<br>
> b = <value optimized out><br>
> i = <value optimized out><br>
> rpl = <value optimized out><br>
> tt = <value optimized out><br>
> foo = <value optimized out><br>
> cbs = <value optimized out><br>
> ---Type <return> to continue, or q <return> to quit---<br>
> __FUNCTION__ = "free_cell"<br>
> #4 0x00007f08ce9319f1 in wait_handler (ti=<value optimized out>,<br>
> wait_tl=<value optimized out>, data=<value optimized out>) at timer.c:645<br>
> p_cell = (struct cell *) 0x7f08aa203e28<br>
> #5 0x0000000000513d8f in timer_main () at timer.c:894<br>
> No locals.<br>
> #6 0x000000000046501b in main_loop () at main.c:1618<br>
> i = 4<br>
> pid = <value optimized out><br>
> si = (struct socket_info *) 0x0<br>
> si_desc = "udp receiver child=3<br>
> sock=XXX.XXX.XXX.XX:XXXX\000\000\000\210�\231\000\000\000\000\000\031",<br>
> '\0' <repeats 15 times>, "\001\000\000\000\000\000\000\000�\215\213",<br>
> '\0' <repeats 13 times>, "\004", '\0' <repeats 15 times>,<br>
> "\b\236\005I�\177\000\000\227%J\000\000\000\000"<br>
> #7 0x0000000000467873 in main (argc=<value optimized out>,<br>
> argv=0x7fff49059e08) at main.c:2398<br>
> cfg_stream = (FILE *) 0x12e1010<br>
> c = <value optimized out><br>
> r = <value optimized out><br>
> tmp = 0x7fff4905ae90 ""<br>
> tmp_len = 32520<br>
> port = <value optimized out><br>
> proto = <value optimized out><br>
> ret = <value optimized out><br>
> seed = 1235801225<br>
> ---Type <return> to continue, or q <return> to quit---<br>
> rfd = 4<br>
> debug_save = <value optimized out><br>
> debug_flag = 0<br>
> dont_fork_cnt = 0<br>
> n_lst = <value optimized out><br>
> p = <value optimized out><br>
> (gdb)<br>
> (gdb) quit<br>
> kamailio2:/var/kamailio# kamailio -V<br>
> version: kamailio 3.1.2 (x86_64/linux) eb24c1-dirty<br>
> flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS,<br>
> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,<br>
> DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,<br>
> USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES<br>
> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,<br>
> MAX_URI_SIZE 1024, BUF_SIZE 65535, PKG_SIZE 8MB<br>
> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.<br>
> id: eb24c1 -dirty<br>
> compiled on 09:35:52 Apr 28 2011 with gcc 4.3.2<br>
</div></div></blockquote></div><br>