Hi, <br><br>I compiled kamailio with MEMDBG = 0 because I didn't want the memory debug in kamailio log, but I didn't know these turns on f_malloc and disabling q_malloc...<br><br>Probably the size of shared memory that I start kamailio is 32MB because I didn't gave a different -m parameter value, exist a command to verify this information?<br>
<br>At moment that crash happened, there were few registered users agents and were being made tests with register and calls with TLS protocol. I got only the two situations that I showed at first email.<br><br><br>Best Regards<br>
<br><div class="gmail_quote">2012/7/30 Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Hello,<br>
<br>
MEMDBG=0 turns on f_malloc, disabling q_malloc. The token F_MALLOC
in the kamailio -V output shows f_malloc is in use.<br>
<br>
q_malloc is more suitable for debugging, as it keep trace of
location in the source code that allocates/frees the memory chunks,
at the expense of a bit more memory usage due to tracking overhead.
q_malloc was the default for 3.1, that's why I asked because the
trace showed lines from f_malloc c code.<br>
<br>
What is the size of shared memory you start kamailio? Is the default
32MB or you give a different -m parameter value? Was the serve very
loaded? How many such situations did you get so far?<br>
<br>
Cheers,<br>
Daniel<div><div class="h5"><br>
<br>
<br>
<div>On 7/30/12 7:16 PM, Bruno Bresciani
wrote:<br>
</div>
<blockquote type="cite">Hi,<br>
<br>
Bellow is output of 'kamailio -V'...<br>
<br>
------------------------------------------------------------------------------------------------------------------<br>
version: kamailio 3.1.2 (i386/linux) 4d9f90<br>
flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS,
USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM,
SHM_MMAP, PKG_MALLOC, F_MALLOC, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER,
USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES<br>
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN
16, MAX_URI_SIZE 1024, BUF_SIZE 65535, PKG_SIZE 4MB<br>
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.<br>
id: 4d9f90<br>
compiled on 10:39:54 Apr 29 2011 with gcc 4.1.2<br>
-------------------------------------------------------------------------------------------------------------------<br>
<br>
Daniel, I don't remember to enable f_malloc... <span lang="en"><span>Actually I
do not know the difference between f_malloc and g_malloc to
choose compile kamailio with f_malloc instead of g_malloc.</span><span> <br>
My commands to compile kamailio were:<br>
<b><br>
$ make PREFIX="/home2/local/kamailio" FLAVOUR=kamailio
include_modules="db_postgres tls" MEMDBG=0 cfg</b></span></span><b><br>
$ make all<br>
$ make install</b><br>
<span lang="en"><span> <br>
If you need more informations, </span></span><span lang="en"><span>I
am available...<br>
<br>
Best Regards<br>
</span></span>
<div class="gmail_quote">
2012/7/30 Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Hello,<br>
<br>
can you provide output of 'kamailio -V'? Did you enable
f_malloc instead of q_malloc which was default in 3.1?<br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
<div>On 7/30/12 1:44 PM, Bruno Bresciani wrote:<br>
</div>
<blockquote type="cite">Thank's for feedback Daniel,<br>
<br>
This core is add to the tracker with ID 247...<br>
<br>
Best Regards<br>
<br>
<br>
<div class="gmail_quote">2012/7/30 Daniel-Constantin
Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Hello,<br>
<br>
it is in my to-do list to investigate, so far I
was not able to reproduce.<br>
<br>
Vacation time together with other travelings
makes a bit more slower process. Please add it
also to the tracker:<br>
<br>
<a href="http://sip-router.org/tracker/" target="_blank">http://sip-router.org/tracker/</a><br>
<br>
A minor release for 3.3 branch is going to be
soon, being there makes sure it will be reviewed
to see if it applies.<br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
<div>On 7/27/12 3:00 PM, Bruno Bresciani
wrote:<br>
</div>
<blockquote type="cite">Hi Users,<br>
<br>
This is my ultimate try to ask for some
help to debug the two cores generated by
fm_realloc() function. If somebody can
give me a feedback about my questions or
doubts I will be very grateful, this user
list of kamailio always help me to solve
my problems.<br>
<br>
Best Regards<br>
<br>
<br>
<div class="gmail_quote">2012/7/23 Bruno
Bresciani <span dir="ltr"><<a href="mailto:bruno.bresciani@gmail.com" target="_blank">bruno.bresciani@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Hi All,<br>
<br>
I imagine that this core doesn't easy
to replicate but somebody can help me
to understand why it was generated?
There is some way to prevent it?<br>
<br>
Best Regards
<div>
<div> <br>
<br>
<br>
<div class="gmail_quote">
2012/7/18 Renan Capaverde <span dir="ltr"><<a href="mailto:renan.capaverde@digitro.com.br" target="_blank">renan.capaverde@digitro.com.br</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>I am having the same
problem on kamailio 3.1.6.
Someone please help.<br>
<br>
Cheers,<br>
Renan<br>
<br>
Em 17/7/2012 14:26, Bruno
Bresciani escreveu:<br>
</div>
<div>
<div>
<blockquote type="cite">Excuse-me
Daniel, <span lang="en"><span>I
forgot</span> <span>this
information...
Kamailio version
3.1.2<br>
<br>
Best Regards<br>
<br>
</span></span><br>
<div class="gmail_quote">
2012/7/17
Daniel-Constantin
Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Hello,<br>
<br>
send also the
version
'kamailio -V' to
match the proper
sources.<br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
<div>On
7/17/12 7:11
PM, Bruno
Bresciani
wrote:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div>Hi all,<br>
<br>
I observed two
at 0x08173bf0
in fm_realloc
(qm=0xb5f7d000,
p=0xb61f0bc0,
size=0) at
mem/f_malloc.c:536.<br>
<br>
I don't know
exactly what
may be caused
this core
because the
debug of
kamailio is
disabled, but
looking at
backtrace
(gdb) and
source code I
could notice a
relationship
with TLS
connection
establishment
(handshake).
Other
important fact
is that core
was generated
by the kamalio
itself with
signal 6
(abort)
because it
can't find the
pointer to
memory
realloc.<br>
I'd like to
know if
someone
already
observed this
issue and if
my analyse
above is
correct?<br>
<br>
Bellow is
backtrace full
with gdb:<br>
<br>
Core was
generated by
`/home2/local/kamailio/sbin/kamailio
-P
/var/run/kamailio.pid'.<br>
Program
terminated
with signal 6,
Aborted.<br>
#0 0x00f30402
in
__kernel_vsyscall
()<br>
(gdb)<br>
(gdb)<br>
(gdb)<br>
(gdb)<br>
(gdb) bt full<br>
#0 0x00f30402
in
__kernel_vsyscall
()<br>
No symbol
table info
available.<br>
#1 0x009fec10
in raise ()
from
/lib/libc.so.6<br>
No symbol
table info
available.<br>
#2 0x00a00521
in abort ()
from
/lib/libc.so.6<br>
No symbol
table info
available.<br>
#3 0x08173bf0
in fm_realloc
(qm=0xb5f7d000,
p=0xb61f0bc0,
size=0) at
mem/f_malloc.c:536<br>
f =
(struct
fm_frag *)
0xb61f0bb8<br>
pf =
<value
optimized
out><br>
orig_size = 18<br>
L=Ãb¬Ã2n =
(struct
fm_frag *)
0xb61f0bd2<br>
ptr =
<value
optimized
out><br>
hash =
23548<br>
#4 0x0051a87f
in ser_realloc
(ptr=0xb61f0bc0,
size=32) at
../../mem/shm_mem.h:266<br>
No locals.<br>
#5 0x00460295
in
OPENSSL_cleanse
() from
/lib/libcrypto.so.6<br>
No symbol
table info
available.<br>
#6 0x004609f7
in
CRYPTO_realloc
() from
/lib/libcrypto.so.6<br>
No symbol
table info
available.<br>
#7 0x003efb22
in sk_insert
() from
/lib/libcrypto.so.6<br>
No symbol
table info
available.<br>
#8 0x003efbaa
in sk_push ()
from
/lib/libcrypto.so.6<br>
No symbol
table info
available.<br>
#9 0x007c2ac3
in
ssl_bytes_to_cipher_list
() from
/lib/libssl.so.6<br>
No symbol
table info
available.<br>
#10 0x007aa9f3
in
ssl3_get_client_hello
() from
/lib/libssl.so.6<br>
No symbol
table info
available.<br>
#11 0x007ab815
in ssl3_accept
() from
/lib/libssl.so.6<br>
No symbol
table info
available.<br>
#12 0x007c1c3a
in SSL_accept
() from
/lib/libssl.so.6<br>
No symbol
table info
available.<br>
#13 0x005278e3
in tls_accept
(c=0xb621af88,
error=0xbf91dfc8)
at
tls_server.c:346<br>
ret =
<value
optimized
out><br>
ssl =
(SSL *)
0xb61c1878<br>
cert =
<value
optimized
out><br>
tls_c
= (struct
tls_extra_data
*) 0xb6208e58<br>
tls_log =
<value
optimized
out><br>
#14 0x0052da8e
in tls_read_f
(c=0xb621af88,
flags=0xbf91e200)
at
tls_server.c:1028<br>
r =
(struct
tcp_req *)
0xb621affc<br>
bytes_free =
4095<br>
bytes_read =
70<br>
read_size =
4095<br>
ssl_error = 0<br>
ssl_read = 0<br>
ssl =
(SSL *)
0xb61c1878<br>
rd_buf
=
"\026\003\001\000A\001\000\000=\003\001Oÿ\v¡ywP\fkÃDÃ<br>
0/\000\a\000\005\001\000ÃÃÃï000\000\026\0009\0008\0005\000\026\000\023\000\n\0003\0002\00<br>
202i<ä<åþ\031ì234 \237ê030Y){lѶ"...06Ã'Ãr9ìë¸{[©\t\205\037\036ÿHPH5µ\t(é32Ã027æò036¾\"\rÔ¤YÃ215uº\027)ö000ñ215\vÃà éEa\023xéJ\206T\204CÃ
wr_buf =
"\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ<br>
?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"...<br>
rd =
{buf =
0xbf90dfa4
"\026\003\001",
pos = 70, used
= 70, size =
65536}<br>
wr = {<br>
buf =
0xbf8fdfa4
"\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ<br>
?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"...,
pos = 0, used
= 0,<br>
size = 65536}<br>
tls_c
= (struct
tls_extra_data
*) 0xb6208e58<br>
enc_rd_buf =
(struct
tls_rd_buf *)
0x0<br>
n = 0<br>
flush_flags =
<value
optimized
out><br>
err_src =
0x541a83 "TLS
read:"<br>
x =
<value
optimized
out><br>
#15 0x08158ade
in
tcp_read_headers
(c=0xb621af88,
read_flags=0xbf91e200)
at
tcp_read.c:406<br>
bytes
= -1<br>
remaining =
<value
optimized
out><br>
p =
<value
optimized
out><br>
r =
(struct
tcp_req *)
0xb621affc<br>
#16 0x08158fd4
in
tcp_read_req
(con=0xb621af88,
bytes_read=0xbf91e204,
read_flags=0xbf91e200)
at
tcp_read.c:871<br>
bytes
= <value
optimized
out><br>
total_bytes =
0<br>
resp =
<value
optimized
out><br>
size =
<value
optimized
out><br>
req =
(struct
tcp_req *)
0xb621affc<br>
dst =
{send_sock =
0x1, to = {s =
{sa_family =
57540, sa_data
=
"\221¿\020\000\000\000\000\000\000\000\020\000\000"},
sin =
{sin_family =
57540,<br>
sin_port
= 49041,
sin_addr =
{s_addr = 16},
sin_zero =
"\000\000\000\000\020\000\000"},
sin6 =
{sin6_family =
57540,
sin6_port =
49041,<br>
sin6_flowinfo
= 16,
sin6_addr =
{in6_u =
{u6_addr8 =
"\000\000\000\000\020\000\000\000\001\000\000\000\001\000\000",
u6_addr16 =
{0, 0, 16, 0,
1, 0,<br>
1,
0}, u6_addr32
= {0, 16, 1,
1}}},
sin6_scope_id
= 12}}, id =
-1080958468,
proto = 4
'\004',
send_flags =
{f = 0 '\0',
blst_imask = 0
'\0'}}<br>
c = 10
'\n'<br>
#17 0x08159a0f
in handle_io
(fm=0x83296b0,
events=1,
idx=-1) at
tcp_read.c:1150<br>
ret =
4<br>
n = 4<br>
read_flags = 1<br>
con =
(struct
tcp_connection
*) 0xb621af88<br>
s = 12<br>
resp =
<value
optimized
out><br>
t =
<value
optimized
out><br>
#18 0x0815d653
in
tcp_receive_loop
(unix_sock=35)
at
io_wait.h:1091<br>
No locals.<br>
---Type
<return>
to continue,
or q
<return>
to quit---<br>
#19 0x0812885b
in
tcp_init_children
() at
tcp_main.c:4819<br>
r = 3<br>
reader_fd_1 =
35<br>
pid =
<value
optimized
out><br>
si =
<value
optimized
out><br>
#20 0x080ad412
in main_loop
() at
main.c:1632<br>
i = 4<br>
pid =
<value
optimized
out><br>
si =
(struct
socket_info *)
0x0<br>
si_desc = "udp
receiver
child=3 sock=<a href="http://192.168.166.31:5060" target="_blank">192.168.166.31:5060</a>\000\221¿\002\000\000\000/\200\220\000\001\000\000\000\000\000H\000¨ä21¿\001\000\000\000°¶è026¶\000\000\000\000\000õ\030s2\b\002\000\000\000âf\b\000õ\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000ÿÿÿÿ¸ä21¿"<br>
#21 0x080afbe7
in main
(argc=3,
argv=dwarf2_read_address:
Corrupted
DWARF
expression.<br>
) at
main.c:2398<br>
cfg_stream =
(FILE *)
0x8a2d008<br>
c =
<value
optimized
out><br>
r = 0<br>
tmp =
0xbf91e674
"\212þ\221¿\233þ\221¿°þ\221¿»þ\221¿Ã\221¿ù1¿\020ÿ\221¿Dÿ\221¿Lÿ\221¿Wÿ\221¿]ÿ\221¿oÿ\221¿{ÿ\221¿\202ÿ\221
tmp_len =
10492229<br>
port =
136192361<br>
proto
= -1080957480<br>
ret =
<value
optimized
out><br>
seed =
21782655<br>
rfd =
4<br>
debug_save =
<value
optimized
out><br>
debug_flag = 0<br>
dont_fork_cnt
= 0<br>
n_lst
= <value
optimized
out><br>
p =
<value
optimized
out><br>
<br>
<br>
Best Regards<br>
<br>
<fieldset></fieldset>
<br>
</div>
</div>
<pre>_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
<a href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a>
<a href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><span><font color="#888888">
</font></span></pre>
<span><font color="#888888">
</font></span></blockquote>
<span><font color="#888888">
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - <a href="http://asipto.com/u/katu" target="_blank">http://asipto.com/u/katu</a>
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - <a href="http://asipto.com/u/kpw" target="_blank">http://asipto.com/u/kpw</a></pre>
</font></span></div>
</blockquote>
</div>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
<a href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a>
<a href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<br>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
SIP Express Router (SER) and
Kamailio (OpenSER) - sr-users
mailing list<br>
<a href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a><br>
<a href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
<a href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a>
<a href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - <a href="http://asipto.com/u/katu" target="_blank">http://asipto.com/u/katu</a>
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - <a href="http://asipto.com/u/kpw" target="_blank">http://asipto.com/u/kpw</a></pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - <a href="http://asipto.com/u/katu" target="_blank">http://asipto.com/u/katu</a>
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - <a href="http://asipto.com/u/kpw" target="_blank">http://asipto.com/u/kpw</a></pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - <a href="http://asipto.com/u/katu" target="_blank">http://asipto.com/u/katu</a>
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - <a href="http://asipto.com/u/kpw" target="_blank">http://asipto.com/u/kpw</a></pre>
</div></div></div>
</blockquote></div><br>