<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hello,<br>
<br>
<div class="moz-cite-prefix">On 7/30/12 9:01 PM, Bruno Bresciani
wrote:<br>
</div>
<blockquote
cite="mid:CAKv6MrYB1kYTuiAKuJNRiEh4rw_AFU2POP4GD+BHD5h9_ru81A@mail.gmail.com"
type="cite">Hi, <br>
<br>
I compiled kamailio with MEMDBG = 0 because I didn't want the
memory debug in kamailio log, but I didn't know these turns on
f_malloc and disabling q_malloc...<br>
</blockquote>
the log messages related to memory operations can be controlled by
global parameters memdbg and memlog.<br>
<br>
<blockquote
cite="mid:CAKv6MrYB1kYTuiAKuJNRiEh4rw_AFU2POP4GD+BHD5h9_ru81A@mail.gmail.com"
type="cite"><br>
Probably the size of shared memory that I start kamailio is 32MB
because I didn't gave a different -m parameter value, exist a
command to verify this information?<br>
</blockquote>
<br>
kamctl fifo get_statistics all<br>
<br>
and see the shared memory total value. It will be interesting to see
available shared memory as well.<br>
<br>
<blockquote
cite="mid:CAKv6MrYB1kYTuiAKuJNRiEh4rw_AFU2POP4GD+BHD5h9_ru81A@mail.gmail.com"
type="cite">
<br>
At moment that crash happened, there were few registered users
agents and were being made tests with register and calls with TLS
protocol. I got only the two situations that I showed at first
email.<br>
</blockquote>
<br>
If you plan to start a new installation, I strongly recommend 3.3
branch, the code is more actual and easier to debug. 3.1 is no
longer an official maintained branch, those being now 3.3 and 3.2.
I'm looking at this issue to be sure it is no longer in latest
stable.<br>
<br>
Cheers,<br>
Daniel<br>
<br>
<blockquote
cite="mid:CAKv6MrYB1kYTuiAKuJNRiEh4rw_AFU2POP4GD+BHD5h9_ru81A@mail.gmail.com"
type="cite"><br>
<br>
Best Regards<br>
<br>
<div class="gmail_quote">2012/7/30 Daniel-Constantin Mierla <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Hello,<br>
<br>
MEMDBG=0 turns on f_malloc, disabling q_malloc. The token
F_MALLOC in the kamailio -V output shows f_malloc is in use.<br>
<br>
q_malloc is more suitable for debugging, as it keep trace of
location in the source code that allocates/frees the memory
chunks, at the expense of a bit more memory usage due to
tracking overhead. q_malloc was the default for 3.1, that's
why I asked because the trace showed lines from f_malloc c
code.<br>
<br>
What is the size of shared memory you start kamailio? Is the
default 32MB or you give a different -m parameter value? Was
the serve very loaded? How many such situations did you get
so far?<br>
<br>
Cheers,<br>
Daniel
<div>
<div class="h5"><br>
<br>
<br>
<div>On 7/30/12 7:16 PM, Bruno Bresciani wrote:<br>
</div>
<blockquote type="cite">Hi,<br>
<br>
Bellow is output of 'kamailio -V'...<br>
<br>
------------------------------------------------------------------------------------------------------------------<br>
version: kamailio 3.1.2 (i386/linux) 4d9f90<br>
flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS,
TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST,
DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC,
USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST,
HAVE_RESOLV_RES<br>
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144,
MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535,
PKG_SIZE 4MB<br>
poll method support: poll, epoll_lt, epoll_et,
sigio_rt, select.<br>
id: 4d9f90<br>
compiled on 10:39:54 Apr 29 2011 with gcc 4.1.2<br>
-------------------------------------------------------------------------------------------------------------------<br>
<br>
Daniel, I don't remember to enable f_malloc... <span
lang="en"><span>Actually I do not know the
difference between f_malloc and g_malloc to choose
compile kamailio with f_malloc instead of
g_malloc.</span><span> <br>
My commands to compile kamailio were:<br>
<b><br>
$ make PREFIX="/home2/local/kamailio"
FLAVOUR=kamailio include_modules="db_postgres
tls" MEMDBG=0 cfg</b></span></span><b><br>
$ make all<br>
$ make install</b><br>
<span lang="en"><span> <br>
If you need more informations, </span></span><span
lang="en"><span>I am available...<br>
<br>
Best Regards<br>
</span></span>
<div class="gmail_quote"> 2012/7/30 Daniel-Constantin
Mierla <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Hello,<br>
<br>
can you provide output of 'kamailio -V'? Did you
enable f_malloc instead of q_malloc which was
default in 3.1?<br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
<div>On 7/30/12 1:44 PM, Bruno Bresciani
wrote:<br>
</div>
<blockquote type="cite">Thank's for feedback
Daniel,<br>
<br>
This core is add to the tracker with ID
247...<br>
<br>
Best Regards<br>
<br>
<br>
<div class="gmail_quote">2012/7/30
Daniel-Constantin Mierla <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:miconda@gmail.com"
target="_blank">miconda@gmail.com</a>></span><br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Hello,<br>
<br>
it is in my to-do list to
investigate, so far I was not able
to reproduce.<br>
<br>
Vacation time together with other
travelings makes a bit more slower
process. Please add it also to the
tracker:<br>
<br>
<a moz-do-not-send="true"
href="http://sip-router.org/tracker/"
target="_blank">http://sip-router.org/tracker/</a><br>
<br>
A minor release for 3.3 branch is
going to be soon, being there makes
sure it will be reviewed to see if
it applies.<br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
<div>On 7/27/12 3:00 PM, Bruno
Bresciani wrote:<br>
</div>
<blockquote type="cite">Hi
Users,<br>
<br>
This is my ultimate try to ask
for some help to debug the two
cores generated by
fm_realloc() function. If
somebody can give me a
feedback about my questions or
doubts I will be very
grateful, this user list of
kamailio always help me to
solve my problems.<br>
<br>
Best Regards<br>
<br>
<br>
<div class="gmail_quote">2012/7/23
Bruno Bresciani <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:bruno.bresciani@gmail.com"
target="_blank">bruno.bresciani@gmail.com</a>></span><br>
<blockquote
class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
Hi All,<br>
<br>
I imagine that this core
doesn't easy to replicate
but somebody can help me
to understand why it was
generated? There is some
way to prevent it?<br>
<br>
Best Regards
<div>
<div> <br>
<br>
<br>
<div
class="gmail_quote">
2012/7/18 Renan
Capaverde <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:renan.capaverde@digitro.com.br" target="_blank">renan.capaverde@digitro.com.br</a>></span><br>
<blockquote
class="gmail_quote"
style="margin:0 0
0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
<div>I am having
the same
problem on
kamailio
3.1.6. Someone
please help.<br>
<br>
Cheers,<br>
Renan<br>
<br>
Em 17/7/2012
14:26, Bruno
Bresciani
escreveu:<br>
</div>
<div>
<div>
<blockquote
type="cite">Excuse-me
Daniel, <span
lang="en"><span>I
forgot</span>
<span>this
information...
Kamailio
version 3.1.2<br>
<br>
Best Regards<br>
<br>
</span></span><br>
<div
class="gmail_quote">
2012/7/17
Daniel-Constantin
Mierla <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span><br>
<blockquote
class="gmail_quote"
style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
Hello,<br>
<br>
send also the
version
'kamailio -V'
to match the
proper
sources.<br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
<div>On
7/17/12 7:11
PM, Bruno
Bresciani
wrote:<br>
</div>
</div>
</div>
<blockquote
type="cite">
<div>
<div>Hi all,<br>
<br>
I observed two
at 0x08173bf0
in fm_realloc
(qm=0xb5f7d000,
p=0xb61f0bc0,
size=0) at
mem/f_malloc.c:536.<br>
<br>
I don't know
exactly what
may be caused
this core
because the
debug of
kamailio is
disabled, but
looking at
backtrace
(gdb) and
source code I
could notice a
relationship
with TLS
connection
establishment
(handshake).
Other
important fact
is that core
was generated
by the kamalio
itself with
signal 6
(abort)
because it
can't find the
pointer to
memory
realloc.<br>
I'd like to
know if
someone
already
observed this
issue and if
my analyse
above is
correct?<br>
<br>
Bellow is
backtrace full
with gdb:<br>
<br>
Core was
generated by
`/home2/local/kamailio/sbin/kamailio
-P
/var/run/kamailio.pid'.<br>
Program
terminated
with signal 6,
Aborted.<br>
#0 0x00f30402
in
__kernel_vsyscall
()<br>
(gdb)<br>
(gdb)<br>
(gdb)<br>
(gdb)<br>
(gdb) bt full<br>
#0 0x00f30402
in
__kernel_vsyscall
()<br>
No symbol
table info
available.<br>
#1 0x009fec10
in raise ()
from
/lib/libc.so.6<br>
No symbol
table info
available.<br>
#2 0x00a00521
in abort ()
from
/lib/libc.so.6<br>
No symbol
table info
available.<br>
#3 0x08173bf0
in fm_realloc
(qm=0xb5f7d000,
p=0xb61f0bc0,
size=0) at
mem/f_malloc.c:536<br>
f =
(struct
fm_frag *)
0xb61f0bb8<br>
pf =
<value
optimized
out><br>
orig_size = 18<br>
L=Ãb¬Ã2n =
(struct
fm_frag *)
0xb61f0bd2<br>
ptr =
<value
optimized
out><br>
hash =
23548<br>
#4 0x0051a87f
in ser_realloc
(ptr=0xb61f0bc0,
size=32) at
../../mem/shm_mem.h:266<br>
No locals.<br>
#5 0x00460295
in
OPENSSL_cleanse
() from
/lib/libcrypto.so.6<br>
No symbol
table info
available.<br>
#6 0x004609f7
in
CRYPTO_realloc
() from
/lib/libcrypto.so.6<br>
No symbol
table info
available.<br>
#7 0x003efb22
in sk_insert
() from
/lib/libcrypto.so.6<br>
No symbol
table info
available.<br>
#8 0x003efbaa
in sk_push ()
from
/lib/libcrypto.so.6<br>
No symbol
table info
available.<br>
#9 0x007c2ac3
in
ssl_bytes_to_cipher_list
() from
/lib/libssl.so.6<br>
No symbol
table info
available.<br>
#10 0x007aa9f3
in
ssl3_get_client_hello
() from
/lib/libssl.so.6<br>
No symbol
table info
available.<br>
#11 0x007ab815
in ssl3_accept
() from
/lib/libssl.so.6<br>
No symbol
table info
available.<br>
#12 0x007c1c3a
in SSL_accept
() from
/lib/libssl.so.6<br>
No symbol
table info
available.<br>
#13 0x005278e3
in tls_accept
(c=0xb621af88,
error=0xbf91dfc8)
at
tls_server.c:346<br>
ret =
<value
optimized
out><br>
ssl =
(SSL *)
0xb61c1878<br>
cert =
<value
optimized
out><br>
tls_c
= (struct
tls_extra_data
*) 0xb6208e58<br>
tls_log =
<value
optimized
out><br>
#14 0x0052da8e
in tls_read_f
(c=0xb621af88,
flags=0xbf91e200)
at
tls_server.c:1028<br>
r =
(struct
tcp_req *)
0xb621affc<br>
bytes_free =
4095<br>
bytes_read =
70<br>
read_size =
4095<br>
ssl_error = 0<br>
ssl_read = 0<br>
ssl =
(SSL *)
0xb61c1878<br>
rd_buf
=
"\026\003\001\000A\001\000\000=\003\001Oÿ\v¡ywP\fkÃDÃ<br>
0/\000\a\000\005\001\000ÃÃÃï000\000\026\0009\0008\0005\000\026\000\023\000\n\0003\0002\00<br>
202i<ä<åþ\031ì234 \237ê030Y){lѶ"...06Ã'Ãr9ìë¸{[©\t\205\037\036ÿHPH5µ\t(é32Ã027æò036¾\"\rÔ¤YÃ215uº\027)ö000ñ215\vÃà éEa\023xéJ\206T\204CÃ
wr_buf =
"\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ<br>
?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"...<br>
rd =
{buf =
0xbf90dfa4
"\026\003\001",
pos = 70, used
= 70, size =
65536}<br>
wr = {<br>
buf =
0xbf8fdfa4
"\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ<br>
?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"...,
pos = 0, used
= 0,<br>
size = 65536}<br>
tls_c
= (struct
tls_extra_data
*) 0xb6208e58<br>
enc_rd_buf =
(struct
tls_rd_buf *)
0x0<br>
n = 0<br>
flush_flags =
<value
optimized
out><br>
err_src =
0x541a83 "TLS
read:"<br>
x =
<value
optimized
out><br>
#15 0x08158ade
in
tcp_read_headers
(c=0xb621af88,
read_flags=0xbf91e200)
at
tcp_read.c:406<br>
bytes
= -1<br>
remaining =
<value
optimized
out><br>
p =
<value
optimized
out><br>
r =
(struct
tcp_req *)
0xb621affc<br>
#16 0x08158fd4
in
tcp_read_req
(con=0xb621af88,
bytes_read=0xbf91e204,
read_flags=0xbf91e200)
at
tcp_read.c:871<br>
bytes
= <value
optimized
out><br>
total_bytes =
0<br>
resp =
<value
optimized
out><br>
size =
<value
optimized
out><br>
req =
(struct
tcp_req *)
0xb621affc<br>
dst =
{send_sock =
0x1, to = {s =
{sa_family =
57540, sa_data
=
"\221¿\020\000\000\000\000\000\000\000\020\000\000"},
sin =
{sin_family =
57540,<br>
sin_port
= 49041,
sin_addr =
{s_addr = 16},
sin_zero =
"\000\000\000\000\020\000\000"},
sin6 =
{sin6_family =
57540,
sin6_port =
49041,<br>
sin6_flowinfo
= 16,
sin6_addr =
{in6_u =
{u6_addr8 =
"\000\000\000\000\020\000\000\000\001\000\000\000\001\000\000",
u6_addr16 =
{0, 0, 16, 0,
1, 0,<br>
1,
0}, u6_addr32
= {0, 16, 1,
1}}},
sin6_scope_id
= 12}}, id =
-1080958468,
proto = 4
'\004',
send_flags =
{f = 0 '\0',
blst_imask = 0
'\0'}}<br>
c = 10
'\n'<br>
#17 0x08159a0f
in handle_io
(fm=0x83296b0,
events=1,
idx=-1) at
tcp_read.c:1150<br>
ret =
4<br>
n = 4<br>
read_flags = 1<br>
con =
(struct
tcp_connection
*) 0xb621af88<br>
s = 12<br>
resp =
<value
optimized
out><br>
t =
<value
optimized
out><br>
#18 0x0815d653
in
tcp_receive_loop
(unix_sock=35)
at
io_wait.h:1091<br>
No locals.<br>
---Type
<return>
to continue,
or q
<return>
to quit---<br>
#19 0x0812885b
in
tcp_init_children
() at
tcp_main.c:4819<br>
r = 3<br>
reader_fd_1 =
35<br>
pid =
<value
optimized
out><br>
si =
<value
optimized
out><br>
#20 0x080ad412
in main_loop
() at
main.c:1632<br>
i = 4<br>
pid =
<value
optimized
out><br>
si =
(struct
socket_info *)
0x0<br>
si_desc = "udp
receiver
child=3 sock=<a
moz-do-not-send="true" href="http://192.168.166.31:5060" target="_blank">192.168.166.31:5060</a>\000\221¿\002\000\000\000/\200\220\000\001\000\000\000\000\000H\000¨ä21¿\001\000\000\000°¶è026¶\000\000\000\000\000õ\030s2\b\002\000\000\000âf\b\000õ\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000ÿÿÿÿ¸ä21¿"<br>
#21 0x080afbe7
in main
(argc=3,
argv=dwarf2_read_address:
Corrupted
DWARF
expression.<br>
) at
main.c:2398<br>
cfg_stream =
(FILE *)
0x8a2d008<br>
c =
<value
optimized
out><br>
r = 0<br>
tmp =
0xbf91e674
"\212þ\221¿\233þ\221¿°þ\221¿»þ\221¿Ã\221¿ù1¿\020ÿ\221¿Dÿ\221¿Lÿ\221¿Wÿ\221¿]ÿ\221¿oÿ\221¿{ÿ\221¿\202ÿ\221
tmp_len =
10492229<br>
port =
136192361<br>
proto
= -1080957480<br>
ret =
<value
optimized
out><br>
seed =
21782655<br>
rfd =
4<br>
debug_save =
<value
optimized
out><br>
debug_flag = 0<br>
dont_fork_cnt
= 0<br>
n_lst
= <value
optimized
out><br>
p =
<value
optimized
out><br>
<br>
<br>
Best Regards<br>
<br>
<fieldset></fieldset>
<br>
</div>
</div>
<pre>_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
<a moz-do-not-send="true" href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a>
<a moz-do-not-send="true" href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><span><font color="#888888">
</font></span></pre>
<span><font
color="#888888">
</font></span></blockquote>
<span><font
color="#888888">
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a moz-do-not-send="true" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a moz-do-not-send="true" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - <a moz-do-not-send="true" href="http://asipto.com/u/katu" target="_blank">http://asipto.com/u/katu</a>
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - <a moz-do-not-send="true" href="http://asipto.com/u/kpw" target="_blank">http://asipto.com/u/kpw</a></pre>
</font></span></div>
</blockquote>
</div>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
<a moz-do-not-send="true" href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a>
<a moz-do-not-send="true" href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<br>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
SIP Express Router
(SER) and Kamailio
(OpenSER) -
sr-users mailing
list<br>
<a
moz-do-not-send="true"
href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a><br>
<a
moz-do-not-send="true"
href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users"
target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
<a moz-do-not-send="true" href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a>
<a moz-do-not-send="true" href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a moz-do-not-send="true" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a moz-do-not-send="true" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - <a moz-do-not-send="true" href="http://asipto.com/u/katu" target="_blank">http://asipto.com/u/katu</a>
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - <a moz-do-not-send="true" href="http://asipto.com/u/kpw" target="_blank">http://asipto.com/u/kpw</a></pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a moz-do-not-send="true" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a moz-do-not-send="true" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - <a moz-do-not-send="true" href="http://asipto.com/u/katu" target="_blank">http://asipto.com/u/katu</a>
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - <a moz-do-not-send="true" href="http://asipto.com/u/kpw" target="_blank">http://asipto.com/u/kpw</a></pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a moz-do-not-send="true" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a moz-do-not-send="true" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - <a moz-do-not-send="true" href="http://asipto.com/u/katu" target="_blank">http://asipto.com/u/katu</a>
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - <a moz-do-not-send="true" href="http://asipto.com/u/kpw" target="_blank">http://asipto.com/u/kpw</a></pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla - <a class="moz-txt-link-freetext" href="http://www.asipto.com">http://www.asipto.com</a>
<a class="moz-txt-link-freetext" href="http://twitter.com/#!/miconda">http://twitter.com/#!/miconda</a> - <a class="moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - <a class="moz-txt-link-freetext" href="http://asipto.com/u/katu">http://asipto.com/u/katu</a>
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - <a class="moz-txt-link-freetext" href="http://asipto.com/u/kpw">http://asipto.com/u/kpw</a></pre>
</body>
</html>