<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>Hi Daniel </div><div><br></div><div>Thanks for your help .</div><div><br></div><div>Did you get a chance to look at this problem. </div><div>Please let me know how did you build the mongo-c-driver ?</div><div><br><br>Regards,<div>Jijo</div></div><div><br>On Oct 2, 2015, at 7:21 AM, Jijo <<a href="mailto:realjijo@gmail.com">realjijo@gmail.com</a>> wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr">I have rebuild the official version(4.3.2) again with the default scripts and now i'm getting diffrent core. <div><br><div>This happens only when i enable TLS module in kamailio. So i believe there is some corruption causing when TLS module in Kamailio and SSL in Mongo Driver is enabled. </div><div><div><br></div><div>I'm running with the offical kamailio script by just enabling mongo modules. It happens for NDB as well.<div><br></div><div>How did you build the mongo driver when you implemented it? May be i can try the same.</div><div><br></div><div>Please find the scripts used for testing.</div><div><br></div><div>
<p class=""><span class="">Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio.pid -m 64 -M 8'.</span></p>
<p class=""><span class="">Program terminated with signal 11, Segmentation fault.</span></p>
<p class=""><span class="">#0 0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08, _aor=0x7fff6de7ef70) at udomain.c:1017</span></p>
<p class=""><span class="">1017<span class=""> </span><span class=""> </span><span class=""> </span>lock_get(_d->table[sl].lock);</span></p>
<p class=""><span class="">Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 glibc-2.12-1.166.el6_7.1.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 nss-softokn-freebl-3.14.3-22.el6_6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64</span></p>
<p class=""><span class="">(gdb) bt</span></p>
<p class=""><span class="">#0 0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08, _aor=0x7fff6de7ef70) at udomain.c:1017</span></p>
<p class=""><span class="">#1 0x00007f85a1d7069b in add_contacts (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08, _a=0x7fff6de7ef70, _mode=0, _use_regid=1) at save.c:831</span></p>
<p class=""><span class="">#2 0x00007f85a1d72602 in save (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08, _cflags=0, _uri=0x0) at save.c:986</span></p>
<p class=""><span class="">#3 0x00007f85a1d5975a in w_save2 (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08 "h^{\234\205\177", _cflags=0x0) at reg_mod.c:414</span></p>
<p class=""><span class="">#4 0x000000000041decb in do_action (h=0x7fff6de7f720, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1059</span></p>
<p class=""><span class="">#5 0x000000000042a553 in run_actions (h=0x7fff6de7f720, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1548</span></p>
<p class=""><span class="">#6 0x000000000042abb8 in run_actions_safe (h=0x7fff6de80a10, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1613</span></p>
<p class=""><span class="">#7 0x0000000000543d50 in rval_get_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, i=0x7fff6de7fbf8, rv=0x7f85a3edf478, cache=0x0) at rvalue.c:912</span></p>
<p class=""><span class="">#8 0x0000000000547f88 in rval_expr_eval_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, res=0x7fff6de7fbf8, rve=0x7f85a3edf470) at rvalue.c:1906</span></p>
<p class=""><span class="">#9 0x000000000054837e in rval_expr_eval_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, res=0x7fff6de80080, rve=0x7f85a3edfb70) at rvalue.c:1914</span></p>
<p class=""><span class="">#10 0x000000000041d927 in do_action (h=0x7fff6de80a10, a=0x7f85a3ee0270, msg=0x7f85a3f054b8) at action.c:1029</span></p>
<p class=""><span class="">#11 0x000000000042a553 in run_actions (h=0x7fff6de80a10, a=0x7f85a3edc990, msg=0x7f85a3f054b8) at action.c:1548</span></p>
<p class=""><span class="">#12 0x000000000041a8c3 in do_action (h=0x7fff6de80a10, a=0x7f85a3ebdd98, msg=0x7f85a3f054b8) at action.c:677</span></p>
<p class=""><span class="">#13 0x000000000042a553 in run_actions (h=0x7fff6de80a10, a=0x7f85a3ebdb08, msg=0x7f85a3f054b8) at action.c:1548</span></p>
<p class=""><span class="">#14 0x000000000042ac80 in run_top_route (a=0x7f85a3ebdb08, msg=0x7f85a3f054b8, c=0x0) at action.c:1634</span></p>
<p class=""><span class="">#15 0x000000000050a9f4 in receive_msg (</span></p>
<p class=""><span class=""> buf=0xa70b00 "REGISTER sip:192.168.2.142 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.2.119:60887;branch=z9hG4bK-524287-1---d670bd2004732b4a;rport\r\nMax-Forwards: 70\r\nContact: <sip:usera@192.168.2.119:60887;rinstance=d9f6274d7"..., len=534, rcv_info=0x7fff6de80d00) at receive.c:196</span></p>
<p class=""><span class="">#16 0x000000000060a4a6 in udp_rcv_loop () at udp_server.c:495</span></p>
<p class=""><span class="">#17 0x00000000004a7fb3 in main_loop () at main.c:1573</span></p>
<p class=""><span class="">#18 0x00000000004ae38e in main (argc=7, argv=0x7fff6de81138) at main.c:2533</span></p>
<p class=""><span class="">(gdb) quit</span></p>
<p class=""><span class="">[root@localhost /</span></p></div><div><br></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 1, 2015 at 12:29 PM, Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
If you recompiled with different souce code or different flags then
the binary is no longer matching the corefile properly.<br>
<br>
Cheers,<br>
Daniel<div><div class="h5"><br>
<br>
<div>On 01/10/15 17:00, Jijo wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Why the backtrace doesn't show the timer function pointer?
Am i missing something?</div>
<div><br>
</div>
<div>Im building the mongo-c-driver as below</div>
<div>get the tar file from <a href="https://github.com/mongodb/mongo-c-driver/releases/download/1.1.11/mongo-c-driver-1.1.11.tar.gz" target="_blank">https://github.com/mongodb/mongo-c-driver/releases/download/1.1.11/mongo-c-driver-1.1.11.tar.gz</a></div>
<div>run the rpm build using the spec file attached. </div>
<div>To disable the ssl, i have modified '--enable-ssl=no' in
the configure command in the spec file</div>
<div>%configure --disable-static --disable-silent-rules
--enable-debug-symbols --enable-man-pages --enable-ssl=yes
--enable-sasl --with-libbson=bundled --enable-optimizations<br>
</div>
<div><br>
</div>
<div>In kamailio im using <span style="font-size:12.8px">db_mongodb.
I'm getting same error for ndb_mongodb as well.</span></div>
<div><span style="font-size:12.8px"><br>
</span></div>
<div><span style="font-size:12.8px">Please find the kamailio
config scripts.</span></div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Oct 1, 2015 at 10:43 AM,
Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank"></a><a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Hello,<br>
<br>
the bracktrace is not useful.<br>
<br>
How did you compile the mongo-c library before and were
there any special parameters you set to the modules in
kamailio config? Are you using db_mongodb or ndb_mongodb?<br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
<br>
<div>On 01/10/15 16:25, Jijo wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Daniel,
<div><br>
</div>
<div>Thanks.. As a sidenote, If i disable ssl from
mongo-c-driver library then i don't have any
crash..</div>
<div><br>
</div>
<div>Something changed in my environment. Im not
able to get the full backtrace. This is what i'm
getting now.</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>aded symbols for /lib64/liblber-2.4.so.2</div>
<div>Reading symbols from
/lib64/libnss_dns-2.12.so...Reading symbols
from
/usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done.</div>
<div>done.</div>
<div>Loaded symbols for /lib64/<a href="http://libnss_dns-2.12.so" target="_blank">libnss_dns-2.12.so</a></div>
<div>Core was generated by
`/usr/local/kamailio_proxy/sbin/kamailio -f
/usr/local/kamailio_proxy/etc/kamail'.</div>
<div>Program terminated with signal 11,
Segmentation fault.</div>
<div>#0 0x00007f69ffffffff in ?? ()</div>
<div>Missing separate debuginfos, use:
debuginfo-install
cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64
cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64
db4-4.7.25-19.el6_6.x86_64
keyutils-libs-1.4-5.el6.x86_64
krb5-libs-1.10.3-42.el6.x86_64
libcom_err-1.41.12-22.el6.x86_64
libcurl-7.19.7-46.el6.x86_64
libidn-1.18-2.el6.x86_64
libselinux-2.0.94-5.8.el6.x86_64
libssh2-1.4.2-1.el6_6.1.x86_64
libunistring-0.9.3-5.el6.x86_64
libxml2-2.7.6-20.el6.x86_64
mysql-libs-5.1.73-5.el6_6.x86_64
nspr-4.10.8-1.el6_6.x86_64
nss-3.18.0-5.3.el6_6.x86_64
nss-util-3.18.0-1.el6_6.x86_64
openldap-2.4.40-5.el6.x86_64
openssl-1.0.1e-42.el6.x86_64
zlib-1.2.3-29.el6.x86_64</div>
<div>(gdb) bt</div>
<div>#0 0x00007f69ffffffff in ?? ()</div>
<div>#1 0x00000000005fef8e in
compat_old_handler (ti=69660042,
tl=0x7f693e736d50, data=0x7f693e736d50) at
timer.c:996</div>
<div>#2 0x00000000005ff94c in slow_timer_main
() at timer.c:1130</div>
<div>#3 0x00000000004a8676 in main_loop () at
main.c:1628</div>
<div>#4 0x00000000004ae38e in main (argc=15,
argv=0x7ffdd0dc8848) at main.c:2533</div>
<div>(gdb) </div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Oct 1, 2015 at
2:37 AM, Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> The
first backtrace is related to pike. Can you
get from gdb of fisrt trace:<br>
<br>
frame 2<br>
p root->entries[b]<br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
<div>On 30/09/15 16:19, Jijo wrote:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div>
<div dir="ltr">I'm observing a
segmentation fault when mongodb
& tls enabled. It doesn't happen
when one of the module is disabled.
<div>
<div><span style="font-size:12.8px">OS:
centos 6.7 kamailio 4.3.2</span><span style="font-size:12.8px"> </span>mongo-c-driver
version 1.1.10.</div>
<div><br>
</div>
<div>It happens immediately after
i register a subscriber. The
location table is updated with
the new data and in parallel
core is generated as well.</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div><br>
</div>
<div>Loaded symbols for /lib64/<a href="http://libnss_dns-2.12.so" target="_blank">libnss_dns-2.12.so</a></div>
<div>Core was generated by
`/usr/local/kamailio_proxy/sbin/kamailio
-f
/usr/local/kamailio_proxy/etc/kamail'.</div>
<div>Program terminated with
signal 11, Segmentation fault.</div>
<div>#0 0x00007fd3adf65e69 in
atomic_cmpxchg_int
(var=0x56444e3055445030,
old=0, new_v=1) at
../../atomic/atomic_x86.h:233</div>
<div>233<span style="white-space:pre-wrap"> </span>ATOMIC_FUNC_CMPXCHG(cmpxchg,
"cmpxchgl %2, %1", int , int)</div>
<div>Missing separate
debuginfos, use:
debuginfo-install
cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64
cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64
db4-4.7.25-19.el6_6.x86_64
keyutils-libs-1.4-5.el6.x86_64
krb5-libs-1.10.3-42.el6.x86_64
libcom_err-1.41.12-22.el6.x86_64
libcurl-7.19.7-46.el6.x86_64
libidn-1.18-2.el6.x86_64
libselinux-2.0.94-5.8.el6.x86_64
libssh2-1.4.2-1.el6_6.1.x86_64
libunistring-0.9.3-5.el6.x86_64
libxml2-2.7.6-20.el6.x86_64
mysql-libs-5.1.73-5.el6_6.x86_64
nspr-4.10.8-1.el6_6.x86_64
nss-3.18.0-5.3.el6_6.x86_64
nss-util-3.18.0-1.el6_6.x86_64
openldap-2.4.40-5.el6.x86_64
openssl-1.0.1e-42.el6.x86_64
zlib-1.2.3-29.el6.x86_64</div>
<div>(gdb) bt</div>
<div>#0 0x00007fd3adf65e69 in
atomic_cmpxchg_int
(var=0x56444e3055445030,
old=0, new_v=1) at
../../atomic/atomic_x86.h:233</div>
<div>#1 0x00007fd3adf65eb6 in
futex_get
(lock=0x56444e3055445030) at
../../futexlock.h:99</div>
<div>#2 0x00007fd3adf66468 in
prv_lock_tree_branch (b=192
'\300') at ip_tree.c:47</div>
<div>#3 0x00007fd3adf664e5 in
lock_tree_branch (b=192
'\300') at ip_tree.c:66</div>
<div>#4 0x00007fd3adf6dda6 in
swap_routine (ticks=123206032,
param=0x0) at pike_funcs.c:293</div>
<div>#5 0x00000000005fef8e in
compat_old_handler
(ti=1971296512,
tl=0x7fd32ccea430,
data=0x7fd32ccea430) at
timer.c:996</div>
<div>#6 0x00000000005ff94c in
slow_timer_main () at
timer.c:1130</div>
<div>#7 0x00000000004a8676 in
main_loop () at main.c:1628</div>
<div>#8 0x00000000004ae38e in
main (argc=11,
argv=0x7fff1d97ec28) at
main.c:2533</div>
<div>(gdb) </div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>On Clean up after the core,
mongo db does segmentation fault
as well</div>
<div><br>
</div>
<div>
<div>Loaded symbols for /lib64/<a href="http://libnss_files-2.12.so" target="_blank">libnss_files-2.12.so</a></div>
<div>Reading symbols from
/lib64/libnss_dns-2.12.so...Reading
symbols from
/usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done.</div>
<div>done.</div>
<div>Loaded symbols for /lib64/<a href="http://libnss_dns-2.12.so" target="_blank">libnss_dns-2.12.so</a></div>
<div>Core was generated by
`/usr/local/kamailio_proxy/sbin/kamailio
-f
/usr/local/kamailio_proxy/etc/kamail'.</div>
<div>Program terminated with
signal 11, Segmentation fault.</div>
<div>#0 __pthread_mutex_destroy
(mutex=0x7fd32cce7be8) at
pthread_mutex_destroy.c:28</div>
<div>28<span style="white-space:pre-wrap"> </span>
if ((mutex->__data.__kind
&
PTHREAD_MUTEX_ROBUST_NORMAL_NP)
== 0</div>
<div>Missing separate
debuginfos, use:
debuginfo-install
cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64
cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64
db4-4.7.25-19.el6_6.x86_64
keyutils-libs-1.4-5.el6.x86_64
krb5-libs-1.10.3-42.el6.x86_64
libcom_err-1.41.12-22.el6.x86_64
libcurl-7.19.7-46.el6.x86_64
libidn-1.18-2.el6.x86_64
libselinux-2.0.94-5.8.el6.x86_64
libssh2-1.4.2-1.el6_6.1.x86_64
libunistring-0.9.3-5.el6.x86_64
libxml2-2.7.6-20.el6.x86_64
mysql-libs-5.1.73-5.el6_6.x86_64
nspr-4.10.8-1.el6_6.x86_64
nss-3.18.0-5.3.el6_6.x86_64
nss-util-3.18.0-1.el6_6.x86_64
openldap-2.4.40-5.el6.x86_64
openssl-1.0.1e-42.el6.x86_64
zlib-1.2.3-29.el6.x86_64</div>
<div>(gdb) bt</div>
<div>#0 __pthread_mutex_destroy
(mutex=0x7fd32cce7be8) at
pthread_mutex_destroy.c:28</div>
<div>#1 0x00007fd32c86f9e6 in
_mongoc_ssl_thread_cleanup ()
at src/mongoc/mongoc-ssl.c:555</div>
<div>#2 _mongoc_ssl_cleanup ()
at src/mongoc/mongoc-ssl.c:106</div>
<div>#3 0x00007fd32c8606a9 in
_mongoc_do_cleanup () at
src/mongoc/mongoc-init.c:127</div>
<div>#4 0x000000344480cdb3 in
pthread_once () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:104</div>
<div>#5 0x00007fd32c8523df in
__do_global_dtors_aux () from
/usr/lib64/libmongoc-1.0.so.0.0.0</div>
<div>#6 0x0000000000000000 in
?? ()</div>
<div>(gdb) </div>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>mongo-c-driver version 1.1.10</div>
<div><br>
</div>
<div>
<div><span style="font-size:12.8px">version:
kamailio 4.3.2
(x86_64/linux) b5980b-dirty</span></div>
<div><span style="font-size:12.8px">flags:
STATS: Off, USE_TCP,
USE_TLS, USE_SCTP,
TLS_HOOKS, USE_RAW_SOCKS,
DISABLE_NAGLE, USE_MCAST,
DNS_IP_HACK, SHM_MEM,
SHM_MMAP, PKG_MALLOC,
F_MALLOC, DBG_F_MALLOC,
USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT,
USE_DNS_CACHE,
USE_DNS_FAILOVER, USE_NAPTR,
USE_DST_BLACKLIST,
HAVE_RESOLV_RES</span></div>
<div><span style="font-size:12.8px">ADAPTIVE_WAIT_LOOPS=1024,
MAX_RECV_BUFFER_SIZE 262144,
MAX_LISTEN 16, MAX_URI_SIZE
1024, BUF_SIZE 65535,
DEFAULT PKG_SIZE 8MB</span></div>
<div><span style="font-size:12.8px">poll
method support: poll,
epoll_lt, epoll_et,
sigio_rt, select.</span></div>
<div><span style="font-size:12.8px">id:
b5980b -dirty</span></div>
<div><span style="font-size:12.8px">compiled
on 13:35:24 Sep 28 2015 with
gcc 4.</span></div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
</div>
</div>
<pre>_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
<a href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a>
<a href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><span><font color="#888888">
</font></span></pre>
<span><font color="#888888"> </font></span></blockquote>
<span><font color="#888888"> <br>
<pre cols="72">--
Daniel-Constantin Mierla
<a href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Book: SIP Routing With Kamailio - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - <a href="http://asipto.com/u/kat" target="_blank">http://asipto.com/u/kat</a></pre>
</font></span></div>
<br>
_______________________________________________<br>
SIP Express Router (SER) and Kamailio
(OpenSER) - sr-users mailing list<br>
<a href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a><br>
<a href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla
<a href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Book: SIP Routing With Kamailio - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - <a href="http://asipto.com/u/kat" target="_blank">http://asipto.com/u/kat</a></pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla
<a href="http://twitter.com/#!/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Book: SIP Routing With Kamailio - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - <a href="http://asipto.com/u/kat" target="_blank">http://asipto.com/u/kat</a></pre>
</div></div></div>
</blockquote></div><br></div>
</div></blockquote><blockquote type="cite"><div><kamailio-local.cfg></div></blockquote><blockquote type="cite"><div><kamailio.cfg></div></blockquote></body></html>