<html><body><div style="color:#000; background-color:#fff; font-family:arial, helvetica, sans-serif;font-size:10pt"><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt;"><span>Hi Daniel </span></div><div style="font-family: arial, helvetica, sans-serif; font-size: 13px; color: rgb(0, 0, 0); background-color: transparent; font-style: normal;"><span><br></span></div><div style="font-family: arial, helvetica, sans-serif; font-size: 13px; color: rgb(0, 0, 0); background-color: transparent; font-style: normal;"><span>Just want to make you aware of something on kamailio 4.0.3 .</span></div><div style="font-family: arial, helvetica, sans-serif; font-size: 13px; color: rgb(0, 0, 0); background-color: transparent; font-style: normal;"><span>I loaded cfgutils and used pkg_summary() . But I run into the following:</span></div><div style="font-family: arial, helvetica, sans-serif; font-size: 13px; color: rgb(0, 0, 0); background-color:
transparent; font-style: normal;"><span><br></span></div><div style="background-color: transparent;"><font size="2">0(14112) NOTICE: <core> [mem/q_malloc.c:437]: qm_free(): qm_free(0x7fbad3882010, 0x7fbad3ab7220), called from <core>: rpc_lookup.c: destroy_rpcs(79)</font></div><div style="background-color: transparent;"><font size="2"> 0(14112) NOTICE: <core> [mem/q_malloc.c:472]: qm_free(): qm_free: freeing frag. 0x7fbad3ab71f0 alloc'ed from <core>: rpc_lookup.c: rpc_hash_add(156)</font></div><div style="background-color: transparent;"><font size="2"> 0(14112) NOTICE: <core> [mem/q_malloc.c:437]: qm_free(): qm_free(0x7fba8923e000, 0x7fba894e62b0), called from cfgutils: cfgutils.c: mod_destroy(806)</font></div><div style="background-color: transparent;"><font size="2"><span></span></font></div><div style="background-color: transparent;"><font size="2"> 0(14112) : <core> [mem/q_malloc.c:466]: qm_free():
BUG: qm_free: freeing already freed pointer (0x7fba894e62b0), called from cfgutils: cfgutils.c: mod_destroy(806), first free cfgutils: cfgutils.c: mod_destroy(806) - aborting</font></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt;"><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt;">Regards,</div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt;">Dragos</div><div style="font-family: arial, helvetica, sans-serif; font-size: 13px; color: rgb(0, 0, 0); background-color: transparent; font-style: normal;"><br></div> <div style="font-family: arial, helvetica, sans-serif; font-size: 10pt;"> <div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;"> <div dir="ltr"> <hr size="1"> <font size="2" face="Arial"> <b><span style="font-weight:bold;">From:</span></b> David Cunningham <dcunningham@voisonics.com><br> <b><span style="font-weight:
bold;">To:</span></b> Daniel-Constantin Mierla <miconda@gmail.com> <br><b><span style="font-weight: bold;">Cc:</span></b> Kamailio (SER) - Users Mailing List <sr-users@lists.sip-router.org> <br> <b><span style="font-weight: bold;">Sent:</span></b> Wednesday, October 2, 2013 9:26 AM<br> <b><span style="font-weight: bold;">Subject:</span></b> Re: [SR-Users] Memory leak in 3.3.4<br> </font> </div> <div class="y_msg_container"><br><div id="yiv6682514051"><div><div dir="ltr"><div><div>Hi Daniel,<br clear="none"><br clear="none"></div>Thank you - I used the restart method and got a whole load of memory information. 10 megabytes of it! I've compressed that and attached it to this email.<br clear="none"><br clear="none"></div>Thanks for your help.<br clear="none">
<br clear="none"></div><div class="yiv6682514051gmail_extra"><br clear="none"><br clear="none"><div class="yiv6682514051yqt4615910786" id="yiv6682514051yqt92476"><div class="yiv6682514051gmail_quote">On 2 October 2013 17:17, Daniel-Constantin Mierla <span dir="ltr"><<a rel="nofollow" shape="rect" ymailto="mailto:miconda@gmail.com" target="_blank" href="mailto:miconda@gmail.com">miconda@gmail.com</a>></span> wrote:<br clear="none">
<blockquote class="yiv6682514051gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div>
Hello,<br clear="none">
<br clear="none">
ahh, right, forgot that it was disabled because it was not safe to
use at runtime for worker processes. You have to use:<br clear="none">
<br clear="none">
kamcmd cfg.set_now_int core mem_dump_pkg <pid><br clear="none">
<br clear="none">
then wait for a bit so that process is receiving a SIP message.<br clear="none">
<br clear="none">
Or, if you restart, the stats are printed in syslog for all
processes.<br clear="none">
<br clear="none">
Another alternative is to call pkg_status() or pkg_summary() from
cfgutils module in config file, like:<br clear="none">
<br clear="none">
if(src_ip==127.0.0.1 && is_method("OPTIONS") &&
$rU=="pkgstats") {<br clear="none">
send_reply("200", "OK");<br clear="none">
pkg_summary();<br clear="none">
exit;<br clear="none">
}<br clear="none">
<br clear="none">
You can use sipsak to send an options request to <a rel="nofollow" shape="rect" href="">"sip:pkgstats@..."</a><br clear="none">
<br clear="none">
Cheers,<br clear="none">
Daniel<div><div class="yiv6682514051h5"><br clear="none">
<br clear="none">
<div>On 10/2/13 2:30 AM, David Cunningham
wrote:<br clear="none">
</div>
<blockquote type="cite">
<div dir="ltr">Hi Daniel,<br clear="none">
<br clear="none">
When I send SIGUSR1 to a worker process nothing is logged. Any
ideas there? The following is the output of "sercmd pkg.stats".<br clear="none">
<br clear="none">
{<br clear="none">
entry: 0<br clear="none">
pid: 14570<br clear="none">
rank: 0<br clear="none">
used: 191100<br clear="none">
free: 3731504<br clear="none">
real_used: 462800<br clear="none">
}<br clear="none">
{<br clear="none">
entry: 1<br clear="none">
pid: 14571<br clear="none">
rank: 1<br clear="none">
used: 202656<br clear="none">
free: 3714236<br clear="none">
real_used: 480068<br clear="none">
}<br clear="none">
{<br clear="none">
entry: 2<br clear="none">
pid: 14572<br clear="none">
rank: 2<br clear="none">
used: 202720<br clear="none">
free: 3714220<br clear="none">
real_used: 480084<br clear="none">
}<br clear="none">
{<br clear="none">
entry: 3<br clear="none">
pid: 14573<br clear="none">
rank: 3<br clear="none">
used: 202664<br clear="none">
free: 3714324<br clear="none">
real_used: 479980<br clear="none">
}<br clear="none">
{<br clear="none">
entry: 4<br clear="none">
pid: 14574<br clear="none">
rank: 4<br clear="none">
used: 202744<br clear="none">
free: 3714100<br clear="none">
real_used: 480204<br clear="none">
}<br clear="none">
{<br clear="none">
entry: 5<br clear="none">
pid: 14575<br clear="none">
rank: 5<br clear="none">
used: 201908<br clear="none">
free: 3715176<br clear="none">
real_used: 479128<br clear="none">
}<br clear="none">
{<br clear="none">
entry: 6<br clear="none">
pid: 14576<br clear="none">
rank: 6<br clear="none">
used: 202572<br clear="none">
free: 3714368<br clear="none">
real_used: 479936<br clear="none">
}<br clear="none">
{<br clear="none">
entry: 7<br clear="none">
pid: 14577<br clear="none">
rank: 7<br clear="none">
used: 202740<br clear="none">
free: 3714488<br clear="none">
real_used: 479816<br clear="none">
}<br clear="none">
{<br clear="none">
entry: 8<br clear="none">
pid: 14578<br clear="none">
rank: 8<br clear="none">
used: 199760<br clear="none">
free: 3717084<br clear="none">
real_used: 477220<br clear="none">
}<br clear="none">
{<br clear="none">
entry: 9<br clear="none">
pid: 14579<br clear="none">
rank: 9<br clear="none">
used: 199440<br clear="none">
free: 3722828<br clear="none">
real_used: 471476<br clear="none">
}<br clear="none">
{<br clear="none">
entry: 10<br clear="none">
pid: 14580<br clear="none">
rank: 10<br clear="none">
used: 199440<br clear="none">
free: 3722828<br clear="none">
real_used: 471476<br clear="none">
}<br clear="none">
{<br clear="none">
entry: 11<br clear="none">
pid: 14581<br clear="none">
rank: 11<br clear="none">
used: 199440<br clear="none">
free: 3722828<br clear="none">
real_used: 471476<br clear="none">
}<br clear="none">
{<br clear="none">
entry: 12<br clear="none">
pid: 14582<br clear="none">
rank: 12<br clear="none">
used: 199440<br clear="none">
free: 3722828<br clear="none">
real_used: 471476<br clear="none">
}<br clear="none">
{<br clear="none">
entry: 13<br clear="none">
pid: 14583<br clear="none">
rank: 13<br clear="none">
used: 199440<br clear="none">
free: 3722828<br clear="none">
real_used: 471476<br clear="none">
}<br clear="none">
{<br clear="none">
entry: 14<br clear="none">
pid: 14584<br clear="none">
rank: 14<br clear="none">
used: 199440<br clear="none">
free: 3722828<br clear="none">
real_used: 471476<br clear="none">
}<br clear="none">
{<br clear="none">
entry: 15<br clear="none">
pid: 14585<br clear="none">
rank: 15<br clear="none">
used: 199440<br clear="none">
free: 3722828<br clear="none">
real_used: 471476<br clear="none">
}<br clear="none">
{<br clear="none">
entry: 16<br clear="none">
pid: 14586<br clear="none">
rank: 16<br clear="none">
used: 199440<br clear="none">
free: 3722828<br clear="none">
real_used: 471476<br clear="none">
}<br clear="none">
{<br clear="none">
entry: 17<br clear="none">
pid: 14587<br clear="none">
rank: -1<br clear="none">
used: 199400<br clear="none">
free: 3722868<br clear="none">
real_used: 471436<br clear="none">
}<br clear="none">
{<br clear="none">
entry: 18<br clear="none">
pid: 14588<br clear="none">
rank: -1<br clear="none">
used: 2105844<br clear="none">
free: 3722868<br clear="none">
real_used: 2337944<br clear="none">
}<br clear="none">
{<br clear="none">
entry: 19<br clear="none">
pid: 14589<br clear="none">
rank: -2<br clear="none">
used: 198552<br clear="none">
free: 3723236<br clear="none">
real_used: 471068<br clear="none">
}<br clear="none">
{<br clear="none">
entry: 20<br clear="none">
pid: 0<br clear="none">
rank: 0<br clear="none">
used: 209756<br clear="none">
free: 3712368<br clear="none">
real_used: 481936<br clear="none">
}<br clear="none">
{<br clear="none">
entry: 21<br clear="none">
pid: 14591<br clear="none">
rank: -4<br clear="none">
used: 253660<br clear="none">
free: 3668704<br clear="none">
real_used: 525600<br clear="none">
}<br clear="none">
<br clear="none">
<br clear="none">
</div>
<div class="yiv6682514051gmail_extra"><br clear="none">
<br clear="none">
<div class="yiv6682514051gmail_quote">On 1 October 2013 22:45,
Daniel-Constantin Mierla <span dir="ltr"><<a rel="nofollow" shape="rect" ymailto="mailto:miconda@gmail.com" target="_blank" href="mailto:miconda@gmail.com">miconda@gmail.com</a>></span> wrote:<br clear="none">
<blockquote class="yiv6682514051gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div> Hello,<br clear="none">
<br clear="none">
send SIGUSR1 to one of the worker processes (e.g., UDP
listener). The main process of kamailio doesn't process
SIP messages, so it is not exposed to leaks.<br clear="none">
<br clear="none">
You can used 'kamcmd pkg.stats" to see statistics about
usage of pkg per each process (older versions have
sercmd). <br clear="none">
<br clear="none">
'kamctl ps' gives the list of processes.<br clear="none">
<br clear="none">
Cheers,<br clear="none">
Daniel
<div>
<div><br clear="none">
<br clear="none">
<div>On 10/1/13 2:35 PM, David Cunningham wrote:<br clear="none">
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>Hi Daniel,<br clear="none">
<br clear="none">
</div>
What I copied and pasted is all that's listed
when I do a kill -SIGUSR1 on the Kamailio master
process. How do I get fuller logs?<br clear="none">
<br clear="none">
</div>
Thanks for your help.<br clear="none">
<br clear="none">
</div>
<div class="yiv6682514051gmail_extra"><br clear="none">
<br clear="none">
<div class="yiv6682514051gmail_quote">On 1 October 2013 21:57,
Daniel-Constantin Mierla <span dir="ltr"><<a rel="nofollow" shape="rect" ymailto="mailto:miconda@gmail.com" target="_blank" href="mailto:miconda@gmail.com">miconda@gmail.com</a>></span>
wrote:<br clear="none">
<blockquote class="yiv6682514051gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Hello,<br clear="none">
<br clear="none">
you gave only the first part of memory log
output. The chunks listed there are from
startup, thus not part of a leak, because they
are not allocated again at runtime.<br clear="none">
<br clear="none">
Can you give the full logs with messages from
qm_status? The leaks are typically listed
towards the end of those messages.<br clear="none">
<br clear="none">
Yesterday someone else reported a potential
leak when using perl module, so it might be
something else there. I found one, which is
related to an error case, but perhaps you
don't get that, otherwise you will notice the
error message.<br clear="none">
<br clear="none">
Cheers,<br clear="none">
Daniel
<div>
<div><br clear="none">
<br clear="none">
On 10/1/13 12:03 PM, David Cunningham
wrote:<br clear="none">
<blockquote class="yiv6682514051gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"> Hello,<br clear="none">
<br clear="none">
Following up on this after a long break.
It seems we are still having a memory
leak. We've added the memory debugging
options as per the documentation with
the following output.<br clear="none">
I'm not sure how to read it though - can
anyone help? Thank you.<br clear="none">
<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
<core> [main.c:754]: Memory status
(pkg):<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: (0x4019b008):<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: heap size= 4194304<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: used= 191100,
used+overhead=462800, free=3731504<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: max used (+overhead)= 480224<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: dumping all alloc'ed.
fragments:<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 0. N address=0x401cf414
frag=0x401cf3fc size=512 used=1<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: str_hash.h:
str_hash_alloc(69)<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 1. N address=0x401cf644
frag=0x401cf62c size=128 used=1<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: str_hash.h:
str_hash_alloc(69)<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 2. N address=0x401cf6f4
frag=0x401cf6dc size=256 used=1<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: counters.c:
cnt_hash_add(331)<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 3. N address=0x401cf824
frag=0x401cf80c size=64 used=1<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: counters.c:
init_counters(121)<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 4. N address=0x401cf894
frag=0x401cf87c size=8 used=1<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: route.c: init_rlist(180)<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 5. N address=0x401cf8cc
frag=0x401cf8b4 size=64 used=1<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: str_hash.h:
str_hash_alloc(69)<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 6. N address=0x401cf93c
frag=0x401cf924 size=24 used=1<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: route.c: route_add(159)<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 7. N address=0x401cf984
frag=0x401cf96c size=8 used=1<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: route.c: init_rlist(180)<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 8. N address=0x401cf9bc
frag=0x401cf9a4 size=64 used=1<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: str_hash.h:
str_hash_alloc(69)<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 9. N address=0x401cfa2c
frag=0x401cfa14 size=24 used=1<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: route.c: route_add(159)<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 10. N address=0x401cfa74
frag=0x401cfa5c size=8 used=1<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: db_id.c: dupl_string(50)<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 11. N address=0x401cfaac
frag=0x401cfa94 size=64 used=1<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: str_hash.h:
str_hash_alloc(69)<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 12. N address=0x401cfb1c
frag=0x401cfb04 size=24 used=1<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: route.c: route_add(159)<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 13. N address=0x401cfb64
frag=0x401cfb4c size=8 used=1<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: route.c: init_rlist(180)<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed<br clear="none">
Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 14. N address=0x401cfb9c
frag=0x401cfb84 size=64 used=1<br clear="none">
<br clear="none">
<br clear="none">
</blockquote>
<br clear="none">
</div>
</div>
<div> -- <br clear="none">
Daniel-Constantin Mierla - <a rel="nofollow" shape="rect" target="_blank" href="http://www.asipto.com/">http://www.asipto.com</a><br clear="none">
<a rel="nofollow" shape="rect" target="_blank" href="http://twitter.com/#%21/miconda">http://twitter.com/#!/miconda</a>
- <a rel="nofollow" shape="rect" target="_blank" href="http://www.linkedin.com/in/miconda">http://www.linkedin.com/in/miconda</a><br clear="none">
</div>
Kamailio Advanced Trainings - Berlin, Nov
25-28; Miami, Nov 18-20, 2013<br clear="none">
- more details about Kamailio trainings at <a rel="nofollow" shape="rect" target="_blank" href="http://www.asipto.com/">http://www.asipto.com</a>
-<br clear="none">
<br clear="none">
</blockquote>
</div>
<br clear="none">
<br clear="all">
<br clear="none">
-- <br clear="none">
David Cunningham, Voisonics<br clear="none">
<a rel="nofollow" shape="rect" target="_blank" href="http://voisonics.com/">http://voisonics.com/</a><br clear="none">
USA: <a rel="nofollow" shape="rect" href="">+1 213 221
1092</a><br clear="none">
UK: <a rel="nofollow" shape="rect" href="">+44 (0) 20
3298 1642</a><br clear="none">
Australia: <a rel="nofollow" shape="rect" href="">+61 (0) 2
8063 9019</a><br clear="none">
</div>
</blockquote>
<br clear="none">
<pre>--
Daniel-Constantin Mierla - <a rel="nofollow" shape="rect" target="_blank" href="http://www.asipto.com/">http://www.asipto.com</a>
<a rel="nofollow" shape="rect" target="_blank" href="http://twitter.com/#%21/miconda">http://twitter.com/#!/miconda</a> - <a rel="nofollow" shape="rect" target="_blank" href="http://www.linkedin.com/in/miconda">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at <a rel="nofollow" shape="rect" target="_blank" href="http://www.asipto.com/">http://www.asipto.com</a> -
</pre>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="none">
<br clear="all">
<br clear="none">
-- <br clear="none">
David Cunningham, Voisonics<br clear="none">
<a rel="nofollow" shape="rect" target="_blank" href="http://voisonics.com/">http://voisonics.com/</a><br clear="none">
USA: <a rel="nofollow" shape="rect" href="">+1 213 221 1092</a><br clear="none">
UK: <a rel="nofollow" shape="rect" href="">+44 (0) 20 3298 1642</a><br clear="none">
Australia: <a rel="nofollow" shape="rect" href="">+61 (0) 2 8063 9019</a><br clear="none">
</div>
</blockquote>
<br clear="none">
<pre>--
Daniel-Constantin Mierla - <a rel="nofollow" shape="rect" target="_blank" href="http://www.asipto.com/">http://www.asipto.com</a>
<a rel="nofollow" shape="rect" target="_blank" href="http://twitter.com/#!/miconda">http://twitter.com/#!/miconda</a> - <a rel="nofollow" shape="rect" target="_blank" href="http://www.linkedin.com/in/miconda">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at <a rel="nofollow" shape="rect" target="_blank" href="http://www.asipto.com/">http://www.asipto.com</a> -
</pre>
</div></div></div>
</blockquote></div></div><br clear="none"><br clear="all"><br clear="none">-- <br clear="none">David Cunningham, Voisonics<br clear="none"><a rel="nofollow" shape="rect" target="_blank" href="http://voisonics.com/">http://voisonics.com/</a><br clear="none">USA: +1 213 221 1092<br clear="none">UK: +44 (0) 20 3298 1642<br clear="none">Australia: +61 (0) 2 8063 9019<br clear="none">
</div></div></div><br><div class="yqt4615910786" id="yqt78022">_______________________________________________<br clear="none">SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list<br clear="none"><a shape="rect" ymailto="mailto:sr-users@lists.sip-router.org" href="mailto:sr-users@lists.sip-router.org">sr-users@lists.sip-router.org</a><br clear="none"><a shape="rect" href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><br clear="none"></div><br><br></div> </div> </div> </div></body></html>