<div dir="ltr">Hello Daniel,<div><br></div><div>Indeed xlog doesn't print in syslog the sip message received and wireshak got it, that means that packet was received in network interface but kamailio didn't received the packet(as xlog didn't print that packet).<br></div><div><br></div><div>Anyway i have configured the following parameters in sysctl</div><div><br></div><div><div style="margin:0px;padding:0px;border:0px"><span style="margin:0px;padding:0px;border:0px;font-family:Arial,sans-serif">net.ipv4.udp_rmem_min = 131072</span><br></div><div style="margin:0px;padding:0px;border:0px"><div style="margin:0px;padding:0px;border:0px;font-family:Arial,sans-serif;font-size:medium"><font style="margin:0px;padding:0px;border:0px">net.ipv4.udp_wmem_min = 131072</font></div><div style="margin:0px;padding:0px;border:0px;font-family:Arial,sans-serif;font-size:medium"><font style="margin:0px;padding:0px;border:0px">net.core.netdev_max_backlog=1000</font></div><div style="margin:0px;padding:0px;border:0px;font-family:Arial,sans-serif;font-size:medium"><font style="margin:0px;padding:0px;border:0px">net.core.rmem_max=67108864</font></div><div style="margin:0px;padding:0px;border:0px;font-family:Arial,sans-serif;font-size:medium"><font style="margin:0px;padding:0px;border:0px">net.ipv4.udp_mem = 192576 256768 385152</font></div><div><font style="margin:0px;padding:0px;border:0px"><br></font></div><div>and also set maxbuffer=134190336 in my kamailio configuration.</div><div><br></div><div>With these configurations I got an improvement but still below of what we need and even the CPU is between 65% and 70% sleeping.<br></div><div><br></div><div>Do you have any suggestion? the traffic average that interface receives now is 200mbs.</div><div><br></div><div>The server network interface is 10GB.</div><div><br></div><div>Any suggestion is welcome.</div><div><br></div><div>Best Regards</div><div>José Seabra</div><div><br></div></div></div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">2015-09-27 17:08 GMT+01:00 Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Hello,<br>
<br>
first, be sure you run latest version of branch 4.3, there was a fix
for dialog since 4.3.2.<br>
<br>
If a process in shown to run recvfrom() that means it waits for
kernel to pass a new upd packet to it -- recvfrom() is a function
from standard lib c.<br>
<br>
Wireshark (like other sniffing tools) is capturing the network
traffic before the kernel passes it to application layer. If
wireshark see it, doesn't mean the application has received it from
kernel, it can still be dropped by firewall or other kernel modules.<br>
<br>
You can add a xlog() at the beginning of request_route to see what
requests are received by kamailio.<br>
<br>
Cheers,<br>
Daniel<div><div><br>
<br>
<div>On 27/09/15 15:52, José Seabra wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hello Danel,
<div>Thank you for your response,</div>
<div><br>
</div>
<div>Well i don't have SELinux enabled and IPTables isn't
running.</div>
<div><br>
</div>
<div>About kamailio configuration, Pike module isn't also
enabled, the only thing that i have enabled to limit the
number of concurrent calls is the dialog module, counting the
number of active dialogs and reject next new invites only
after it has reached the 10.000 simultaneous calls.</div>
<div><br>
</div>
<div>The strange behavior is that kamailio only stop processing
some sip messages like Byes and 200 Ok after more or less 40
seconds after siptester start sending calls.<br>
</div>
<div><br>
</div>
<div>And i can see several cases in wireshark that kamailio
receives BYES/200OK from interface B(Carriers interface) but
it isn't forward to Interface A(customer interface), if i make
for example 20cps i don't have this issue, all packets are
handled correctly and there isn't 408 requests timeouts.</div>
<div><br>
</div>
<div><br>
</div>
<div>Best Regards</div>
<div>José Seabra</div>
<div><br>
</div>
<div> </div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2015-09-27 13:52 GMT+01:00
Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank"></a><a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> The tmp variable is
not intialized by default, so it can be anything therefore
the out of bounds value. It used only when printing some
errors, so it didn't get a proper value as no such error
was printed for you case.<br>
<br>
The backtrace shows the process in in recvfrom(), so it
waits for traffic from network. That is the case when the
processing of former sip message was finished and now
Kamailio waits to receive another one.<br>
<br>
If you run centos, be sure you don't have limits in
selinux rules - better disable it completely and test
again. Also, be sure you don't have limits in the
firewall, especially contrack module and sip firewall
modules in the kernel.<br>
<br>
In kamailio.cfg, if you have pike module, be sure you
don't drop good packet with this module.<br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
<div>On 25/09/15 19:53, José Seabra wrote:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div>
<div dir="ltr">
<div>Adding more information to my last email,
when kamailio starts receiving calls, after more
or less 40 seconds it stops processing some sip
messages.</div>
<div><br>
</div>
I'm using the kamailio version 4.3.2
<div><br>
</div>
<div>Best Regards</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2015-09-25 18:34
GMT+01:00 José Seabra <span dir="ltr"><<a href="mailto:joseseabra4@gmail.com" target="_blank"></a><a href="mailto:joseseabra4@gmail.com" target="_blank">joseseabra4@gmail.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">I have one kamailio server
listening in 2 network interfaces, kamailio
receives sip messages from interface A and
then send it to destine through interface B.
<div>When we start kamailio service then
run gdb for each kamailio network
interface process, execute the command "bt
full" the result is the following:</div>
<div><br>
<div><b>tmp = 0x5f801b994c4715bd
<Address 0x5f801b994c4715bd out of
bounds></b><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>interface - A</div>
<div><br>
</div>
<div> bt full</div>
<div>#0 0x00007fb01bfa40c3 in recvfrom
() from /lib64/libc.so.6</div>
<div>No symbol table info available.</div>
<div>#1 0x000000000047f2ba in
udp_rcv_loop () at udp_server.c:421</div>
<div> len = 13</div>
<div> buf = '\000' <repeats
65535 times></div>
<div> tmp = 0x5f801b994c4715bd
<Address 0x5f801b994c4715bd out of
bounds></div>
<div> from = 0x7faf9bf38bb8</div>
<div> fromlen = 16</div>
<div> ri = {src_ip = {af =
2309178400, len = 32765, u = {addrl =
{5458819, 4284160}, addr32 = {5458819,
0, 4284160, 0}, addr16 = {19331, 83,
0, 0, 24320, 65, 0, 0}, </div>
<div> addr =
"\203KS\000\000\000\000\000\000_A\000\000\000\000"}},
dst_ip = {af = 2, len = 4, u = {addrl
= {80026686, 0}, addr32 = {80026686,
0, 0, 0}, </div>
<div> addr16 = {7230, 1221,
0, 0, 0, 0, 0, 0}, addr =
">\034\305\004", '\000' <repeats
11 times>}}, src_port = 55576,
dst_port = 5060, proto_reserved1 = 0, </div>
<div> proto_reserved2 = 0,
src_su = {s = {sa_family = 32687,
sa_data =
"\000\000\200D\243\211\375\177\000\000\213\337D"},
sin = {sin_family = 32687, sin_port =
0, </div>
<div> sin_addr = {s_addr =
2309178496}, sin_zero =
"\375\177\000\000\213\337D"}, sin6 =
{sin6_family = 32687, sin6_port = 0,
sin6_flowinfo = 2309178496, </div>
<div> sin6_addr = {__in6_u
= {__u6_addr8 =
"\375\177\000\000\213\337D\000\000\000\000\000p",
<incomplete sequence \372\233>,
__u6_addr16 = {32765, 0, 57227, </div>
<div> 68, 0, 0,
21616, 39930}, __u6_addr32 = {32765,
4513675, 0, 2616874096}}},
sin6_scope_id = 32687}}, bind_address
= 0x7faf9becd7d8, proto = 1 '\001'}</div>
<div> __FUNCTION__ =
"udp_rcv_loop"</div>
<div>#2 0x0000000000525eae in main_loop
() at main.c:1573</div>
<div> i = 0</div>
<div> pid = 0</div>
<div> si = 0x7faf9becd7d8</div>
<div> si_desc = "udp receiver
child=0 sock=<a href="http://62.28.197.4:5060" target="_blank">62.28.197.4:5060</a>\000\000\000\000\000\000\001\000\000\000\000\000\000\000\260\244r\023n\000\000\000\071\263q\000\000\000\000\000\020\243q\000\000\000\000\000\b\000\000\000\001\000\000\000\260\373\371\233\257\177\000\000\220E\243\211\375\177\000\000|\351O\000\000\000\000\000\000\266\253\023\257\177\000\000\260\244r\023\257\177\000"</div>
<div>---Type <return> to continue,
or q <return> to quit---</div>
<div> nrprocs = 32</div>
<div> __FUNCTION__ = "main_loop"</div>
<div>#3 0x000000000052c395 in main
(argc=11, argv=0x7ffd89a34848) at
main.c:2533</div>
<div> cfg_stream = 0x2296010</div>
<div> c = -1</div>
<div> r = 0</div>
<div> tmp = 0x7ffd89a35f61 ""</div>
<div> tmp_len = 0</div>
<div> port = 0</div>
<div> proto = 32688</div>
<div> options = 0x725010
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"</div>
<div> ret = -1</div>
<div> seed = 1641321572</div>
<div> rfd = 4</div>
<div> debug_save = 0</div>
<div> debug_flag = 0</div>
<div> dont_fork_cnt = 0</div>
<div> n_lst = 0x7ffd89a34710</div>
<div> p = 0x7fb01bf4c646
<intel_check_word+166>
"H\205\300t\025\017\266P\003\017\266\312D9\371\017\204",
<incomplete sequence \305></div>
<div> __FUNCTION__ = "main"</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>interface B</div>
<div>bt full</div>
<div>#0 0x00007fb01bfa40c3 in recvfrom
() from /lib64/libc.so.6</div>
<div>No symbol table info available.</div>
<div>#1 0x000000000047f2ba in
udp_rcv_loop () at udp_server.c:421</div>
<div> len = 45</div>
<div><br>
</div>
<div> buf = '\000' <repeats
65535 times></div>
<div> tmp = 0x13cb2f24296000e9
<Address 0x13cb2f24296000e9 out of
bounds></div>
<div> from = 0x7faf9bf38bb8</div>
<div> fromlen = 16</div>
<div> ri = {src_ip = {af =
2309178400, len = 32765, u = {addrl =
{5458819, 4284160}, addr32 = {5458819,
0, 4284160, 0}, addr16 = {19331, 83,
0, 0, 24320, 65, 0, 0}, addr =
"\203KS\000\000\000\000\000\000_A\000\000\000\000"}}, </div>
<div> dst_ip = {af = 2, len =
4, u = {addrl = {2764381246, 0},
addr32 = {2764381246, 0, 0, 0}, addr16
= {7230, 42181, 0, 0, 0, 0, 0, 0},
addr = ">\034Ť", '\000' <repeats
11 times>}}, src_port = 56136,
dst_port = 5060, </div>
<div> proto_reserved1 = 0,
proto_reserved2 = 0, src_su = {s =
{sa_family = 32687, sa_data =
"\000\000\200D\243\211\375\177\000\000\213\337D"},
sin = {sin_family = 32687, sin_port =
0, sin_addr = {s_addr = 2309178496}, </div>
<div> sin_zero =
"\375\177\000\000\213\337D"}, sin6 =
{sin6_family = 32687, sin6_port = 0,
sin6_flowinfo = 2309178496, sin6_addr
= {__in6_u = {</div>
<div> __u6_addr8 =
"\375\177\000\000\213\337D\000\000\000\000\000p",
<incomplete sequence \372\233>,
__u6_addr16 = {32765, 0, 57227, 68, 0,
0, 21616, 39930}, __u6_addr32 =
{32765, 4513675, 0, 2616874096}}}, </div>
<div> sin6_scope_id =
32687}}, bind_address =
0x7faf9becda08, proto = 1 '\001'}</div>
<div> __FUNCTION__ =
"udp_rcv_loop"</div>
<div>#2 0x0000000000525eae in main_loop
() at main.c:1573</div>
<div>---Type <return> to continue,
or q <return> to quit---</div>
<div> i = 0</div>
<div> pid = 0</div>
<div> si = 0x7faf9becda08</div>
<div> si_desc = "udp receiver
child=0 sock=<a href="http://62.28.197.164:5060" target="_blank">62.28.197.164:5060</a>\000\000\000\000\001\000\000\000\000\000\000\000\260\244r\023n\000\000\000\071\263q\000\000\000\000\000\020\243q\000\000\000\000\000\b\000\000\000\001\000\000\000\260\373\371\233\257\177\000\000\220E\243\211\375\177\000\000|\351O\000\000\000\000\000\000\266\253\023\257\177\000\000\260\244r\023\257\177\000"</div>
<div> nrprocs = 32</div>
<div> __FUNCTION__ = "main_loop"</div>
<div>#3 0x000000000052c395 in main
(argc=11, argv=0x7ffd89a34848) at
main.c:2533</div>
<div> cfg_stream = 0x2296010</div>
<div> c = -1</div>
<div> r = 0</div>
<div> tmp = 0x7ffd89a35f61 ""</div>
<div> tmp_len = 0</div>
<div> port = 0</div>
<div> proto = 32688</div>
<div> options = 0x725010
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"</div>
<div>---Type <return> to continue,
or q <return> to quit---</div>
<div> ret = -1</div>
<div> seed = 1641321572</div>
<div> rfd = 4</div>
<div> debug_save = 0</div>
<div> debug_flag = 0</div>
<div> dont_fork_cnt = 0</div>
<div> n_lst = 0x7ffd89a34710</div>
<div> p = 0x7fb01bf4c646
<intel_check_word+166>
"H\205\300t\025\017\266P\003\017\266\312D9\371\017\204",
<incomplete sequence \305></div>
<div> __FUNCTION__ = "main"</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>[root@c4-srv1 kamailio]# kamctl
stats shmem</div>
<div>shmem:fragments = 2552</div>
<div>shmem:free_size = 2082764040</div>
<div>shmem:max_used_size = 66449600</div>
<div>shmem:real_used_size = 64719608</div>
<div>shmem:total_size = 2147483648</div>
<div>shmem:used_size = 54355992</div>
</div>
<div><br>
</div>
<div>Can anyone explain the meaning of </div>
<div><b>tmp = 0x5f801b994c4715bd
<Address 0x5f801b994c4715bd out of
bounds></b></div>
<div><br>
</div>
<div>Best Regards</div>
<span><font color="#888888">
<div>José Seabra<br clear="all">
<div><br>
</div>
</div>
</font></span></div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div>Cumprimentos
<div>José Seabra</div>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
</div>
</div>
<pre>_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
<a href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a>
<a href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><span><font color="#888888">
</font></span></pre>
<span><font color="#888888"> </font></span></blockquote>
<span><font color="#888888"> <br>
<pre cols="72">--
Daniel-Constantin Mierla
<a href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Book: SIP Routing With Kamailio - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - <a href="http://asipto.com/u/kat" target="_blank">http://asipto.com/u/kat</a></pre>
</font></span></div>
<br>
_______________________________________________<br>
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
mailing list<br>
<a href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a><br>
<a href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div>Cumprimentos
<div>José Seabra</div>
</div>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla
<a href="http://twitter.com/#!/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Book: SIP Routing With Kamailio - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - <a href="http://asipto.com/u/kat" target="_blank">http://asipto.com/u/kat<br clear="all"><div><br></div>-- <br><div class="gmail_signature">Cumprimentos<div>José Seabra</div></div>
</a><a href="http://store.itcenter.com.pt/" style="margin-left:10px;width:23px;min-height:23px;display:block" target="_blank"><img src="http://www.itcenter.com.pt/email/assinatura-email-itcenter_store.jpg" alt="ITCENTER Store" title="ITCENTER Store"></a><a href="http://support.itcenter.com.pt/otrs/customer.pl" style="margin-left:10px;width:23px;min-height:23px;display:block" target="_blank"><img src="http://www.itcenter.com.pt/email/assinatura-email-itcenter_helpdesk.jpg" alt="ITCENTER Helpdesk" title="ITCENTER Helpdesk"></a><table style="border-collapse:collapse"><tbody><tr width="100" style="float:right"><td style="float:right;margin-left:5px;padding:0px"><a href="https://www.facebook.com/itcenterpt" style="margin-left:5px;width:23px;min-height:23px;display:block" target="_blank"><img src="http://www.itcenter.com.pt/email/assinatura-email-itcenter_facebook.jpg" alt="ITCENTER Facebook"></a></td><td style="float:right;margin-left:5px;padding:0px"><a href="https://www.linkedin.com/company/itcenter" style="margin-left:5px;width:23px;min-height:23px;display:block" target="_blank"><img src="http://www.itcenter.com.pt/email/assinatura-email-itcenter_linkedin.jpg" alt="ITCENTER Linkedin"></a></td><td style="float:right;padding:0px"><a href="https://twitter.com/ITCenter_PT" style="margin-left:5px;width:23px;min-height:23px;display:block" target="_blank"><img src="http://www.itcenter.com.pt/email/assinatura-email-itcenter_twitter.jpg" alt="ITCENTER Twitter"></a><div><br></div></td></tr></tbody></table></pre></div></div></div></blockquote></div>
</div></div>