<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN" "http://www.w3.org/TR/REC-html40/strict.dtd"><html><head><meta name="qrichtext" content="1" /><style type="text/css">p, li { white-space: pre-wrap; }</style></head><body style=" font-family:'Lucida Console'; font-size:9pt; font-weight:400; font-style:normal;">On Dienstag, 30. Juni 2009, Munder Albert (CI/ISE) wrote:<br>
> [..]<br>
> We are running OpenSER in a pilot project and<br>
> unfortunately have some stability problems. <br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>Hallo Albert,<br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>> * Appr. 5000 subscriber accounts<br>
> * Appr. 1200 simultaneously registered users<br>
> * Signalling encrypted with TLS<br>
> * Media data encrypted with SRTP<br>
> * Clients: softphones and hardphones<br>
> * Re-registration time for clients: 3600 sec<br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>I've not that much experience with TCP, but don't think that this numbers should be a problem in a setup like this.<br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>> OpenSER configuration<br>
> · Works as stateful SIP Proxy<br>
> 1 mySQL database<br>
> 2 Version 1.3.4.-TLS<br>
> 3 Tcp_children: 100 --> is it recommended to increase this number?<br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>This are quite a lot of children, but ok.<br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>> 4 Udp_children: 20<br>
> 5 Tcp_connection_timeout: 3600<br>
> 6 Shared memory:<br>
> · -m 512 when error occurred<br>
> 1 Now set to 1024<br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>How much PKG_MEM do you use? The default value?<br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>> Problems<br>
> * Shared memory consumption<br>
> Shared memory usage is permanently increasing (about 50 MB per day)<br>
> Application already crashed twice<br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>This could be a memory leak, what modules do you use? And do you use any proprietary modules? You could use the memory debugging to further investigate this: http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory<br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>> First messages were, these, repeated thousands of times (5915 times):<br>
> Jun 17 08:54:52 si-.... /usr/local/sbin/openser[13921]:<br>
> ERROR:core:tcpconn_new: shared memory allocation failure Jun 17 08:54:52<br>
> si-... /usr/local/sbin/openser[13921]: ERROR:core:handle_new_connect:<br>
> tcpconn_new failed, closing socket And a few of these also (7613 times):<br>
> Jun 17 08:57:24 si-... /usr/local/sbin/openser[13880]:<br>
> ERROR:core:tls_accept: some error in SSL: Jun 17 08:57:24 si-...<br>
> /usr/local/sbin/openser[13880]: ERROR:core:tls_print_errstack:<br>
> error:1409C041:SSL routines:SSL3_SETUP_BUFFERS:malloc failure<br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>This are caused from insufficient memory conditions. I can't comment on the TCP and TLS errors. But before really starting to investigate this problem, would it be possible for you to use a more recent version, e.g. kamailio 1.5.1 for testing?<br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>> * TCP errors, lost SIP messages<br>
><br>
> Examples from error messages:<br>
> 14.100 times in log file from 17.06.09<br>
> Jun 17 04:03:15 si-... /usr/local/sbin/openser[13863]:<br>
> ERROR:core:tcp_blocking_connect: poll error: flags 18 Jun 17 04:03:15<br>
> si-... /usr/local/sbin/openser[13863]: ERROR:core:tcp_blocking_connect:<br>
> failed to retrieve SO_ERROR (111) Connection refused Jun 17 04:03:15 si-...<br>
> /usr/local/sbin/openser[13863]: ERROR:core:tcpconn_connect:<br>
> tcp_blocking_connect failed Jun 17 04:03:15 si-...<br>
> /usr/local/sbin/openser[13863]: ERROR:core:tcp_send: connect failed Jun 17<br>
> 04:03:15 si-.. /usr/local/sbin/openser[13863]: ERROR:tm:msg_send: tcp_send<br>
> failed Jun 17 04:03:15 si-... /usr/local/sbin/openser[13863]:<br>
> ERROR:tm:t_forward_nonack: sending request failed<br>
><br>
> Appears at least 20 000 times; and in the day of the last shared memory<br>
> errors, it was 225.794 times in the log file (note that the number in<br>
> parenthesis is usually 1 or 2, but on that day it has reached 6): Jun 17<br>
> 09:01:27 si-.... /usr/local/sbin/openser[13921]: WARNING:core:send2child:<br>
> no free tcp receiver, connection passed to the leastbusy one (6) Jun 17<br>
> 09:01:27 si-... /usr/local/sbin/openser[13921]: WARNING:core:send2child: no<br>
> free tcp receiver, connection passed to the leastbusy one (5)<br>
><br>
> * Certificate validation problems<br>
> TCP traffic is currently significantly increased by some ( appr. 70)<br>
> clients which failed to validate the TLS certificate. Registration is<br>
> repeated every 5 sec.<br>
><br>
> Circa 30 thousand per day (on that day, it was 37.162 times in log)<br>
> Jun 17 04:03:10 si-024lc008 /usr/local/sbin/openser[13801]:<br>
> ERROR:core:tls_accept: some error in SSL: Jun 17 04:03:10 si-024lc008<br>
> /usr/local/sbin/openser[13801]: ERROR:core:tls_print_errstack:<br>
> error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca<br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>Best regards,<br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>Henning</p></body></html>