<div dir="ltr">Hi,<br>Yes, we are using PRTG to get call and registration stats using SNMP.<br>Thanks for the tip anyway as we haven't set SNMP monitoring yet in all of our servers.<br><br><div class="gmail_quote">On Tue, Sep 23, 2008 at 11:44 PM, Jeffrey Magder <span dir="ltr"><<a href="mailto:jmagder@somanetworks.com">jmagder@somanetworks.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">I noticed that you have loaded the snmpstats module in your<br>
configuration file. Are you polling SNMPStats with SNMP at all? If<br>
not, it is possible that its interprocess buffer is filling up. The<br>
solution in this case would be to either periodically poll the module,<br>
or disable the module if you aren't using it.<br>
<div><div></div><div class="Wj3C7c"><br>
On Tue, 2008-09-23 at 16:31 +0900, mayamatakeshi wrote:<br>
> Hello,<br>
> we have openser 1.3.3 running in production (current rev.: 4943).<br>
> For 3 times in 50 days we had to restart openser to correct pkg memory<br>
> problem.<br>
> After some time logging messages like this:<br>
> /openser.log:Aug 19 10:39:18 ipx022 /usr/local/sbin/openser[16991]:<br>
> ERROR:core:new_credentials: no pkg memory left,<br>
> openser will eventually run out of pkg memory and refuse all<br>
> subsequent requests.<br>
><br>
> We are trying to recreate this in our lab so that we can follow memory<br>
> troubleshooting instructions at<br>
> <a href="http://kamailio.net/dokuwiki/doku.php/troubleshooting:memory" target="_blank">http://kamailio.net/dokuwiki/doku.php/troubleshooting:memory</a>, but so<br>
> far we were unable to do it even when generating millions of calls and<br>
> registration transactions (we are using SIPp to generate normal call<br>
> flows and even abnormal call flows detected when reading openser.log,<br>
> like 'invalid cseq for aor', malformed SIP messages etc).<br>
> And this is much more than in our production environment, with just<br>
> 600 subscribers and about 2000 calls a day.<br>
><br>
> The frequency the problem happens is increasing with the number of<br>
> subscribers, so we are performing periodic restart of openser<br>
> (actually, what we do is to switch over to the standby server). We<br>
> already recompiled openser with pkg memory pool size set to 4MB so<br>
> that this will not have to be done frequently.<br>
><br>
> Since we cannot recreate this in our lab, we suspect there is a<br>
> situation happening in production that might not be having been<br>
> properly handled by openser.cfg. So my question is: would it be<br>
> possible to an overlooked detail in openser.cfg to cause pkg memory<br>
> problem?<br>
><br>
><br>
> In case someone could take a look at it, here's our cfg file:<br>
><br>
> ####### Global Parameters #########<br>
><br>
> debug=0<br>
> log_stderror=no<br>
> log_facility=LOG_LOCAL0<br>
><br>
> fork=yes<br>
> children=4<br>
><br>
> /* uncomment the following lines to enable debugging */<br>
> #debug=6<br>
> #fork=no<br>
> #log_stderror=yes<br>
><br>
> /* uncomment the next line to disable TCP (default on) */<br>
> disable_tcp=yes<br>
><br>
> /* uncomment the next line to enable the auto temporary blacklisting<br>
> of<br>
> not available destinations (default disabled) */<br>
> #disable_dns_blacklist=no<br>
><br>
> /* uncomment the next line to enable IPv6 lookup after IPv4 dns<br>
> lookup failures (default disabled) */<br>
> #dns_try_ipv6=yes<br>
><br>
> /* uncomment the next line to disable the auto discovery of local<br>
> aliases<br>
> based on revers DNS on IPs (default on) */<br>
> #auto_aliases=no<br>
><br>
> /* uncomment the following lines to enable TLS support (default off)<br>
> */<br>
> #disable_tls = no<br>
> #listen = tls:your_IP:5061<br>
> #tls_verify_server = 1<br>
> #tls_verify_client = 1<br>
> #tls_require_client_certificate = 0<br>
> #tls_method = TLSv1<br>
> #tls_certificate = "/usr/local/etc/openser/tls/user/user-cert.pem"<br>
> #tls_private_key = "/usr/local/etc/openser/tls/user/user-privkey.pem"<br>
> #tls_ca_list = "/usr/local/etc/openser/tls/user/user-calist.pem"<br>
><br>
><br>
> port=5060<br>
><br>
> /* uncomment and configure the following line if you want openser to<br>
> bind on a specific interface/port/proto (default bind on all<br>
> available) */<br>
> #listen=udp:<a href="http://202.173.5.181:5060" target="_blank">202.173.5.181:5060</a><br>
><br>
><br>
> ####### Modules Section ########<br>
><br>
> #set module path<br>
> mpath="/usr/local/lib/openser/modules/"<br>
><br>
> /* uncomment next line for MySQL DB support */<br>
> loadmodule "mysql.so"<br>
> loadmodule "sl.so"<br>
> loadmodule "tm.so"<br>
> loadmodule "rr.so"<br>
> loadmodule "maxfwd.so"<br>
> loadmodule "usrloc.so"<br>
> loadmodule "registrar.so"<br>
> loadmodule "textops.so"<br>
> loadmodule "mi_fifo.so"<br>
> loadmodule "uri_db.so"<br>
> loadmodule "uri.so"<br>
> loadmodule "xlog.so"<br>
> loadmodule "acc.so"<br>
> loadmodule "carrierroute.so"<br>
> loadmodule "nathelper.so"<br>
> loadmodule "dialog.so"<br>
> loadmodule "snmpstats.so"<br>
> /* uncomment next lines for MySQL based authentication support<br>
> NOTE: a DB (like mysql) module must be also loaded */<br>
> loadmodule "auth.so"<br>
> loadmodule "auth_db.so"<br>
> loadmodule "lcr.so"<br>
> /* uncomment next line for aliases support<br>
> NOTE: a DB (like mysql) module must be also loaded */<br>
> loadmodule "alias_db.so"<br>
> /* uncomment next line for multi-domain support<br>
> NOTE: a DB (like mysql) module must be also loaded<br>
> NOTE: be sure and enable multi-domain support in all used modules<br>
> (see "multi-module params" section ) */<br>
> loadmodule "domain.so"<br>
> /* uncomment the next two lines for presence server support<br>
> NOTE: a DB (like mysql) module must be also loaded */<br>
> #loadmodule "presence.so"<br>
> #loadmodule "presence_xml.so"<br>
> loadmodule "uac.so"<br>
> loadmodule "avpops.so"<br>
> # ----------------- setting module-specific parameters ---------------<br>
><br>
><br>
> # ----- mi_fifo params -----<br>
> modparam("mi_fifo", "fifo_name", "/tmp/openser_fifo")<br>
><br>
><br>
> # ----- rr params -----<br>
> # add value to ;lr param to cope with most of the UAs<br>
> modparam("rr", "enable_full_lr", 1)<br>
> # do not append from tag to the RR (no need for this script)<br>
> modparam("rr", "append_fromtag", 1)<br>
><br>
><br>
> # ----- rr params -----<br>
> modparam("registrar", "method_filtering", 1)<br>
> /* uncomment the next line to disable parallel forking via location */<br>
> # modparam("registrar", "append_branches", 0)<br>
> /* uncomment the next line not to allow more than 10 contacts per AOR<br>
> */<br>
> modparam("registrar", "max_contacts", 10)<br>
> modparam("registrar", "min_expires", 30)<br>
> modparam("registrar", "max_expires", 40)<br>
> modparam("registrar", "default_expires", 35)<br>
> # ----- uri_db params -----<br>
> /* by default we disable the DB support in the module as we do not<br>
> need it<br>
> in this configuration */<br>
> modparam("uri_db", "use_uri_table", 0)<br>
> modparam("uri_db", "db_url",<br>
> "mysql://openser:openserrw@localhost/openser")<br>
> modparam("uri_db", "use_domain", 1)<br>
><br>
> # ----- acc params -----<br>
> /* what sepcial events should be accounted ? */<br>
> modparam("acc", "early_media", 1)<br>
> modparam("acc", "report_ack", 1)<br>
> modparam("acc", "report_cancels", 1)<br>
> /* by default ww do not adjust the direct of the sequential requests.<br>
> if you enable this parameter, be sure the enable "append_fromtag"<br>
> in "rr" module */<br>
> modparam("acc", "detect_direction", 0)<br>
> /* account triggers (flags) */<br>
> modparam("acc", "failed_transaction_flag", 3)<br>
> modparam("acc", "log_flag", 1)<br>
> modparam("acc", "log_missed_flag", 2)<br>
> /* uncomment the following lines to enable DB accounting also */<br>
> modparam("acc", "db_flag", 1)<br>
> modparam("acc", "db_missed_flag", 2)<br>
><br>
><br>
> # ----- usrloc params -----<br>
> #modparam("usrloc", "db_mode", 0)<br>
> /* uncomment the following lines if you want to enable DB persistency<br>
> for location entries */<br>
> modparam("usrloc", "db_mode", 2)<br>
> modparam("usrloc", "db_url",<br>
> "mysql://openser:openserrw@localhost/openser")<br>
> modparam("usrloc", "use_domain", 1)<br>
><br>
> # ----- auth_db params -----<br>
> /* uncomment the following lines if you want to enable the DB based<br>
> authentication */<br>
> modparam("auth_db", "calculate_ha1", yes)<br>
> modparam("auth_db", "password_column", "password")<br>
> modparam("auth_db", "db_url",<br>
> "mysql://openser:openserrw@localhost/openser")<br>
> modparam("auth_db", "load_credentials", "$avp(s:rpid)=rpid;<br>
> $avp(s:blocked)=subscriber_status")<br>
><br>
><br>
> # ----- alias_db params -----<br>
> /* uncomment the following lines if you want to enable the DB based<br>
> aliases */<br>
> modparam("alias_db", "db_url",<br>
> "mysql://openser:openserrw@localhost/openser")<br>
> modparam("alias_db", "use_domain", 0)<br>
><br>
> # ----- domain params -----<br>
> /* uncomment the following lines to enable multi-domain detection<br>
> support */<br>
> modparam("domain", "db_url",<br>
> "mysql://openser:openserrw@localhost/openser")<br>
> modparam("domain", "db_mode", 1) # Use caching<br>
><br>
><br>
> # ----- multi-module params -----<br>
> /* uncomment the following line if you want to enable multi-domain<br>
> support<br>
> in the modules (dafault off) */<br>
> #modparam("alias_db|auth_db|usrloc|uri_db", "use_domain", 1)<br>
><br>
><br>
> # ----- presence params -----<br>
> /* uncomment the following lines if you want to enable presence */<br>
> #modparam("presence|presence_xml", "db_url",<br>
> # "mysql://openser:openserrw@localhost/openser")<br>
> #modparam("presence_xml", "force_active", 1)<br>
> #modparam("presence", "server_address", "sip:<a href="http://192.168.1.2:5060" target="_blank">192.168.1.2:5060</a>")<br>
><br>
> # ----- carrieroute params -----<br>
> modparam("carrierroute", "db_url",<br>
> "mysql://openser:openserrw@localhost/openser")<br>
> modparam("carrierroute", "config_source", "db")<br>
> modparam("carrierroute", "use_domain", 1)<br>
> # ----- NatHelper -----<br>
> #para versao a partir da versao 1.2 eh necessario esse paramtro para<br>
> nao dar erro qdo usa a funcao "fix_nated_register();"<br>
> modparam("nathelper|registrar", "received_avp", "$avp(i:42)")<br>
> modparam("nathelper", "rtpproxy_sock", "udp:<a href="http://127.0.0.1:22222" target="_blank">127.0.0.1:22222</a>")<br>
><br>
><br>
> # ----- LCR -----<br>
> modparam("lcr", "db_url",<br>
> "mysql://openser:openserrw@localhost/openser")<br>
> modparam("lcr|tm", "fr_inv_timer_avp", "$avp(i:704)")<br>
> modparam("lcr", "gw_uri_avp", "$avp(i:709)")<br>
> modparam("^auth$|lcr", "rpid_avp", "$avp(i:302)")<br>
> modparam("lcr", "contact_avp", "$avp(i:711)")<br>
> modparam("lcr", "ruri_user_avp", "$avp(i:500)")<br>
> modparam("lcr", "dm_flag", 25)<br>
><br>
> # ----- Dialog ----<br>
> modparam("dialog", "dlg_flag", 4)<br>
><br>
> # ----- SnmpStat -----<br>
> modparam("snmpstats", "sipEntityType", "registrarServer")<br>
> modparam("snmpstats", "sipEntityType", "proxyServer")<br>
> modparam("snmpstats", "MsgQueueMinorThreshold", 2000)<br>
> modparam("snmpstats", "MsgQueueMajorThreshold", 5000)<br>
> modparam("snmpstats", "dlg_minor_threshold", 500)<br>
> modparam("snmpstats", "dlg_major_threshold", 750)<br>
> modparam("snmpstats", "snmpgetPath","/usr/bin/")<br>
> modparam("snmpstats", "snmpCommunity","public")<br>
><br>
><br>
><br>
> ####### Routing Logic ########<br>
><br>
><br>
> # main request routing logic<br>
><br>
> route{<br>
><br>
> if (!mf_process_maxfwd_header("10")) {<br>
> sl_send_reply("483","Too Many Hops");<br>
> exit;<br>
> }<br>
><br>
> ##nat<br>
> route(2);<br>
><br>
> if (has_totag()) {<br>
> # sequential request withing a dialog should<br>
> # take the path determined by record-routing<br>
> if (loose_route()) {<br>
> if (is_method("BYE")) {<br>
> setflag(1); # do accouting ...<br>
> setflag(3); # ... even if the transaction fails<br>
> }<br>
> route(1);<br>
> } else {<br>
> /* uncomment the following lines if you want to enable<br>
> presence */<br>
> ##if (is_method("SUBSCRIBE") && $rd ==<br>
> "your.server.ip.address") {<br>
> ## # in-dialog subscribe requests<br>
> ## route(2);<br>
> ## exit;<br>
> ##}<br>
> if ( is_method("ACK") ) {<br>
> if ( t_check_trans() ) {<br>
> # non loose-route, but stateful ACK; must be an<br>
> ACK after a 487 or e.g. 404 from upstream server<br>
> #t_relay();<br>
> #exit;<br>
> route(1);<br>
> } else {<br>
> # ACK without matching transaction ... ignore and<br>
> discard.\n");<br>
> exit;<br>
> }<br>
> }<br>
> sl_send_reply("404","Not here");<br>
> }<br>
> exit;<br>
> }<br>
><br>
> #initial requests<br>
> setflag(4); #for dialog statistics<br>
><br>
><br>
> # CANCEL processing<br>
> if (is_method("CANCEL"))<br>
> {<br>
> if (t_check_trans()) route(1);<br>
> # t_relay();<br>
> # exit;<br>
> }<br>
><br>
> #t_check_trans();<br>
><br>
> if (is_method("PUBLISH|SUBSCRIBE|REFER|OPTIONS|MESSAGE"))<br>
> {<br>
> sl_send_reply("405", "Method not allowed");<br>
> exit;<br>
> }<br>
><br>
> # authenticate if from local subscriber (uncomment to enable auth)<br>
> if (!(method=="REGISTER") && (!from_gw()))<br>
> {<br>
> if (!proxy_authorize("", "subscriber")) {<br>
> proxy_challenge("", "0");<br>
> exit;<br>
> }<br>
> if (!check_from()) {<br>
> sl_send_reply("403","Forbidden auth ID");<br>
> exit;<br>
> }else if (avp_check("$avp(s:blocked)", "eq/0")) {<br>
> sl_send_reply("603","Subscriber disabled");<br>
> exit;<br>
> }else if (avp_check("$avp(s:blocked)", "eq/1")) {<br>
> sl_send_reply("603","Subscriber with outgoing<br>
> blocked");<br>
> exit;<br>
> }<br>
><br>
> consume_credentials();<br>
> # caller authenticated<br>
> }<br>
><br>
> # record routing<br>
> if (!is_method("REGISTER|MESSAGE"))<br>
> record_route();<br>
><br>
> # account only INVITEs<br>
> if (is_method("INVITE")) {<br>
> setflag(1); # do accouting<br>
> }<br>
><br>
> if (is_method("REGISTER"))<br>
> {<br>
> # authenticate the REGISTER requests (uncomment to enable<br>
> auth)<br>
> if (!proxy_authorize("", "subscriber"))<br>
> {<br>
> proxy_challenge("", "0");<br>
> exit;<br>
> }<br>
><br>
> if (!check_to())<br>
> {<br>
> sl_send_reply("403","Forbidden auth ID");<br>
> exit;<br>
> }else if (avp_check("$avp(s:blocked)", "eq/0")) {<br>
> sl_send_reply("403","Subscriber disabled");<br>
> exit;<br>
> }<br>
><br>
> if (!save("location"))<br>
> sl_reply_error();<br>
><br>
> exit;<br>
> }<br>
><br>
> if ($rU==NULL) {<br>
> # request with no Username in RURI<br>
> sl_send_reply("484","Address Incomplete");<br>
> exit;<br>
> }<br>
><br>
> # apply DB based aliases (uncomment to enable)<br>
> ##alias_db_lookup("dbaliases");<br>
> #if the call came from a known gateway it is not authenticated and<br>
> we cannot use the function check_from()<br>
> if (from_gw()) {<br>
> route(4);<br>
> }else if (!check_from()) {#if the check_from() returns false the<br>
> call is not from a subscriber<br>
> route(4);<br>
> } else {#it is a subscriber, route using flip domain<br>
> xlog("L_INFO", "routing using carrierroute $rm to $ru\n");<br>
> if (!cr_user_rewrite_uri("$fu", "flip"))<br>
> {<br>
> t_newtran();<br>
> t_reply("404", "No Route");<br>
> exit;<br>
> }<br>
> #replaces from by it's default DID<br>
> uac_replace_from("sip:$avp(s:rpid)@$fd");<br>
> }<br>
><br>
> # when routing via usrloc, log the missed calls also<br>
> setflag(2);<br>
><br>
> route(1);<br>
> }<br>
><br>
> route[1] {<br>
> xlog("L_INFO", "ROUTE_1 $rm to $ru\n");<br>
> if (subst_uri('/(sip:.*);nat=yes/\1/'))<br>
> {<br>
> setflag(6);<br>
> };<br>
><br>
> if (isflagset(5)||isflagset(6)) {<br>
> route(3);<br>
> }<br>
><br>
> if (!t_relay()) {<br>
> sl_reply_error();<br>
> };<br>
> exit;<br>
> }<br>
><br>
> route[2] {<br>
> xlog("L_INFO", "ROUTE_2 $rm to $ru\n");<br>
> if (method=="REGISTER") {<br>
> fix_nated_register();<br>
> } else if (!from_gw()){<br>
> fix_nated_contact();<br>
> };<br>
> setflag(5);<br>
> }<br>
> route[3] {<br>
> xlog("L_INFO", "ROUTE_3 $rm to $ru\n");<br>
> if (is_method("BYE|CANCEL")) {<br>
> unforce_rtp_proxy();<br>
> } else if (is_method("INVITE")) {<br>
> xlog("L_INFO", "FORCE RTP w/ parameter.\n");<br>
> force_rtp_proxy("r");<br>
> t_on_failure("1");<br>
> };<br>
> if (isflagset(5))<br>
> search_append('Contact:.*sip:[^>[:cntrl:]]*',<br>
> ';nat=yes');<br>
> t_on_reply("1");<br>
> }<br>
><br>
> route[4] {<br>
><br>
> xlog("L_INFO", "uri does exist $rm to $ru \n");<br>
> if (alias_db_lookup("dbaliases")){<br>
><br>
> if (!lookup("location")) {<br>
> switch ($retcode) {<br>
> case -1:<br>
> t_newtran();<br>
> t_reply("404", "Subscriber not<br>
> online");<br>
> exit;<br>
> case -2:<br>
> sl_send_reply("405", "Method Not<br>
> Allowed");<br>
> exit;<br>
> }<br>
> }<br>
> }else{#check if did is blocked<br>
> $rU = "(BLK)" + $rU;<br>
> if (alias_db_lookup("dbaliases")){<br>
> sl_send_reply("403", "DID blocked");<br>
> exit;<br>
><br>
> }else{# if it is not a valid DID nor a blocked DID tries to<br>
> route it using peering domain<br>
> if (!cr_rewrite_uri("peering", "call_id"))<br>
> {<br>
> t_newtran();<br>
> t_reply("404", "Peering Not Found");<br>
> exit;<br>
> }<br>
><br>
> }<br>
> }<br>
> }<br>
><br>
> failure_route[1] {<br>
> xlog("L_INFO", "FAILURE $rm to $ru\n");<br>
> if (isflagset(6)||isflagset(5)) {<br>
> unforce_rtp_proxy();<br>
> }<br>
> }<br>
><br>
> onreply_route[1] {<br>
> xlog("L_INFO", "ONREPLY_1 - Status $rs from $si $rm .\n");<br>
> if (is_method("INVITE")) {<br>
> if ((isflagset(5)||isflagset(6)) &&<br>
> status=~"(183)|(2[0-9][0-9])") {<br>
> force_rtp_proxy();<br>
> }<br>
> search_append('Contact:.*sip:[^>[:cntrl:]]*',<br>
> ';nat=yes');<br>
><br>
> if (!from_gw()){ #if (isflagset(6)) {<br>
> xlog("L_INFO", "ONREPLY_1 - ! from gw.\n");<br>
> fix_nated_contact();<br>
> }<br>
> exit;<br>
> }<br>
> }<br>
><br>
> Regards,<br>
> takeshi<br>
><br>
><br>
><br>
</div></div>> _______________________________________________<br>
> Users mailing list<br>
> <a href="mailto:Users@lists.kamailio.org">Users@lists.kamailio.org</a><br>
> <a href="http://lists.kamailio.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.kamailio.org/cgi-bin/mailman/listinfo/users</a><br>
<br>
</blockquote></div><br></div>