<div dir="ltr"><br><div class="gmail_quote">On Sun, Oct 5, 2008 at 5:28 PM, Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com">miconda@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hello,<br>
<br>
mayamatakeshi wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="Ih2E3d">
<br>
On Fri, Sep 26, 2008 at 6:24 PM, mayamatakeshi <<a href="mailto:mayamatakeshi@gmail.com" target="_blank">mayamatakeshi@gmail.com</a> <mailto:<a href="mailto:mayamatakeshi@gmail.com" target="_blank">mayamatakeshi@gmail.com</a>>> wrote:<br>
<br>
<br>
On Tue, Sep 23, 2008 at 6:00 PM, Daniel-Constantin Mierla<br></div><div><div></div><div class="Wj3C7c">
<<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a> <mailto:<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>>> wrote:<br>
<br>
Hello,<br>
<br>
<br>
On 09/23/08 10:31, mayamatakeshi wrote:<br>
<br>
Hello,<br>
we have openser 1.3.3 running in production (current rev.:<br>
4943).<br>
For 3 times in 50 days we had to restart openser to<br>
correct pkg memory problem.<br>
<br>
openser 1.3.3 was released 3 weeks ago, so I guess you were<br>
running previous version before, but it happened again since<br>
you upgraded to 1.3.3, right?<br>
<br>
<br>
After some time logging messages like this:<br>
/openser.log:Aug 19 10:39:18 ipx022<br>
/usr/local/sbin/openser[16991]:<br>
ERROR:core:new_credentials: no pkg memory left,<br>
openser will eventually run out of pkg memory and refuse<br>
all subsequent requests.<br>
<br>
We are trying to recreate this in our lab so that we can<br>
follow memory troubleshooting instructions at<br>
<a href="http://kamailio.net/dokuwiki/doku.php/troubleshooting:memory" target="_blank">http://kamailio.net/dokuwiki/doku.php/troubleshooting:memory</a>,<br>
but so far we were unable to do it even when generating<br>
millions of calls and registration transactions (we are<br>
using SIPp to generate normal call flows and even abnormal<br>
call flows detected when reading openser.log, like<br>
'invalid cseq for aor', malformed SIP messages etc).<br>
<br>
We can spot memory leaks even the "out of memory" message is<br>
not printed. Just archive the logs (the most important is the<br>
shut down time) and made them available for download so they<br>
can be investigated.<br>
<br>
There could be two reasons:<br>
- there is memory leak but happens in some cases that you<br>
don't reproduce in lab, but they are in the production environment<br>
- you get memory fragmentation<br>
<br>
Let's see first the debug messages...<br>
<br>
<br>
Hello,<br>
here are the link for openser.log and cfg files:<br>
<a href="http://www.yousendit.com/download/bVlEV0o4R3NoeWJIRGc9PQ" target="_blank">http://www.yousendit.com/download/bVlEV0o4R3NoeWJIRGc9PQ</a><br>
<br>
After compilation with debug flags for memory manager, I left<br>
openser running in production for 24 hours. Then, I moved all<br>
traffic to another host and waited for more than 30 minutes before<br>
stopping openser.<br>
In the openser.cfg, I set debug=2. If you need, I can run it again<br>
with a higher value (but I hope it doesn't have to be too high,<br>
due to overhead concerns).<br>
<br>
Sorry, I forgot to tell one thing: the last revision that showed this problem was 4809, so we reverted back to that revision before performing the above.<br>
</div></div></blockquote>
to understand that you couldn't reproduce with latest svn version? So you had to get a previous version?<br>
</blockquote></div><br>Hi,<br>no, the reason for reversion is that the latest version running in production will not show the problem because we adopted preventive reset to minimize impact to customer calls. So I don't know yet if it shows this problem or not. <br>
So I collected the logs using a revision that I was sure could recreate the problem.<br><br>But here's some developments on my investigation:<br>Up to now, I was trying to recreate the problem using VirtualMachines running the same OS (Fedora 5) as in production. It never happened there, even after 30 million of calls.<br>
But we eventually were able to test openser 1.3 using a production machine with the same spec as the ones showing the problem and we were able to generate pkg memory problem using a simple outgoing SIPp scenario. The problem always happens after we reach around 28.000 calls and we confirmed the amount of calls needed to cause the problem grows linearly with the amount of pkg memory (after increase of pkg memory pool by 4, problem started to happen only after around 128.000 calls).<br>
However, we also tried the same tests with kamailio 1.4 (rev. 5017) on that machine and we could not recreate the problem after 1.5 million calls, so we are thinking in just upgrade to 1.4 after other scenarios show everything else is working.<br>
<br>But I don't know why the problem cannot be recreated using the VMs: the only significant difference is that the productions machines have 4 NICs that are bound in 2 pairs (1 for private ip and another for public ip) while the VMs have just one NIC.<br>
<br>I hope upgrading to 1.4 will solve everything, however, since nobody is complaining about having openser stopping after 28.000 calls, I still believe we have some problem in the openser.cfg itself. I'll check it after we put kamailio 1.4 in production.<br>
<br>regards,<br>takeshi<br></div>