<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN" "http://www.w3.org/TR/REC-html40/strict.dtd"><html><head><meta name="qrichtext" content="1" /><style type="text/css">p, li { white-space: pre-wrap; }</style></head><body style=" font-family:'Lucida Console'; font-size:9pt; font-weight:400; font-style:normal;">On Montag, 9. November 2009, Robin Vleij wrote:<br>
> > do you experience any problems in your setup when you use a reasonable<br>
> > SHM mem size? In my experience the size of the SHM memory (as displayed<br>
><br>
> I've had problems finding a "reasonable" shm mem size. :) Standard is<br>
> like 32MB, which runs out quickly when customers do "funny stuff" (read:<br>
> loops). Now I'm compiling with #define SHM_MEM_SIZE 4*32. 128MB should<br>
> be enough to hold pretty long.<br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>Hi Robin,<br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>btw, there is no need to re-compile the server just to change this setting, its a normal daemon binary parameter. 128 MB should be really fine, given the load you quoted.<br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>> So there's no immediate memory problem or<br>
> crashes (when it's full, it gets errors and stops processing traffic the<br>
> right way). But right now for example, after a "funny" customer, I'm<br>
> seeing over 40mb per child in top (16 children). That won't go down<br>
> anymore, so we'll have to see how long it holds.<br>
> What do you suggest for SHM sizes?<br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>With today memory sizes/ prizes you could use for example 512 MB, which should give you plenty of room even in really abnormal load conditions. And as its shared, you'll have still plenty of room for e.g. the database.<br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>> > machine has been completely passive over a longer time, it will not<br>
> > reclaim this memory. On a certain test system for example there is one<br>
> > process that has 11MB SHM at the moment, even if its completely idle.<br>
><br>
> OK. We often run very long on 10-20MB per process (all processes have<br>
> about the same, at least the children that process UDP), but like today<br>
> when someone has a problem and it becomes sip-spaghetti it jumps up to<br>
> 40MB and then continues to slowly rise from there. Doesn't feel good to<br>
> be able to hit some kind of roof with the same traffic load.<br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>You mentioned the the loops a few times, normally they should be pretty fast detected by max forward counter checks and additionally by diversion header checks?<br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>> > If you've a real memory leak in shared memory then after a certain time<br>
> > interval the server will report memory allocation errors. Otherwise i<br>
> > don't think its something to worry about.<br>
><br>
> It does, if I don't make the limit higher. So say that I'm running on<br>
> 32, then if I would hit that after some weeks uptime it would start<br>
> reporting memory allocation errors in different parts of my config and<br>
> stop doing important stuff. I also reproduced this assigning a small<br>
> amount to a dev machine and then sending 20cps to the machine.<br>
><br>
> On a test machine I have like 4 processes all using 600kb or so, then<br>
> after 20 calls it'll go up to something like<br>
><br>
> 31409 root 15 0 94672 1936 1052 R 0.0 0.7 0:00.00 kamailio<br>
> 31408 root 15 0 94784 3072 2068 S 0.0 1.2 0:00.00 kamailio<br>
> 31407 root 15 0 94784 3072 2068 S 0.0 1.2 0:00.00 kamailio<br>
> 31406 root 25 0 94672 5428 4556 S 0.0 2.1 0:00.02 kamailio<br>
><br>
> And go back in size only a little after 15-20 minutes or so (often a bit<br>
> faster is load is low).<br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>With the memory debugging you could dump all the allocations during runtime, but they are a bit hard to read for a non-developer. But this way you could reproduce "call by call" how your server behave and how the situation develops.<br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>> If this is a leak, it'll be almost impossible to find. I can't run<br>
> production with memlog or debug on, and in dev it's quite hard to<br>
> reproduce it seems. Not sure what to expect. :)<br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>If you'd have a leak in a common used code path, then you'll run out of memory pretty fast, like in a few days. If your servers are stable (like some weeks or month) with the setting you use at the moment, i don't think there is much to worry. <br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>Regards,<br>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p>Henning</p></body></html>