<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hello,<br>
<br>
<div class="moz-cite-prefix">On 11/22/12 8:51 AM, Uri Shacked wrote:<br>
</div>
<blockquote
cite="mid:CAMMbDhT7nLMHuz0_tG75nsoOktBCy6iR015Mvb=rOrDpsK3JNQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
Hi,</div>
<div> </div>
<div>>> Btw, is the reload time constant now? Even if you
run couple of times?</div>
<div> </div>
<div>yes, the reload time is constant - 56 sec. i tested for 100
reloads in an hour and it was OK.</div>
</div>
</blockquote>
<br>
ok - it's 56sec or 36sec?<br>
<br>
<blockquote
cite="mid:CAMMbDhT7nLMHuz0_tG75nsoOktBCy6iR015Mvb=rOrDpsK3JNQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div> </div>
<div>>> What are the values for 'kamctl mi get_statistics
shmem:'?</div>
<div> </div>
<div>i configured kamailio to start with 4Gb and after reload
the shmem (real_used) take around 30% of it.</div>
<div>but, after 20 reloads it grows in 1%. so, after the 100
reloads the real_used take around 34%-35% of shmem.</div>
<div>i made the choise to compile again with f_malloc and not
use mem_join. the reloads are faster, it uses less real_size
(12% and not 30%) and the increasment of it is around 1% for 5
reloads (i do 5 reloads a week). i will keep track on it and
update.</div>
<div> </div>
<div>thanks a lot.</div>
<div> </div>
<div>Btw, what do you think? would you use f_malloc with no
mem_join or q_malloc with join?</div>
</div>
</blockquote>
<br>
Can you try with master branch? I switched to q_malloc with no debug
info. That means the overhead should be lower and the joining
faster.<br>
<br>
You can do it also with 3.3, you have to edit Makefile.defs:<br>
- be sure MEMDBG=0<br>
- replace the next block:<br>
<br>
ifeq ($(MEMDBG), 1)<br>
C_DEFS+= -DDBG_QM_MALLOC<br>
C_DEFS+= -DMEM_JOIN_FREE<br>
else<br>
C_DEFS+= -DF_MALLOC<br>
C_DEFS+= -DMEM_JOIN_FREE<br>
endif<br>
<br>
with:<br>
<br>
ifeq ($(MEMDBG), 1)<br>
C_DEFS+= -DDBG_QM_MALLOC<br>
C_DEFS+= -DMEM_JOIN_FREE<br>
else<br>
C_DEFS+= -DMEM_JOIN_FREE<br>
endif<br>
<br>
Practically is removal of line C_DEFS+= -DF_MALLOC<br>
<br>
Then recompile and reinstall..<br>
<br>
Cheers,<br>
Daniel<br>
<blockquote
cite="mid:CAMMbDhT7nLMHuz0_tG75nsoOktBCy6iR015Mvb=rOrDpsK3JNQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div> </div>
<div>BR,</div>
<div>Uri</div>
<div><br>
</div>
<div class="gmail_quote">
On Wed, Nov 21, 2012 at 8:55 PM, Daniel-Constantin Mierla <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote style="margin:0px 0px 0px
0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"
class="gmail_quote">
<div text="#000000" bgcolor="#FFFFFF"> Hello,
<div class="im"><br>
<br>
<div>On 11/21/12 1:33 PM, Uri Shacked wrote:<br>
</div>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hi,</div>
<div> </div>
<div class="im">
<div>I recompiled with MEMDBG=1 and installed.</div>
<div>here are the results for reloading 5 million
rows with MTREE:</div>
<div> </div>
<div>mem_join=1 -->takes 56 seconds and the
real_used_size of shmem is around 1.2Gb.</div>
<div>mem_join=0 --> takes 10 seconds and the
real_used_size of shmem is around 2Gb<br>
</div>
<div>does it seems normal?</div>
<div>56 seconds is a lot of time......</div>
</div>
</div>
</blockquote>
the join is done for the free operation, meaning that most
of the time is spent when freeing the old tree from
memory. The new values will be used after loading the
database records, then the old tree is destroyed (this
involves the join operation). Also, the sip routing is not
affected, loading the new records and destroying old
memory tree is done in the MI/RPC process.<br>
<br>
In other words, while the MI/RPC process takes care of
loading new data and destroying the old one, the SIP
routing is not affected at all.<br>
<br>
Even when the reload command is executed, the old tree is
used until all the records are loaded in a new tree. At
that moment, the pointer to the active tree is changed
from the old tree to the new tree (a very simple sequence
of assignments, very fast). Routing will use the new tree
and the Mi/RPC process starts destroying the old tree.
<div class="im"><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div> </div>
<div>by the way, when the f_malloc was used, the
size of the real_used shmem was twice smaller.</div>
</div>
</blockquote>
<br>
</div>
The overhead when storing small values is significant for
q_malloc, each fragment keeping references (pointers) to
file name and line where it was allocated and freed. In
addition it keeps information to get to previews and next
fragment, resulting in faster join.<br>
<br>
It is some space to improve, in order to make less
overhead (like a compile time option not to keep info
about file and line of malloc/free). I will think about
what can be done for the next release.<br>
<br>
Btw, is the reload time constant now? Even if you run
couple of times?<br>
<br>
What are the values for 'kamctl mi get_statistics shmem:'?<br>
<br>
Cheers,<br>
Daniel
<div>
<div class="h5"><br>
<blockquote type="cite">
<div dir="ltr">
<div> </div>
<div>Thanks.<br>
</div>
<div class="gmail_quote">On Tue, Nov 20, 2012 at
9:45 PM, Daniel-Constantin Mierla <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:miconda@gmail.com"
target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote style="margin:0px 0px 0px
0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"
class="gmail_quote">
<div text="#000000" bgcolor="#FFFFFF"> Hello,
<div><br>
<br>
<div>On 11/20/12 7:34 PM, Uri Shacked
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hi,</div>
<div> </div>
<div>can you be a litle more specific
of the steps of the install and
where do i make the changes?</div>
</div>
</blockquote>
<br>
</div>
in the source tree, edit the file
Makefile.defs and set:<br>
<br>
MEMDBG=1<br>
<br>
then run:<br>
<br>
make all<br>
make install
<div><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>some words of what is the diff
between f_malloc and q_malloc will
be great :-).</div>
</div>
</blockquote>
<br>
</div>
q_malloc is more debugging purposes, keeping
more information for each chunk, therefore
the overhead is a bit higher than with
f_malloc, but because keeps more details, it
is faster to find the fragments that can be
joined.<br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div> </div>
<div>thanks,</div>
<div>Uri<br>
<br>
</div>
<div class="gmail_quote">On Tue, Nov
20, 2012 at 6:26 PM,
Daniel-Constantin Mierla <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:miconda@gmail.com"
target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote style="margin:0px 0px
0px
0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"
class="gmail_quote">
<div text="#000000"
bgcolor="#FFFFFF"> Hello,<br>
<br>
ok, I will look over it. At
this moment the f_malloc
(which is enabled for 3.3) has
a pretty inefficient mem join
implementation, can you try
with q_malloc? Edit
Makefile.defs and set:<br>
<br>
MEMDBG=1<br>
<br>
Then compile and install.<br>
<br>
The join operation should be
faster, let's see if you get
blocking issues with this one.<br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
<div>On 11/20/12 2:57 PM,
Uri Shacked wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Daniel hi,</div>
<div> </div>
<div>I attached 2 txt
files.</div>
<div>One with
mem_join=1, the
other with
mem_join=0, and the
info you asked for.</div>
<div>Let me know if it
is OK.</div>
<div> </div>
<div>Thanks,</div>
<div>Uri<br>
<br>
</div>
<div
class="gmail_quote">On
Mon, Nov 19, 2012 at
10:50 AM,
Daniel-Constantin
Mierla <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span>
wrote:<br>
<blockquote
style="margin:0px
0px 0px
0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"
class="gmail_quote">
<div
text="#000000"
bgcolor="#FFFFFF">
Hello,<br>
<br>
if you set
memjoin to 0, do
you see any
difference?<br>
<br>
Can you try
again (with
memjoin 1 as
well as 0) and
send the output
of:<br>
<br>
kamctl mi
get_statistics
shmem:<br>
<br>
before executing
the reload
commands?<br>
<br>
When it gets to
100%, can you
see which
process is using
the cpu and
attach to it
with:<br>
<br>
gdb
/path/to/kamailio
PID<br>
<br>
then do:<br>
<br>
bt full<br>
<br>
and send output
here?<br>
<br>
Cheers,<br>
Daniel
<div>
<div><br>
<br>
<div>On
11/18/12 4:09
PM, Uri
Shacked wrote:<br>
</div>
</div>
</div>
<blockquote
type="cite">
<div>
<div>
<div dir="ltr">
<div>After
some testing I
notice the
following:</div>
<div>First
reload of 5
million
records after
kamailio
started took
about 9 sec.</div>
<div>Second
reload (4
minutes after
the first one)
took 60 sec.</div>
<div>The third
one (again
about 4
minutes after
the
secind) got
kamailio to
use 100% cpu
and after 13
minutes! i
killed it.....</div>
<div> </div>
<div>I can
understand
that the
memory manger
works harder,
still, any
ideas on how
to use
mem_join and
keep on
reloading
data.</div>
<div>(in real
life our data
loads 5
million
records once a
day when
almost no
traffic. still
after a few
days it
stops...)</div>
<div> </div>
<div>Thanks,</div>
<div>Uri</div>
<div><br>
<br>
</div>
<div
class="gmail_quote">On
Sun, Nov 18,
2012 at 11:52
AM, Uri
Shacked <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:ushacked@gmail.com" target="_blank">ushacked@gmail.com</a>></span>
wrote:<br>
<blockquote
style="margin:0px
0px 0px
0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"
class="gmail_quote">
<div dir="ltr">
<div><font
face="Times
New Roman"
size="3"> </font>
<p
style="margin:0cm
0cm
0pt;line-height:normal"
class="MsoNormal"><span>Hi,</span></p>
<font
face="Times
New Roman"
size="3"> </font>
<p
style="margin:0cm
0cm
0pt;line-height:normal"
class="MsoNormal"><span>I am using MTREE and DIALPLAN modules to load
lots of info
to kamailio.
(6 million
rows).</span></p>
<font
face="Times
New Roman"
size="3"> </font>
<p
style="margin:0cm
0cm
0pt;line-height:normal"
class="MsoNormal"><span>When kamailio was running with 3.2.1 (no
mem_join=1
option), the
used size was
increasing but
the process of
loading the
data was fast
eanough.</span></p>
<font
face="Times
New Roman"
size="3"> </font>
<p
style="margin:0cm
0cm
0pt;line-height:normal"
class="MsoNormal"><span>I upgraded to 3.3.2 and set mem_join=1. Now the
loading
process take
about 10 time
longer and
sometimes
stops kamailio
from
responding to
traffic.</span></p>
<font
face="Times
New Roman"
size="3"> </font>
<p
style="margin:0cm
0cm
0pt;line-height:normal"
class="MsoNormal"><span>Any ideas?</span></p>
<font
face="Times
New Roman"
size="3"> </font>
<p
style="margin:0cm
0cm
0pt;line-height:normal"
class="MsoNormal"><span>Thanks,</span></p>
<font
face="Times
New Roman"
size="3"> </font>
<p
style="margin:0cm
0cm
0pt;line-height:normal"
class="MsoNormal"><span>Uri</span></p>
<font
face="Times
New Roman"
size="3"> </font></div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
</div>
</div>
<pre>_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
<a moz-do-not-send="true" href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a>
<a moz-do-not-send="true" href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><span><font color="#888888">
</font></span></pre>
<span><font
color="#888888">
</font></span></blockquote>
<span><font
color="#888888">
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a moz-do-not-send="true" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a moz-do-not-send="true" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a></pre>
</font></span></div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a moz-do-not-send="true" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a moz-do-not-send="true" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a></pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a moz-do-not-send="true" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a moz-do-not-send="true" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a></pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<pre cols="72">--
Daniel-Constantin Mierla - <a moz-do-not-send="true" href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a moz-do-not-send="true" href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a></pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla - <a class="moz-txt-link-freetext" href="http://www.asipto.com">http://www.asipto.com</a>
<a class="moz-txt-link-freetext" href="http://twitter.com/#!/miconda">http://twitter.com/#!/miconda</a> - <a class="moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda">http://www.linkedin.com/in/miconda</a></pre>
</body>
</html>