<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Hello,</p>
<p>yes, the db related allocated chunks look to be quite high in
number. That can be possible if there are db results with lots of
records, but in this case there are more related to column
structure than row structure.</p>
<p>Tracing back in the history of commits, I noticed a place where
db result is not freed if no row is returned by query, in the
presence module, patch done by you. It was not backported to 4.3,
iirc, but if you did it in your branch, try with the latest patch
pushed few minutes ago in master and 4.4.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<br>
<div class="moz-cite-prefix">On 19/09/16 20:20, Phil Lavin wrote:<br>
</div>
<blockquote
cite="mid:VI1PR03MB18409FF40CAAAB21B54DE77389F40@VI1PR03MB1840.eurprd03.prod.outlook.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Courier New \,serif";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;
mso-fareast-language:EN-US;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New",serif;
color:black;
mso-fareast-language:EN-GB;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;
mso-fareast-language:EN-US;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;
mso-fareast-language:EN-GB;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle24
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle25
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style="color:windowtext">Nevermind –
I found this in the docs.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif;color:windowtext">kamcmd cfg.set_now_int
core mem_dump_pkg 17645<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New",serif;color:windowtext">kamcmd cfg.set_now_int
core mem_dump_pkg 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">There were
27526 allocations. I have taken and parsed the output,
summing the memory usage, grouping it by file/function.
Parsed output is at
<a moz-do-not-send="true"
href="https://gist.github.com/phil-lavin/d2607a4eed36065e01ed42c43a0d27ad">https://gist.github.com/phil-lavin/d2607a4eed36065e01ed42c43a0d27ad</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">Do any of
those stick out to you as being far higher than they should
be? My feeling is that it’s related to the db_* stuff.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">Cheers<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span
style="color:windowtext;mso-fareast-language:EN-GB"
lang="EN-US">From:</span></b><span
style="color:windowtext;mso-fareast-language:EN-GB"
lang="EN-US"> sr-users
[<a class="moz-txt-link-freetext" href="mailto:sr-users-bounces@lists.sip-router.org">mailto:sr-users-bounces@lists.sip-router.org</a>]
<b>On Behalf Of </b>Phil Lavin<br>
<b>Sent:</b> 19 September 2016 18:15<br>
<b>To:</b> <a class="moz-txt-link-abbreviated" href="mailto:miconda@gmail.com">miconda@gmail.com</a>; Kamailio (SER) - Users
Mailing List <a class="moz-txt-link-rfc2396E" href="mailto:sr-users@lists.sip-router.org"><sr-users@lists.sip-router.org></a><br>
<b>Subject:</b> Re: [SR-Users] OOM Errors<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:windowtext">4.3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span
style="color:windowtext;mso-fareast-language:EN-GB"
lang="EN-US">From:</span></b><span
style="color:windowtext;mso-fareast-language:EN-GB"
lang="EN-US"> Daniel-Constantin Mierla [<a
moz-do-not-send="true" href="mailto:miconda@gmail.com">mailto:miconda@gmail.com</a>]
<br>
<b>Sent:</b> 19 September 2016 17:47<br>
<b>To:</b> Phil Lavin <<a moz-do-not-send="true"
href="mailto:phil.lavin@cloudcall.com">phil.lavin@cloudcall.com</a>>;
Kamailio (SER) - Users Mailing List <<a
moz-do-not-send="true"
href="mailto:sr-users@lists.sip-router.org">sr-users@lists.sip-router.org</a>><br>
<b>Subject:</b> Re: [SR-Users] OOM Errors<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p>The rpc command is in 4.4, what version are you using?<span
style="font-size:12.0pt;mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p>Cheers,<br>
Daniel<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 19/09/16 18:11, Phil Lavin wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="color:windowtext">Having
adding some graphing, I see this memory issue on another
production Kamailio server. A few of the processes are
using significantly more pkg memory than others. What I
assume is happening is that memory is leaking and will
eventually start hitting OOM errors. See <a
moz-do-not-send="true"
href="https://gist.github.com/phil-lavin/27f63b472ace53dc956125a47d210bc9">https://gist.github.com/phil-lavin/27f63b472ace53dc956125a47d210bc9</a>
for pkg memory stats output.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext">When was
corex.pkg_summary added? I’m getting not found error:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal">error: 500 - command corex.pkg_summary
not found<o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span
style="color:windowtext;mso-fareast-language:EN-GB"
lang="EN-US">From:</span></b><span
style="color:windowtext;mso-fareast-language:EN-GB"
lang="EN-US"> sr-users [<a moz-do-not-send="true"
href="mailto:sr-users-bounces@lists.sip-router.org">mailto:sr-users-bounces@lists.sip-router.org</a>]
<b>On Behalf Of </b>Phil Lavin<br>
<b>Sent:</b> 19 September 2016 14:36<br>
<b>To:</b> <a moz-do-not-send="true"
href="mailto:miconda@gmail.com">miconda@gmail.com</a>;
Kamailio (SER) - Users Mailing List
<a moz-do-not-send="true"
href="mailto:sr-users@lists.sip-router.org"><sr-users@lists.sip-router.org></a><br>
<b>Subject:</b> Re: [SR-Users] OOM Errors</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext">Thanks,
Daniel.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext">Kamailio
has been restarted now so there’s not much of any use in
the pkg memory output.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext">I’m going
to add some monitoring/graphing on this and see if there’s
an upward trend in usage.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span
style="color:windowtext;mso-fareast-language:EN-GB"
lang="EN-US">From:</span></b><span
style="color:windowtext;mso-fareast-language:EN-GB"
lang="EN-US"> sr-users [<a moz-do-not-send="true"
href="mailto:sr-users-bounces@lists.sip-router.org">mailto:sr-users-bounces@lists.sip-router.org</a>]
<b>On Behalf Of </b>Daniel-Constantin Mierla<br>
<b>Sent:</b> 19 September 2016 13:41<br>
<b>To:</b> Kamailio (SER) - Users Mailing List <<a
moz-do-not-send="true"
href="mailto:sr-users@lists.sip-router.org">sr-users@lists.sip-router.org</a>><br>
<b>Subject:</b> Re: [SR-Users] OOM Errors</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p>Hello,<o:p></o:p></p>
<p>it seems that the pkg is out of free space, not shm. You
can check it with:<o:p></o:p></p>
<p>kamcmd pkg.stats<o:p></o:p></p>
<p>and look for the same pid that printed the log messages.<o:p></o:p></p>
<p>To troubleshoot, it is good to print the pkg summary, see:<o:p></o:p></p>
<p> - <a moz-do-not-send="true"
href="https://www.kamailio.org/docs/modules/stable/modules/corex.html#corex.rpc.pkg_summary">https://www.kamailio.org/docs/modules/stable/modules/corex.html#corex.rpc.pkg_summary</a><o:p></o:p></p>
<p>Then get the log messages from syslog printed by the
command and send them over here.<o:p></o:p></p>
<p>Cheers,<br>
Daniel<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<p class="MsoNormal">On 19/09/16 14:02, Phil Lavin wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Hi all,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Kamailio stopped performing some
functions earlier today, citing OOM errors. An excerpt
from the log is below.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">SHM usage, at the time, looked fine.
Also see below for that output.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">A restart of Kamailio resolved the
issue. Is this SHM related or OS memory manager related?<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">root@ua-proxy-01:~# kamctl stats
shmem</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">shmem:fragments = 910</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">shmem:free_size = 104131224</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">shmem:max_used_size = 51590800</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">shmem:real_used_size = 30086504</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">shmem:total_size = 134217728</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">shmem:used_size = 19558800</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">Sep 19 11:35:56 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[data_lump.c:161]: insert_new_lump_before(): out of
memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">Sep 19 11:40:10 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[msg_translator.c:2390]: build_res_buf_from_sip_req():
out of memory; needs 423</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">Sep 19 11:40:16 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[msg_translator.c:2390]: build_res_buf_from_sip_req():
out of memory; needs 453</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">Sep 19 11:42:46 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[data_lump.c:136]: insert_new_lump_after(): out of
memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">Sep 19 11:42:46 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core> [ut.c:163]:
as_asciiz(): Out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">Sep 19 11:42:57 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core> [ut.c:163]:
as_asciiz(): Out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">Sep 19 11:43:12 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[data_lump.c:361]: anchor_lump(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">Sep 19 11:43:12 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[data_lump.c:361]: anchor_lump(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">Sep 19 11:43:12 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[data_lump.c:306]: del_lump(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">Sep 19 11:43:12 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[data_lump.c:306]: del_lump(): out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">Sep 19 11:43:12 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[msg_translator.c:326]: received_builder(): out of
memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">Sep 19 11:43:12 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[msg_translator.c:326]: received_builder(): out of
memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">Sep 19 11:43:12 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[msg_translator.c:326]: received_builder(): out of
memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">Sep 19 11:43:16 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[data_lump.c:136]: insert_new_lump_after(): out of
memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">Sep 19 11:43:16 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core> [ut.c:163]:
as_asciiz(): Out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">Sep 19 11:44:18 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[data_lump.c:161]: insert_new_lump_before(): out of
memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">Sep 19 11:45:07 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core> [ut.c:163]:
as_asciiz(): Out of memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">Sep 19 11:45:30 ua-proxy-01
/sbin/kamailio[20218]: ERROR: acc [acc_extra.c:267]:
extra2strar(): extra2strar: out of memory.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">Sep 19 11:45:43 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[data_lump.c:186]: insert_subst_lump_after(): out of
memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif">Sep 19 11:45:52 ua-proxy-01
/sbin/kamailio[20218]: ERROR: <core>
[data_lump.c:161]: insert_new_lump_before(): out of
memory</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New ,serif",serif"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span
style="font-size:12.0pt"><br>
<br>
</span><o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list<o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="mailto:sr-users@lists.sip-router.org">sr-users@lists.sip-router.org</a><o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span
style="font-size:12.0pt"> </span><o:p></o:p></p>
<pre>-- <o:p></o:p></pre>
<pre>Daniel-Constantin Mierla<o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="http://twitter.com/#%21/miconda">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" href="http://www.linkedin.com/in/miconda">http://www.linkedin.com/in/miconda</a><o:p></o:p></pre>
<pre>Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - <a moz-do-not-send="true" href="http://www.asipto.com">http://www.asipto.com</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif;mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<pre>-- <o:p></o:p></pre>
<pre>Daniel-Constantin Mierla<o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="http://twitter.com/#%21/miconda">http://twitter.com/#!/miconda</a> - <a moz-do-not-send="true" href="http://www.linkedin.com/in/miconda">http://www.linkedin.com/in/miconda</a><o:p></o:p></pre>
<pre>Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - <a moz-do-not-send="true" href="http://www.asipto.com">http://www.asipto.com</a><o:p></o:p></pre>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla
<a class="moz-txt-link-freetext" href="http://twitter.com/#!/miconda">http://twitter.com/#!/miconda</a> - <a class="moz-txt-link-freetext" href="http://www.linkedin.com/in/miconda">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - <a class="moz-txt-link-freetext" href="http://www.asipto.com">http://www.asipto.com</a></pre>
</body>
</html>