<div dir="ltr">Hi Shankar, <div><br></div><div>I will setup and test case and try to reproduce as soon as I can. I suspect I will be able to get back to you early next week only</div><div><br></div><div>Cheers</div><div>Jason</div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jan 23, 2014 at 8:48 AM, Shankar <span dir="ltr"><<a href="mailto:shankar.rk@plintron.com" target="_blank">shankar.rk@plintron.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Hi,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">From our repeated load tests what we can conclude irrespective of the number of simultaneous calls, there is always this error “t_continue: transaction not found” occurring. <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">If I run say 20 cps, then after running 5000 calls, we observe exactly 20 calls failing with the above error. We doubt that there is something happening during a particular point in time (for a second) which impacts the saving of those new transactions into shared memory. <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">For 10 cps run, we observe exactly 10 call failures. We repeated with different cps and found that error is exactly equal to the cps being run. <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Any configuration we are missing. Anyone can help?<u></u><u></u></span></p>
<div><div class="h5"><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Shankar<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<div><div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in"><p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Shankar [mailto:<a href="mailto:shankar.rk@plintron.com" target="_blank">shankar.rk@plintron.com</a>] <br>
<b>Sent:</b> Tuesday, January 21, 2014 3:09 PM<br><b>To:</b> 'Jason Penton'; 'SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List'<br><b>Subject:</b> FW: [SR-Users] FW: Regd. t_suspend() and t_continue()<u></u><u></u></span></p>
</div></div><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Hi Jason,<u></u><u></u></span></p><p class="MsoNormal">
<span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Below is our config, <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">route[LOCATION] {<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> if(is_method("INVITE"))<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> {<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> if(!route(FROMCSCF))<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> {<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> setflag(FLT_ACC);<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> setflag(FLT_ACCFAILED);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> dlg_manage();<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> dlg_setflag("4");<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> async_route("RESUME", "1");<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> exit;<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> }<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> }<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">}<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">route[RESUME]<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">{<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> route(TO_LOCATION); // here t_relay to REGISTRAR is done for user lookup.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> exit;<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">}<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Shankar<u></u><u></u></span></p><div style="border:none;border-bottom:solid windowtext 1.0pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p></div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p>Date: Tue, 21 Jan 2014 11:14:21 +0200<u></u><u></u></p><p>From: Jason Penton <<a href="mailto:jason.penton@smilecoms.com" target="_blank">jason.penton@smilecoms.com</a>><u></u><u></u></p><p>To: "Kamailio (SER) - Users Mailing List"<u></u><u></u></p>
<p> <<a href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a>><u></u><u></u></p><p>Subject: Re: [SR-Users] FW: Regd. t_suspend() and t_continue()<u></u><u></u></p><p>Message-ID:<u></u><u></u></p>
<p> <<a href="mailto:CAE=KcrghqJHgnGDxqS1fYvUzM=HqRAcKWfEAsNJjm8xUDCq68w@mail.gmail.com" target="_blank">CAE=KcrghqJHgnGDxqS1fYvUzM=HqRAcKWfEAsNJjm8xUDCq68w@mail.gmail.com</a>><u></u><u></u></p><p>Content-Type: text/plain; charset="iso-8859-1"<u></u><u></u></p>
<p><u></u> <u></u></p><p>We use it heavily, but not using the async module - we use it directly from the IMS code.<u></u><u></u></p><p><u></u> <u></u></p><p>Can you please provide your config (or a relevant snippet) file so I can see what exactly you are testing/trying to do<u></u><u></u></p>
<p><u></u> <u></u></p><p>Cheers<u></u><u></u></p><p>jason<u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<div><div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in"><p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Shankar [<a href="mailto:shankar.rk@plintron.com" target="_blank">mailto:shankar.rk@plintron.com</a>] <br>
<b>Sent:</b> Tuesday, January 21, 2014 2:25 PM<br><b>To:</b> 'SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List'<br><b>Subject:</b> RE: [SR-Users] FW: Regd. t_suspend() and t_continue()<u></u><u></u></span></p>
</div></div><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Hi,<u></u><u></u></span></p><p class="MsoNormal">
<span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Anyone who had used t_suspend() and t_continue() can share the performance details?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I tried async module with one sec sleep time. I tried only 5 calls per second but still it was not successful. After sometime I see below logs,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d;background:yellow">Jan 21 13:51:55 PLT-RA-RD-W167A PCscf[16520]: ERROR: tm [t_suspend.c:128]: t_continue(): ERROR: t_continue: transaction not found<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d;background:yellow">Jan 21 13:52:49 PLT-RA-RD-W167A last message repeated 15 times<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d;background:yellow">Jan 21 13:59:38 PLT-RA-RD-W167A last message repeated 12 times<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d;background:yellow">Jan 21 14:13:03 PLT-RA-RD-W167A last message repeated 5 times</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Any configuration changes can help here?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Shankar<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<div><div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in"><p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Shankar [<a href="mailto:shankar.rk@plintron.com" target="_blank">mailto:shankar.rk@plintron.com</a>] <br>
<b>Sent:</b> Wednesday, January 15, 2014 1:26 PM<br><b>To:</b> 'Jason Penton'<br><b>Cc:</b> 'SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List'<br><b>Subject:</b> RE: [SR-Users] FW: Regd. t_suspend() and t_continue()<u></u><u></u></span></p>
</div></div><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Hi Jason,<u></u><u></u></span></p><p class="MsoNormal">
<span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I am using 4.0.2 <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Shankar<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in"><p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Jason Penton [<a href="mailto:jason.penton@smilecoms.com" target="_blank">mailto:jason.penton@smilecoms.com</a>] <br>
<b>Sent:</b> Wednesday, January 15, 2014 1:21 PM<br><b>To:</b> Shankar<br><b>Cc:</b> SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List<br><b>Subject:</b> Re: [SR-Users] FW: Regd. t_suspend() and t_continue()<u></u><u></u></span></p>
</div><p class="MsoNormal"><u></u> <u></u></p><div><p class="MsoNormal">Hi Shankar,<u></u><u></u></p><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal">What version of Kamailio are you running? Kamailio -V<u></u><u></u></p>
</div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal">Cheers<u></u><u></u></p></div><div><p class="MsoNormal">Jason<u></u><u></u></p></div></div><div><p class="MsoNormal" style="margin-bottom:12.0pt">
<u></u> <u></u></p><div><p class="MsoNormal">On Wed, Jan 15, 2014 at 6:58 AM, Shankar <<a href="mailto:shankar.rk@plintron.com" target="_blank">shankar.rk@plintron.com</a>> wrote:<u></u><u></u></p><div><div><p>Hi Jason,<u></u><u></u></p>
<p> <u></u><u></u></p><p>Please find below my response inline,<u></u><u></u></p><div><p> <u></u><u></u></p><p> <u></u><u></u></p><p>I have some questions for you as we have used suspend/continue quite a lot in the IMS code and don't have any leaks.<u></u><u></u></p>
<p> <u></u><u></u></p><p>Firstly, why are you using pkg_mem for your hash_id and label? Remember that you will be in 2 different processes in the suspend and continue portions of the code... so pkg_mem will not work - you should use shm_mem instead.<u></u><u></u></p>
<p> <u></u><u></u></p></div><p><span style="background:yellow">[Shankar] We use pkg_mem because we are invoking t_continue from the same process ( using thread ).</span><u></u><u></u></p><div><p> <u></u><u></u></p><p>Secondly, how are you using top to tell that you have a leak? Kamailio's memory is internally managed.<u></u><u></u></p>
<p> <u></u><u></u></p></div><p><span style="background:yellow">[Shankar] After running for say 20minutes, we get out of shared memory error. Also in top output we observed incremental increase in the shared usage of shared memory for the process.</span><u></u><u></u></p>
<div><p> <u></u><u></u></p><p>Cheers<u></u><u></u></p><p>Jason<u></u><u></u></p><p> <u></u><u></u></p><p> <u></u><u></u></p><p>On Mon, Jan 13, 2014 at 1:29 PM, Shankar <<a href="mailto:shankar.rk@plintron.com" target="_blank">shankar.rk@plintron.com</a>> wrote:<u></u><u></u></p>
<p> <u></u><u></u></p><p>> Re-sending without the attachment.<u></u><u></u></p><p>> <u></u><u></u></p><p>> <u></u><u></u></p><p>> <u></u><u></u></p></div><p>> *From:* Shankar [<a href="mailto:shankar.rk@plintron.com" target="_blank">mailto:shankar.rk@plintron.com</a>]<u></u><u></u></p>
<p>> *Sent:* Monday, January 13, 2014 4:57 PM<u></u><u></u></p><p>> *To:* '<a href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a>'<u></u><u></u></p><p>> *Subject:* Regd. t_suspend() and t_continue()<u></u><u></u></p>
<div><div><p>> <u></u><u></u></p><p>> <u></u><u></u></p><p>> <u></u><u></u></p><p>> Hi,<u></u><u></u></p><p>> <u></u><u></u></p><p>> <u></u><u></u></p><p>> <u></u><u></u></p><p>> We are trying out the t_suspend() and t_continue() in our test setup. <u></u><u></u></p>
<p>> We are facing memory leak ( both shm and pkg as per top command results).<u></u><u></u></p><p>> <u></u><u></u></p><p>> <u></u><u></u></p><p>> <u></u><u></u></p><p>> Please find below the scenario,<u></u><u></u></p>
<p>> <u></u><u></u></p><p>> <u></u><u></u></p><p>> <u></u><u></u></p><p>> 1) Do a t_newtran()<u></u><u></u></p><p>> <u></u><u></u></p><p>> 2) Allocate pkg memory for hashid and label.<u></u><u></u></p>
<p>> <u></u><u></u></p><p>> 3) Call t_suspend()<u></u><u></u></p><p>> <u></u><u></u></p><p>> 4) Do t_continue() when async result is available<u></u><u></u></p><p>> <u></u><u></u></p><p>> 5) De-allocate pkg memory reserved for hashid and label<u></u><u></u></p>
<p>> <u></u><u></u></p><p>> 6) Do a t_relay() which forwards the sip message to another sip node.<u></u><u></u></p><p>> <u></u><u></u></p><p>> <u></u><u></u></p><p>> <u></u><u></u></p><p>> In the step (6) above, we see t_newtran() allocates one more time <u></u><u></u></p>
<p>> shared memory for the same transaction.<u></u><u></u></p><p>> <u></u><u></u></p><p>> <u></u><u></u></p><p>> <u></u><u></u></p><p>> We tried t_release() after step (4) to release the transaction as<u></u><u></u></p>
<p>> t_relay() anyways allocates new shared memory. Nothing helped.<u></u><u></u></p><p>> <u></u><u></u></p><p>> <u></u><u></u></p><p>> <u></u><u></u></p><p>> Please let me know what are the logs you would require to debug the same.<u></u><u></u></p>
<p>> I am attaching syslog for this run.<u></u><u></u></p><p>> <u></u><u></u></p><p>> <u></u><u></u></p><p>> <u></u><u></u></p><p>> Regards,<u></u><u></u></p><p>> <u></u><u></u></p><p>> Shankar<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p></div></div></div></div></div><p class="MsoNormal"><br><br clear="all"><u></u><u></u></p><div><p class="MsoNormal"><u></u> <u></u></p></div><p class="MsoNormal">-- <u></u><u></u></p>
<div><div><table border="0" cellspacing="0" cellpadding="0" width="600" style="width:6.25in"><tbody><tr><td valign="top" style="padding:0in 0in 0in 0in"></td></tr></tbody></table><p class="MsoNormal"><span><u></u> <u></u></span></p>
<table border="0" cellspacing="0" cellpadding="0"><tbody><tr><td style="padding:7.5pt 0in 0in 11.25pt"><p class="MsoNormal" style="line-height:13.5pt"><strong><span style="font-family:"Arial","sans-serif";color:#75b343">Jason Penton</span></strong><span style="font-family:"Arial","sans-serif";color:#75b343"><u></u><u></u></span></p>
</td></tr><tr><td style="padding:7.5pt 0in 0in 11.25pt"><p class="MsoNormal" style="line-height:13.5pt"><strong><span style="font-family:"Arial","sans-serif";color:#75b343">Senior Manager: Applications and Services</span></strong><span style="font-family:"Arial","sans-serif";color:#75b343"><u></u><u></u></span></p>
</td></tr><tr><td style="padding:7.5pt 0in 0in 11.25pt"><p class="MsoNormal" style="line-height:13.5pt"><strong><span style="font-family:"Arial","sans-serif";color:#86919a">Smile Communications Pty (Ltd)</span></strong><span style="font-family:"Arial","sans-serif";color:#86919a"><u></u><u></u></span></p>
</td></tr><tr><td style="padding:0in 0in 0in 0in"><table border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%"><tbody><tr><td width="27%" style="width:27.0%;padding:7.5pt 0in 0in 11.25pt"><p class="MsoNormal" style="line-height:13.5pt">
<strong><span style="font-family:"Arial","sans-serif";color:#86919a">Mobile:</span></strong><span style="font-family:"Arial","sans-serif";color:#86919a"><u></u><u></u></span></p></td>
<td width="73%" style="width:73.0%;padding:7.5pt 0in 0in 7.5pt"><p class="MsoNormal" style="line-height:13.5pt"><span style="font-family:"Arial","sans-serif";color:#86919a"><a href="tel:%2B27%20%280%29%2083%20283%207000" value="+27832837000" target="_blank">+27 (0) 83 283 7000</a><u></u><u></u></span></p>
</td></tr><tr><td style="padding:7.5pt 0in 0in 11.25pt"><p class="MsoNormal" style="line-height:13.5pt"><strong><span style="font-family:"Arial","sans-serif";color:#86919a">Skype:</span></strong><span style="font-family:"Arial","sans-serif";color:#86919a"><u></u><u></u></span></p>
</td><td style="padding:7.5pt 0in 0in 7.5pt"><p class="MsoNormal" style="line-height:13.5pt"><span style="font-family:"Arial","sans-serif";color:#86919a">jason.barry.penton<u></u><u></u></span></p></td>
</tr></tbody></table></td></tr><tr><td style="padding:7.5pt 0in 0in 11.25pt"><p class="MsoNormal" style="line-height:13.5pt"><span style="font-family:"Arial","sans-serif";color:#75b343"><a href="mailto:name.surname@smilecoms.com" target="_blank"><span style="color:#75b343;text-decoration:none">jason.penton@smilecoms.com</span></a><u></u><u></u></span></p>
</td></tr><tr><td style="padding:7.5pt 0in 0in 11.25pt"><p class="MsoNormal" style="line-height:13.5pt"><span style="font-family:"Arial","sans-serif";color:#75b343"><a href="http://www.smilecoms.com/" target="_blank"><span style="color:#75b343">www.smilecoms.com</span></a><br>
<img border="0" width="420" height="144" src="http://196.33.227.129/~smlcoms/sigs/pty/images/smile_signature_07_09.jpg"><u></u><u></u></span></p></td></tr></tbody></table></div><p class="MsoNormal"><u></u> <u></u></p></div>
</div><pre>This email is subject to the disclaimer of Smile Communications at <a href="http://www.smilecoms.com/disclaimer" target="_blank">http://www.smilecoms.com/disclaimer</a><u></u><u></u></pre><pre><u></u> <u></u></pre>
<pre><u></u> <u></u></pre><pre><u></u> <u></u></pre></div></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><div><table border="0" cellspacing="0" cellpadding="0" width="600" style="width:450pt"><tbody><tr><td valign="top" style="padding:0cm"><p></p></td>
</tr></tbody></table><table width="320px" border="0" cellpadding="0" cellspacing="0" style="font-family:monospace"><tbody><tr><td style="font-family:Arial,sans-serif;font-size:16px;color:rgb(117,179,67);line-height:18px;padding-left:15px;padding-top:10px">
<strong>Jason Penton</strong></td></tr><tr><td style="font-family:Arial,sans-serif;font-size:16px;color:rgb(117,179,67);line-height:18px;padding-left:15px;padding-top:10px"><strong>Senior Manager: Applications and Services</strong></td>
</tr><tr><td style="font-family:Arial,sans-serif;font-size:16px;color:rgb(134,145,154);line-height:18px;padding-left:15px;padding-top:10px"><strong>Smile Communications Pty (Ltd)</strong></td></tr><tr><td><table cellpadding="0" cellspacing="0" width="100%">
<tbody><tr><td width="27%" style="font-family:Arial,sans-serif;font-size:16px;color:rgb(134,145,154);line-height:18px;padding-left:15px;padding-top:10px"><strong>Mobile:</strong></td><td width="73%" style="font-family:Arial,sans-serif;font-size:16px;color:rgb(134,145,154);line-height:18px;padding-left:10px;padding-top:10px">
+27 (0) 83 283 7000</td></tr><tr><td style="font-family:Arial,sans-serif;font-size:16px;color:rgb(134,145,154);line-height:18px;padding-left:15px;padding-top:10px"><strong>Skype:</strong></td><td style="font-family:Arial,sans-serif;font-size:16px;color:rgb(134,145,154);line-height:18px;padding-left:10px;padding-top:10px">
jason.barry.penton</td></tr></tbody></table></td></tr><tr><td style="font-family:Arial,sans-serif;font-size:16px;color:rgb(117,179,67);line-height:18px;padding-left:15px;padding-top:10px"><a href="mailto:name.surname@smilecoms.com" style="font-family:Arial;color:rgb(117,179,67);text-decoration:none" target="_blank">jason.penton@smilecoms.com</a></td>
</tr><tr><td style="font-family:Arial,sans-serif;font-size:16px;color:rgb(117,179,67);line-height:18px;padding-left:15px;padding-top:10px"><a href="http://www.smilecoms.com/" style="font-family:Arial;color:rgb(117,179,67)" target="_blank">www.smilecoms.com</a><br>
<img src="http://196.33.227.129/~smlcoms/sigs/pty/images/smile_signature_07_09.jpg" width="420" height="144"><br></td></tr></tbody></table></div><br></div>
</div>
<pre>This email is subject to the disclaimer of Smile Communications at http://www.smilecoms.com/disclaimer