<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD><TITLE>Message</TITLE>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.2800.1476" name=GENERATOR></HEAD>
<BODY>
<DIV><SPAN class=233392912-12042005><FONT face=Arial color=#0000ff
size=2>Forgive my ignorance, why is NAT a bottleneck? Are you refering to CPU
load or managibility? With alot of load balancers, NAT is the only way to
go.</FONT></SPAN></DIV>
<BLOCKQUOTE dir=ltr style="MARGIN-RIGHT: 0px">
<DIV></DIV>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left><FONT
face=Tahoma size=2>-----Original Message-----<BR><B>From:</B> Tina
[mailto:kramarv@yahoo.com] <BR><B>Sent:</B> Monday, April 11, 2005 11:37
AM<BR><B>To:</B> Java Rockx<BR><B>Cc:</B>
serusers@lists.iptel.org<BR><B>Subject:</B> Re: [Serusers] still no help - usrloc
synchronization<BR><BR></FONT></DIV>
<DIV>Pretty good news, Paul. I've only tested IP tunneling. It worked for me.
The problem I described was born in my thoughts. I've also read some
complaints from SIP-LVS users. The only configuration where Call-Id stickness
does not necessary is LVS-NAT. Unfortunately, NAT becomes bottleneck very
fast.</DIV>
<DIV><BR><BR><B><I>Java Rockx <javarockx@gmail.com></I></B> wrote:</DIV>
<BLOCKQUOTE class=replbq
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #1010ff 2px solid">Tina,<BR><BR>I
wish I had more information on that, but that LVS stuff is a black art to
me. One of our engineers whipped up an LVS configuration that seems quite
happy with the Call-ID "stickness" you described. All I can say right now is
that it is apparently possible to do this with 100% open source. (i'm
keeping my fingers crossed) :-)<BR><BR>Regards,<BR>Paul<BR><BR>
<DIV><SPAN class=gmail_quote>On Apr 10, 2005 11:03 PM, <B
class=gmail_sendername>Tina</B> <<A
href="mailto:kramarv@yahoo.com">kramarv@yahoo.com</A>> wrote:</SPAN>
<BLOCKQUOTE class=gmail_quote
style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">
<DIV>Paul,</DIV>
<DIV>thanks a lot, it helps a lot. The thing I do not understand - how you
are going to solve call-id stickness (see my another post). Since we're
also use LVS I had to find something proprietary, I am afraid this is not
a best solution, but the only I have today.</DIV>
<DIV><SPAN class=e id=q_1032f368bae8ae67_1>
<DIV>KRs,</DIV>
<DIV>Tina<BR><BR><B><I>Java Rockx <<A
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:javarockx@gmail.com"
target=_blank>javarockx@gmail.com</A>></I></B> wrote:</DIV>
<BLOCKQUOTE
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: rgb(16,16,255) 2px solid">Tina,<BR><BR>I
really don't know how the LVS server is configured because our network
guy that we acquired from RedHat set all that stuff up. I do know that
we basically have this set up:<BR><BR>I hope the formatting goes well
:-)<BR><BR>+-----------+
+----------+ +-------+
+---------+ +--------------------+<BR>| internet |------|
Cisco |------| LVS |-----| Cisco |----|
Application |<BR>| cloud
| | 3600
|
|
| | 3600 | | & DB
Servers |<BR>+-----------+
+----------+ +--------+
+---------+ +--------------------+<BR><BR>So the
"Application & DB Servers" box represents multiple servers (all are
dedicated to their service). These include SER, MySQL, Apache,
configuration management, monitoring, RTP proxies, etc.<BR><BR>MySQL is
active-active so we have two-way replication. We only have 2 MySQL
servers. And this is all we plan on ever having as MySQL is more than
capable of handling anything ser can throw at it (given the right
hardware).<BR><BR>Each ser server is 100% oblivious to other SER servers
except for when handling REGISTER messages. Here is basically the
ser.cfg for each SER server:<BR><BR>listen=<A
onclick="return top.js.OpenExtLink(window,event,this)"
href="http://10.3.0.221/" target=_blank>10.3.0.221</A> # this will be <A
onclick="return top.js.OpenExtLink(window,event,this)"
href="http://10.3.0.222/" target=_blank>10.3.0.222</A> on the peer sip
proxy<BR>modparam("usrloc", "db_mode", 1) # write-through<BR><BR>route
{<BR> if (method=="REGISTER")
{<BR>
if (src_ip==<A onclick="return top.js.OpenExtLink(window,event,this)"
href="http://10.3.0.221/" target=_blank>10.3.0.221</A>) { # ip of peer
ser
proxy<BR>
save_memory("location");<BR>
break;<BR>
} else
{<BR>
save("location");<BR>
t_replicate("<A onclick="return top.js.OpenExtLink(window,event,this)"
href="http://10.3.0.222/"
target=_blank>10.3.0.222</A>");<BR>
break;<BR>
};<BR> };<BR>}<BR><BR>We have
identifed a deficiency in the usrloc module urecord.c file whereby when
using write-though mode the save_memory() function still attempt to
update MySQL. This is a very bad thing because it causes a primary key
violation on the location table.<BR><BR>Yesterday I posted a patch to
serdev to fix this, but today we identified another case where
save_memory() tried to insert in to the location table. I'll post a
revised patch to serdev as soon as I get a chance.<BR><BR>So here is
what happens. Assume I have two sip routers and two MySQL servers
as:<BR><BR>sip-01 is <A
onclick="return top.js.OpenExtLink(window,event,this)"
href="http://10.3.0.221/" target=_blank>10.3.0.221</A><BR>sip-02 is <A
onclick="re turn top.js.OpenExtLink(window,event,this)"
href="http://10.3.0.222/" target=_blank>10.3.0.222</A><BR>db-01 is <A
onclick="return top.js.OpenExtLink(window,event,this)"
href="http://10.2.0.21/" target=_blank>10.2.0.21</A><BR>db-02 is <A
onclick="return top.js.OpenExtLink(window,event,this)"
href="http://10.2.0.22/" target=_blank>10.2.0.22</A><BR><BR>Now let's
just assume that sip-01 is connected to db-01 and sip-02 is connected to
db-02.<BR><BR>When a SIP UA connects to a sip router (via LVS), assume
sip-01 in this example and REGISTERs, sip-01 will call save("location")
which updates/inserts to the location table. sip-01 then calls
t_replicate to sip-02. <BR><BR>sip-02 recieves the REGISTER message and
calls save_memory("location") to update its in-memory cache only. It
should never hit db-02.<BR><BR>Now db-01 eventually (within a second or
two) replicates to db-02 and now sip-02 is good to go, even in the event
of a restart.<BR><BR>Our difficulties mostly surrounded the fact th at
ser was inserting the physical IP of eth0 on the server as the top VIA
rather than the VIP as required. We solved this by using
record_route_preset() and passing the VIP as the IP. We also had to
bring up a dummy interface in order to get ser to honor the
VIP.<BR><BR>Hope this helps.<BR><BR>Regards,<BR>Paul<BR><BR><BR>
<DIV><SPAN class=gmail_quote>On Apr 8, 2005 7:19 PM, <B
class=gmail_sendername>Tina</B> <<A
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:kramarv@yahoo.com" target=_blank>kramarv@yahoo.com</A>>
wrote:</SPAN>
<BLOCKQUOTE class=gmail_quote
style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">
<DIV>Paul,</DIV>
<DIV>we are using LVS as well. It creates template connection
according to source Ip (e.g. UAC) and forwards to one of the tunneled
IP. I tested this a couple of weeks ago, ser was happy and inserted
VIP into via header, the only issue is to route the calls into right
server.</DIV>
<DIV>I agree, there is no such thing as SIP-level clustering. However,
how would you solve location problem by DB replication - 1) usrloc
does not update from DB....2) even though, I am afraid it can become
bottleneck and lock down users ...</DIV>
<DIV> </DIV>
<DIV>KRs,</DIV>
<DIV>Tina</DIV>
<DIV><SPAN>
<DIV> </DIV>
<DIV><BR><B><I>Java Rockx <<A
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:javarockx@gmail.com"
target=_blank>javarockx@gmail.com</A>></I></B> wrote:</DIV>
<BLOCKQUOTE
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: rgb(16,16,255) 2px solid">See
my inline comments.<BR><BR>Regards,<BR>Paul<BR><BR>
<DIV><SPAN class=gmail_quote>On Apr 8, 2005 1:08 PM, <B
class=gmail_sendername>Greger V. Teigre</B> <<A
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:greger@teigre.com"
target=_blank>greger@teigre.com</A>> wrote:</SPAN>
<BLOCKQUOTE class=gmail_quote
style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">
<DIV>Hi Tina,</DIV>
<DIV> </DIV>
<DIV>> I enjoy reading your posts, thanks a lot for the great
work you are<BR>> doing for ser users! <BR></DIV>
<DIV>Thanks :-)</DIV>
<DIV><BR>> I am going to consider that kind of scenario as well
(load balancer<BR>> returning Virtual IP address). I think for
such configuration server<BR>> side has to keep session
information and always forward a call to the<BR>> right
server. <BR></DIV>
<DIV>I believe the load balancer can be in front
and transparantly translate addresses in the SIP
messages:</DIV>
<DIV>User agent always sees a.b.c.d</DIV>
<DIV>
|</DIV>
<DIV>Load balancer with public IP a.b.c.d</DIV>
<DIV>
|
|</DIV>
<DIV>ser1 (<A
onclick="return top.js.OpenExtLink(window,event,this)"
href="http://10.0.0.2/"
target=_blank>10.0.0.2</A>)
ser2 (<A onclick="return top.js.OpenExtLink(window,event,this)"
href="http://10.0.0.3/" target=_blank>10.0.0.3</A>)</DIV>
<DIV> </DIV>
<DIV>
<DIV>Paul: You have a similar load-balancing solution, right? Do
you use a commerical load balancer that you can reveal the name
of? :-)</DIV></DIV></BLOCKQUOTE>
<DIV><BR>Our platform is 100% open source. Our load balancer is LVS.
I have no idea how it actually works. One if our engineers from
RedHat set all that networking stuff up and it's way beyond
me.<BR> </DIV><BR>
<BLOCKQUOTE class=gmail_quote
style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">
<DIV>The load balancer need to understand SIP (sort of) and use
something (ex. IP address of user agent) to select SER server to
send to and rewrite incoming to <A
onclick="return top.js.OpenExtLink(window,event,this)"
href="http://10.0.0.2/" target=_blank>10.0.0.2</A> and <A
onclick="return top.js.OpenExtLink(window,event,this)"
href="http://10.0.0.3/" target=_blank>10.0.0.3</A> respectively
and outgoing to a.b.c.d. ser1 and ser2 will happily believe
they are alone and communicate directly with user agent using
their own private addresses. Both should know about all
users (usrloc). Paul just recently posted a small patch for
a setup where they use two-way replication between
two</DIV></BLOCKQUOTE>
<DIV><BR>That small patch will need a slight revision. We discovered
through further testing that that patch only fixed attempts to
INSERT into the location table. We didn't cover the UPDATE route,
unfortunately. I believe Andrew has that patch done and is testing
it. So I'll post it to serdev as soon as we know it works -
possibly later today.<BR> </DIV><BR>
<BLOCKQUOTE class=gmail_quote
style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">
<DIV>mysql servers and where each SER instance will save all
REGISTERs it receives from user agents, but only save to memory
those replicated from the other SER peer. Klaus pointed out that
the same can be done by not doing mysql replication, but have one
DB for each SER and just use t_replicate() and save_noreply() (for
two servers, for more than two you need to use
forward_tcp()).</DIV>
<DIV> </DIV>
<DIV>I have heard complaints that this approach will generate
too much network traffic. Also, t_replicate() does not
guarantee delivery of the replication (forward_tcp is better, but
not fool proof). So, the "best" solution would
probably be to hope that Paul's</DIV></BLOCKQUOTE>
<DIV><BR>I fully agree. Replication at the sip proxy level, IMHO, is
not a good idea. From a purist point of view, replication is not a
function of a SIP router - nor should it be. It is a function of the
database and we believe that the MySQL replication is a better
approach from a speed and reliabilty point of view - not to mention
that it simplifies the tasks of the sip router.<BR> </DIV><BR>
<BLOCKQUOTE class=gmail_quote
style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">
<DIV>recent TCP/UDP patch for network access to FIFO commands get
included in the CVS. Then, a new replication module should use
XMLRPC or whatever protocol the patch is using to forward (in a
guaranteed mode) the usrloc information, which then would be
injected directly into SER's memory, just like save() would, and
of course in whatever db_mode you would like. The
replicaton module would have to keep a queue (of course, flushed
to a DB or something) in case of network problems, so that the
replication would be done (sooner or later) regardless of
problems.</DIV>
<DIV> Of course, this is a major undertaking, so
for most replication scenarioes, I would think the forward_tcp() +
save_noreply() would work just fine.</DIV>
<DIV> </DIV>
<DIV>g-)</DIV>
<DIV><BR>> "Greger V. Teigre" <<A
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:greger@teigre.com"
target=_blank>greger@teigre.com</A>> wrote:<BR>> I was
thinking about a load balancing scenario where the load<BR>>
balancer will replace the IP addresses. <BR>> g-)<BR>>
<BR>> ---- Original Message ----<BR>> From: Tina<BR>> To:
Greger V. Teigre<BR>> Sent: Wednesday, April 06, 2005 09:45
PM<BR>> Subject: Re: [Serusers] still no help - usrloc
synchronization<BR>> <BR>>> Thank you for givingme the
scenario with "restricted IP" NAT, I am<BR>>> starting to
find some acceptable solution.<BR>>> Unfortunately,
"one-public-IP" approach is not free from problems<BR>>>
also. If your SIP router inserts this "one-public-IP" into the
VIA<BR>>> header, the reply routing goes via wrong SIP
server...<BR>>> If your SIP server inserts its
real-IP-address - the scenario<BR>>> mentioned above is s
till not resolved.<BR>>> Any comments?<BR>>>
Tina<BR>>> <BR>>> "Greger V. Teigre" <<A
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:greger@teigre.com"
target=_blank>greger@teigre.com</A>> wrote:<BR>>> See
inline.<BR>>> <BR>>>> If you use DNS server for
load balancing... the client receives one<BR>>>> of your
domain IP addresses according to SRV. I don't see
the<BR>>>> problem <BR>>>> with a call here,
cause UAC asks t! he address only once (before<BR>>>>
sending INVITE). UAC already has the IP for BYE/reINVITEs. So
why<BR>>>> would you replicate INVITEs?<BR>>>
<BR>>> I would never replicate INVITEs, I would just make
sure that they are<BR>>> proxied through the correct SER
server (i.e. IP).<BR>>> <BR>>> The problems depends on
your setup. If you have SERs with different<BR>>> IPs, ex
UA1 has registered with server A and UA2 ha s registered
with<BR>>> server B: If UA2 wants to call UA1 and UA is
behind an IP restricted<BR>>> NAT, server A is stored in the
NAT table of the NAT in front of UA1.<BR>>> If server B
sends an INVITE to UA1, the INVITE will be refused by<BR>>>
UA1's NAT.<BR>>> This is why a "one
public IP" in front of a load balancing<BR>>> cluster
probably is a good way to go.<BR>>> <BR>>>> If you
use IPVS/LVS... I believe you can force SER to insert
it's<BR>>>> public IP into VIA,! so there is no problem
with replies. With<BR>>>> regard <BR>>>> to
another requests, I believe load balancer keeps
connection<BR>>>> template, then when another request
comes it would be forwarded to<BR>>>> the same
ser.<BR>>> <BR>>> Yes. There are different "keys" to
use to load balance SIP messages.<BR>>> One good way from a
NAT point of view is to use originating IP<BR>> >
address. What you must remember is that the problem is not
on the<BR>>> server side, but on the client side. The
NAT will in many situations<BR>>> stop incoming UDP packets
if the originating ip:port is not already<BR>>> stored in
the NAT table. The Via header does not matter for the
NAT.<BR>>> g-)<BR>>> <BR>>>> Any
comments?<BR>>>> <BR>>>> "Greger V. Teigre"
<<A onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:greger@teigre.com"
target=_blank>greger@teigre.com</A>> wrote:<BR>>>>
Yes, I believe that is so. But still you get a problem if the NAT
is<BR>>>> restricted, port-restricted or symmetric... The
best would be to<BR>>>> load>> balance and always
make sure that a given client is handled<BR>>>> through a
given <BR>>>> SER (REGISTER and INVITEs). That includes
forwarding INVITEs from<BR>>>> one <BR>>>> SER
to<BR>>>> another... OR you must load balance in front of
your servers with<BR>>>> one <BR>>>>
common<BR>>>> public IP.<BR>>>>
g-)<BR>>>> <BR>>>> Matt Schulte
wrote:<BR>>>>> Ack, I didn't even think about NAT.
Would these be added before it<BR>>>>> gets sent off
to the second proxy? ie:<BR>>>>> <BR>>>>>
if (!src_ip==<A
onclick="return top.js.OpenExtLink(window,event,this)"
href="http://blah.netlogic.net/"
target=_blank>blah.netlogic.net</A>) {<BR>>>>>
add_rcv_param();<BR>>>>> t_replicate("<A
onclick="return top.js.OpenExtLink(window,event,this)"
href="http://blah.netlogic.net/"
target=_blank>blah.netlogic.net</A>", "999");<BR>>>>>
};<BR>>>>> <BR>>>>> -----Original
Message-----<BR>>>>> From: Greger V. Teigre [mailto:<A
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:greger@tei gre.com"
target=_blank>greger@teigre.com</A>]<BR>>>>> Sent:
Tuesday, April 05, 2005 7:49 AM<BR>>>>> To: Matt
Schulte; <A onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:kramarv@yahoo.com"
target=_blank>kramarv@yahoo.com</A><BR>>>>> ! ; Cc: <A
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:serusers@lists.iptel.org"
target=_blank>serusers@lists.iptel.org</A><BR>>>>> Subject:
! Re: [Serusers] still no help - usrloc
synchronization<BR>>>>> <BR>>>>>
<BR>>>>> Well, you still have the NAT issues unless
you do load balancing<BR>>>>> and
your<BR>>>>> SER servers have the same public
IP.<BR>>>>> Have you looked at 0.9.0 nathelper
function add_rcv_param() ? It<BR>>>>> will add
received info to the contact header for the other SER
to<BR>>>>> process. Haven't really tried
yet...<BR>>>>> g-)<BR>>>>>
<BR>>>>> Matt Schulte wrote:<BR>>>>>>
I'm starting to lean this direction, using t_replicate and all.
I<BR>>>>>> could never get usrloc (db mode) to
function properly..<BR>>>>>> t_replicate
is<BR>>>>> <BR>>>>>> a dirty but very
effective workaround.<BR>>>>>>
<BR>>>>>> -----Original
Message-----<BR>>>>>> From: Greger V. Teigre
[mailto:<A onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:greger@teigre.com"
target=_blank>greger@teigre.com</A>]<BR>>>>>> Sent:
Saturday, April 02,! 2005 1:33 AM<BR>>>>>> To: <A
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:kramarv@yahoo.com"
target=_blank>kramarv@yahoo.com</A><BR>>>>>> Cc: <A
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:serusers@lists.iptel.org"
target=_blank>serusers@lists.iptel.org</A><BR>>>>>>
Subject: Re: [Serusers] still ! no help - usrloc
synchronization<BR>>>>>> <BR>>>>>>
<BR>>>>>> Have a look at this
thread:<BR>>>>>> <A
onclick="return top.js.OpenExtLink(window,event,this)"
href="http://lists.iptel.org/pipermail/serusers/2005-January/014669.html"
target=_blank>http://lists.iptel.org/pipermail/serusers/2005-January/014669.html</A><BR>>>>>>
g-)<BR>>>>>> <BR>>>>>> Java Rockx
wrote:<BR>>>>>>>
Tina,<BR>>>>>>> <BR>>>>>>> I
thought I saw you post the other day that you did not want
to<BR>>>>>>> use t_replicate(), however, this is
probably your best bet to<BR>>>>>>> getting
this<BR>>>>>> <BR>>>>>>> to work,
IMHO.<BR>>>>>>> <BR>>>>>>>
Regards,<BR>>>>>>>
Paul<BR>>>>>>> <BR>>>>>>> On
Apr 1, 2005 4:08 PM, Tina wrote:<BR>>>>>> !
>><BR>>>>>>> ! ;> Hi, please help me,
I'm stuck with it!!!!!<BR>>>>>>>> I am trying
to set up several sers with a shared MySQL
database<BR>>>>>>>> for location
service.<BR>>>>>>>>
<BR>>>>>>>> I set in each
ser.cfg:<BR>>>>>>>>
<BR>>>>>>>> modparam("usrloc", "db_mode",
2)<BR>>>>>>>>
modparam("usrloc",<BR>>>>>>>>
"db_url","<A>sql://ser:heslo@192.168.25.163/ser</A>")<BR>>>>>>>>
<BR>>>>>>>> and the servers are not
synchronized.<BR>>>>>>>> The I
set<BR>>>>>>>> modparam("usrloc", "db_mode",
2)<BR>>>>>>>>
<BR>>>>>>>> <BR>>>>>>>>
made UAC (Xlite) register to one of the
servers.<BR>>>>>>>> I see it via usrloc, but
there is no record in "location"
mySQL<BR>>>>>>>> table....So others do not
see the client and I'm unable to
make<BR>>>>>>>>
calls....<BR>>>>>>>>
<BR>>>>>>>> <BR>>>>>>>>
Please help how to work with usrloc and
mySQL...<BR>>>>>>>>
<BR>>>>>>>>
Tina,<BR>>>>>>>> software
engineer<BR>>>>>>>>
<BR>>>>>>>>
________________________________<BR>>>>>>>>
Do you Yahoo!?<BR>>>>>>>> Better first dates.
More second dates. Yahoo!
Personals<BR>>>>>>>>
<BR>>>>>>>> <BR>>>>>>>>
_______________________________________________<BR>>>>>>>>
Serusers mailing list<BR>>>>>>>> <A
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:serusers@lists.iptel.org"
target=_blank>serusers@lists.iptel.org</A><BR>>>>>>>>
<A onclick="return top.js.OpenExtLink(window,event,this)"
href="http://lists.iptel.org/mailman/listinfo/serusers"
target=_blank>http://lists.iptel.org/mailman/listinfo/serusers</A><BR>>>>>>>>
<BR>>>>>>>> <BR>>>>>>>>
<BR>>>>>>> <BR>>>>>>>
_______________________________________________<BR>>>>>>>
Serusers mailing list<BR>>>>>>> <A
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:serusers@lists.iptel.org"
target=_blank>serusers@lists.iptel.org</A><BR>>>>>>>
<A onclick="return top.js.OpenExtLink(window,event,this)"
href="http://lists.iptel.org/mailman/listinfo/serusers"
target=_blank>http://lists.iptel.org/mailman/listinfo/serusers</A>
<BR>>>>>> <BR>>>>>>
_______________________________________________<BR>>>>>>
Serusers mailing list<BR>>>>>> <A
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:serusers@lists.iptel.org"
target=_blank>serusers@lists.iptel.org</A> <A
onclick="return top.js.OpenExtLink(window,event,this)"
href="http://lists.iptel.org/mailman/listinfo/serusers"
target=_blank>http://lists.iptel.org/mailman/listinfo/serusers</A><BR>>>>
<BR>>>> <BR>>>> <BR>>>>
<BR>>>> Yahoo! Messenger<BR>>>> Show us what our
next emoticon should look like. Join the fun.<BR>>>
<BR>>> <BR>>> Do you Yahoo!?<BR>>> Better first
dates. More second dates. Yahoo! Personals<BR>> <BR>>
<BR>> Yahoo! Messenger<BR>> Show us what our next emoticon
should look like. Join the
fun.</DIV><BR>_______________________________________________<BR>Serusers
mailing list<BR><A
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:serusers@lists.iptel.org"
target=_blank>serusers@lists.iptel.org</A><BR><A
onclick="return top.js.OpenExtLink(window,event,this)"
href="http://lists.iptel.org/mailman/listinfo/serusers"
target=_blank>http://lists.iptel.org/mailman/listinfo/serusers</A><BR><BR><BR></BLOCKQUOTE></DIV><BR></BLOCKQUOTE></SPAN></DIV><SPAN>
<P></P>
<HR SIZE=1>
Do you Yahoo!?<BR><A
onclick="return top.js.OpenExtLink(window,event,this)"
href="http://us.rd.yahoo.com/my/navbar/sethp/*http://www.yahoo.com/r/hs"
target=_blank>Make Yahoo! your home page</A>
<P></P></SPAN></BLOCKQUOTE></DIV><BR></BLOCKQUOTE></SPAN></DIV><SPAN
class=ad>
<P>__________________________________________________<BR>Do You
Yahoo!?<BR>Tired of spam? Yahoo! Mail has the best spam protection around
<BR><A onclick="return top.js.OpenExtLink(window,event,this)"
href="http://mail.yahoo.com/" target=_blank>http://mail.yahoo.com</A>
</P></SPAN></BLOCKQUOTE></DIV><BR></BLOCKQUOTE>
<P>__________________________________________________<BR>Do You
Yahoo!?<BR>Tired of spam? Yahoo! Mail has the best spam protection around
<BR>http://mail.yahoo.com </P></BLOCKQUOTE></BODY></HTML>