From david at lang.hm  Mon Sep 15 07:44:22 2008
From: david at lang.hm (david at lang.hm)
Date: Sun, 14 Sep 2008 22:44:22 -0700 (PDT)
Subject: [relp] UDP relp
Message-ID: <alpine.DEB.1.10.0809142230450.19489@asgard.lang.hm>

I joined after the thread died down, but I have a couple of comments

1. relp needs to implement timeouts anyway, even on TCP, so I don't see it 
a major problem to hae to have them for UDP.

2. you asked why would anyone want RELP over UDP and I have a couple of 
thoughts

first off if the messages are sent via udp there is no need to have the 
connect calls, which simplifies using relp slightly

second, in cases where you need more then one server to recieve and 
process messages due to volume, there are some techniques that are 
available if each message is seperate that aren't available if they are 
inside a single TCP connection.

if you modify the sender to use a different source port for each message 
(a simple close/open of the udp socket will let the OS do this for you), 
then you can use a shared IP address across the recieving machines, along 
with some iptables magic to send the logs to multiple machines but only 
have one of them process the logs (see http://www.linux-ha.org/ClusterIP
for a detailed example). this approach can also handle tcp connections, 
but you would have to have multiple tcp connections to do it, which would 
be more complications.

since the acknowledgement and timeouts are being done at the application 
layer anyway, there isn't really much benifit from having tcp do them as 
well.

finally, sending the messages out via udp and tracking the responses at 
the application layer puts us only a short distance away from being able 
to use multicast to send the message to multiple systems and wait for 
acknowledgements from all of them.

In my case I need to send my logs to four different sets of machines.

Set 1. archiveing (recieves messages and stores them in case we need to 
dig them up in the next three years)

set 2. commercial real-time alerting (management is more comfortable with 
commercial software)

set 3. opensource real-time alerting (us techies think that we can do more 
with opensource software then a commercial black-box approach)

set 4. database based search tool for the recent past

now, we can do this today by having four output queues and sending each 
message four times, but as the traffic rate climbs we will start to run 
into limits that will cause problems (if nothing else this means we can 
only do ~200Mb with gig-ethernet), being able to send the message once and 
get four acknowlegements back would be a significant savings.

David Lang


From david at lang.hm  Mon Sep 15 07:44:22 2008
From: david at lang.hm (david at lang.hm)
Date: Sun, 14 Sep 2008 22:44:22 -0700 (PDT)
Subject: [relp] UDP relp
Message-ID: <alpine.DEB.1.10.0809142230450.19489@asgard.lang.hm>

I joined after the thread died down, but I have a couple of comments

1. relp needs to implement timeouts anyway, even on TCP, so I don't see it 
a major problem to hae to have them for UDP.

2. you asked why would anyone want RELP over UDP and I have a couple of 
thoughts

first off if the messages are sent via udp there is no need to have the 
connect calls, which simplifies using relp slightly

second, in cases where you need more then one server to recieve and 
process messages due to volume, there are some techniques that are 
available if each message is seperate that aren't available if they are 
inside a single TCP connection.

if you modify the sender to use a different source port for each message 
(a simple close/open of the udp socket will let the OS do this for you), 
then you can use a shared IP address across the recieving machines, along 
with some iptables magic to send the logs to multiple machines but only 
have one of them process the logs (see http://www.linux-ha.org/ClusterIP
for a detailed example). this approach can also handle tcp connections, 
but you would have to have multiple tcp connections to do it, which would 
be more complications.

since the acknowledgement and timeouts are being done at the application 
layer anyway, there isn't really much benifit from having tcp do them as 
well.

finally, sending the messages out via udp and tracking the responses at 
the application layer puts us only a short distance away from being able 
to use multicast to send the message to multiple systems and wait for 
acknowledgements from all of them.

In my case I need to send my logs to four different sets of machines.

Set 1. archiveing (recieves messages and stores them in case we need to 
dig them up in the next three years)

set 2. commercial real-time alerting (management is more comfortable with 
commercial software)

set 3. opensource real-time alerting (us techies think that we can do more 
with opensource software then a commercial black-box approach)

set 4. database based search tool for the recent past

now, we can do this today by having four output queues and sending each 
message four times, but as the traffic rate climbs we will start to run 
into limits that will cause problems (if nothing else this means we can 
only do ~200Mb with gig-ethernet), being able to send the message once and 
get four acknowlegements back would be a significant savings.

David Lang