[rsyslog] Thinking about syslog forwarding infrastructure

[Rainer Gerhards]

Hi Matt,

On Tue, 2008-07-29 at 17:28 -0500, Matt Hellman wrote:
> In the interest of brevity, I'm leaving out details...if you need
> more, just ask.
> 
> We have a security event monitoring system that processes probably in
> the neighborhood of 100 million syslog messages per day (I know
> precisely how many events it has processed, but it doesn't break them
> down by protocol).  In some of our WAN sites, I would like to
> implement a local system that will receive all the local syslog
> messages and ship some/all back to the main collector on our LAN. The
> main collector on the LAN would receive the bulk of its message from
> other LAN systems.  Then the main collect would ship some/all to the
> SEM environment. I'm leaning towards using rsyslog for this task and
> have a few questions:
> 
> 1) What kind of system [rough estimate] would I need for the main
> collector if assume 200 million syslog messages per day and peak that
> is triple that average rate (~7000 eps)?

Quite honestly: I don't know. Which rules you carry out has a big
effect. But I have no real good big deployment numbers. The old game:
everyone is interested in them, no-one conveys them (hint: let me know
if you have some ;)).

> 
> 2) Can I enable rate limiting in a way that will:
> 	A1) start dropping messages beyond a given threshold

you can do that

> 
> 	A2) start intelligently dropping messages beyond a given threshold
> (i.e. start dropping events matching this regex)

not yet, but an interesting idea

> 
> 	B) allow me to alert someone that this is occurring (is written to
> log file, etc)

mmhhh... not really. That's another interesting idea, and it should be
simple to enable. It conveys that to the debug log, but does not emit a
user message.

In any case, I think there are a couple of docs you need to read and
*understand* for this scale of deployment. Ask if you do not understand
them - I have written them and may have left too much out just out of
habit ;)

First and foremost, you must understand rsyslog queues. They handle all
queueing and rate limiting. The relevant doc is:

http://www.rsyslog.com/doc-queues.html

Then, I suggest to have a quick look at some use cases. I probably is a
good idea to read the queue doc once, go over the use cases and the
re-read the queue doc with the use cases on your mind:

http://www.rsyslog.com/doc-queues.html
http://www.rsyslog.com/doc-rsyslog_reliable_forwarding.html
http://wiki.rsyslog.com/index.php/OffPeakHours

Finally, IMHO you need to think about syslog reliability and the service
level you can expect. This is not only true for rsyslog, but the other
folks don't tell you about the reliability issues. Read this:

http://blog.gerhards.net/2008/04/on-unreliability-of-plain-tcp-syslog.html

I hope this gets you started.

Rainer

> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog