[rsyslog] Thinking about syslog forwarding infrastructure

[Matt Hellman]

Thanks for the reply Rainer.

>> 1) What kind of system [rough estimate] would I need for the main
>> collector if assume 200 million syslog messages per day and peak that
>> is triple that average rate (~7000 eps)?
>
> Quite honestly: I don't know. Which rules you carry out has a big
> effect. But I have no real good big deployment numbers. The old game:
> everyone is interested in them, no-one conveys them (hint: let me know
> if you have some ;)).

crap. well, I can probably test this easily enough myself.  I just
feel better knowing that someone has already done it.

>>       A2) start intelligently dropping messages beyond a given threshold
>> (i.e. start dropping events matching this regex)
>
> not yet, but an interesting idea

well, regex wouldn't be the only "intelligent" way to drop messages.
I suppose anything that isn't arbitrary might be considered
intelligent. Currently this is done based on priority, which won't
work well for us because we use a product (Snare) that converts
windows events into syslog that all have the same priority.  FWIW,
this is a common way for SEM products to collect Windows events.

>>       B) allow me to alert someone that this is occurring (is written to
>> log file, etc)
>
> mmhhh... not really. That's another interesting idea, and it should be
> simple to enable. It conveys that to the debug log, but does not emit a
> user message.

I was thinking about this and I don't necessarily need the product to
emit something directly to a user, if that's what you mean.  I plan to
buffer to disk. Can I create a process to monitor the queue files or
something --warning: I have printed but at best skimmed many of the
docs you reference;-)

> In any case, I think there are a couple of docs you need to read and
> *understand* for this scale of deployment. Ask if you do not understand
> them - I have written them and may have left too much out just out of
> habit ;)

re: doc links.  Thanks. I was being lazy and trying to avoid having to
read them prematurely;-) I think I'm too the point where I believe
rsyslog can theoretically deliver on my requirements though. It's time
to dig in.