[rsyslog] syslog server and reports

[david at lang.hm]

On Sun, 6 Sep 2009, Israel Garcia wrote:

> I have some debian lenny servers sending their logs (via TCP) to a
> central rsyslog server.
> Every remote servers has at /etc/rsyslog.conf:
>
> *.*   @@IP_CENTRAL_SERVER
>
> So, I can see in the central syslog server all  logs without problems.
> I'm looking for a single and simple report, like logwatch for example
> who process all logs and send me in ONE mail  or on ONE html page all
> resume info of all logs. I tried with logwatch and I didn't get this
> report I'm looking for.
>
> My question is?
> Is there any tool, script, app, etc which I run on the syslog server
> and give me the information of all servers in a way as simple as
> possible? Maybe in a single resume mail separated by a line for
> example?

there are a lot of products and projects out there to analyse logs and 
generate reports.

the problem is that what I am interested in seeing in a report may or may 
not match what you are interested in seeing.

also, most of this effort is taking place within originizations that have 
large volumes of logs, so distilling it down to a single report or e-mail 
requires that a lot of detail gets left out (and that goes back to exactly 
what you are interested in seeing)

when you say you want one page that shows you 'everything', what is it 
that you want to see?

are there particular messages that you want to see if they show up even 
once? or are you interested in simplifying log messages into categories 
and seeing how many messages in each category you have.

do you only care about the logs showing up sometime during the day? or are 
you interested in the trending of how many logs you get each second 
throughout the day (or anything in between)

unfortunantly the result of all these questions probably means that you 
will need to customize whatever you use to exactly the report that you 
want.

large companies can spend millions of dollars on systems and software to 
alert, report, and query their logs.

I am currently getting ~300M log messages/day and I distill it down to a 
single e-mail report that I look at (and generate additional reports with 
subsets of the data for other people to look at).


the best advice I ever got was to use the approach termed 'artificial 
ignorance'

start off with all your logs

for any log type that you can categorize create a summary of that log type 
(even if it's an unimportant log, count it because the number of times an 
unimportant thing happens can be important)

look at what's left and repeat the process

after several iterations of this you end up with the vast majority of your 
logs summarized and a report of "what's left", any new messages that you 
have never seen before  (which usually mean they are important) show up in 
the "what's left" bucket and tend to stand out

you do need to keep on top of this, upgrades to systems, new installs, 
etc cause new logs to show up, if you categorize and summarize them your 
final report stays small, if you let things slide for several months the 
final report can end up very large (and therefor useless)

David Lang