[rsyslog] does rsyslog supports data analytic

[david at lang.hm]

On Thu, 10 Sep 2009, Josh Zhao wrote:

> hi all,      i want to receive each syslog msg then input it into my special
> processing module.after processing the data,output the new data into
> database.of course,the raw data we must keep it into files. can anyone give
> me some suggestions?

would you not just list two destinations, one to the place you want the 
raw data archived and one to the processing module?



I have a very high volume of logs (>300M/day), so I roll the logs every 5 
min with this script

#!/bin/sh
#

PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin

umask 022
year=`date +%Y`
month=`date +%m`
day=`date +%d`
fdate=`date +%Y%m%d.%H%M`
logroot=/var/log
logroll=$logroot/oldlogs

cd $logroot
mkdir -p $logroll/$year/$month/$day >/dev/null 2>/dev/null
mv messages messages.$fdate
mv messages.$fdate $logroll/$year/$month/$day/messages.$fdate
mv /usr/local/bin/ita/system/itascan1a-p/winlogs 
/usr/local/bin/ita/system/itascan1a-p/winlogs.0
pkill -HUP syslogd
pkill -HUP syslog-ng
#pkill win-dump
gzip -9 $logroll/$year/$month/$day/messages.$fdate



>  PS: i browse the git source code, but i can't understand why the
> Experimental-lockfree<http://git.adiscon.com/?p=rsyslog.git;a=shortlog;h=refs/heads/Experimental-lockfree>
> is
> not adopted?

I believe that it boils down to complications in being sure that there are 
no bugs, and the fact that even without that there has been a LOT of room 
for improvement from the early 3.x timeframe to the current 5.x version.

I expect that after the current round of improvements are settled that 
aspect of things will get reexamined.

David Lang