[rsyslog] filter logger tags from syslog

[david at lang.hm]

On Thu, 26 Nov 2009, Jose Sanchez wrote:

> Hello,
>
> Thanks again for the great response.
> It's actually working! rsyslog is removing the "logger:" thing and all 
> the nasty stuff from it automatically, how come? Is it because we are 
> not adding any tag in the template? Im still not understanding how 
> rsyslog removes the logger thing.
>
> Ok, Im now getting the proper output and like David said Im now getting 
> issues with filtering the apache logs with all the rsyslog messages.

> I've tried to use the following filter but for some reason is not 
> working and Im not 100% if this is the best solution to use,
>
> This is how I had set it up,
>
> $template line,"%msg%\n"
> if $msg contains 'GET' then /var/log/apache.test.log;line
> *.* /var/log/test.log;line
>
> Not sure if Im on the right path, any help will be appreciated.
> I have also tried the "if" sentence without specifying the template name.


when rsyslog receives a message it parses it. the message over the wire 
hasn't changed (still has the timestamp, servername, logger: etc), but 
rsyslog puts those parts into the seperate variables and puts what is left 
of the message into the %msg% variable.

so when you change the template from the default of
%timestamp% %hostname% %syslogtag%%msg%
to just
%msg%

the log file has just the part you care about.

now for the filtering.

you could do
:%programname, isequal, "logger" /var/log/apache.test.log;line

(as I understand it, this format is a bit more efficiant for rsyslog than 
the equivalent of 
if $programname eq "logger" then /var/log/apache.test.log;line
)

I would actually suggest that you use the perl script that you posted, and 
filter for programname equal to "apache_syslog", filtering on just 
'logger' means that you can't use logger for anything else.

you don't want to just filter for 'GET' as there are a bunch of log files 
that won't have GET in them

David Lang


> Many Thanks.
>
> --- On Thu, 11/26/09, david at lang.hm <david at lang.hm> wrote:
>
>> From: david at lang.hm <david at lang.hm>
>> Subject: Re: [rsyslog] filter logger tags from syslog
>> To: "rsyslog-users" <rsyslog at lists.adiscon.com>
>> Date: Thursday, November 26, 2009, 6:38 PM
>> On Thu, 26 Nov 2009, Jose Sanchez
>> wrote:
>>
>>> Hello,
>>>
>>> I appreciate all the responses.
>>> Im not sure how can I can acconplish options 1) and 2)
>> automatically.
>>> For option 3) the thing is I need "combined" log type
>> so I cannot reform this.
>>>
>>> Im trying to centralize an access_log file from one
>> server to the rsyslog server and I need to completely remove
>> the tags I mentioned on my previous post.
>>> I have also tried using a perl script mentioned at the
>> botton of this email, but it salso arriving with a tag,
>> "apache_syslog:" as showed below,
>>>
>>> "apache_syslog: XXX.XXX.XXX.XXX - -
>> [26/Nov/2009:18:23:02 -0600] \"GET /.."
>>>
>>> Basically, this log will be parsed by awstats which is
>> pretty much stricted with the log format so that's why I
>> need a clean log sent from the apache server to the rsyslog
>> server.
>>
>> don't forget that you need to filter these messages into a
>> seperate file,
>> otherwise you will have your apache combined log messages
>> mixed with other
>> syslog messages (which will really confuse awstats)
>>
>> option 1 is what Rainer suggested
>>
>> option 2 is to run the log through another step before
>> awstats runs,
>> something along the lines of
>>
>> cut -c 16- file |cut -f 3- -d ' ' |awstats
>>
>> the first cut removes the timestamp (always 15 characters,
>> but with a
>> variable number of spaces in it), the second cut removes
>> the servername
>> and the syslog tag ('logger:' in your first example)
>>
>> David Lang
>>
>>> Thank you very much for all the help.
>>>
>>> Below is the Perl script:
>>>
>>> #!/usr/local/bin/perl
>>> # script: apache-access-logger
>>>
>>> use Sys::Syslog;
>>> $SERVER_NAME = shift || '';
>>>
>>> $PRIORITY = 'info';
>>> $FACILITY = 'local1';
>>>
>>> Sys::Syslog::setlogsock('unix');
>>> openlog ($SERVER_NAME,'ndelay', $FACILITY);
>>>
>>> while (<>) {
>>>   chomp;
>>>   syslog($PRIORITY,$_);
>>> }
>>> closelog;
>>>
>>> --- On Thu, 11/26/09, david at lang.hm <david at lang.hm>
>> wrote:
>>>
>>>> From: david at lang.hm <david at lang.hm>
>>>> Subject: Re: [rsyslog] filter logger tags from
>> syslog
>>>> To: "rsyslog-users" <rsyslog at lists.adiscon.com>
>>>> Date: Thursday, November 26, 2009, 2:21 AM
>>>> On Wed, 25 Nov 2009, Jose Sanchez
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I've been using classic syslog for
>> centralizing apache
>>>> access logs from one server to a remote syslog
>> server, the
>>>> thing is syslog adds some nasty tags before the
>> lines in the
>>>> access logs and I cant get them off, ie:
>>>>>
>>>>> "Nov 25 21:25:37 server1 logger:"
>>>>>
>>>>> I would like to know if rsyslog has the option
>> to
>>>> filter this kind of stuff, I just want to have the
>> logs sent
>>>> to the syslog server exactly like I was saving
>> them in a
>>>> local access.log file.
>>>>>
>>>>> Thanks in advance.
>>>>
>>>> 'logger:' is added by the logger program that
>> apache is
>>>> using to send the
>>>> logs to syslog.
>>>>
>>>> a properly formatted syslog message will include
>> a
>>>> timestamp and what
>>>> server it came from (note that the apache logs do
>> _not_
>>>> tell you what
>>>> virtual server the log comes from, it usually uses
>> a
>>>> different file for
>>>> each log, so when you mix them into syslog you
>> won't be
>>>> able to tell them
>>>> apart)
>>>>
>>>> so you have three basic options
>>>>
>>>> 1. let logger do it's default thing and then use
>> a
>>>> formatting command to
>>>> strip off the 'syslogie' parts to get back to the
>> apache
>>>> default in the
>>>> file
>>>>
>>>> 2. leave the 'syslogie' parts in when you write it
>> to a
>>>> file and have your
>>>> analysis tool strip them out
>>>>
>>>> 3. reformat the apache log message so that you put
>> useful
>>>> information in
>>>> the 'syslogie' parts of the message.
>>>>
>>>> you can move the timestamp to the beginning (you
>> can do
>>>> this with or
>>>> without the timezone, the format obviously
>> differs
>>>> slightly)
>>>>
>>>> you can put the name of the virtual host in the
>> server
>>>> field
>>>>
>>>> you can replace 'logger:' with something like
>> apache[80]:
>>>> or apache[443]:
>>>>
>>>> I am going to be setting up something along the
>> lines of #3
>>>> in the next
>>>> few weeks. I figure I will also want to tinker
>> with other
>>>> things in the
>>>> log message. there are items that apache can log,
>> but does
>>>> not log by
>>>> default (I believe that how long it took to
>> process the
>>>> request is one of
>>>> these), and also since syslog defaults to limiting
>> log
>>>> messages to 1-2K
>>>> (depending on your impementation), there are some
>> fields
>>>> that I would want
>>>> to move late in the message so that if they get
>> very long
>>>> they don't cause
>>>> other fields to be lost due to truncation (URL and
>> referrer
>>>> fields can be
>>>> several K long by themselves)
>>>>
>>>> David Lang
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>