From mbiebl at gmail.com  Mon Mar  1 00:00:57 2010
From: mbiebl at gmail.com (Michael Biebl)
Date: Mon, 1 Mar 2010 00:00:57 +0100
Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103981@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com>
	<aa76cdac1002251244y2be4e04egc7e0e3eb57494431@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com>
	<aa76cdac1002260355l26269784y8fb8ccd45caeaa12@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com>
	<aa76cdac1002260443p4c7744e2ha1c8c2372c741ebf@mail.gmail.com>
	<aa76cdac1002280515g72c07527mae8ebc08d4ecd404@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103980@GRFEXC.intern.adiscon.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103981@GRFEXC.intern.adiscon.com>
Message-ID: <aa76cdac1002281500g3d50db29rcd20fea1fbf08d48@mail.gmail.com>

2010/2/28 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>> OK, rather than trying to figure out what may be going on here, I'll
>> see that
>> I copy the old omfile code and create a new ompipe. So that will behave
>> exactly like it did before. The new optimizations are not so important
>> for
>> pipes.
>
> I have done this now. I would appreciate if you could checkout the v4-stable
> git head and try with that one. It should now have the exact same behavior as
> in 4.4.2 (the pipe code is now the same!). I tested on Debian and it worked
> like before.

I use $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
in my rsyslog.conf, but the new ompipe module does not seem to respect
that setting but always uses the high-resolution time format.

Other than that, I did not (yet) experience any blank xconsole windows
with v4.6.0-5-gf12a199 (git describe)

Just curious, when I fire up xconsole after boot, I get two log
messages immediately. Where do those log messages come from?

Michael


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From david at lang.hm  Mon Mar  1 03:15:57 2010
From: david at lang.hm (david at lang.hm)
Date: Sun, 28 Feb 2010 18:15:57 -0800 (PST)
Subject: [rsyslog] Feedback requested: inconsistency in config statements
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710397F@GRFEXC.intern.adiscon.com>
References: <001201cab868$d148da7c$100013ac@intern.adiscon.com>
	<alpine.DEB.2.00.1002280426420.15263@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA710397F@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.2.00.1002281346000.15263@asgard.lang.hm>

On Sun, 28 Feb 2010, Rainer Gerhards wrote:

> Hi David,
>
> yes, that makes much more sense to me. Let's call that filter expression tree
> for the time being (the parse tree used for normalization has some similar
> ideas, but a totally different scope and semantics).
>
> While I now see where you are heading, I fail to see how this could
> sufficiently easy be implemented for a real (and complex) configuration.
>
> Let's look at this example conf:
>
> mail.* /Var/log/mail.log
> if programname startswith abc log to /abc
> if programname startswith 'acc' and MSG contains 'error' log to /acc
> # we don't want debug messages from here on
> *.debug ~
> if programname startswith 'acc' or (programname contains 'bde' and (facility
>> 1 and
> facility < 5)) log to @1.1.1.1
> if programname startswith bcd or MSG startswith '123' or MSG contains
> 'error2' log to /bcd
> *.* /var/log/catchall
>
> I simply cannot envision how you would store this as a tree that does not
> look like what we have today.

Ok, Here is what I came up with.

0-1,0-6
   --programname startswith abc log to /abc
   | programname startswith 'acc' and MSG contains 'error' log to /acc
    \programname startswith 'acc' log to @1.1.1.1
   --programname contains 'bde' log to @1.1.1.1
   --MSG startswith '123' log to/bcd
   --MSG contains 'error2' log to /bcd
   --log to /var/log/catchall
0-1,7
   --programname startswith abc log to /abc
    \programname startswith 'acc' and MSG contains 'error' log to /acc
2,0-6
   --log to /var/log/mail.log
   --programname startswith abc log to /abc
   | programname startswith 'acc' and MSG contains 'error' log to /acc
   | programname startswith 'acc' log to @1.1.1.1
    \programname startswith bcd log to /bcd
   --programname contains 'bde' log to @1.1.1.1
   --MSG startswith '123' log to/bcd
   --MSG contains 'error2' log to /bcd
   --log to /var/log/catchall
2,7
   --log to /var/log/mail.log
   --programname startswith abc log to /abc
    \programname startswith 'acc' and MSG contains 'error' log to /acc
3-4,0-6
   --programname startswith abc log to /abc
   | programname startswith 'acc' and MSG contains 'error' log to /acc
   | programname startswith 'acc' log to @1.1.1.1
    \programname startswith bcd log to /bcd
   --programname contains 'bde' log to @1.1.1.1
   --MSG startswith '123' log to/bcd
   --MSG contains 'error2' log to /bcd
   --log to /var/log/catchall
3-4,7
   --programname startswith abc log to /abc
    \programname startswith 'acc' and MSG contains 'error' log to /acc
5-23,0-6
   --programname startswith abc log to /abc
   | programname startswith 'acc' and MSG contains 'error' log to /acc
    \programname startswith 'acc' log to @1.1.1.1
   --programname contains 'bde' log to @1.1.1.1
   --MSG startswith '123' log to/bcd
   --MSG contains 'error2' log to /bcd
   --log to /var/log/catchall
5-23,7
   --programname startswith abc log to /abc
    \programname startswith 'acc' and MSG contains 'error' log to /acc

This looks very ugly and is definantly more complicated, but I think it 
still ends up being a slight (but not drastic) win.

for the initial ruleset you have to check a minimum of 4 tests and a 
maximum of 13 tests (debug messages have 4-5 tests, all others have 12-13)

Assuming the facility/priority lookup can be a table lookup, for the 
tree-based ruleset you have a minimum of 2 tests and a maximum of 7 tests, 
in each case one of the tests is a multi-option comparison which is 
slightly more expensive than a single 'startswith' test, but should be 
cheaper than two 'startswith' tests.

This definantly trades off start time complexity for runtime efficiancy.

David Lang


and here is the long, ugly process I went through to derive this.



first normalize the tests (replace 'or' with two lines, change facility 
and priority to ranges of numbers)

facility = 2 log to /Var/log/mail.log
programname startswith abc log to /abc
programname startswith 'acc' and MSG contains 'error' log to /acc
severity 0..6 and programname startswith 'acc' log to @1.1.1.1
severity 0..6 and programname startswith bcd log to /bcd
facility 2..4 and severity 0..6 and programname contains 'bde' log to @1.1.1.1
severity 0..6 and MSG startswith '123' log to/bcd
severity 0..6 and MSG contains 'error2' log to /bcd
severity 0..6 and log to /var/log/catchall

I then add the full facility/priority ranges


now normalize the facility/severity tests (split overlaps)

we have the following ranges for facility
0-1
2
3-4
5-23

and for severity
0-6
7

to shorten the lines in the e-mail I will letter the remaining 
tests/actions

A) log to /Var/log/mail.log
B) programname startswith abc log to /abc
C) programname startswith 'acc' and MSG contains 'error' log to /acc
D) programname startswith 'acc' log to @1.1.1.1
E) programname startswith bcd log to /bcd
F) programname contains 'bde' log to @1.1.1.1
G) MSG startswith '123' log to/bcd
H) MSG contains 'error2' log to /bcd
I) log to /var/log/catchall

so the tree then becomes (straight translation no reordering)

2,0-6 A
2,7 A
0-1,0-6 B
0-1,7 B
2,0-6 B
2,7 B
3-4,0-6 B
3-4,7 B
5-23,0-6 B
5-23,7 B
0-1,0-6 C
0-1,7 C
2,0-6 C
2,7 C
3-4,0-6 C
3-4,7 C
5-23,0-6 C
5-23,7 C
0-1,0-6 D
2,0-6 D
3-4,0-6 D
5-23,0-6 D
2,0-6 E
3-4,0-6 E
0-1,0-6 F
2,0-6 F
3-4,0-6 F
5-23,0-6 F
0-1,0-6 G
2,0-6 G
3-4,0-6 G
5-23,0-6 G
0-1,0-6 H
2,0-6 H
3-4,0-6 H
5-23,0-6 H
0-1,0-6 I
2,0-6 I
3-4,0-6 I
5-23,0-6 I

now to order them

0-1,0-6 B
0-1,0-6 C
0-1,0-6 D
0-1,0-6 F
0-1,0-6 G
0-1,0-6 H
0-1,0-6 I
0-1,7 B
0-1,7 C
2,0-6 A
2,0-6 B
2,0-6 I
2,0-6 H
2,0-6 G
2,0-6 F
2,0-6 E
2,0-6 D
2,0-6 C
2,7 A
2,7 B
2,7 C
3-4,0-6 B
3-4,0-6 C
3-4,0-6 D
3-4,0-6 E
3-4,0-6 F
3-4,0-6 G
3-4,0-6 H
3-4,0-6 I
3-4,7 B
3-4,7 C
5-23,0-6 B
5-23,0-6 C
5-23,0-6 D
5-23,0-6 F
5-23,0-6 G
5-23,0-6 H
5-23,0-6 I
5-23,7 B
5-23,7 C

I then simplify this to

0-1,0-6 B,C,D,F,G,H,I
0-1,7 B,C
2,0-6 A,B,C,D,E,F,G,H,I
2,7 A,B,C
3-4,0-6 B,C,D,E,F,G,H,I
3-4,7 B,C
5-23,0-6 B,C,D,F,G,H,I
5-23,7 B,C

looking at the lettered tests

A) log to /Var/log/mail.log
B) programname startswith abc log to /abc
C) programname startswith 'acc' and MSG contains 'error' log to /acc
D) programname startswith 'acc' log to @1.1.1.1
E) programname startswith bcd log to /bcd
F) programname contains 'bde' log to @1.1.1.1
G) MSG startswith '123' log to/bcd
H) MSG contains 'error2' log to /bcd
I) log to /var/log/catchall

the only ones that can be combined are B-E

so if you make tests

J) B,C
K) B,C,D
L) B,C,D,E

the results simplify to

0-1,0-6 K,F,G,H,I
0-1,7 J
2,0-6 A,L,F,G,H,I
2,7 A,J
3-4,0-6 L,F,G,H,I
3-4,7 J
5-23,0-6 K,F,G,H,I
5-23,7 J

expanded out to show all tests

0-1,0-6
   --programname startswith abc log to /abc
   | programname startswith 'acc' and MSG contains 'error' log to /acc
    \programname startswith 'acc' log to @1.1.1.1
   --programname contains 'bde' log to @1.1.1.1
   --MSG startswith '123' log to/bcd
   --MSG contains 'error2' log to /bcd
   --log to /var/log/catchall
0-1,7
   --programname startswith abc log to /abc
    \programname startswith 'acc' and MSG contains 'error' log to /acc
2,0-6
   --log to /var/log/mail.log
   --programname startswith abc log to /abc
   | programname startswith 'acc' and MSG contains 'error' log to /acc
   | programname startswith 'acc' log to @1.1.1.1
    \programname startswith bcd log to /bcd
   --programname contains 'bde' log to @1.1.1.1
   --MSG startswith '123' log to/bcd
   --MSG contains 'error2' log to /bcd
   --log to /var/log/catchall
2,7
   --log to /var/log/mail.log
   --programname startswith abc log to /abc
    \programname startswith 'acc' and MSG contains 'error' log to /acc
3-4,0-6
   --programname startswith abc log to /abc
   | programname startswith 'acc' and MSG contains 'error' log to /acc
   | programname startswith 'acc' log to @1.1.1.1
    \programname startswith bcd log to /bcd
   --programname contains 'bde' log to @1.1.1.1
   --MSG startswith '123' log to/bcd
   --MSG contains 'error2' log to /bcd
   --log to /var/log/catchall
3-4,7
   --programname startswith abc log to /abc
    \programname startswith 'acc' and MSG contains 'error' log to /acc
5-23,0-6
   --programname startswith abc log to /abc
   | programname startswith 'acc' and MSG contains 'error' log to /acc
    \programname startswith 'acc' log to @1.1.1.1
   --programname contains 'bde' log to @1.1.1.1
   --MSG startswith '123' log to/bcd
   --MSG contains 'error2' log to /bcd
   --log to /var/log/catchall
5-23,7
   --programname startswith abc log to /abc
    \programname startswith 'acc' and MSG contains 'error' log to /acc

> Rainer
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Sunday, February 28, 2010 2:02 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] Feedback requested: inconsistency in config
>> statements
>>
>> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
>>
>>> Quickly from the mobile: pri based filters require *excatly* one
>> single word (!) lookup currently. You can't beat that.
>>>
>>> I am not totally sure how you think about parse trees. In my view,
>> parsin and filteing are two different things./stages. Maybe we have
>> different concepts on our minds?
>>
>> I am probably using the wrong terminology here.
>>
>> say you have a set of rules that say
>>
>> if programname startswith abc log to /abc
>> if programname startswith acc log to /acc
>> if programname startswith acc log to @1.1.1.1
>> if programname startswith bcd log to /bcd
>>
>> and assuming that these were the only possible programnames for
>> simplicity
>> in the explination.
>>
>> the filtering logic would go somthing like this
>>
>> the rules would compile into a tree
>>
>> -a-b -> /abc
>>    -c -> /acc, at 1.1.1.1
>> -b   -> /bcd
>>
>> receive programname abc
>> I have no facility/Pri filter rules
>> I have no time filter rules
>> I have no system filter rules
>> I have programname filter rules
>>    look at the first character of the programname
>>     it's "a", I have more than one rule that fits that.
>>      look at the second character of the programname
>>       it's a "b", There are no other decisions to make,
>>         invoke the action /abc
>>
>> receive progranmane bcd
>> I have no facility/PRI filter rules
>> I have no time filter rules
>> I have no system filter rules
>> I have programname filter rules
>>    look at the first character of the programname
>>     it's a "b", There are no other decisions to make,
>>       invoke the action /bcd
>>
>> receive programname acc
>> I have no facility/Pri filter rules
>> I have no time filter rules
>> I have no system filter rules
>> I have programname filter rules
>>    look at the first character of the programname
>>     it's "a", I have more than one rule that fits that.
>>      look at the second character of the programname
>>       it's a "c", There are no other decisions to make,
>>         invoke the action /acc and the action @1.1.1.1
>>
>> similarly the facility/pri filtering rules would be either compiled
>> into a
>> tree, or (since it's 256 entries total) into a lookup table with
>> pointers
>> to the list of actions to take for that entry
>>
>> the idea is to spend the time at startup to create the tree that
>> represents the ruleset in order to speed up the processing of each
>> individual message.
>>
>> the real tree would be a bit more complicated than I describe here as
>> it
>> needs to have 'anything else' entries between the known entries (which
>> means that it would not be able to shortcut quite as much as I
>> describe),
>> and have provision for 'do a more complicated thing (like regex) here
>> and
>> if it matches continue'. but except for regex matches, this would make
>> processing the rules pretty much independant of how many rules there
>> were
>> or how complex the ruleset is.
>>
>> there doe snot need to be one single programname filter tree, if you
>> had a
>> rule that said
>> if pri = info and programname startswith def then def
>>
>> the pri tree/table would have an entry for info that would point to a
>> filter tree for programname that just had the check for def in it
>>
>> am I makeing any more sense now?
>>
>> David Lang
>>
>>> rainer
>>>
>>> ----- Urspr?ngliche Nachricht -----
>>> Von: "david at lang.hm" <david at lang.hm>
>>> An: "rsyslog-users" <rsyslog at lists.adiscon.com>
>>> Gesendet: 28.02.10 11:28
>>> Betreff: Re: [rsyslog] Feedback requested: inconsistency in config
>> statements
>>>
>>> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
>>>
>>>>>
>>>>> if the type is depriciated then the destination would be another
>>>>> config_option (which you would then look up)
>>>>>
>>>>> if the type is 'ignore' then it would just be skipped over
>> (possibly
>>>>> with
>>>>> a warning being logged that the config line is being ignored)
>>>>>
>>>>> this would also clean up some of the current cases where a boolean
>>>>> option
>>>>> doesn't honor on/off and true/false.
>>>>
>>>> True/false is NOT a valid boolean option. See
>>>>
>>>>
>> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=18
>> 4c0d87
>>>> 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308
>>>
>>> I was not remembering correctly then, maby it was yes/no vs on/off. I
>> know
>>> I ran into something where what was documented didn't work and I
>> changed
>>> it to another version of a boolean answer and it fixed the problem.
>>>
>>>>> For the second half of the config language (telling rsyslog what to
>> do
>>>>> with the logs it has received) also has several variations in place
>> and
>>>>> is
>>>>> showing issues.
>>>>>
>>>>> However I think that the right answer here is to end up
>> implementing
>>>>> something like the parsing trees and then compile the other config
>>>>> language options into that tree to be consistant (and fast) under
>> the
>>>>> covers, no matter which way the instructions are written (except
>> when
>>>>> you
>>>>> have to use regex matches)
>>>>
>>>> I don't fully agree here with you. For example, the traditional PRI
>> based
>>>> filter is lightning fast. I don't see any way it could be nearly as
>> fast with
>>>> any unified approach. Right now, we have a "unified" filter
>> structure, but it
>>>> contains three filter cases, each one adding potential power at the
>> price of
>>>> decreased speed. I think we need to keep different types of filters
>> in order
>>>> to have some lightning-fast ones. But more of this could be done
>> under the
>>>> hood.
>>>
>>> My guess/expectation is that the tree-based parsing would be about
>> the
>>> same speed as the traditional PRI based filter for choices made based
>> on
>>> PRI, slowing down for other types of conditions only in that more of
>> the
>>> message may need to be scanned (and if there is no selection at a
>> level,
>>> skipping that level should be lightning fast). As a result, a set of
>>> filteres based soley on programname would be as fast as the current
>> PRI
>>> filters.
>>>
>>> David Lang
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From reader at newsguy.com  Mon Mar  1 04:01:34 2010
From: reader at newsguy.com (Harry Putnam)
Date: Sun, 28 Feb 2010 21:01:34 -0600
Subject: [rsyslog] How rsyslog derives hostname
Message-ID: <87hbp0eodt.fsf@newsguy.com>

Running rsyslog-3.22.1

NOTE: Windbag alert.. its a bit long.
(And still more windyness in a PS at the end)

I know newer version are available but 3.22.1 is the newest version my
linux distro (gentoo 2010) offers.

Maybe the mechanism that caused the phenomena I observed has changed
in the later versions.

I'm curious to know how rsyslog derives the hostname.

Before explaining the phenomena I noticed, just be clear that this
problem arose strictly from a blunder on my part.  So, this is not a
complaint about rsyslog.  Just wondering if it points to what could be
a problem.

Due to a blatant blunder on my part, I miss-typed the hosts name
in the /etc/hosts file.  At the 127 line I spelled the hosts name wrong.

    127.0.0.1       logsrv.local.lan   *=>logsvr<=*     localhost

The actual name is `logsrv'
    logsrv
not logsvr.  (I transposed the v and r.)

In this code below from my (no doubt, poorly written)  rsyslog.conf 

[
  This first part contains some fairly standard code.  Mostly
  unchanged from the default /etc/rsyslog.conf that gets installed.
 [ blah ]
]
  [then my own code below]
  [...]

  ## write all localhost output to a debug file
  $template Ldebug,"/var/log/debug.log"
  if \
      $source == 'logsrv'\
  then  -?Ldebug

  ## write all data from remote hosts to:
  ## HOSTNAME/HOSTNAME.log for each client
  $template RDDF,"/var/log/%hostname%/%hostname%.log"
  if \
          $source  != 'logsrv' \
  then    -?RDDF
  [...]

Note: I used the actual name (logsrv) in the code, since I wasn't sure
which of rsyslogs' variables would contain that info instead of
`localhost' or `0.0.0.0' or `127.0.0.1` ... etc.

And understand that this rsyslog.conf was being used in an
experimental environment while I familiarized myself with rsyslog.
 
The idea was to direct all localhost log output to /var/log/debug.log
(in addition to the more normal defaults in rsyslog.conf not shown here).
And log data coming from remote clients to /var/log/dynaDIR/dynaFILE
Where DIR and FILE are both named dynamically after the host sending
the log data.

It worked just like I had hoped.

But I accidentally found what might be seen as a weakness in rsyslog.

I'm not experienced enough to really judge something like this. And
not adept enough at reading source code to see what the mechanism is.

Apparently rsyslog preferred the miss-typed name in /etc/hosts over
the name returned by gethostbyname or the newer getaddrinfo, both
appear to be involved in the source code.

With the mistaken spelling in /etc/hosts, rsyslog (correctly) saw the
mistaken name inside `$source' as not matching `logsrv' in my code, so
wrote to:

  /var/log/logsvr/logsvr.log

Once I got the typo fixed, rsyslog wrote all that same stuff to
/var/log/debug.log as expected.

So it makes me wonder how robust the mechanism for getting the
localhost name is.  rsyslog appears to use gethostbyname and the
newer getaddrinfo.

Further... I'm not sure where cmds like gethostbyname (and similar
ones) get the info either... for all I know they may just grep
/etc/hosts.

I do know that even while I had the mis-spelled name in /etc/hosts, the
`hostname' command still returned the correct name.  So the `hostname'
cmd must rely on something else.

At any rate, rsyslog apparently turns to grepping the /etc/hosts
file at some point, since in this case the log output was directed to a
directory and file named after the TYPO in /etc/hosts.

It took me a few minutes to figure out where the bad name was coming
from since the `hostname' cmd returned the correct name (logsrv).

Some linux distros have a special file that tells the OS what its name
is. 

On my distro (gentoo linux) it's a file at /etc/conf.d/hostname, that
is dedicated to nothing more than giving the host a name.

On Debian its /etc/hostname

Other distros I've used have a few different ways of getting that
information into play, but I haven't used any linux distros that
relied on /etc/hosts.

Is having rsyslog rely on /etc/hosts as the final arbiter about the
host name a good plan?

ps - two asides:
 1) Any suggestions for better code are welcome
 2) Can anyone tell me what the significance of the question mark
    preceding the log file variable name is:

  $template RDDF,"/var/log/%hostname%/%hostname%.log"
  if \
          $source  != 'logsrv' \
  then    -?RDDF
          ------
           here

I know what the dash sinifies but not the question mark. I just
copied what I saw in the examples and it seems to work.
          



From rgerhards at hq.adiscon.com  Mon Mar  1 07:45:48 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 1 Mar 2010 07:45:48 +0100
Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released
References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com><aa76cdac1002251244y2be4e04egc7e0e3eb57494431@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com><aa76cdac1002260355l26269784y8fb8ccd45caeaa12@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com><aa76cdac1002260443p4c7744e2ha1c8c2372c741ebf@mail.gmail.com><aa76cdac1002280515g72c07527mae8ebc08d4ecd404@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103980@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103981@GRFEXC.intern.adiscon.com>
	<aa76cdac1002281500g3d50db29rcd20fea1fbf08d48@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103983@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Monday, March 01, 2010 12:01 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released
> 
> 2010/2/28 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> >> OK, rather than trying to figure out what may be going on here, I'll
> >> see that
> >> I copy the old omfile code and create a new ompipe. So that will
> behave
> >> exactly like it did before. The new optimizations are not so
> important
> >> for
> >> pipes.
> >
> > I have done this now. I would appreciate if you could checkout the
> v4-stable
> > git head and try with that one. It should now have the exact same
> behavior as
> > in 4.4.2 (the pipe code is now the same!). I tested on Debian and it
> worked
> > like before.
> 
> I use $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
> in my rsyslog.conf, but the new ompipe module does not seem to respect
> that setting but always uses the high-resolution time format.

I see - that was a side-effect that pipes used the same output module like
regular pipes. But I have now re-established it, please try the newest git
v4-stable branch.

> 
> Other than that, I did not (yet) experience any blank xconsole windows
> with v4.6.0-5-gf12a199 (git describe)
> 
> Just curious, when I fire up xconsole after boot, I get two log
> messages immediately. Where do those log messages come from?

It seems to stem back to the API usage in the old code. The old code opens
the pipe and tries to write. It disables writing only when the API returns it
would block. The new code used create() instead of open, and that immediately
returned an error if nobody was listening.

It looks like with the old sequence, the OS permits to write a handful of
bytes before it returns EAGAIN. However, I would not take that behavior for
granted: I guess it may be different under different platforms and/or
versions.

Rainer


From mrdemeanour at jackpot.uk.net  Mon Mar  1 08:41:17 2010
From: mrdemeanour at jackpot.uk.net (Mr. Demeanour)
Date: Mon, 01 Mar 2010 07:41:17 +0000
Subject: [rsyslog] How rsyslog derives hostname
In-Reply-To: <87hbp0eodt.fsf@newsguy.com>
References: <87hbp0eodt.fsf@newsguy.com>
Message-ID: <4B8B6F9D.5040803@jackpot.uk.net>

Harry Putnam wrote:
> 
> Further... I'm not sure where cmds like gethostbyname (and similar
> ones) get the info either... for all I know they may just grep
> /etc/hosts.

man resolv.conf (i.e. it's configurable).

-- 
Jack.


From rgerhards at hq.adiscon.com  Mon Mar  1 10:39:27 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 1 Mar 2010 10:39:27 +0100
Subject: [rsyslog] How rsyslog derives hostname
References: <87hbp0eodt.fsf@newsguy.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710398A@GRFEXC.intern.adiscon.com>

I need to re-read this more careful, but for now let me say that rsyslog does
not touch /etc/hosts at all. It exlusively relies on what is returned by the
OS. But the "source" property is not locally generated, it contains whatever
the sender placed into the relevant field.

Please also see this article:

http://www.rsyslog.com/doc-syslog_parsing.html

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Harry Putnam
> Sent: Monday, March 01, 2010 4:02 AM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] How rsyslog derives hostname
> 
> Running rsyslog-3.22.1
> 
> NOTE: Windbag alert.. its a bit long.
> (And still more windyness in a PS at the end)
> 
> I know newer version are available but 3.22.1 is the newest version my
> linux distro (gentoo 2010) offers.
> 
> Maybe the mechanism that caused the phenomena I observed has changed
> in the later versions.
> 
> I'm curious to know how rsyslog derives the hostname.
> 
> Before explaining the phenomena I noticed, just be clear that this
> problem arose strictly from a blunder on my part.  So, this is not a
> complaint about rsyslog.  Just wondering if it points to what could be
> a problem.
> 
> Due to a blatant blunder on my part, I miss-typed the hosts name
> in the /etc/hosts file.  At the 127 line I spelled the hosts name
> wrong.
> 
>     127.0.0.1       logsrv.local.lan   *=>logsvr<=*     localhost
> 
> The actual name is `logsrv'
>     logsrv
> not logsvr.  (I transposed the v and r.)
> 
> In this code below from my (no doubt, poorly written)  rsyslog.conf
> 
> [
>   This first part contains some fairly standard code.  Mostly
>   unchanged from the default /etc/rsyslog.conf that gets installed.
>  [ blah ]
> ]
>   [then my own code below]
>   [...]
> 
>   ## write all localhost output to a debug file
>   $template Ldebug,"/var/log/debug.log"
>   if \
>       $source == 'logsrv'\
>   then  -?Ldebug
> 
>   ## write all data from remote hosts to:
>   ## HOSTNAME/HOSTNAME.log for each client
>   $template RDDF,"/var/log/%hostname%/%hostname%.log"
>   if \
>           $source  != 'logsrv' \
>   then    -?RDDF
>   [...]
> 
> Note: I used the actual name (logsrv) in the code, since I wasn't sure
> which of rsyslogs' variables would contain that info instead of
> `localhost' or `0.0.0.0' or `127.0.0.1` ... etc.
> 
> And understand that this rsyslog.conf was being used in an
> experimental environment while I familiarized myself with rsyslog.
> 
> The idea was to direct all localhost log output to /var/log/debug.log
> (in addition to the more normal defaults in rsyslog.conf not shown
> here).
> And log data coming from remote clients to /var/log/dynaDIR/dynaFILE
> Where DIR and FILE are both named dynamically after the host sending
> the log data.
> 
> It worked just like I had hoped.
> 
> But I accidentally found what might be seen as a weakness in rsyslog.
> 
> I'm not experienced enough to really judge something like this. And
> not adept enough at reading source code to see what the mechanism is.
> 
> Apparently rsyslog preferred the miss-typed name in /etc/hosts over
> the name returned by gethostbyname or the newer getaddrinfo, both
> appear to be involved in the source code.
> 
> With the mistaken spelling in /etc/hosts, rsyslog (correctly) saw the
> mistaken name inside `$source' as not matching `logsrv' in my code, so
> wrote to:
> 
>   /var/log/logsvr/logsvr.log
> 
> Once I got the typo fixed, rsyslog wrote all that same stuff to
> /var/log/debug.log as expected.
> 
> So it makes me wonder how robust the mechanism for getting the
> localhost name is.  rsyslog appears to use gethostbyname and the
> newer getaddrinfo.
> 
> Further... I'm not sure where cmds like gethostbyname (and similar
> ones) get the info either... for all I know they may just grep
> /etc/hosts.
> 
> I do know that even while I had the mis-spelled name in /etc/hosts, the
> `hostname' command still returned the correct name.  So the `hostname'
> cmd must rely on something else.
> 
> At any rate, rsyslog apparently turns to grepping the /etc/hosts
> file at some point, since in this case the log output was directed to a
> directory and file named after the TYPO in /etc/hosts.
> 
> It took me a few minutes to figure out where the bad name was coming
> from since the `hostname' cmd returned the correct name (logsrv).
> 
> Some linux distros have a special file that tells the OS what its name
> is.
> 
> On my distro (gentoo linux) it's a file at /etc/conf.d/hostname, that
> is dedicated to nothing more than giving the host a name.
> 
> On Debian its /etc/hostname
> 
> Other distros I've used have a few different ways of getting that
> information into play, but I haven't used any linux distros that
> relied on /etc/hosts.
> 
> Is having rsyslog rely on /etc/hosts as the final arbiter about the
> host name a good plan?
> 
> ps - two asides:
>  1) Any suggestions for better code are welcome
>  2) Can anyone tell me what the significance of the question mark
>     preceding the log file variable name is:
> 
>   $template RDDF,"/var/log/%hostname%/%hostname%.log"
>   if \
>           $source  != 'logsrv' \
>   then    -?RDDF
>           ------
>            here
> 
> I know what the dash sinifies but not the question mark. I just
> copied what I saw in the examples and it seems to work.
> 
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Mon Mar  1 14:27:57 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 1 Mar 2010 14:27:57 +0100
Subject: [rsyslog] Feedback requested: inconsistency in config statements
References: <001201cab868$d148da7c$100013ac@intern.adiscon.com><alpine.DEB.2.00.1002280426420.15263@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA710397F@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.00.1002281346000.15263@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103992@GRFEXC.intern.adiscon.com>

Hi David,

thanks for the explanation, I now see where you are heading too. I have not
dug into all details, but find the post very interesting and insightful. I'll
archive it so that I have it at hand when I can take the route to actually
implement such an approach. All in all, it still looks rather complex to me.
I guess it would be useful to have a guy with experience in writing compiler
code optimizers at hand when I tackle that beast ;)

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Monday, March 01, 2010 3:16 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Feedback requested: inconsistency in config
> statements
> 
> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
> 
> > Hi David,
> >
> > yes, that makes much more sense to me. Let's call that filter
> expression tree
> > for the time being (the parse tree used for normalization has some
> similar
> > ideas, but a totally different scope and semantics).
> >
> > While I now see where you are heading, I fail to see how this could
> > sufficiently easy be implemented for a real (and complex)
> configuration.
> >
> > Let's look at this example conf:
> >
> > mail.* /Var/log/mail.log
> > if programname startswith abc log to /abc
> > if programname startswith 'acc' and MSG contains 'error' log to /acc
> > # we don't want debug messages from here on
> > *.debug ~
> > if programname startswith 'acc' or (programname contains 'bde' and
> (facility
> >> 1 and
> > facility < 5)) log to @1.1.1.1
> > if programname startswith bcd or MSG startswith '123' or MSG contains
> > 'error2' log to /bcd
> > *.* /var/log/catchall
> >
> > I simply cannot envision how you would store this as a tree that does
> not
> > look like what we have today.
> 
> Ok, Here is what I came up with.
> 
> 0-1,0-6
>    --programname startswith abc log to /abc
>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>     \programname startswith 'acc' log to @1.1.1.1
>    --programname contains 'bde' log to @1.1.1.1
>    --MSG startswith '123' log to/bcd
>    --MSG contains 'error2' log to /bcd
>    --log to /var/log/catchall
> 0-1,7
>    --programname startswith abc log to /abc
>     \programname startswith 'acc' and MSG contains 'error' log to /acc
> 2,0-6
>    --log to /var/log/mail.log
>    --programname startswith abc log to /abc
>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>    | programname startswith 'acc' log to @1.1.1.1
>     \programname startswith bcd log to /bcd
>    --programname contains 'bde' log to @1.1.1.1
>    --MSG startswith '123' log to/bcd
>    --MSG contains 'error2' log to /bcd
>    --log to /var/log/catchall
> 2,7
>    --log to /var/log/mail.log
>    --programname startswith abc log to /abc
>     \programname startswith 'acc' and MSG contains 'error' log to /acc
> 3-4,0-6
>    --programname startswith abc log to /abc
>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>    | programname startswith 'acc' log to @1.1.1.1
>     \programname startswith bcd log to /bcd
>    --programname contains 'bde' log to @1.1.1.1
>    --MSG startswith '123' log to/bcd
>    --MSG contains 'error2' log to /bcd
>    --log to /var/log/catchall
> 3-4,7
>    --programname startswith abc log to /abc
>     \programname startswith 'acc' and MSG contains 'error' log to /acc
> 5-23,0-6
>    --programname startswith abc log to /abc
>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>     \programname startswith 'acc' log to @1.1.1.1
>    --programname contains 'bde' log to @1.1.1.1
>    --MSG startswith '123' log to/bcd
>    --MSG contains 'error2' log to /bcd
>    --log to /var/log/catchall
> 5-23,7
>    --programname startswith abc log to /abc
>     \programname startswith 'acc' and MSG contains 'error' log to /acc
> 
> This looks very ugly and is definantly more complicated, but I think it
> still ends up being a slight (but not drastic) win.
> 
> for the initial ruleset you have to check a minimum of 4 tests and a
> maximum of 13 tests (debug messages have 4-5 tests, all others have 12-
> 13)
> 
> Assuming the facility/priority lookup can be a table lookup, for the
> tree-based ruleset you have a minimum of 2 tests and a maximum of 7
> tests,
> in each case one of the tests is a multi-option comparison which is
> slightly more expensive than a single 'startswith' test, but should be
> cheaper than two 'startswith' tests.
> 
> This definantly trades off start time complexity for runtime
> efficiancy.
> 
> David Lang
> 
> 
> and here is the long, ugly process I went through to derive this.
> 
> 
> 
> first normalize the tests (replace 'or' with two lines, change facility
> and priority to ranges of numbers)
> 
> facility = 2 log to /Var/log/mail.log
> programname startswith abc log to /abc
> programname startswith 'acc' and MSG contains 'error' log to /acc
> severity 0..6 and programname startswith 'acc' log to @1.1.1.1
> severity 0..6 and programname startswith bcd log to /bcd
> facility 2..4 and severity 0..6 and programname contains 'bde' log to
> @1.1.1.1
> severity 0..6 and MSG startswith '123' log to/bcd
> severity 0..6 and MSG contains 'error2' log to /bcd
> severity 0..6 and log to /var/log/catchall
> 
> I then add the full facility/priority ranges
> 
> 
> now normalize the facility/severity tests (split overlaps)
> 
> we have the following ranges for facility
> 0-1
> 2
> 3-4
> 5-23
> 
> and for severity
> 0-6
> 7
> 
> to shorten the lines in the e-mail I will letter the remaining
> tests/actions
> 
> A) log to /Var/log/mail.log
> B) programname startswith abc log to /abc
> C) programname startswith 'acc' and MSG contains 'error' log to /acc
> D) programname startswith 'acc' log to @1.1.1.1
> E) programname startswith bcd log to /bcd
> F) programname contains 'bde' log to @1.1.1.1
> G) MSG startswith '123' log to/bcd
> H) MSG contains 'error2' log to /bcd
> I) log to /var/log/catchall
> 
> so the tree then becomes (straight translation no reordering)
> 
> 2,0-6 A
> 2,7 A
> 0-1,0-6 B
> 0-1,7 B
> 2,0-6 B
> 2,7 B
> 3-4,0-6 B
> 3-4,7 B
> 5-23,0-6 B
> 5-23,7 B
> 0-1,0-6 C
> 0-1,7 C
> 2,0-6 C
> 2,7 C
> 3-4,0-6 C
> 3-4,7 C
> 5-23,0-6 C
> 5-23,7 C
> 0-1,0-6 D
> 2,0-6 D
> 3-4,0-6 D
> 5-23,0-6 D
> 2,0-6 E
> 3-4,0-6 E
> 0-1,0-6 F
> 2,0-6 F
> 3-4,0-6 F
> 5-23,0-6 F
> 0-1,0-6 G
> 2,0-6 G
> 3-4,0-6 G
> 5-23,0-6 G
> 0-1,0-6 H
> 2,0-6 H
> 3-4,0-6 H
> 5-23,0-6 H
> 0-1,0-6 I
> 2,0-6 I
> 3-4,0-6 I
> 5-23,0-6 I
> 
> now to order them
> 
> 0-1,0-6 B
> 0-1,0-6 C
> 0-1,0-6 D
> 0-1,0-6 F
> 0-1,0-6 G
> 0-1,0-6 H
> 0-1,0-6 I
> 0-1,7 B
> 0-1,7 C
> 2,0-6 A
> 2,0-6 B
> 2,0-6 I
> 2,0-6 H
> 2,0-6 G
> 2,0-6 F
> 2,0-6 E
> 2,0-6 D
> 2,0-6 C
> 2,7 A
> 2,7 B
> 2,7 C
> 3-4,0-6 B
> 3-4,0-6 C
> 3-4,0-6 D
> 3-4,0-6 E
> 3-4,0-6 F
> 3-4,0-6 G
> 3-4,0-6 H
> 3-4,0-6 I
> 3-4,7 B
> 3-4,7 C
> 5-23,0-6 B
> 5-23,0-6 C
> 5-23,0-6 D
> 5-23,0-6 F
> 5-23,0-6 G
> 5-23,0-6 H
> 5-23,0-6 I
> 5-23,7 B
> 5-23,7 C
> 
> I then simplify this to
> 
> 0-1,0-6 B,C,D,F,G,H,I
> 0-1,7 B,C
> 2,0-6 A,B,C,D,E,F,G,H,I
> 2,7 A,B,C
> 3-4,0-6 B,C,D,E,F,G,H,I
> 3-4,7 B,C
> 5-23,0-6 B,C,D,F,G,H,I
> 5-23,7 B,C
> 
> looking at the lettered tests
> 
> A) log to /Var/log/mail.log
> B) programname startswith abc log to /abc
> C) programname startswith 'acc' and MSG contains 'error' log to /acc
> D) programname startswith 'acc' log to @1.1.1.1
> E) programname startswith bcd log to /bcd
> F) programname contains 'bde' log to @1.1.1.1
> G) MSG startswith '123' log to/bcd
> H) MSG contains 'error2' log to /bcd
> I) log to /var/log/catchall
> 
> the only ones that can be combined are B-E
> 
> so if you make tests
> 
> J) B,C
> K) B,C,D
> L) B,C,D,E
> 
> the results simplify to
> 
> 0-1,0-6 K,F,G,H,I
> 0-1,7 J
> 2,0-6 A,L,F,G,H,I
> 2,7 A,J
> 3-4,0-6 L,F,G,H,I
> 3-4,7 J
> 5-23,0-6 K,F,G,H,I
> 5-23,7 J
> 
> expanded out to show all tests
> 
> 0-1,0-6
>    --programname startswith abc log to /abc
>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>     \programname startswith 'acc' log to @1.1.1.1
>    --programname contains 'bde' log to @1.1.1.1
>    --MSG startswith '123' log to/bcd
>    --MSG contains 'error2' log to /bcd
>    --log to /var/log/catchall
> 0-1,7
>    --programname startswith abc log to /abc
>     \programname startswith 'acc' and MSG contains 'error' log to /acc
> 2,0-6
>    --log to /var/log/mail.log
>    --programname startswith abc log to /abc
>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>    | programname startswith 'acc' log to @1.1.1.1
>     \programname startswith bcd log to /bcd
>    --programname contains 'bde' log to @1.1.1.1
>    --MSG startswith '123' log to/bcd
>    --MSG contains 'error2' log to /bcd
>    --log to /var/log/catchall
> 2,7
>    --log to /var/log/mail.log
>    --programname startswith abc log to /abc
>     \programname startswith 'acc' and MSG contains 'error' log to /acc
> 3-4,0-6
>    --programname startswith abc log to /abc
>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>    | programname startswith 'acc' log to @1.1.1.1
>     \programname startswith bcd log to /bcd
>    --programname contains 'bde' log to @1.1.1.1
>    --MSG startswith '123' log to/bcd
>    --MSG contains 'error2' log to /bcd
>    --log to /var/log/catchall
> 3-4,7
>    --programname startswith abc log to /abc
>     \programname startswith 'acc' and MSG contains 'error' log to /acc
> 5-23,0-6
>    --programname startswith abc log to /abc
>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>     \programname startswith 'acc' log to @1.1.1.1
>    --programname contains 'bde' log to @1.1.1.1
>    --MSG startswith '123' log to/bcd
>    --MSG contains 'error2' log to /bcd
>    --log to /var/log/catchall
> 5-23,7
>    --programname startswith abc log to /abc
>     \programname startswith 'acc' and MSG contains 'error' log to /acc
> 
> > Rainer
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >> Sent: Sunday, February 28, 2010 2:02 PM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] Feedback requested: inconsistency in config
> >> statements
> >>
> >> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
> >>
> >>> Quickly from the mobile: pri based filters require *excatly* one
> >> single word (!) lookup currently. You can't beat that.
> >>>
> >>> I am not totally sure how you think about parse trees. In my view,
> >> parsin and filteing are two different things./stages. Maybe we have
> >> different concepts on our minds?
> >>
> >> I am probably using the wrong terminology here.
> >>
> >> say you have a set of rules that say
> >>
> >> if programname startswith abc log to /abc
> >> if programname startswith acc log to /acc
> >> if programname startswith acc log to @1.1.1.1
> >> if programname startswith bcd log to /bcd
> >>
> >> and assuming that these were the only possible programnames for
> >> simplicity
> >> in the explination.
> >>
> >> the filtering logic would go somthing like this
> >>
> >> the rules would compile into a tree
> >>
> >> -a-b -> /abc
> >>    -c -> /acc, at 1.1.1.1
> >> -b   -> /bcd
> >>
> >> receive programname abc
> >> I have no facility/Pri filter rules
> >> I have no time filter rules
> >> I have no system filter rules
> >> I have programname filter rules
> >>    look at the first character of the programname
> >>     it's "a", I have more than one rule that fits that.
> >>      look at the second character of the programname
> >>       it's a "b", There are no other decisions to make,
> >>         invoke the action /abc
> >>
> >> receive progranmane bcd
> >> I have no facility/PRI filter rules
> >> I have no time filter rules
> >> I have no system filter rules
> >> I have programname filter rules
> >>    look at the first character of the programname
> >>     it's a "b", There are no other decisions to make,
> >>       invoke the action /bcd
> >>
> >> receive programname acc
> >> I have no facility/Pri filter rules
> >> I have no time filter rules
> >> I have no system filter rules
> >> I have programname filter rules
> >>    look at the first character of the programname
> >>     it's "a", I have more than one rule that fits that.
> >>      look at the second character of the programname
> >>       it's a "c", There are no other decisions to make,
> >>         invoke the action /acc and the action @1.1.1.1
> >>
> >> similarly the facility/pri filtering rules would be either compiled
> >> into a
> >> tree, or (since it's 256 entries total) into a lookup table with
> >> pointers
> >> to the list of actions to take for that entry
> >>
> >> the idea is to spend the time at startup to create the tree that
> >> represents the ruleset in order to speed up the processing of each
> >> individual message.
> >>
> >> the real tree would be a bit more complicated than I describe here
> as
> >> it
> >> needs to have 'anything else' entries between the known entries
> (which
> >> means that it would not be able to shortcut quite as much as I
> >> describe),
> >> and have provision for 'do a more complicated thing (like regex)
> here
> >> and
> >> if it matches continue'. but except for regex matches, this would
> make
> >> processing the rules pretty much independant of how many rules there
> >> were
> >> or how complex the ruleset is.
> >>
> >> there doe snot need to be one single programname filter tree, if you
> >> had a
> >> rule that said
> >> if pri = info and programname startswith def then def
> >>
> >> the pri tree/table would have an entry for info that would point to
> a
> >> filter tree for programname that just had the check for def in it
> >>
> >> am I makeing any more sense now?
> >>
> >> David Lang
> >>
> >>> rainer
> >>>
> >>> ----- Urspr?ngliche Nachricht -----
> >>> Von: "david at lang.hm" <david at lang.hm>
> >>> An: "rsyslog-users" <rsyslog at lists.adiscon.com>
> >>> Gesendet: 28.02.10 11:28
> >>> Betreff: Re: [rsyslog] Feedback requested: inconsistency in config
> >> statements
> >>>
> >>> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
> >>>
> >>>>>
> >>>>> if the type is depriciated then the destination would be another
> >>>>> config_option (which you would then look up)
> >>>>>
> >>>>> if the type is 'ignore' then it would just be skipped over
> >> (possibly
> >>>>> with
> >>>>> a warning being logged that the config line is being ignored)
> >>>>>
> >>>>> this would also clean up some of the current cases where a
> boolean
> >>>>> option
> >>>>> doesn't honor on/off and true/false.
> >>>>
> >>>> True/false is NOT a valid boolean option. See
> >>>>
> >>>>
> >>
> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=18
> >> 4c0d87
> >>>> 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308
> >>>
> >>> I was not remembering correctly then, maby it was yes/no vs on/off.
> I
> >> know
> >>> I ran into something where what was documented didn't work and I
> >> changed
> >>> it to another version of a boolean answer and it fixed the problem.
> >>>
> >>>>> For the second half of the config language (telling rsyslog what
> to
> >> do
> >>>>> with the logs it has received) also has several variations in
> place
> >> and
> >>>>> is
> >>>>> showing issues.
> >>>>>
> >>>>> However I think that the right answer here is to end up
> >> implementing
> >>>>> something like the parsing trees and then compile the other
> config
> >>>>> language options into that tree to be consistant (and fast) under
> >> the
> >>>>> covers, no matter which way the instructions are written (except
> >> when
> >>>>> you
> >>>>> have to use regex matches)
> >>>>
> >>>> I don't fully agree here with you. For example, the traditional
> PRI
> >> based
> >>>> filter is lightning fast. I don't see any way it could be nearly
> as
> >> fast with
> >>>> any unified approach. Right now, we have a "unified" filter
> >> structure, but it
> >>>> contains three filter cases, each one adding potential power at
> the
> >> price of
> >>>> decreased speed. I think we need to keep different types of
> filters
> >> in order
> >>>> to have some lightning-fast ones. But more of this could be done
> >> under the
> >>>> hood.
> >>>
> >>> My guess/expectation is that the tree-based parsing would be about
> >> the
> >>> same speed as the traditional PRI based filter for choices made
> based
> >> on
> >>> PRI, slowing down for other types of conditions only in that more
> of
> >> the
> >>> message may need to be scanned (and if there is no selection at a
> >> level,
> >>> skipping that level should be lightning fast). As a result, a set
> of
> >>> filteres based soley on programname would be as fast as the current
> >> PRI
> >>> filters.
> >>>
> >>> David Lang
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From david at lang.hm  Mon Mar  1 20:39:09 2010
From: david at lang.hm (david at lang.hm)
Date: Mon, 1 Mar 2010 11:39:09 -0800 (PST)
Subject: [rsyslog] Feedback requested: inconsistency in config statements
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103992@GRFEXC.intern.adiscon.com>
References: <001201cab868$d148da7c$100013ac@intern.adiscon.com><alpine.DEB.2.00.1002280426420.15263@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA710397F@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.00.1002281346000.15263@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7103992@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.2.00.1003011130460.5131@asgard.lang.hm>

On Mon, 1 Mar 2010, Rainer Gerhards wrote:

> Hi David,
>
> thanks for the explanation, I now see where you are heading too. I have not
> dug into all details, but find the post very interesting and insightful. I'll
> archive it so that I have it at hand when I can take the route to actually
> implement such an approach. All in all, it still looks rather complex to me.
> I guess it would be useful to have a guy with experience in writing compiler
> code optimizers at hand when I tackle that beast ;)

you may also want to dig into yacc/bison as they are designed to help 
people build parser trees.

I am assuming that this is the type of thing that syslog-ng is doing to be 
able to efficiantly handle a lot of log processing.

I currently have a script that looks for a couple hundred programnames and 
puts the data into ~50 different files depending on what that name is. 
Given that the output side of rsyslog is currently the bottleneck, I would 
not want to try and implement that in rsyslog today, but if rsyslog gained 
the ability to handle this sort of ruleset efficiantly it would greatly 
simplify my life.

David Lang

> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Monday, March 01, 2010 3:16 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] Feedback requested: inconsistency in config
>> statements
>>
>> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
>>
>>> Hi David,
>>>
>>> yes, that makes much more sense to me. Let's call that filter
>> expression tree
>>> for the time being (the parse tree used for normalization has some
>> similar
>>> ideas, but a totally different scope and semantics).
>>>
>>> While I now see where you are heading, I fail to see how this could
>>> sufficiently easy be implemented for a real (and complex)
>> configuration.
>>>
>>> Let's look at this example conf:
>>>
>>> mail.* /Var/log/mail.log
>>> if programname startswith abc log to /abc
>>> if programname startswith 'acc' and MSG contains 'error' log to /acc
>>> # we don't want debug messages from here on
>>> *.debug ~
>>> if programname startswith 'acc' or (programname contains 'bde' and
>> (facility
>>>> 1 and
>>> facility < 5)) log to @1.1.1.1
>>> if programname startswith bcd or MSG startswith '123' or MSG contains
>>> 'error2' log to /bcd
>>> *.* /var/log/catchall
>>>
>>> I simply cannot envision how you would store this as a tree that does
>> not
>>> look like what we have today.
>>
>> Ok, Here is what I came up with.
>>
>> 0-1,0-6
>>    --programname startswith abc log to /abc
>>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>>     \programname startswith 'acc' log to @1.1.1.1
>>    --programname contains 'bde' log to @1.1.1.1
>>    --MSG startswith '123' log to/bcd
>>    --MSG contains 'error2' log to /bcd
>>    --log to /var/log/catchall
>> 0-1,7
>>    --programname startswith abc log to /abc
>>     \programname startswith 'acc' and MSG contains 'error' log to /acc
>> 2,0-6
>>    --log to /var/log/mail.log
>>    --programname startswith abc log to /abc
>>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>>    | programname startswith 'acc' log to @1.1.1.1
>>     \programname startswith bcd log to /bcd
>>    --programname contains 'bde' log to @1.1.1.1
>>    --MSG startswith '123' log to/bcd
>>    --MSG contains 'error2' log to /bcd
>>    --log to /var/log/catchall
>> 2,7
>>    --log to /var/log/mail.log
>>    --programname startswith abc log to /abc
>>     \programname startswith 'acc' and MSG contains 'error' log to /acc
>> 3-4,0-6
>>    --programname startswith abc log to /abc
>>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>>    | programname startswith 'acc' log to @1.1.1.1
>>     \programname startswith bcd log to /bcd
>>    --programname contains 'bde' log to @1.1.1.1
>>    --MSG startswith '123' log to/bcd
>>    --MSG contains 'error2' log to /bcd
>>    --log to /var/log/catchall
>> 3-4,7
>>    --programname startswith abc log to /abc
>>     \programname startswith 'acc' and MSG contains 'error' log to /acc
>> 5-23,0-6
>>    --programname startswith abc log to /abc
>>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>>     \programname startswith 'acc' log to @1.1.1.1
>>    --programname contains 'bde' log to @1.1.1.1
>>    --MSG startswith '123' log to/bcd
>>    --MSG contains 'error2' log to /bcd
>>    --log to /var/log/catchall
>> 5-23,7
>>    --programname startswith abc log to /abc
>>     \programname startswith 'acc' and MSG contains 'error' log to /acc
>>
>> This looks very ugly and is definantly more complicated, but I think it
>> still ends up being a slight (but not drastic) win.
>>
>> for the initial ruleset you have to check a minimum of 4 tests and a
>> maximum of 13 tests (debug messages have 4-5 tests, all others have 12-
>> 13)
>>
>> Assuming the facility/priority lookup can be a table lookup, for the
>> tree-based ruleset you have a minimum of 2 tests and a maximum of 7
>> tests,
>> in each case one of the tests is a multi-option comparison which is
>> slightly more expensive than a single 'startswith' test, but should be
>> cheaper than two 'startswith' tests.
>>
>> This definantly trades off start time complexity for runtime
>> efficiancy.
>>
>> David Lang
>>
>>
>> and here is the long, ugly process I went through to derive this.
>>
>>
>>
>> first normalize the tests (replace 'or' with two lines, change facility
>> and priority to ranges of numbers)
>>
>> facility = 2 log to /Var/log/mail.log
>> programname startswith abc log to /abc
>> programname startswith 'acc' and MSG contains 'error' log to /acc
>> severity 0..6 and programname startswith 'acc' log to @1.1.1.1
>> severity 0..6 and programname startswith bcd log to /bcd
>> facility 2..4 and severity 0..6 and programname contains 'bde' log to
>> @1.1.1.1
>> severity 0..6 and MSG startswith '123' log to/bcd
>> severity 0..6 and MSG contains 'error2' log to /bcd
>> severity 0..6 and log to /var/log/catchall
>>
>> I then add the full facility/priority ranges
>>
>>
>> now normalize the facility/severity tests (split overlaps)
>>
>> we have the following ranges for facility
>> 0-1
>> 2
>> 3-4
>> 5-23
>>
>> and for severity
>> 0-6
>> 7
>>
>> to shorten the lines in the e-mail I will letter the remaining
>> tests/actions
>>
>> A) log to /Var/log/mail.log
>> B) programname startswith abc log to /abc
>> C) programname startswith 'acc' and MSG contains 'error' log to /acc
>> D) programname startswith 'acc' log to @1.1.1.1
>> E) programname startswith bcd log to /bcd
>> F) programname contains 'bde' log to @1.1.1.1
>> G) MSG startswith '123' log to/bcd
>> H) MSG contains 'error2' log to /bcd
>> I) log to /var/log/catchall
>>
>> so the tree then becomes (straight translation no reordering)
>>
>> 2,0-6 A
>> 2,7 A
>> 0-1,0-6 B
>> 0-1,7 B
>> 2,0-6 B
>> 2,7 B
>> 3-4,0-6 B
>> 3-4,7 B
>> 5-23,0-6 B
>> 5-23,7 B
>> 0-1,0-6 C
>> 0-1,7 C
>> 2,0-6 C
>> 2,7 C
>> 3-4,0-6 C
>> 3-4,7 C
>> 5-23,0-6 C
>> 5-23,7 C
>> 0-1,0-6 D
>> 2,0-6 D
>> 3-4,0-6 D
>> 5-23,0-6 D
>> 2,0-6 E
>> 3-4,0-6 E
>> 0-1,0-6 F
>> 2,0-6 F
>> 3-4,0-6 F
>> 5-23,0-6 F
>> 0-1,0-6 G
>> 2,0-6 G
>> 3-4,0-6 G
>> 5-23,0-6 G
>> 0-1,0-6 H
>> 2,0-6 H
>> 3-4,0-6 H
>> 5-23,0-6 H
>> 0-1,0-6 I
>> 2,0-6 I
>> 3-4,0-6 I
>> 5-23,0-6 I
>>
>> now to order them
>>
>> 0-1,0-6 B
>> 0-1,0-6 C
>> 0-1,0-6 D
>> 0-1,0-6 F
>> 0-1,0-6 G
>> 0-1,0-6 H
>> 0-1,0-6 I
>> 0-1,7 B
>> 0-1,7 C
>> 2,0-6 A
>> 2,0-6 B
>> 2,0-6 I
>> 2,0-6 H
>> 2,0-6 G
>> 2,0-6 F
>> 2,0-6 E
>> 2,0-6 D
>> 2,0-6 C
>> 2,7 A
>> 2,7 B
>> 2,7 C
>> 3-4,0-6 B
>> 3-4,0-6 C
>> 3-4,0-6 D
>> 3-4,0-6 E
>> 3-4,0-6 F
>> 3-4,0-6 G
>> 3-4,0-6 H
>> 3-4,0-6 I
>> 3-4,7 B
>> 3-4,7 C
>> 5-23,0-6 B
>> 5-23,0-6 C
>> 5-23,0-6 D
>> 5-23,0-6 F
>> 5-23,0-6 G
>> 5-23,0-6 H
>> 5-23,0-6 I
>> 5-23,7 B
>> 5-23,7 C
>>
>> I then simplify this to
>>
>> 0-1,0-6 B,C,D,F,G,H,I
>> 0-1,7 B,C
>> 2,0-6 A,B,C,D,E,F,G,H,I
>> 2,7 A,B,C
>> 3-4,0-6 B,C,D,E,F,G,H,I
>> 3-4,7 B,C
>> 5-23,0-6 B,C,D,F,G,H,I
>> 5-23,7 B,C
>>
>> looking at the lettered tests
>>
>> A) log to /Var/log/mail.log
>> B) programname startswith abc log to /abc
>> C) programname startswith 'acc' and MSG contains 'error' log to /acc
>> D) programname startswith 'acc' log to @1.1.1.1
>> E) programname startswith bcd log to /bcd
>> F) programname contains 'bde' log to @1.1.1.1
>> G) MSG startswith '123' log to/bcd
>> H) MSG contains 'error2' log to /bcd
>> I) log to /var/log/catchall
>>
>> the only ones that can be combined are B-E
>>
>> so if you make tests
>>
>> J) B,C
>> K) B,C,D
>> L) B,C,D,E
>>
>> the results simplify to
>>
>> 0-1,0-6 K,F,G,H,I
>> 0-1,7 J
>> 2,0-6 A,L,F,G,H,I
>> 2,7 A,J
>> 3-4,0-6 L,F,G,H,I
>> 3-4,7 J
>> 5-23,0-6 K,F,G,H,I
>> 5-23,7 J
>>
>> expanded out to show all tests
>>
>> 0-1,0-6
>>    --programname startswith abc log to /abc
>>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>>     \programname startswith 'acc' log to @1.1.1.1
>>    --programname contains 'bde' log to @1.1.1.1
>>    --MSG startswith '123' log to/bcd
>>    --MSG contains 'error2' log to /bcd
>>    --log to /var/log/catchall
>> 0-1,7
>>    --programname startswith abc log to /abc
>>     \programname startswith 'acc' and MSG contains 'error' log to /acc
>> 2,0-6
>>    --log to /var/log/mail.log
>>    --programname startswith abc log to /abc
>>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>>    | programname startswith 'acc' log to @1.1.1.1
>>     \programname startswith bcd log to /bcd
>>    --programname contains 'bde' log to @1.1.1.1
>>    --MSG startswith '123' log to/bcd
>>    --MSG contains 'error2' log to /bcd
>>    --log to /var/log/catchall
>> 2,7
>>    --log to /var/log/mail.log
>>    --programname startswith abc log to /abc
>>     \programname startswith 'acc' and MSG contains 'error' log to /acc
>> 3-4,0-6
>>    --programname startswith abc log to /abc
>>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>>    | programname startswith 'acc' log to @1.1.1.1
>>     \programname startswith bcd log to /bcd
>>    --programname contains 'bde' log to @1.1.1.1
>>    --MSG startswith '123' log to/bcd
>>    --MSG contains 'error2' log to /bcd
>>    --log to /var/log/catchall
>> 3-4,7
>>    --programname startswith abc log to /abc
>>     \programname startswith 'acc' and MSG contains 'error' log to /acc
>> 5-23,0-6
>>    --programname startswith abc log to /abc
>>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>>     \programname startswith 'acc' log to @1.1.1.1
>>    --programname contains 'bde' log to @1.1.1.1
>>    --MSG startswith '123' log to/bcd
>>    --MSG contains 'error2' log to /bcd
>>    --log to /var/log/catchall
>> 5-23,7
>>    --programname startswith abc log to /abc
>>     \programname startswith 'acc' and MSG contains 'error' log to /acc
>>
>>> Rainer
>>>> -----Original Message-----
>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>>> Sent: Sunday, February 28, 2010 2:02 PM
>>>> To: rsyslog-users
>>>> Subject: Re: [rsyslog] Feedback requested: inconsistency in config
>>>> statements
>>>>
>>>> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
>>>>
>>>>> Quickly from the mobile: pri based filters require *excatly* one
>>>> single word (!) lookup currently. You can't beat that.
>>>>>
>>>>> I am not totally sure how you think about parse trees. In my view,
>>>> parsin and filteing are two different things./stages. Maybe we have
>>>> different concepts on our minds?
>>>>
>>>> I am probably using the wrong terminology here.
>>>>
>>>> say you have a set of rules that say
>>>>
>>>> if programname startswith abc log to /abc
>>>> if programname startswith acc log to /acc
>>>> if programname startswith acc log to @1.1.1.1
>>>> if programname startswith bcd log to /bcd
>>>>
>>>> and assuming that these were the only possible programnames for
>>>> simplicity
>>>> in the explination.
>>>>
>>>> the filtering logic would go somthing like this
>>>>
>>>> the rules would compile into a tree
>>>>
>>>> -a-b -> /abc
>>>>    -c -> /acc, at 1.1.1.1
>>>> -b   -> /bcd
>>>>
>>>> receive programname abc
>>>> I have no facility/Pri filter rules
>>>> I have no time filter rules
>>>> I have no system filter rules
>>>> I have programname filter rules
>>>>    look at the first character of the programname
>>>>     it's "a", I have more than one rule that fits that.
>>>>      look at the second character of the programname
>>>>       it's a "b", There are no other decisions to make,
>>>>         invoke the action /abc
>>>>
>>>> receive progranmane bcd
>>>> I have no facility/PRI filter rules
>>>> I have no time filter rules
>>>> I have no system filter rules
>>>> I have programname filter rules
>>>>    look at the first character of the programname
>>>>     it's a "b", There are no other decisions to make,
>>>>       invoke the action /bcd
>>>>
>>>> receive programname acc
>>>> I have no facility/Pri filter rules
>>>> I have no time filter rules
>>>> I have no system filter rules
>>>> I have programname filter rules
>>>>    look at the first character of the programname
>>>>     it's "a", I have more than one rule that fits that.
>>>>      look at the second character of the programname
>>>>       it's a "c", There are no other decisions to make,
>>>>         invoke the action /acc and the action @1.1.1.1
>>>>
>>>> similarly the facility/pri filtering rules would be either compiled
>>>> into a
>>>> tree, or (since it's 256 entries total) into a lookup table with
>>>> pointers
>>>> to the list of actions to take for that entry
>>>>
>>>> the idea is to spend the time at startup to create the tree that
>>>> represents the ruleset in order to speed up the processing of each
>>>> individual message.
>>>>
>>>> the real tree would be a bit more complicated than I describe here
>> as
>>>> it
>>>> needs to have 'anything else' entries between the known entries
>> (which
>>>> means that it would not be able to shortcut quite as much as I
>>>> describe),
>>>> and have provision for 'do a more complicated thing (like regex)
>> here
>>>> and
>>>> if it matches continue'. but except for regex matches, this would
>> make
>>>> processing the rules pretty much independant of how many rules there
>>>> were
>>>> or how complex the ruleset is.
>>>>
>>>> there doe snot need to be one single programname filter tree, if you
>>>> had a
>>>> rule that said
>>>> if pri = info and programname startswith def then def
>>>>
>>>> the pri tree/table would have an entry for info that would point to
>> a
>>>> filter tree for programname that just had the check for def in it
>>>>
>>>> am I makeing any more sense now?
>>>>
>>>> David Lang
>>>>
>>>>> rainer
>>>>>
>>>>> ----- Urspr?ngliche Nachricht -----
>>>>> Von: "david at lang.hm" <david at lang.hm>
>>>>> An: "rsyslog-users" <rsyslog at lists.adiscon.com>
>>>>> Gesendet: 28.02.10 11:28
>>>>> Betreff: Re: [rsyslog] Feedback requested: inconsistency in config
>>>> statements
>>>>>
>>>>> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
>>>>>
>>>>>>>
>>>>>>> if the type is depriciated then the destination would be another
>>>>>>> config_option (which you would then look up)
>>>>>>>
>>>>>>> if the type is 'ignore' then it would just be skipped over
>>>> (possibly
>>>>>>> with
>>>>>>> a warning being logged that the config line is being ignored)
>>>>>>>
>>>>>>> this would also clean up some of the current cases where a
>> boolean
>>>>>>> option
>>>>>>> doesn't honor on/off and true/false.
>>>>>>
>>>>>> True/false is NOT a valid boolean option. See
>>>>>>
>>>>>>
>>>>
>> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=18
>>>> 4c0d87
>>>>>> 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308
>>>>>
>>>>> I was not remembering correctly then, maby it was yes/no vs on/off.
>> I
>>>> know
>>>>> I ran into something where what was documented didn't work and I
>>>> changed
>>>>> it to another version of a boolean answer and it fixed the problem.
>>>>>
>>>>>>> For the second half of the config language (telling rsyslog what
>> to
>>>> do
>>>>>>> with the logs it has received) also has several variations in
>> place
>>>> and
>>>>>>> is
>>>>>>> showing issues.
>>>>>>>
>>>>>>> However I think that the right answer here is to end up
>>>> implementing
>>>>>>> something like the parsing trees and then compile the other
>> config
>>>>>>> language options into that tree to be consistant (and fast) under
>>>> the
>>>>>>> covers, no matter which way the instructions are written (except
>>>> when
>>>>>>> you
>>>>>>> have to use regex matches)
>>>>>>
>>>>>> I don't fully agree here with you. For example, the traditional
>> PRI
>>>> based
>>>>>> filter is lightning fast. I don't see any way it could be nearly
>> as
>>>> fast with
>>>>>> any unified approach. Right now, we have a "unified" filter
>>>> structure, but it
>>>>>> contains three filter cases, each one adding potential power at
>> the
>>>> price of
>>>>>> decreased speed. I think we need to keep different types of
>> filters
>>>> in order
>>>>>> to have some lightning-fast ones. But more of this could be done
>>>> under the
>>>>>> hood.
>>>>>
>>>>> My guess/expectation is that the tree-based parsing would be about
>>>> the
>>>>> same speed as the traditional PRI based filter for choices made
>> based
>>>> on
>>>>> PRI, slowing down for other types of conditions only in that more
>> of
>>>> the
>>>>> message may need to be scanned (and if there is no selection at a
>>>> level,
>>>>> skipping that level should be lightning fast). As a result, a set
>> of
>>>>> filteres based soley on programname would be as fast as the current
>>>> PRI
>>>>> filters.
>>>>>
>>>>> David Lang
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Mon Mar  1 20:58:51 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 1 Mar 2010 20:58:51 +0100
Subject: [rsyslog] Feedback requested: inconsistency in config statements
References: <001201cab868$d148da7c$100013ac@intern.adiscon.com><alpine.DEB.2.00.1002280426420.15263@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA710397F@GRFEXC.intern.adiscon.com><alpine.DEB.2.00.1002281346000.15263@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA7103992@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.00.1003011130460.5131@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103998@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Monday, March 01, 2010 8:39 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Feedback requested: inconsistency in config
> statements
> 
> On Mon, 1 Mar 2010, Rainer Gerhards wrote:
> 
> > Hi David,
> >
> > thanks for the explanation, I now see where you are heading too. I
> have not
> > dug into all details, but find the post very interesting and
> insightful. I'll
> > archive it so that I have it at hand when I can take the route to
> actually
> > implement such an approach. All in all, it still looks rather complex
> to me.
> > I guess it would be useful to have a guy with experience in writing
> compiler
> > code optimizers at hand when I tackle that beast ;)
> 
> you may also want to dig into yacc/bison as they are designed to help
> people build parser trees.

I know yacc but not bison. Yacc creates decent single-threading parsers out
of a fixed set of rules. I never thought about using them for message
parsing, because of the ample limitations it has (most notably unable to run
on multiple threads!).

But we need to keep in mind that now are talking about the filter
expressions. That is a different beast. The thing to look at probably would
be the gcc code optimizer, but I have to admit that is too large an effort
for me right now. A good (dragon?;)) book on compiler optimization technics
for expression trees would probably the right path to take.

> I am assuming that this is the type of thing that syslog-ng is doing to
> be
> able to efficiantly handle a lot of log processing.

I hardly believe so. I don't see how this could work out well...

> I currently have a script that looks for a couple hundred programnames
> and
> puts the data into ~50 different files depending on what that name is.
> Given that the output side of rsyslog is currently the bottleneck, I
> would
> not want to try and implement that in rsyslog today, but if rsyslog
> gained
> the ability to handle this sort of ruleset efficiantly it would greatly
> simplify my life.

Rather than trying to optimize the whole expression engine, this could
probably done very easy with a normalizing parser. As part of the
normalization, it could also create a meta-property "filename" that than is
used inside the rule engine. I'd assume that the effort to create that is by
far less than optimizing the expression tree. *This* is part of my approach
of log normalizing.

Rianer
> 
> David Lang
> 
> > Rainer
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >> Sent: Monday, March 01, 2010 3:16 AM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] Feedback requested: inconsistency in config
> >> statements
> >>
> >> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
> >>
> >>> Hi David,
> >>>
> >>> yes, that makes much more sense to me. Let's call that filter
> >> expression tree
> >>> for the time being (the parse tree used for normalization has some
> >> similar
> >>> ideas, but a totally different scope and semantics).
> >>>
> >>> While I now see where you are heading, I fail to see how this could
> >>> sufficiently easy be implemented for a real (and complex)
> >> configuration.
> >>>
> >>> Let's look at this example conf:
> >>>
> >>> mail.* /Var/log/mail.log
> >>> if programname startswith abc log to /abc
> >>> if programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>> # we don't want debug messages from here on
> >>> *.debug ~
> >>> if programname startswith 'acc' or (programname contains 'bde' and
> >> (facility
> >>>> 1 and
> >>> facility < 5)) log to @1.1.1.1
> >>> if programname startswith bcd or MSG startswith '123' or MSG
> contains
> >>> 'error2' log to /bcd
> >>> *.* /var/log/catchall
> >>>
> >>> I simply cannot envision how you would store this as a tree that
> does
> >> not
> >>> look like what we have today.
> >>
> >> Ok, Here is what I came up with.
> >>
> >> 0-1,0-6
> >>    --programname startswith abc log to /abc
> >>    | programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>     \programname startswith 'acc' log to @1.1.1.1
> >>    --programname contains 'bde' log to @1.1.1.1
> >>    --MSG startswith '123' log to/bcd
> >>    --MSG contains 'error2' log to /bcd
> >>    --log to /var/log/catchall
> >> 0-1,7
> >>    --programname startswith abc log to /abc
> >>     \programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >> 2,0-6
> >>    --log to /var/log/mail.log
> >>    --programname startswith abc log to /abc
> >>    | programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>    | programname startswith 'acc' log to @1.1.1.1
> >>     \programname startswith bcd log to /bcd
> >>    --programname contains 'bde' log to @1.1.1.1
> >>    --MSG startswith '123' log to/bcd
> >>    --MSG contains 'error2' log to /bcd
> >>    --log to /var/log/catchall
> >> 2,7
> >>    --log to /var/log/mail.log
> >>    --programname startswith abc log to /abc
> >>     \programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >> 3-4,0-6
> >>    --programname startswith abc log to /abc
> >>    | programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>    | programname startswith 'acc' log to @1.1.1.1
> >>     \programname startswith bcd log to /bcd
> >>    --programname contains 'bde' log to @1.1.1.1
> >>    --MSG startswith '123' log to/bcd
> >>    --MSG contains 'error2' log to /bcd
> >>    --log to /var/log/catchall
> >> 3-4,7
> >>    --programname startswith abc log to /abc
> >>     \programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >> 5-23,0-6
> >>    --programname startswith abc log to /abc
> >>    | programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>     \programname startswith 'acc' log to @1.1.1.1
> >>    --programname contains 'bde' log to @1.1.1.1
> >>    --MSG startswith '123' log to/bcd
> >>    --MSG contains 'error2' log to /bcd
> >>    --log to /var/log/catchall
> >> 5-23,7
> >>    --programname startswith abc log to /abc
> >>     \programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>
> >> This looks very ugly and is definantly more complicated, but I think
> it
> >> still ends up being a slight (but not drastic) win.
> >>
> >> for the initial ruleset you have to check a minimum of 4 tests and a
> >> maximum of 13 tests (debug messages have 4-5 tests, all others have
> 12-
> >> 13)
> >>
> >> Assuming the facility/priority lookup can be a table lookup, for the
> >> tree-based ruleset you have a minimum of 2 tests and a maximum of 7
> >> tests,
> >> in each case one of the tests is a multi-option comparison which is
> >> slightly more expensive than a single 'startswith' test, but should
> be
> >> cheaper than two 'startswith' tests.
> >>
> >> This definantly trades off start time complexity for runtime
> >> efficiancy.
> >>
> >> David Lang
> >>
> >>
> >> and here is the long, ugly process I went through to derive this.
> >>
> >>
> >>
> >> first normalize the tests (replace 'or' with two lines, change
> facility
> >> and priority to ranges of numbers)
> >>
> >> facility = 2 log to /Var/log/mail.log
> >> programname startswith abc log to /abc
> >> programname startswith 'acc' and MSG contains 'error' log to /acc
> >> severity 0..6 and programname startswith 'acc' log to @1.1.1.1
> >> severity 0..6 and programname startswith bcd log to /bcd
> >> facility 2..4 and severity 0..6 and programname contains 'bde' log
> to
> >> @1.1.1.1
> >> severity 0..6 and MSG startswith '123' log to/bcd
> >> severity 0..6 and MSG contains 'error2' log to /bcd
> >> severity 0..6 and log to /var/log/catchall
> >>
> >> I then add the full facility/priority ranges
> >>
> >>
> >> now normalize the facility/severity tests (split overlaps)
> >>
> >> we have the following ranges for facility
> >> 0-1
> >> 2
> >> 3-4
> >> 5-23
> >>
> >> and for severity
> >> 0-6
> >> 7
> >>
> >> to shorten the lines in the e-mail I will letter the remaining
> >> tests/actions
> >>
> >> A) log to /Var/log/mail.log
> >> B) programname startswith abc log to /abc
> >> C) programname startswith 'acc' and MSG contains 'error' log to /acc
> >> D) programname startswith 'acc' log to @1.1.1.1
> >> E) programname startswith bcd log to /bcd
> >> F) programname contains 'bde' log to @1.1.1.1
> >> G) MSG startswith '123' log to/bcd
> >> H) MSG contains 'error2' log to /bcd
> >> I) log to /var/log/catchall
> >>
> >> so the tree then becomes (straight translation no reordering)
> >>
> >> 2,0-6 A
> >> 2,7 A
> >> 0-1,0-6 B
> >> 0-1,7 B
> >> 2,0-6 B
> >> 2,7 B
> >> 3-4,0-6 B
> >> 3-4,7 B
> >> 5-23,0-6 B
> >> 5-23,7 B
> >> 0-1,0-6 C
> >> 0-1,7 C
> >> 2,0-6 C
> >> 2,7 C
> >> 3-4,0-6 C
> >> 3-4,7 C
> >> 5-23,0-6 C
> >> 5-23,7 C
> >> 0-1,0-6 D
> >> 2,0-6 D
> >> 3-4,0-6 D
> >> 5-23,0-6 D
> >> 2,0-6 E
> >> 3-4,0-6 E
> >> 0-1,0-6 F
> >> 2,0-6 F
> >> 3-4,0-6 F
> >> 5-23,0-6 F
> >> 0-1,0-6 G
> >> 2,0-6 G
> >> 3-4,0-6 G
> >> 5-23,0-6 G
> >> 0-1,0-6 H
> >> 2,0-6 H
> >> 3-4,0-6 H
> >> 5-23,0-6 H
> >> 0-1,0-6 I
> >> 2,0-6 I
> >> 3-4,0-6 I
> >> 5-23,0-6 I
> >>
> >> now to order them
> >>
> >> 0-1,0-6 B
> >> 0-1,0-6 C
> >> 0-1,0-6 D
> >> 0-1,0-6 F
> >> 0-1,0-6 G
> >> 0-1,0-6 H
> >> 0-1,0-6 I
> >> 0-1,7 B
> >> 0-1,7 C
> >> 2,0-6 A
> >> 2,0-6 B
> >> 2,0-6 I
> >> 2,0-6 H
> >> 2,0-6 G
> >> 2,0-6 F
> >> 2,0-6 E
> >> 2,0-6 D
> >> 2,0-6 C
> >> 2,7 A
> >> 2,7 B
> >> 2,7 C
> >> 3-4,0-6 B
> >> 3-4,0-6 C
> >> 3-4,0-6 D
> >> 3-4,0-6 E
> >> 3-4,0-6 F
> >> 3-4,0-6 G
> >> 3-4,0-6 H
> >> 3-4,0-6 I
> >> 3-4,7 B
> >> 3-4,7 C
> >> 5-23,0-6 B
> >> 5-23,0-6 C
> >> 5-23,0-6 D
> >> 5-23,0-6 F
> >> 5-23,0-6 G
> >> 5-23,0-6 H
> >> 5-23,0-6 I
> >> 5-23,7 B
> >> 5-23,7 C
> >>
> >> I then simplify this to
> >>
> >> 0-1,0-6 B,C,D,F,G,H,I
> >> 0-1,7 B,C
> >> 2,0-6 A,B,C,D,E,F,G,H,I
> >> 2,7 A,B,C
> >> 3-4,0-6 B,C,D,E,F,G,H,I
> >> 3-4,7 B,C
> >> 5-23,0-6 B,C,D,F,G,H,I
> >> 5-23,7 B,C
> >>
> >> looking at the lettered tests
> >>
> >> A) log to /Var/log/mail.log
> >> B) programname startswith abc log to /abc
> >> C) programname startswith 'acc' and MSG contains 'error' log to /acc
> >> D) programname startswith 'acc' log to @1.1.1.1
> >> E) programname startswith bcd log to /bcd
> >> F) programname contains 'bde' log to @1.1.1.1
> >> G) MSG startswith '123' log to/bcd
> >> H) MSG contains 'error2' log to /bcd
> >> I) log to /var/log/catchall
> >>
> >> the only ones that can be combined are B-E
> >>
> >> so if you make tests
> >>
> >> J) B,C
> >> K) B,C,D
> >> L) B,C,D,E
> >>
> >> the results simplify to
> >>
> >> 0-1,0-6 K,F,G,H,I
> >> 0-1,7 J
> >> 2,0-6 A,L,F,G,H,I
> >> 2,7 A,J
> >> 3-4,0-6 L,F,G,H,I
> >> 3-4,7 J
> >> 5-23,0-6 K,F,G,H,I
> >> 5-23,7 J
> >>
> >> expanded out to show all tests
> >>
> >> 0-1,0-6
> >>    --programname startswith abc log to /abc
> >>    | programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>     \programname startswith 'acc' log to @1.1.1.1
> >>    --programname contains 'bde' log to @1.1.1.1
> >>    --MSG startswith '123' log to/bcd
> >>    --MSG contains 'error2' log to /bcd
> >>    --log to /var/log/catchall
> >> 0-1,7
> >>    --programname startswith abc log to /abc
> >>     \programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >> 2,0-6
> >>    --log to /var/log/mail.log
> >>    --programname startswith abc log to /abc
> >>    | programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>    | programname startswith 'acc' log to @1.1.1.1
> >>     \programname startswith bcd log to /bcd
> >>    --programname contains 'bde' log to @1.1.1.1
> >>    --MSG startswith '123' log to/bcd
> >>    --MSG contains 'error2' log to /bcd
> >>    --log to /var/log/catchall
> >> 2,7
> >>    --log to /var/log/mail.log
> >>    --programname startswith abc log to /abc
> >>     \programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >> 3-4,0-6
> >>    --programname startswith abc log to /abc
> >>    | programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>    | programname startswith 'acc' log to @1.1.1.1
> >>     \programname startswith bcd log to /bcd
> >>    --programname contains 'bde' log to @1.1.1.1
> >>    --MSG startswith '123' log to/bcd
> >>    --MSG contains 'error2' log to /bcd
> >>    --log to /var/log/catchall
> >> 3-4,7
> >>    --programname startswith abc log to /abc
> >>     \programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >> 5-23,0-6
> >>    --programname startswith abc log to /abc
> >>    | programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>     \programname startswith 'acc' log to @1.1.1.1
> >>    --programname contains 'bde' log to @1.1.1.1
> >>    --MSG startswith '123' log to/bcd
> >>    --MSG contains 'error2' log to /bcd
> >>    --log to /var/log/catchall
> >> 5-23,7
> >>    --programname startswith abc log to /abc
> >>     \programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>
> >>> Rainer
> >>>> -----Original Message-----
> >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >>>> Sent: Sunday, February 28, 2010 2:02 PM
> >>>> To: rsyslog-users
> >>>> Subject: Re: [rsyslog] Feedback requested: inconsistency in config
> >>>> statements
> >>>>
> >>>> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
> >>>>
> >>>>> Quickly from the mobile: pri based filters require *excatly* one
> >>>> single word (!) lookup currently. You can't beat that.
> >>>>>
> >>>>> I am not totally sure how you think about parse trees. In my
> view,
> >>>> parsin and filteing are two different things./stages. Maybe we
> have
> >>>> different concepts on our minds?
> >>>>
> >>>> I am probably using the wrong terminology here.
> >>>>
> >>>> say you have a set of rules that say
> >>>>
> >>>> if programname startswith abc log to /abc
> >>>> if programname startswith acc log to /acc
> >>>> if programname startswith acc log to @1.1.1.1
> >>>> if programname startswith bcd log to /bcd
> >>>>
> >>>> and assuming that these were the only possible programnames for
> >>>> simplicity
> >>>> in the explination.
> >>>>
> >>>> the filtering logic would go somthing like this
> >>>>
> >>>> the rules would compile into a tree
> >>>>
> >>>> -a-b -> /abc
> >>>>    -c -> /acc, at 1.1.1.1
> >>>> -b   -> /bcd
> >>>>
> >>>> receive programname abc
> >>>> I have no facility/Pri filter rules
> >>>> I have no time filter rules
> >>>> I have no system filter rules
> >>>> I have programname filter rules
> >>>>    look at the first character of the programname
> >>>>     it's "a", I have more than one rule that fits that.
> >>>>      look at the second character of the programname
> >>>>       it's a "b", There are no other decisions to make,
> >>>>         invoke the action /abc
> >>>>
> >>>> receive progranmane bcd
> >>>> I have no facility/PRI filter rules
> >>>> I have no time filter rules
> >>>> I have no system filter rules
> >>>> I have programname filter rules
> >>>>    look at the first character of the programname
> >>>>     it's a "b", There are no other decisions to make,
> >>>>       invoke the action /bcd
> >>>>
> >>>> receive programname acc
> >>>> I have no facility/Pri filter rules
> >>>> I have no time filter rules
> >>>> I have no system filter rules
> >>>> I have programname filter rules
> >>>>    look at the first character of the programname
> >>>>     it's "a", I have more than one rule that fits that.
> >>>>      look at the second character of the programname
> >>>>       it's a "c", There are no other decisions to make,
> >>>>         invoke the action /acc and the action @1.1.1.1
> >>>>
> >>>> similarly the facility/pri filtering rules would be either
> compiled
> >>>> into a
> >>>> tree, or (since it's 256 entries total) into a lookup table with
> >>>> pointers
> >>>> to the list of actions to take for that entry
> >>>>
> >>>> the idea is to spend the time at startup to create the tree that
> >>>> represents the ruleset in order to speed up the processing of each
> >>>> individual message.
> >>>>
> >>>> the real tree would be a bit more complicated than I describe here
> >> as
> >>>> it
> >>>> needs to have 'anything else' entries between the known entries
> >> (which
> >>>> means that it would not be able to shortcut quite as much as I
> >>>> describe),
> >>>> and have provision for 'do a more complicated thing (like regex)
> >> here
> >>>> and
> >>>> if it matches continue'. but except for regex matches, this would
> >> make
> >>>> processing the rules pretty much independant of how many rules
> there
> >>>> were
> >>>> or how complex the ruleset is.
> >>>>
> >>>> there doe snot need to be one single programname filter tree, if
> you
> >>>> had a
> >>>> rule that said
> >>>> if pri = info and programname startswith def then def
> >>>>
> >>>> the pri tree/table would have an entry for info that would point
> to
> >> a
> >>>> filter tree for programname that just had the check for def in it
> >>>>
> >>>> am I makeing any more sense now?
> >>>>
> >>>> David Lang
> >>>>
> >>>>> rainer
> >>>>>
> >>>>> ----- Urspr?ngliche Nachricht -----
> >>>>> Von: "david at lang.hm" <david at lang.hm>
> >>>>> An: "rsyslog-users" <rsyslog at lists.adiscon.com>
> >>>>> Gesendet: 28.02.10 11:28
> >>>>> Betreff: Re: [rsyslog] Feedback requested: inconsistency in
> config
> >>>> statements
> >>>>>
> >>>>> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
> >>>>>
> >>>>>>>
> >>>>>>> if the type is depriciated then the destination would be
> another
> >>>>>>> config_option (which you would then look up)
> >>>>>>>
> >>>>>>> if the type is 'ignore' then it would just be skipped over
> >>>> (possibly
> >>>>>>> with
> >>>>>>> a warning being logged that the config line is being ignored)
> >>>>>>>
> >>>>>>> this would also clean up some of the current cases where a
> >> boolean
> >>>>>>> option
> >>>>>>> doesn't honor on/off and true/false.
> >>>>>>
> >>>>>> True/false is NOT a valid boolean option. See
> >>>>>>
> >>>>>>
> >>>>
> >>
> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=18
> >>>> 4c0d87
> >>>>>> 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308
> >>>>>
> >>>>> I was not remembering correctly then, maby it was yes/no vs
> on/off.
> >> I
> >>>> know
> >>>>> I ran into something where what was documented didn't work and I
> >>>> changed
> >>>>> it to another version of a boolean answer and it fixed the
> problem.
> >>>>>
> >>>>>>> For the second half of the config language (telling rsyslog
> what
> >> to
> >>>> do
> >>>>>>> with the logs it has received) also has several variations in
> >> place
> >>>> and
> >>>>>>> is
> >>>>>>> showing issues.
> >>>>>>>
> >>>>>>> However I think that the right answer here is to end up
> >>>> implementing
> >>>>>>> something like the parsing trees and then compile the other
> >> config
> >>>>>>> language options into that tree to be consistant (and fast)
> under
> >>>> the
> >>>>>>> covers, no matter which way the instructions are written
> (except
> >>>> when
> >>>>>>> you
> >>>>>>> have to use regex matches)
> >>>>>>
> >>>>>> I don't fully agree here with you. For example, the traditional
> >> PRI
> >>>> based
> >>>>>> filter is lightning fast. I don't see any way it could be nearly
> >> as
> >>>> fast with
> >>>>>> any unified approach. Right now, we have a "unified" filter
> >>>> structure, but it
> >>>>>> contains three filter cases, each one adding potential power at
> >> the
> >>>> price of
> >>>>>> decreased speed. I think we need to keep different types of
> >> filters
> >>>> in order
> >>>>>> to have some lightning-fast ones. But more of this could be done
> >>>> under the
> >>>>>> hood.
> >>>>>
> >>>>> My guess/expectation is that the tree-based parsing would be
> about
> >>>> the
> >>>>> same speed as the traditional PRI based filter for choices made
> >> based
> >>>> on
> >>>>> PRI, slowing down for other types of conditions only in that more
> >> of
> >>>> the
> >>>>> message may need to be scanned (and if there is no selection at a
> >>>> level,
> >>>>> skipping that level should be lightning fast). As a result, a set
> >> of
> >>>>> filteres based soley on programname would be as fast as the
> current
> >>>> PRI
> >>>>> filters.
> >>>>>
> >>>>> David Lang
> >>>>> _______________________________________________
> >>>>> rsyslog mailing list
> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>> http://www.rsyslog.com
> >>>>> _______________________________________________
> >>>>> rsyslog mailing list
> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>> http://www.rsyslog.com
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.com
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >>>
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From lanas at securenet.net  Tue Mar  2 02:43:31 2010
From: lanas at securenet.net (lanas)
Date: Mon, 1 Mar 2010 20:43:31 -0500
Subject: [rsyslog] Sending time in remote syslog - test
Message-ID: <20100301204331.1686d17d@mistral.stie>

Hello,

  I did a test using %timereported% in a rsyslog running on a server,
to see if the time of a syslog msg sent from a unit was shown using
the local time of the unit and not the time it was received.

  To easily see the difference I out the time 2 hours back on the
unit.  The unit is running syslogd.  That is, the original syslogd,
not the ng.

  The unit running ran syslogd with the following in its
configuration:

*.*     @10.200.19.162

  The logger utility was used to create log msgs locally on the unit.
These log msgs are then sent to rsyslog on the remote server.

  The workstation was running rsyslog with either:

#$template precise,"%timereported% from: %HOSTNAME% %syslogtag%:%msg%\n"
$template precise,"%TIMESTAMP% from: %HOSTNAME% %syslogtag%:%msg%\n"

  And tests were done with the following commented or not:

$ActionFileDefaultTemplate precise

  (I am not sure if the 'precise' template definition must be used
with $ActionFileDefaultTemplate to take effect)

  The following was then found on the unit after executing the logger
utility:

Feb 27 10:55:34 localhost syslogd 1.4.1#18RR6: restart.
Feb 27 10:55:35 localhost kernel: klogd 1.4.1#18RR6, log source =
                /proc/kmsg started.
Feb 27 10:55:44 localhost root: TEST7

  And these messages were received by rsyslog (after restarting
rsyslog to have its version info included here) :

Feb 27 12:52:32 from: localhost rsyslogd:: [origin software="rsyslogd"
    swVersion="3.22.1" x-pid="6909" x-info="http://www.rsyslog.com"]
    (re)start
Feb 27 12:54:38 from: syslogd 1.4.1#18RR6:: restart.
Feb 27 12:54:39 from: brouter kernel:: klogd 1.4.1#18RR6, log source =
                /proc/kmsg started.
Feb 27 12:54:48 from: brouter root:: TEST7

  The TEST7 msg did now show the unit's time.

  Now, I installed rsyslog at the unit as a drop-in syslog replacement
w/o any configuration change and when rsyslog is used, the local unit
time is received and shown by the server.

  Can we presume that syslogd first generation does not send the time
to a remote syslogging facility ?

Regards,

Al



From rgerhards at hq.adiscon.com  Tue Mar  2 10:03:20 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 2 Mar 2010 10:03:20 +0100
Subject: [rsyslog] Sending time in remote syslog - test
References: <20100301204331.1686d17d@mistral.stie>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71039A1@GRFEXC.intern.adiscon.com>

you can use 

$template raw,"%rawmsg%\n"

to see exactly what the remote box sends. If I recall correctly, sysklogd
sends timestamps, but I may be wrong

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of lanas
> Sent: Tuesday, March 02, 2010 2:44 AM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] Sending time in remote syslog - test
> 
> Hello,
> 
>   I did a test using %timereported% in a rsyslog running on a server,
> to see if the time of a syslog msg sent from a unit was shown using
> the local time of the unit and not the time it was received.
> 
>   To easily see the difference I out the time 2 hours back on the
> unit.  The unit is running syslogd.  That is, the original syslogd,
> not the ng.
> 
>   The unit running ran syslogd with the following in its
> configuration:
> 
> *.*     @10.200.19.162
> 
>   The logger utility was used to create log msgs locally on the unit.
> These log msgs are then sent to rsyslog on the remote server.
> 
>   The workstation was running rsyslog with either:
> 
> #$template precise,"%timereported% from: %HOSTNAME%
> %syslogtag%:%msg%\n"
> $template precise,"%TIMESTAMP% from: %HOSTNAME% %syslogtag%:%msg%\n"
> 
>   And tests were done with the following commented or not:
> 
> $ActionFileDefaultTemplate precise
> 
>   (I am not sure if the 'precise' template definition must be used
> with $ActionFileDefaultTemplate to take effect)
> 
>   The following was then found on the unit after executing the logger
> utility:
> 
> Feb 27 10:55:34 localhost syslogd 1.4.1#18RR6: restart.
> Feb 27 10:55:35 localhost kernel: klogd 1.4.1#18RR6, log source =
>                 /proc/kmsg started.
> Feb 27 10:55:44 localhost root: TEST7
> 
>   And these messages were received by rsyslog (after restarting
> rsyslog to have its version info included here) :
> 
> Feb 27 12:52:32 from: localhost rsyslogd:: [origin software="rsyslogd"
>     swVersion="3.22.1" x-pid="6909" x-info="http://www.rsyslog.com"]
>     (re)start
> Feb 27 12:54:38 from: syslogd 1.4.1#18RR6:: restart.
> Feb 27 12:54:39 from: brouter kernel:: klogd 1.4.1#18RR6, log source =
>                 /proc/kmsg started.
> Feb 27 12:54:48 from: brouter root:: TEST7
> 
>   The TEST7 msg did now show the unit's time.
> 
>   Now, I installed rsyslog at the unit as a drop-in syslog replacement
> w/o any configuration change and when rsyslog is used, the local unit
> time is received and shown by the server.
> 
>   Can we presume that syslogd first generation does not send the time
> to a remote syslogging facility ?
> 
> Regards,
> 
> Al
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From mbiebl at gmail.com  Tue Mar  2 14:11:06 2010
From: mbiebl at gmail.com (Michael Biebl)
Date: Tue, 2 Mar 2010 14:11:06 +0100
Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103983@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com>
	<9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com>
	<aa76cdac1002260355l26269784y8fb8ccd45caeaa12@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com>
	<aa76cdac1002260443p4c7744e2ha1c8c2372c741ebf@mail.gmail.com>
	<aa76cdac1002280515g72c07527mae8ebc08d4ecd404@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103980@GRFEXC.intern.adiscon.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103981@GRFEXC.intern.adiscon.com>
	<aa76cdac1002281500g3d50db29rcd20fea1fbf08d48@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103983@GRFEXC.intern.adiscon.com>
Message-ID: <aa76cdac1003020511n5378077ag313de591d05cb4fa@mail.gmail.com>

2010/3/1 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
>> Sent: Monday, March 01, 2010 12:01 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released
>>
>> I use $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>> in my rsyslog.conf, but the new ompipe module does not seem to respect
>> that setting but always uses the high-resolution time format.
>
> I see - that was a side-effect that pipes used the same output module like
> regular pipes. But I have now re-established it, please try the newest git
> v4-stable branch.

Hi Rainer,

seems current v4-stable works fine now wrt ActionFileDefaultTemplate and ompipe.

Thanks,
Michael


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Tue Mar  2 14:14:08 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 2 Mar 2010 14:14:08 +0100
Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released
References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com><aa76cdac1002260355l26269784y8fb8ccd45caeaa12@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com><aa76cdac1002260443p4c7744e2ha1c8c2372c741ebf@mail.gmail.com><aa76cdac1002280515g72c07527mae8ebc08d4ecd404@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103980@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103981@GRFEXC.intern.adiscon.com><aa76cdac1002281500g3d50db29rcd20fea1fbf08d48@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103983@GRFEXC.intern.adiscon.com>
	<aa76cdac1003020511n5378077ag313de591d05cb4fa@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71039B4@GRFEXC.intern.adiscon.com>

> seems current v4-stable works fine now wrt ActionFileDefaultTemplate
> and ompipe.

Thanks for letting me know. I'll release that as 4.6.1, but wait for a couple
of other things so that they can make a new release together.

Rainer

> 
> Thanks,
> Michael




From david at lang.hm  Wed Mar  3 07:14:30 2010
From: david at lang.hm (david at lang.hm)
Date: Tue, 2 Mar 2010 22:14:30 -0800 (PST)
Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71039B4@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com><aa76cdac1002260355l26269784y8fb8ccd45caeaa12@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com><aa76cdac1002260443p4c7744e2ha1c8c2372c741ebf@mail.gmail.com><aa76cdac1002280515g72c07527mae8ebc08d4ecd404@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103980@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103981@GRFEXC.intern.adiscon.com><aa76cdac1002281500g3d50db29rcd20fea1fbf08d48@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103983@GRFEXC.intern.adiscon.com>
	<aa76cdac1003020511n5378077ag313de591d05cb4fa@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA71039B4@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.2.00.1003022213520.15263@asgard.lang.hm>

slightly off topic, but still related to releases, how close do you think 
we are to a new stable 5.x release?

David Lang

On Tue, 2 Mar 2010, Rainer Gerhards wrote:

>> seems current v4-stable works fine now wrt ActionFileDefaultTemplate
>> and ompipe.
>
> Thanks for letting me know. I'll release that as 4.6.1, but wait for a couple
> of other things so that they can make a new release together.
>
> Rainer
>
>>
>> Thanks,
>> Michael
>
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Wed Mar  3 07:17:11 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 3 Mar 2010 07:17:11 +0100
Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released
References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com><aa76cdac1002260355l26269784y8fb8ccd45caeaa12@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com><aa76cdac1002260443p4c7744e2ha1c8c2372c741ebf@mail.gmail.com><aa76cdac1002280515g72c07527mae8ebc08d4ecd404@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103980@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103981@GRFEXC.intern.adiscon.com><aa76cdac1002281500g3d50db29rcd20fea1fbf08d48@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103983@GRFEXC.intern.adiscon.com><aa76cdac1003020511n5378077ag313de591d05cb4fa@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA71039B4@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.00.1003022213520.15263@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71039B8@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Wednesday, March 03, 2010 7:15 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released
> 
> slightly off topic, but still related to releases, how close do you
> think
> we are to a new stable 5.x release?

very, very close. I originally intended to release it Monday, but an issue
with 4.6.0 came up that, if possible at all, I would like to integrate.
Yesterday, a bug was filed that points into the same direction:

http://bugzilla.adiscon.com/show_bug.cgi?id=182

If I can't find it quickly enough, I'll probably "timeout" next Monday.

Rainer


From reader at newsguy.com  Wed Mar  3 17:43:07 2010
From: reader at newsguy.com (Harry Putnam)
Date: Wed, 03 Mar 2010 10:43:07 -0600
Subject: [rsyslog] How rsyslog derives hostname
References: <87hbp0eodt.fsf@newsguy.com> <4B8B6F9D.5040803@jackpot.uk.net>
Message-ID: <87ocj59x0k.fsf@newsguy.com>

"Mr. Demeanour" <mrdemeanour at jackpot.uk.net> writes:

> Harry Putnam wrote:
>> 
>> Further... I'm not sure where cmds like gethostbyname (and similar
>> ones) get the info either... for all I know they may just grep
>> /etc/hosts.
>
> man resolv.conf (i.e. it's configurable).

I'm not sure how that would return the miss-typed name in /etc/hosts
There is not mention of the hosts file in man resolve.conf what so ever.

If I set a miss-typed name in /etc/hosts

It appears in rsyslog logs   But /etc/resolv.conf only point to
these nameservers

  nameserver 68.87.72.130 <=A comcast nameserver

  nameserver 192.168.0.20 <= My local router which uses the 
                             nameserver above

Neither of these nameservers will be able to resolve my localhosts
name.  It's not a real domain just a home lan.

So something is searching /etc/hosts and giving the info to rsyslog.

However the command `hostname' will return the correct
hostname... not the miss-typed host in /etc/hosts.

Apparently the command `hostname' uses a different mechanism.
I.E scanning /etc/conf.d/hostname... it seems like rsyslog should do
the same to discover the localhosts name.

I see from Gerhards response:
> I need to re-read this more careful, but for now let me say that
> rsyslog does not touch /etc/hosts at all. It exlusively relies on
> what is returned by the OS. But the "source" property is not locally
> generated, it contains whatever the sender placed into the relevant
> field.

That apparently the `sender' (my localhost) is telling rsyslog to use
the miss-typed host name in /etc/hosts.

He says rsyslog exclusively relies on what is returned by the os. Yes
but by what mechanism?

For example: rsyslog could rely on the output of the `hostname' cmd.
But apparently calls something else.

  If it calls resolv.conf... those nameservers will not return
  anything useful.



From rgerhards at hq.adiscon.com  Wed Mar  3 17:49:55 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 3 Mar 2010 17:49:55 +0100
Subject: [rsyslog] How rsyslog derives hostname
References: <87hbp0eodt.fsf@newsguy.com> <4B8B6F9D.5040803@jackpot.uk.net>
	<87ocj59x0k.fsf@newsguy.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71039C8@GRFEXC.intern.adiscon.com>

> He says rsyslog exclusively relies on what is returned by the os. Yes
> but by what mechanism?

gethostname(), what according to the man page is the mechanism to use ;)

http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/net.c;h=ab431f7cddfda4
1b689e6ab4819b5ffe031e52fb;hb=HEAD#l1331

Rainer


From reader at newsguy.com  Wed Mar  3 17:54:09 2010
From: reader at newsguy.com (Harry Putnam)
Date: Wed, 03 Mar 2010 10:54:09 -0600
Subject: [rsyslog] How rsyslog derives hostname
References: <87hbp0eodt.fsf@newsguy.com>
	<9B6E2A8877C38245BFB15CC491A11DA710398A@GRFEXC.intern.adiscon.com>
Message-ID: <87hbox9wi6.fsf@newsguy.com>

"Rainer Gerhards" <rgerhards at hq.adiscon.com> writes:

> I need to re-read this more careful, but for now let me say that rsyslog does
> not touch /etc/hosts at all. It exlusively relies on what is returned by the
> OS. But the "source" property is not locally generated, it contains whatever
> the sender placed into the relevant field.

Good info ... thanks.  So my host is sending the miss-typed host name
to rsyslog by having it in the log line 
   DATE hostname => blah

So I guess the question is how does my host derive the hostname it
sticks into log messages.

So thanks... and I already know the localhost will scan the /etc/hosts
file when resolving the localhost name... so there is the culprit.

Of course the culprit is really me, who miss-typed the host name.

Thanks.



From reader at newsguy.com  Wed Mar  3 18:00:46 2010
From: reader at newsguy.com (Harry Putnam)
Date: Wed, 03 Mar 2010 11:00:46 -0600
Subject: [rsyslog] How rsyslog derives hostname
References: <87hbp0eodt.fsf@newsguy.com> <4B8B6F9D.5040803@jackpot.uk.net>
	<87ocj59x0k.fsf@newsguy.com>
	<9B6E2A8877C38245BFB15CC491A11DA71039C8@GRFEXC.intern.adiscon.com>
Message-ID: <87d3zl9w75.fsf@newsguy.com>

"Rainer Gerhards" <rgerhards at hq.adiscon.com> writes:

>> He says rsyslog exclusively relies on what is returned by the os. Yes
>> but by what mechanism?
>
> gethostname(), what according to the man page is the mechanism to use ;)
>
> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/net.c;h=ab431f7cddfda4
> 1b689e6ab4819b5ffe031e52fb;hb=HEAD#l1331

Looks pretty complicated... gack.

Thanks... Apparently somewhere in the workings of gethostname() the
/etc/hosts file is scanned.

Thanks.




From rgerhards at hq.adiscon.com  Wed Mar  3 18:02:59 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 3 Mar 2010 18:02:59 +0100
Subject: [rsyslog] How rsyslog derives hostname
References: <87hbp0eodt.fsf@newsguy.com><9B6E2A8877C38245BFB15CC491A11DA710398A@GRFEXC.intern.adiscon.com>
	<87hbox9wi6.fsf@newsguy.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71039C9@GRFEXC.intern.adiscon.com>

> > I need to re-read this more careful, but for now let me say that
> rsyslog does
> > not touch /etc/hosts at all. It exlusively relies on what is returned
> by the
> > OS. But the "source" property is not locally generated, it contains
> whatever
> > the sender placed into the relevant field.
> 
> Good info ... thanks.  So my host is sending the miss-typed host name
> to rsyslog by having it in the log line
>    DATE hostname => blah

I still intended to do a write-up in the cookbook, but more important things
got into the way. So here quickly:

You need to be careful when using hostname to build pathes. Bad guys may put
malicous sequences into it. There are property replacer options to solve
that, primarily removing slashes (something along the lines of "secpath" or
so).

If possible, it is better to use fromhost-ip or fromhost, but that is only
the last sender, so not really useful in a relay chain.

Rainer

> 
> So I guess the question is how does my host derive the hostname it
> sticks into log messages.
> 
> So thanks... and I already know the localhost will scan the /etc/hosts
> file when resolving the localhost name... so there is the culprit.
> 
> Of course the culprit is really me, who miss-typed the host name.
> 
> Thanks.
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Wed Mar  3 18:03:33 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 3 Mar 2010 18:03:33 +0100
Subject: [rsyslog] How rsyslog derives hostname
References: <87hbp0eodt.fsf@newsguy.com>
	<4B8B6F9D.5040803@jackpot.uk.net><87ocj59x0k.fsf@newsguy.com><9B6E2A8877C38245BFB15CC491A11DA71039C8@GRFEXC.intern.adiscon.com>
	<87d3zl9w75.fsf@newsguy.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71039CA@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Harry Putnam
> Sent: Wednesday, March 03, 2010 6:01 PM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] How rsyslog derives hostname
> 
> "Rainer Gerhards" <rgerhards at hq.adiscon.com> writes:
> 
> >> He says rsyslog exclusively relies on what is returned by the os.
> Yes
> >> but by what mechanism?
> >
> > gethostname(), what according to the man page is the mechanism to use
> ;)
> >
> >
> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/net.c;h=ab431f7c
> ddfda4
> > 1b689e6ab4819b5ffe031e52fb;hb=HEAD#l1331
> 
> Looks pretty complicated... gack.
> 
> Thanks... Apparently somewhere in the workings of gethostname() the
> /etc/hosts file is scanned.

My (uneducated) guess is that this is distro/version specific.

Raienr


From iamsayan at gmail.com  Wed Mar  3 18:19:52 2010
From: iamsayan at gmail.com (Sayan Chowdhury)
Date: Wed, 3 Mar 2010 22:49:52 +0530
Subject: [rsyslog] How rsyslog derives hostname
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71039CA@GRFEXC.intern.adiscon.com>
References: <87hbp0eodt.fsf@newsguy.com> <4B8B6F9D.5040803@jackpot.uk.net>
	<87ocj59x0k.fsf@newsguy.com>
	<9B6E2A8877C38245BFB15CC491A11DA71039C8@GRFEXC.intern.adiscon.com>
	<87d3zl9w75.fsf@newsguy.com>
	<9B6E2A8877C38245BFB15CC491A11DA71039CA@GRFEXC.intern.adiscon.com>
Message-ID: <87a1ae541003030919j632dac91n75c67717248a21c5@mail.gmail.com>

Hello Harry,
Look at your /etc/host.conf file, it's most probably set up to look at the
/etc/hosts file before the name servers.
Regards,
Sayan

On Wed, Mar 3, 2010 at 10:33 PM, Rainer Gerhards
<rgerhards at hq.adiscon.com>wrote:

> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Harry Putnam
> > Sent: Wednesday, March 03, 2010 6:01 PM
> > To: rsyslog at lists.adiscon.com
> > Subject: Re: [rsyslog] How rsyslog derives hostname
> >
> > "Rainer Gerhards" <rgerhards at hq.adiscon.com> writes:
> >
> > >> He says rsyslog exclusively relies on what is returned by the os.
> > Yes
> > >> but by what mechanism?
> > >
> > > gethostname(), what according to the man page is the mechanism to use
> > ;)
> > >
> > >
> > http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/net.c;h=ab431f7c
> > ddfda4
> > > 1b689e6ab4819b5ffe031e52fb;hb=HEAD#l1331
> >
> > Looks pretty complicated... gack.
> >
> > Thanks... Apparently somewhere in the workings of gethostname() the
> > /etc/hosts file is scanned.
>
> My (uneducated) guess is that this is distro/version specific.
>
> Raienr
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From mrdemeanour at jackpot.uk.net  Wed Mar  3 19:18:16 2010
From: mrdemeanour at jackpot.uk.net (Mr. Demeanour)
Date: Wed, 03 Mar 2010 18:18:16 +0000
Subject: [rsyslog] How rsyslog derives hostname
In-Reply-To: <87ocj59x0k.fsf@newsguy.com>
References: <87hbp0eodt.fsf@newsguy.com> <4B8B6F9D.5040803@jackpot.uk.net>
	<87ocj59x0k.fsf@newsguy.com>
Message-ID: <4B8EA7E8.9020604@jackpot.uk.net>

Harry Putnam wrote:
> "Mr. Demeanour" <mrdemeanour at jackpot.uk.net> writes:
> 
>> Harry Putnam wrote:
>>> Further... I'm not sure where cmds like gethostbyname (and
>>> similar ones) get the info either... for all I know they may just
>>> grep /etc/hosts.
>> man resolv.conf (i.e. it's configurable).
> 
> I'm not sure how that would return the miss-typed name in /etc/hosts 
> There is not mention of the hosts file in man resolve.conf what so
> ever.

Sorry, my response was both too terse and inaccurate. I should have
pointed you instead at /etc/nsswitch.conf, which the name-resolution
library (i.e. gethostbyname()) uses to figure-out how it's supposed to
resolve hostnames into addresses.

There's a line in there that starts with "hosts: ", usually followed on
the same line by "files" and "dns" in that order (and often accompanied
by other bits). That says that to resolve "hosts", you should first look
at "files", then at "dns". In this context, "files" means "/etc/hosts".

So if your nsswitch.conf mentions "files" in the "hosts:" line before
"dns", that is telling the local name-resolution library to look at
/etc/hosts, and use any answer it can get from there in preference to a
real DNS lookup.

I hope that's a bit more helpful!

-- 
Jack.


From tbergfeld at hq.adiscon.com  Thu Mar  4 11:47:36 2010
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Thu, 4 Mar 2010 11:47:36 +0100
Subject: [rsyslog] rsyslog 4.6.1 (v4-stable) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71039DD@GRFEXC.intern.adiscon.com>

Hi all,
We have just released rsyslog 4.6.1.

Rsyslog 4.6.1 is a bug fixing release. Most importantly, it fixes an issue
with the build system that could potentially result in segfaults, especially
on 32 bit machines. Please review the ChangeLog for details.

This is a recommended update for all v4-stable users.

ChangeLog:

http://www.rsyslog.com/Article445.phtml

Download:

http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-196.phtml


As always, feedback is appreciated.

Best regards,
Tom Bergfeld
--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com


From lanas at securenet.net  Thu Mar  4 12:22:13 2010
From: lanas at securenet.net (lanas)
Date: Thu, 4 Mar 2010 06:22:13 -0500
Subject: [rsyslog] Specifying the source address
Message-ID: <20100304062213.5e5c5a3e@mistral.stie>

Hello,

  Is it possible to specify the source address of rsyslog log messages
when logging to a remote host ?

Thanks.


From david at lang.hm  Thu Mar  4 21:13:10 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 4 Mar 2010 12:13:10 -0800 (PST)
Subject: [rsyslog] How rsyslog derives hostname
In-Reply-To: <87a1ae541003030919j632dac91n75c67717248a21c5@mail.gmail.com>
References: <87hbp0eodt.fsf@newsguy.com> <4B8B6F9D.5040803@jackpot.uk.net>
	<87ocj59x0k.fsf@newsguy.com>
	<9B6E2A8877C38245BFB15CC491A11DA71039C8@GRFEXC.intern.adiscon.com>
	<87d3zl9w75.fsf@newsguy.com>
	<9B6E2A8877C38245BFB15CC491A11DA71039CA@GRFEXC.intern.adiscon.com>
	<87a1ae541003030919j632dac91n75c67717248a21c5@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1003041212550.5131@asgard.lang.hm>

or there may be a services-switch file

David Lang

On Wed, 3 Mar 2010, Sayan Chowdhury wrote:

> Hello Harry,
> Look at your /etc/host.conf file, it's most probably set up to look at the
> /etc/hosts file before the name servers.
> Regards,
> Sayan
>
> On Wed, Mar 3, 2010 at 10:33 PM, Rainer Gerhards
> <rgerhards at hq.adiscon.com>wrote:
>
>>> -----Original Message-----
>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>> bounces at lists.adiscon.com] On Behalf Of Harry Putnam
>>> Sent: Wednesday, March 03, 2010 6:01 PM
>>> To: rsyslog at lists.adiscon.com
>>> Subject: Re: [rsyslog] How rsyslog derives hostname
>>>
>>> "Rainer Gerhards" <rgerhards at hq.adiscon.com> writes:
>>>
>>>>> He says rsyslog exclusively relies on what is returned by the os.
>>> Yes
>>>>> but by what mechanism?
>>>>
>>>> gethostname(), what according to the man page is the mechanism to use
>>> ;)
>>>>
>>>>
>>> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/net.c;h=ab431f7c
>>> ddfda4
>>>> 1b689e6ab4819b5ffe031e52fb;hb=HEAD#l1331
>>>
>>> Looks pretty complicated... gack.
>>>
>>> Thanks... Apparently somewhere in the workings of gethostname() the
>>> /etc/hosts file is scanned.
>>
>> My (uneducated) guess is that this is distro/version specific.
>>
>> Raienr
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From reader at newsguy.com  Fri Mar  5 02:39:14 2010
From: reader at newsguy.com (Harry Putnam)
Date: Thu, 04 Mar 2010 19:39:14 -0600
Subject: [rsyslog] How rsyslog derives hostname
References: <87hbp0eodt.fsf@newsguy.com> <4B8B6F9D.5040803@jackpot.uk.net>
	<87ocj59x0k.fsf@newsguy.com>
	<9B6E2A8877C38245BFB15CC491A11DA71039C8@GRFEXC.intern.adiscon.com>
	<87d3zl9w75.fsf@newsguy.com>
	<9B6E2A8877C38245BFB15CC491A11DA71039CA@GRFEXC.intern.adiscon.com>
	<87a1ae541003030919j632dac91n75c67717248a21c5@mail.gmail.com>
Message-ID: <87hbov8s3h.fsf@newsguy.com>

> Hello Harry,
> Look at your /etc/host.conf file, it's most probably set up to look at the
> /etc/hosts file before the name servers.
> Regards,
> Sayan

Yup... that seems to be where its getting set at... thanks.
from /etc/host.conf
[...]
  # This keyword specifies how host lookups are to be performed. It
  # should be followed by one or more lookup methods, separated by
  # commas.  Valid methods are bind, hosts, and nis.
  #
order hosts, bind
[...]



From tbergfeld at hq.adiscon.com  Mon Mar  8 15:29:13 2010
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Mon, 8 Mar 2010 15:29:13 +0100
Subject: [rsyslog] rsyslog 5.4.0 (v5-stable) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103A33@GRFEXC.intern.adiscon.com>

Hi all,
Today, we released rsyslog 5.4.0, a new v5-stable branch. This version begins
a new stable series based on the 5.3.x series, which has been proven rather
well in practice. Most importantly, the previous 5.2.x stable series had some
serious issues. The new 5.4.0 contains fixes for all known problems. The main
new feature is speed: several optimizations were done, including support for
epoll in tcp listeners.

Users of 5.2.x versions are strongly encouraged to upgrade to 5.4.0.


ChangeLog:

http://www.rsyslog.com/Article447.phtml

Download:

http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-197.phtml


As always, feedback is appreciated.

Best regards,
Tom Bergfeld
--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .


From mbiebl at gmail.com  Tue Mar  9 01:25:48 2010
From: mbiebl at gmail.com (Michael Biebl)
Date: Tue, 9 Mar 2010 01:25:48 +0100
Subject: [rsyslog] rsyslog 5.4.0 (v5-stable) released
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103A33@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA7103A33@GRFEXC.intern.adiscon.com>
Message-ID: <aa76cdac1003081625w6dac4ec3raf85bde7491c8338@mail.gmail.com>

Hi everyone!

2010/3/8 Tom Bergfeld <tbergfeld at hq.adiscon.com>:
> Hi all,
> Today, we released rsyslog 5.4.0, a new v5-stable branch. This version begins

So, the help output of rsyslogd has

To run rsyslogd in native mode, use "rsyslogd -c5 <other options>"

Maybe I'm just missing some link to the documentation, but is it
documented somewhere, which features are enabled/available when using
-c5 compared to using -c4 or -c3.
Or put it differently: What's the difference between using -c3 and -c5
for 5.4.0?

Michael


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From pgollucci at p6m7g8.com  Tue Mar  9 02:20:44 2010
From: pgollucci at p6m7g8.com (Philip M. Gollucci)
Date: Tue, 9 Mar 2010 01:20:44 +0000
Subject: [rsyslog] Fwd: cvs commit: ports/sysutils/rsyslog5 Makefile
 distinfo pkg-plist ports/sysutils/rsyslog5/files rsyslogd.in
Message-ID: <4B95A26C.6080301@p6m7g8.com>

fwiw, now in freebsd ports.

-------- Original Message --------
Subject: cvs commit: ports/sysutils/rsyslog5 Makefile distinfo pkg-plist 
         ports/sysutils/rsyslog5/files rsyslogd.in
Date: Mon, 8 Mar 2010 23:15:09 +0000 (UTC)
From: Philip M. Gollucci <pgollucci at FreeBSD.org>
To: ports-committers at FreeBSD.org, cvs-ports at FreeBSD.org, cvs-all at FreeBSD.org

pgollucci    2010-03-08 23:15:09 UTC

   FreeBSD ports repository

   Modified files:
     sysutils/rsyslog5    Makefile distinfo pkg-plist
     sysutils/rsyslog5/files rsyslogd.in
   Log:
   - Update to 5.4.0 [1]
   - Remove -a and old school sockets from rc.d script [myself]

   PR:             ports/144566 [1]
   Submitted by:   Cristiano Rolim Pereira <cristianorolim at hotmail.com> 
(maintainer) [1]

   Feature safe:   yes

   Revision  Changes    Path
   1.13      +1 -1      ports/sysutils/rsyslog5/Makefile
   1.13      +3 -3      ports/sysutils/rsyslog5/distinfo
   1.3       +1 -54     ports/sysutils/rsyslog5/files/rsyslogd.in
   1.8       +1 -0      ports/sysutils/rsyslog5/pkg-plist

http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/Makefile.diff?r1=1.12&r2=1.13&f=h
| --- ports/sysutils/rsyslog5/Makefile	2010/01/28 20:51:59	1.12
| +++ ports/sysutils/rsyslog5/Makefile	2010/03/08 23:15:09	1.13
| @@ -2,11 +2,11 @@
|  # Date created:     29 December 2008
|  # Whom:             Cristiano Rolim Pereira <cristianorolim at hotmail.com>
|  #
| -# $FreeBSD: 
/usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/Makefile,v 1.12 
2010/01/28 20:51:59 glarkin Exp $
| +# $FreeBSD: 
/usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/Makefile,v 1.13 
2010/03/08 23:15:09 pgollucci Exp $
|  #
|
|  PORTNAME=	rsyslog
| -PORTVERSION=	5.3.7
| +PORTVERSION=	5.4.0
|  CATEGORIES=	sysutils
|  MASTER_SITES=	http://download.rsyslog.com/rsyslog/
|  .ifdef MNAME
http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/distinfo.diff?r1=1.12&r2=1.13&f=h
| --- ports/sysutils/rsyslog5/distinfo	2010/01/28 20:51:59	1.12
| +++ ports/sysutils/rsyslog5/distinfo	2010/03/08 23:15:09	1.13
| @@ -1,3 +1,3 @@
| -MD5 (rsyslog-5.3.7.tar.gz) = 1a9b5c76bfea2d0ec28630b8f033d5e2
| -SHA256 (rsyslog-5.3.7.tar.gz) = 
cab7d39d3979e18c23bd0726ffad714c283d1158dea6c611f23eb1718299c029
| -SIZE (rsyslog-5.3.7.tar.gz) = 2109777
| +MD5 (rsyslog-5.4.0.tar.gz) = 291882229d50496f42bd63174076dd37
| +SHA256 (rsyslog-5.4.0.tar.gz) = 
d9cd21d2fcd45fcae65eb0a51927c40315cca02afdc62478abd950febfcf7228
| +SIZE (rsyslog-5.4.0.tar.gz) = 2124201
http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/files/rsyslogd.in.diff?r1=1.2&r2=1.3&f=h
| --- ports/sysutils/rsyslog5/files/rsyslogd.in	2009/12/18 20:44:28	1.2
| +++ ports/sysutils/rsyslog5/files/rsyslogd.in	2010/03/08 23:15:09	1.3
| @@ -1,6 +1,6 @@
|  #!/bin/sh
|  #
| -# $FreeBSD: 
/usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/files/rsyslogd.in,v 
1.2 2009/12/18 20:44:28 miwi Exp $
| +# $FreeBSD: 
/usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/files/rsyslogd.in,v 
1.3 2010/03/08 23:15:09 pgollucci Exp $
|  #
|
|
| @@ -20,59 +20,6 @@ load_rc_config $name
|  pidfile="${rsyslogd_pidfile}"
|  command_args="-i ${pidfile} -f ${rsyslogd_config}"
|  required_files="${rsyslogd_config}"
| -start_precmd="rsyslogd_precmd"
|  extra_commands="reload"
|
| -sockfile="/var/run/rsyslogd.sockets"
| -evalargs="rc_flags=\"\`set_socketlist\` \$rc_flags\""
| -altlog_proglist="named"
| -
| -rsyslogd_precmd()
| -{
| -	local _l _ldir
| -
| -	#	Transitional symlink for old binaries
| -	#
| -	if [ ! -L /dev/log ]; then
| -		ln -sf /var/run/log /dev/log
| -	fi
| -	rm -f /var/run/log
| -
| -	#	Create default list of syslog sockets to watch
| -	#
| -	( umask 022 ; > $sockfile )
| -
| -	#	If running named(8) or ntpd(8) chrooted, added appropriate
| -	#	syslog socket to list of sockets to watch.
| -	#
| -	for _l in $altlog_proglist; do
| -		eval _ldir=\$${_l}_chrootdir
| -		if checkyesno `set_rcvar $_l` && [ -n "$_ldir" ]; then
| -			echo "${_ldir}/var/run/log" >> $sockfile
| -		fi
| -	done
| -
| -	#	If other sockets have been provided, change run_rc_command()'s
| -	#	internal copy of $rsyslogd_flags to force use of specific
| -	#	rsyslogd sockets.
| -	#
| -	if [ -s $sockfile ]; then
| -		echo "/var/run/log" >> $sockfile
| -		eval $evalargs
| -	fi
| -
| -	return 0
| -}
| -
| -set_socketlist()
| -{
| -	local _s _socketargs
| -
| -	_socketargs=
| -	for _s in `cat $sockfile | tr '\n' ' '` ; do
| -		_socketargs="-a $_s $_socketargs"
| -	done
| -	echo $_socketargs
| -}
| -
|  run_rc_command "$1"
http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/pkg-plist.diff?r1=1.7&r2=1.8&f=h
| --- ports/sysutils/rsyslog5/pkg-plist	2010/01/07 04:46:04	1.7
| +++ ports/sysutils/rsyslog5/pkg-plist	2010/03/08 23:15:09	1.8
| @@ -126,6 +126,7 @@ sbin/rsyslogd
|  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_mysql.html
|  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_ng_comparison.html
|  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_packages.html
| +%%PORTDOCS%%%%DOCSDIR%%/rsyslog_pgsql.html
|  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_php_syslog_ng.html
|  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_recording_pri.html
|  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_reliable_forwarding.html


From rgerhards at hq.adiscon.com  Tue Mar  9 07:38:43 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 9 Mar 2010 07:38:43 +0100
Subject: [rsyslog] Fwd: cvs commit: ports/sysutils/rsyslog5 Makefile
	distinfo pkg-plist ports/sysutils/rsyslog5/files rsyslogd.in
References: <4B95A26C.6080301@p6m7g8.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103A45@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Philip M. Gollucci
> Sent: Tuesday, March 09, 2010 2:21 AM
> To: rsyslog-users
> Subject: [rsyslog] Fwd: cvs commit: ports/sysutils/rsyslog5 Makefile
> distinfo pkg-plist ports/sysutils/rsyslog5/files rsyslogd.in
> 
> fwiw, now in freebsd ports.

excellent, this is *very* useful!

Thanks,
Rainer

> 
> -------- Original Message --------
> Subject: cvs commit: ports/sysutils/rsyslog5 Makefile distinfo pkg-
> plist
>          ports/sysutils/rsyslog5/files rsyslogd.in
> Date: Mon, 8 Mar 2010 23:15:09 +0000 (UTC)
> From: Philip M. Gollucci <pgollucci at FreeBSD.org>
> To: ports-committers at FreeBSD.org, cvs-ports at FreeBSD.org, cvs-
> all at FreeBSD.org
> 
> pgollucci    2010-03-08 23:15:09 UTC
> 
>    FreeBSD ports repository
> 
>    Modified files:
>      sysutils/rsyslog5    Makefile distinfo pkg-plist
>      sysutils/rsyslog5/files rsyslogd.in
>    Log:
>    - Update to 5.4.0 [1]
>    - Remove -a and old school sockets from rc.d script [myself]
> 
>    PR:             ports/144566 [1]
>    Submitted by:   Cristiano Rolim Pereira <cristianorolim at hotmail.com>
> (maintainer) [1]
> 
>    Feature safe:   yes
> 
>    Revision  Changes    Path
>    1.13      +1 -1      ports/sysutils/rsyslog5/Makefile
>    1.13      +3 -3      ports/sysutils/rsyslog5/distinfo
>    1.3       +1 -54     ports/sysutils/rsyslog5/files/rsyslogd.in
>    1.8       +1 -0      ports/sysutils/rsyslog5/pkg-plist
> 
> http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/Makefile.diff?r1=1.12
> &r2=1.13&f=h
> | --- ports/sysutils/rsyslog5/Makefile	2010/01/28 20:51:59	1.12
> | +++ ports/sysutils/rsyslog5/Makefile	2010/03/08 23:15:09	1.13
> | @@ -2,11 +2,11 @@
> |  # Date created:     29 December 2008
> |  # Whom:             Cristiano Rolim Pereira
> <cristianorolim at hotmail.com>
> |  #
> | -# $FreeBSD:
> /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/Makefile,v 1.12
> 2010/01/28 20:51:59 glarkin Exp $
> | +# $FreeBSD:
> /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/Makefile,v 1.13
> 2010/03/08 23:15:09 pgollucci Exp $
> |  #
> |
> |  PORTNAME=	rsyslog
> | -PORTVERSION=	5.3.7
> | +PORTVERSION=	5.4.0
> |  CATEGORIES=	sysutils
> |  MASTER_SITES=	http://download.rsyslog.com/rsyslog/
> |  .ifdef MNAME
> http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/distinfo.diff?r1=1.12
> &r2=1.13&f=h
> | --- ports/sysutils/rsyslog5/distinfo	2010/01/28 20:51:59	1.12
> | +++ ports/sysutils/rsyslog5/distinfo	2010/03/08 23:15:09	1.13
> | @@ -1,3 +1,3 @@
> | -MD5 (rsyslog-5.3.7.tar.gz) = 1a9b5c76bfea2d0ec28630b8f033d5e2
> | -SHA256 (rsyslog-5.3.7.tar.gz) =
> cab7d39d3979e18c23bd0726ffad714c283d1158dea6c611f23eb1718299c029
> | -SIZE (rsyslog-5.3.7.tar.gz) = 2109777
> | +MD5 (rsyslog-5.4.0.tar.gz) = 291882229d50496f42bd63174076dd37
> | +SHA256 (rsyslog-5.4.0.tar.gz) =
> d9cd21d2fcd45fcae65eb0a51927c40315cca02afdc62478abd950febfcf7228
> | +SIZE (rsyslog-5.4.0.tar.gz) = 2124201
> http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/files/rsyslogd.in.dif
> f?r1=1.2&r2=1.3&f=h
> | --- ports/sysutils/rsyslog5/files/rsyslogd.in	2009/12/18
> 20:44:28	1.2
> | +++ ports/sysutils/rsyslog5/files/rsyslogd.in	2010/03/08
> 23:15:09	1.3
> | @@ -1,6 +1,6 @@
> |  #!/bin/sh
> |  #
> | -# $FreeBSD:
> /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/files/rsyslogd.i
> n,v
> 1.2 2009/12/18 20:44:28 miwi Exp $
> | +# $FreeBSD:
> /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/files/rsyslogd.i
> n,v
> 1.3 2010/03/08 23:15:09 pgollucci Exp $
> |  #
> |
> |
> | @@ -20,59 +20,6 @@ load_rc_config $name
> |  pidfile="${rsyslogd_pidfile}"
> |  command_args="-i ${pidfile} -f ${rsyslogd_config}"
> |  required_files="${rsyslogd_config}"
> | -start_precmd="rsyslogd_precmd"
> |  extra_commands="reload"
> |
> | -sockfile="/var/run/rsyslogd.sockets"
> | -evalargs="rc_flags=\"\`set_socketlist\` \$rc_flags\""
> | -altlog_proglist="named"
> | -
> | -rsyslogd_precmd()
> | -{
> | -	local _l _ldir
> | -
> | -	#	Transitional symlink for old binaries
> | -	#
> | -	if [ ! -L /dev/log ]; then
> | -		ln -sf /var/run/log /dev/log
> | -	fi
> | -	rm -f /var/run/log
> | -
> | -	#	Create default list of syslog sockets to watch
> | -	#
> | -	( umask 022 ; > $sockfile )
> | -
> | -	#	If running named(8) or ntpd(8) chrooted, added appropriate
> | -	#	syslog socket to list of sockets to watch.
> | -	#
> | -	for _l in $altlog_proglist; do
> | -		eval _ldir=\$${_l}_chrootdir
> | -		if checkyesno `set_rcvar $_l` && [ -n "$_ldir" ]; then
> | -			echo "${_ldir}/var/run/log" >> $sockfile
> | -		fi
> | -	done
> | -
> | -	#	If other sockets have been provided, change
> run_rc_command()'s
> | -	#	internal copy of $rsyslogd_flags to force use of specific
> | -	#	rsyslogd sockets.
> | -	#
> | -	if [ -s $sockfile ]; then
> | -		echo "/var/run/log" >> $sockfile
> | -		eval $evalargs
> | -	fi
> | -
> | -	return 0
> | -}
> | -
> | -set_socketlist()
> | -{
> | -	local _s _socketargs
> | -
> | -	_socketargs=
> | -	for _s in `cat $sockfile | tr '\n' ' '` ; do
> | -		_socketargs="-a $_s $_socketargs"
> | -	done
> | -	echo $_socketargs
> | -}
> | -
> |  run_rc_command "$1"
> http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/pkg-
> plist.diff?r1=1.7&r2=1.8&f=h
> | --- ports/sysutils/rsyslog5/pkg-plist	2010/01/07 04:46:04	1.7
> | +++ ports/sysutils/rsyslog5/pkg-plist	2010/03/08 23:15:09	1.8
> | @@ -126,6 +126,7 @@ sbin/rsyslogd
> |  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_mysql.html
> |  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_ng_comparison.html
> |  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_packages.html
> | +%%PORTDOCS%%%%DOCSDIR%%/rsyslog_pgsql.html
> |  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_php_syslog_ng.html
> |  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_recording_pri.html
> |  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_reliable_forwarding.html
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Tue Mar  9 09:00:01 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 9 Mar 2010 09:00:01 +0100
Subject: [rsyslog] rsyslog 5.4.0 (v5-stable) released
References: <9B6E2A8877C38245BFB15CC491A11DA7103A33@GRFEXC.intern.adiscon.com>
	<aa76cdac1003081625w6dac4ec3raf85bde7491c8338@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103A49@GRFEXC.intern.adiscon.com>

Hi Michael,

The -c option is transitive. So for the diff between -c3 and -c5, you need to
check the compatibility notes for v4 and v5. These are:

http://www.rsyslog.com/doc-v4compatibility.html
http://www.rsyslog.com/doc-v5compatibility.html

I should add to the notes that -c obviously only modifies defaults, and not
program behavior at large. So even with -c3, for example, rsyslog v5 will not
be able to run on zero workers.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Tuesday, March 09, 2010 1:26 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 5.4.0 (v5-stable) released
> 
> Hi everyone!
> 
> 2010/3/8 Tom Bergfeld <tbergfeld at hq.adiscon.com>:
> > Hi all,
> > Today, we released rsyslog 5.4.0, a new v5-stable branch. This
> version begins
> 
> So, the help output of rsyslogd has
> 
> To run rsyslogd in native mode, use "rsyslogd -c5 <other options>"
> 
> Maybe I'm just missing some link to the documentation, but is it
> documented somewhere, which features are enabled/available when using
> -c5 compared to using -c4 or -c3.
> Or put it differently: What's the difference between using -c3 and -c5
> for 5.4.0?
> 
> Michael
> 
> 
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From pgollucci at p6m7g8.com  Thu Mar 11 00:09:51 2010
From: pgollucci at p6m7g8.com (Philip M. Gollucci)
Date: Wed, 10 Mar 2010 23:09:51 +0000
Subject: [rsyslog] rsyslogd 'clients' take out machines if rsyslogd 'server'
	is down
Message-ID: <4B9826BF.7070202@p6m7g8.com>

rsyslog.conf [client version]:
-------------------------------

....

$WorkDirectory /var/spool/rsyslog # location for work (spool) file
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

....
## Local
....

## Remote
*.* :omrelp:cl.tld:2514


If anything on the 'centralized server' fails like mysql is down, or 
rsyslogd is down, or a table is wrong, or insert permissions are wrong, 
or diskio is *really* *really* bad, or the box is 'off' --- about 45 
minutes later. All 50 or so of the clients talking to 'block' and 
suddenly spin in a tight loop and are not killable without -9.

I'm sure its something filling up b/c the timing is nearly to the second 
of the 45 minute mark [given a relatively constant amount of load].

I assume that changing
$ActionResumeRetryCount -1
to
$ActionResumeRetryCount 5

or some other arbirtarily low number in all clients would fix this.  As 
whatever ram buffers are filling up will stop doing that.  If not, maybe 
falling back from RELP to TCP would do the trick [I'd be okay with 
message loss to prevent downtime].

I'm rather more concerned though that it doesn't start queueing to disk 
the spool dir when the ram buffers are full instead of 'crashing' and 
taking the box or all process on it with it.



2542.938673679:800b021c0: rsyslogd 5.3.7 startup, compatibility mode 5, 
module path ''

on FreeBSD 8.0-RELENG-p2








-- 
------------------------------------------------------------------------
1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70  3F8C 75B8 8FFB DB9B 8C1C
Philip M. Gollucci (pgollucci at p6m7g8.com) c: 703.336.9354
VP Apache Infrastructure; Member, Apache Software Foundation
Committer,                        FreeBSD Foundation
Consultant,                       P6M7G8 Inc.
Sr. System Admin,                 Ridecharge Inc.

Work like you don't need the money,
love like you'll never get hurt,
and dance like nobody's watching.


From mangoo at wpkg.org  Thu Mar 18 21:21:06 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Thu, 18 Mar 2010 21:21:06 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
Message-ID: <4BA28B32.6090201@wpkg.org>

I have postgres logging to a separate file through rsyslog with this line:

local0.*                -/var/log/postgresql/postgresql_log


It gets to /var/log/syslog as well, thanks to this line:

*.*;auth,authpriv.none	-/var/log/syslog


As postgres is set to log quite heavily, I would rather not log postgres into /var/log/syslog.

I've read some filtering examples, but I couldn't find one on excluding a given program from a file which is logging everything.

Could anyone point me to a proper example?


Postgres adds such entries through syslog:

Mar 19 04:14:16 sun postgres[388]: [261-1] 2010-03-19 04:14:16 SGT LOG:  duration: 0.037 ms  statement: ROLLBACK


-- 
Tomasz Chmielewski
http://wpkg.org


From sledz at DResearch.DE  Thu Mar 18 21:06:58 2010
From: sledz at DResearch.DE (Sledz, Steffen)
Date: Thu, 18 Mar 2010 21:06:58 +0100
Subject: [rsyslog] patch for architectures without atomic builtins
Message-ID: <DB561678F08F554EB0D9F21B70E268A505AE8B@exchange.intern.dresearch.de>

Hi,

we want to use rsyslogd on an embedded system with an armv5te cpu.
Unfortunately the gcc/libc we have to use for this system do not support
atomic builtins which are vital for rsyslog at the moment. :(

Therefor we created a patch which emulates the atomic builtins
with posix semaphores. This patch should fit to the 5.4.x and
5.5.x versions.

We would be glad if this patch could help to bring rsyslog to
more platforms.

Steffen

From david at lang.hm  Thu Mar 18 23:54:56 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 15:54:56 -0700 (PDT)
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <4BA28B32.6090201@wpkg.org>
References: <4BA28B32.6090201@wpkg.org>
Message-ID: <alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>

On Thu, 18 Mar 2010, Tomasz Chmielewski wrote:

> I have postgres logging to a separate file through rsyslog with this line:
>
> local0.*                -/var/log/postgresql/postgresql_log
>
>
> It gets to /var/log/syslog as well, thanks to this line:
>
> *.*;auth,authpriv.none	-/var/log/syslog
>
>
> As postgres is set to log quite heavily, I would rather not log postgres into /var/log/syslog.
>
> I've read some filtering examples, but I couldn't find one on excluding a given program from a file which is logging everything.
>
> Could anyone point me to a proper example?
>
>
> Postgres adds such entries through syslog:
>
> Mar 19 04:14:16 sun postgres[388]: [261-1] 2010-03-19 04:14:16 SGT LOG:  duration: 0.037 ms  statement: ROLLBACK

you could add

:programname, :isequal, "postgres" ~

ahead of your line to go to /var/log/syslog so that anything with the 
programname of postgres gets dropped first

another thing you could do (if you know that all postgres logs are local0) 
is

local0.* ~

to throw away all logs with facility local0

David Lang


From david at lang.hm  Thu Mar 18 23:55:48 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 15:55:48 -0700 (PDT)
Subject: [rsyslog] patch for architectures without atomic builtins
In-Reply-To: <DB561678F08F554EB0D9F21B70E268A505AE8B@exchange.intern.dresearch.de>
References: <DB561678F08F554EB0D9F21B70E268A505AE8B@exchange.intern.dresearch.de>
Message-ID: <alpine.DEB.2.00.1003181555170.10411@asgard.lang.hm>

On Thu, 18 Mar 2010, Sledz, Steffen wrote:

> Hi,
>
> we want to use rsyslogd on an embedded system with an armv5te cpu.
> Unfortunately the gcc/libc we have to use for this system do not support
> atomic builtins which are vital for rsyslog at the moment. :(
>
> Therefor we created a patch which emulates the atomic builtins
> with posix semaphores. This patch should fit to the 5.4.x and
> 5.5.x versions.
>
> We would be glad if this patch could help to bring rsyslog to
> more platforms.
>
> Steffen

the list strips off attachments.

can you re-send with the patch inline?

David Lang


From mangoo at wpkg.org  Fri Mar 19 00:02:41 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Fri, 19 Mar 2010 00:02:41 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
Message-ID: <4BA2B111.7090604@wpkg.org>

Am 18.03.2010 23:54, david at lang.hm wrote:

>> Postgres adds such entries through syslog:
>>
>> Mar 19 04:14:16 sun postgres[388]: [261-1] 2010-03-19 04:14:16 SGT LOG:  duration: 0.037 ms  statement: ROLLBACK
> 
> you could add
> 
> :programname, :isequal, "postgres" ~
> 
> ahead of your line to go to /var/log/syslog so that anything with the
> programname of postgres gets dropped first

Meaning, like this:

:programname, :isequal, "postgres" ~ *.*;auth,authpriv.none	-/var/log/syslog

?


-- 
Tomasz Chmielewski
http://wpkg.org


From david at lang.hm  Fri Mar 19 00:08:29 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 16:08:29 -0700 (PDT)
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <4BA2B111.7090604@wpkg.org>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
Message-ID: <alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>

On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:

> Am 18.03.2010 23:54, david at lang.hm wrote:
>
>>> Postgres adds such entries through syslog:
>>>
>>> Mar 19 04:14:16 sun postgres[388]: [261-1] 2010-03-19 04:14:16 SGT LOG:  duration: 0.037 ms  statement: ROLLBACK
>>
>> you could add
>>
>> :programname, :isequal, "postgres" ~
>>
>> ahead of your line to go to /var/log/syslog so that anything with the
>> programname of postgres gets dropped first
>
> Meaning, like this:
>
> :programname, :isequal, "postgres" ~ *.*;auth,authpriv.none	-/var/log/syslog

no,

:programname, :isequal, "postgres" ~
*.*;auth,authpriv.none	-/var/log/syslog

each of these is a seperate statement. the ~ as a destination tells 
rsyslog not to proceed further through the rules for this log entry.

David Lang


From mangoo at wpkg.org  Fri Mar 19 00:19:59 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Fri, 19 Mar 2010 00:19:59 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
Message-ID: <4BA2B51F.1060204@wpkg.org>

Am 19.03.2010 00:08, david at lang.hm wrote:

> no,
>
> :programname, :isequal, "postgres" ~
> *.*;auth,authpriv.none -/var/log/syslog
>
> each of these is a seperate statement. the ~ as a destination tells
> rsyslog not to proceed further through the rules for this log entry.

Doesn't work for me.

If I put it like above, nothing gets appended to /var/log/syslog anymore 
(or any other defined log file).

rsyslogd is running as a process.

If it matters, I'm using 64 bit Debian Lenny (rsyslog 3.18.6).


-- 
Tomasz Chmielewski
http://wpkg.org


From david at lang.hm  Fri Mar 19 00:23:22 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 16:23:22 -0700 (PDT)
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <4BA2B51F.1060204@wpkg.org>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
Message-ID: <alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>

On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:

> Am 19.03.2010 00:08, david at lang.hm wrote:
>
>> no,
>> 
>> :programname, :isequal, "postgres" ~
>> *.*;auth,authpriv.none -/var/log/syslog
>> 
>> each of these is a seperate statement. the ~ as a destination tells
>> rsyslog not to proceed further through the rules for this log entry.
>
> Doesn't work for me.
>
> If I put it like above, nothing gets appended to /var/log/syslog anymore (or 
> any other defined log file).
>
> rsyslogd is running as a process.
>
> If it matters, I'm using 64 bit Debian Lenny (rsyslog 3.18.6).

try it with ' instead of " (I get those mixed up at least 50% of the time, 
rsyslog accepts one and not the other and I can never remember which one)

David Lang


From mangoo at wpkg.org  Fri Mar 19 00:40:46 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Fri, 19 Mar 2010 00:40:46 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
Message-ID: <4BA2B9FE.2030008@wpkg.org>

Am 19.03.2010 00:23, david at lang.hm wrote:
> On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:
>
>> Am 19.03.2010 00:08, david at lang.hm wrote:
>>
>>> no,
>>>
>>> :programname, :isequal, "postgres" ~
>>> *.*;auth,authpriv.none -/var/log/syslog
>>>
>>> each of these is a seperate statement. the ~ as a destination tells
>>> rsyslog not to proceed further through the rules for this log entry.
>>
>> Doesn't work for me.
>>
>> If I put it like above, nothing gets appended to /var/log/syslog
>> anymore (or any other defined log file).
>>
>> rsyslogd is running as a process.
>>
>> If it matters, I'm using 64 bit Debian Lenny (rsyslog 3.18.6).
>
> try it with ' instead of " (I get those mixed up at least 50% of the
> time, rsyslog accepts one and not the other and I can never remember
> which one)

OK, with ' it started to log again, but still postgres gets to both 
/var/log/syslog and /var/log/postgresql/postgresql_log.

So, not as expected.

-- 
Tomasz Chmielewski
http://wpkg.org


From david at lang.hm  Fri Mar 19 00:52:45 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 16:52:45 -0700 (PDT)
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <4BA2B9FE.2030008@wpkg.org>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org>
Message-ID: <alpine.DEB.2.00.1003181652290.10411@asgard.lang.hm>

On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:

> Am 19.03.2010 00:23, david at lang.hm wrote:
>> On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:
>> 
>>> Am 19.03.2010 00:08, david at lang.hm wrote:
>>> 
>>>> no,
>>>> 
>>>> :programname, :isequal, "postgres" ~
>>>> *.*;auth,authpriv.none -/var/log/syslog
>>>> 
>>>> each of these is a seperate statement. the ~ as a destination tells
>>>> rsyslog not to proceed further through the rules for this log entry.
>>> 
>>> Doesn't work for me.
>>> 
>>> If I put it like above, nothing gets appended to /var/log/syslog
>>> anymore (or any other defined log file).
>>> 
>>> rsyslogd is running as a process.
>>> 
>>> If it matters, I'm using 64 bit Debian Lenny (rsyslog 3.18.6).
>> 
>> try it with ' instead of " (I get those mixed up at least 50% of the
>> time, rsyslog accepts one and not the other and I can never remember
>> which one)
>
> OK, with ' it started to log again, but still postgres gets to both 
> /var/log/syslog and /var/log/postgresql/postgresql_log.
>
> So, not as expected.

what does the raw log message look like again?

David Lang


From mangoo at wpkg.org  Fri Mar 19 00:58:37 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Fri, 19 Mar 2010 00:58:37 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <alpine.DEB.2.00.1003181652290.10411@asgard.lang.hm>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org>
	<alpine.DEB.2.00.1003181652290.10411@asgard.lang.hm>
Message-ID: <4BA2BE2D.9020205@wpkg.org>

Am 19.03.2010 00:52, david at lang.hm wrote:

>> OK, with ' it started to log again, but still postgres gets to both 
>> /var/log/syslog and /var/log/postgresql/postgresql_log.
>>
>> So, not as expected.
> 
> what does the raw log message look like again?

Like these ones:

Mar 19 07:47:32 db10 postgres[4539]: [87-1] 2010-03-19 07:47:32 SGT LOG:  duration: 0.234 ms  statement: COMMIT
Mar 19 07:47:32 db10 postgres[4539]: [88-1] 2010-03-19 07:47:32 SGT LOG:  duration: 0.042 ms  statement: BEGIN; SET TRANSACTION ISOLATION LEVEL READ 


where postgres[number] can differ.

-- 
Tomasz Chmielewski
http://wpkg.org


From david at lang.hm  Fri Mar 19 01:08:11 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 17:08:11 -0700 (PDT)
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <4BA2BE2D.9020205@wpkg.org>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org>
	<alpine.DEB.2.00.1003181652290.10411@asgard.lang.hm>
	<4BA2BE2D.9020205@wpkg.org>
Message-ID: <alpine.DEB.2.00.1003181706510.10411@asgard.lang.hm>

On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:

> Am 19.03.2010 00:52, david at lang.hm wrote:
>
>>> OK, with ' it started to log again, but still postgres gets to both
>>> /var/log/syslog and /var/log/postgresql/postgresql_log.
>>>
>>> So, not as expected.
>>
>> what does the raw log message look like again?
>
> Like these ones:
>
> Mar 19 07:47:32 db10 postgres[4539]: [87-1] 2010-03-19 07:47:32 SGT LOG:  duration: 0.234 ms  statement: COMMIT
> Mar 19 07:47:32 db10 postgres[4539]: [88-1] 2010-03-19 07:47:32 SGT LOG:  duration: 0.042 ms  statement: BEGIN; SET TRANSACTION ISOLATION LEVEL READ
>
>
> where postgres[number] can differ.

hmm, in these cases postgres should be what's in programname

taking a different slant, IIRC you used local0.* to capture the postgres 
logs into one file, if in the line after that you do

local0.* ~

that should eliminate those logs from going into any files after that.

David Lang


From mangoo at wpkg.org  Fri Mar 19 01:42:29 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Fri, 19 Mar 2010 01:42:29 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <alpine.DEB.2.00.1003181706510.10411@asgard.lang.hm>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org>
	<alpine.DEB.2.00.1003181652290.10411@asgard.lang.hm>
	<4BA2BE2D.9020205@wpkg.org>
	<alpine.DEB.2.00.1003181706510.10411@asgard.lang.hm>
Message-ID: <4BA2C875.4060208@wpkg.org>

Am 19.03.2010 01:08, david at lang.hm wrote:

>> where postgres[number] can differ.
>
> hmm, in these cases postgres should be what's in programname
>
> taking a different slant, IIRC you used local0.* to capture the postgres
> logs into one file, if in the line after that you do
>
> local0.* ~
>
> that should eliminate those logs from going into any files after that.

This one works, but that'll filter out heartbeat, pgpool and others 
useful, which get to syslog.


-- 
Tomasz Chmielewski
http://wpkg.org


From mangoo at wpkg.org  Fri Mar 19 01:44:41 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Fri, 19 Mar 2010 01:44:41 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <4BA2B9FE.2030008@wpkg.org>
References: <4BA28B32.6090201@wpkg.org>	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>	<4BA2B111.7090604@wpkg.org>	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>	<4BA2B51F.1060204@wpkg.org>	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org>
Message-ID: <4BA2C8F9.4050607@wpkg.org>

Am 19.03.2010 00:40, Tomasz Chmielewski wrote:
> Am 19.03.2010 00:23, david at lang.hm wrote:
>> On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:
>>
>>> Am 19.03.2010 00:08, david at lang.hm wrote:
>>>
>>>> no,
>>>>
>>>> :programname, :isequal, "postgres" ~
>>>> *.*;auth,authpriv.none -/var/log/syslog
>>>>
>>>> each of these is a seperate statement. the ~ as a destination tells
>>>> rsyslog not to proceed further through the rules for this log entry.
>>>
>>> Doesn't work for me.
>>>
>>> If I put it like above, nothing gets appended to /var/log/syslog
>>> anymore (or any other defined log file).
>>>
>>> rsyslogd is running as a process.
>>>
>>> If it matters, I'm using 64 bit Debian Lenny (rsyslog 3.18.6).
>>
>> try it with ' instead of " (I get those mixed up at least 50% of the
>> time, rsyslog accepts one and not the other and I can never remember
>> which one)
> 
> OK, with ' it started to log again, but still postgres gets to both
> /var/log/syslog and /var/log/postgresql/postgresql_log.
> 
> So, not as expected.

And here is why:

Mar 19 07:37:26 sun rsyslogd: error: invalid compare operation ':isequal' - ignoring selector
Mar 19 07:37:26 sun rsyslogd: error -3003 compare value property - ignoring selector
Mar 19 07:37:26 sun rsyslogd: the last error occured in /etc/rsyslog.conf, line 58
Mar 19 07:37:26 sun rsyslogd: warning: selector line without actions will be discarded


So, if it's " - it doesn't log anything.

If it's ' - it doesn't like the rule.


-- 
Tomasz Chmielewski
http://wpkg.org



From joeschmo2000 at gmail.com  Fri Mar 19 03:14:00 2010
From: joeschmo2000 at gmail.com (joe schmo)
Date: Thu, 18 Mar 2010 22:14:00 -0400
Subject: [rsyslog] CentOS Issues
Message-ID: <a60a2fa91003181914kd27e298se6a8e3913563e4ca@mail.gmail.com>

Hey all,

So I had been using another syslog server for a long time and recently
decided to make the move over to rsyslog and phplogcon so that we could have
web searchable logs (I already run RANCID w/ CVS and cvsweb enabled so it
seemed only fitting to get our syslog server on the webz too)

I went to the forums with these questions and they suggested I ask the
mailing list. That was actually my second destination anyway because I had
read several messages from the archive before heading to the forums.

I have a few questions that span a few different versions...first I need to
apologize in advance if any of these are noob questions, but I have searched
for them and I can't find any answers that help me.

I am running CentOS 5.3 Final with all latest updates off the main repos.
rsyslog 2.0.6 running currently w/ phplogmon and mysql.

1) I have a copy of rsyslog 2.0.6 up and running without any issues (as it
is the only one I can currently get working...more on that in a sec), other
than the fact that it can't receive udp messages without manual
intervention. The only way I can get it to accept udp (network) syslog
messages is to stop the service, and then run the rsyslog -r514 from
command-line manually. Is there anyway to get it to use this option by
default? I need it to start the service with that option enabled. And the
"just run it manually" answers I have received in other forums aren't
acceptable for me.

2) An extension of the above question: when I try to do the "obvious"
solution to the udp issue and use the imudp module, I realized there IS NO
imudp.so module in the plugins folder. Where would I even get this module
for 2.0.6? The most I can find is a copy of imudp.c and its accompanying
make files, but trying to make only outputs errors associated with missing
files that belong to the C developers package (which I had installed some
time ago off an official CentOS repo, no updates available).

3) Why is it that when I follow all the instructions for compiling and
installing the newer versions of rsyslog (3.0.0+), I can get all the way
through the process (I searched and found a way to correct the i686 GCC
errors) but then it isn't actually installed? Does this have to do with
CentOS 5.3 support or am I missing a step here? After doing the make
install, etc, there is no rsyslog.conf file in /etc/ like when I install
2.0.6, nor any of the other accompanying directories, except the
/usr/lib/rsyslog/ folder, which is fully populated with all the correct
files.

Any guidance would be GREATLY appreciated as I am getting very frustrated
with the amount of time I have put into trying to get everything working
correctly. And I thought setting up RANCID and cvsweb was a bit of a chore!
:)

Thanks in advance!

-Joe


From david at lang.hm  Fri Mar 19 05:20:51 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 21:20:51 -0700 (PDT)
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <4BA2C8F9.4050607@wpkg.org>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org> <4BA2C8F9.4050607@wpkg.org>
Message-ID: <alpine.DEB.2.00.1003182119540.3821@asgard.lang.hm>

On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:

> Am 19.03.2010 00:40, Tomasz Chmielewski wrote:
>> Am 19.03.2010 00:23, david at lang.hm wrote:
>>> On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:
>>>
>>>> Am 19.03.2010 00:08, david at lang.hm wrote:
>>>>
>>>>> no,
>>>>>
>>>>> :programname, :isequal, "postgres" ~
>>>>> *.*;auth,authpriv.none -/var/log/syslog
>>>>>
>>>>> each of these is a seperate statement. the ~ as a destination tells
>>>>> rsyslog not to proceed further through the rules for this log entry.
>>>>
>>>> Doesn't work for me.
>>>>
>>>> If I put it like above, nothing gets appended to /var/log/syslog
>>>> anymore (or any other defined log file).
>>>>
>>>> rsyslogd is running as a process.
>>>>
>>>> If it matters, I'm using 64 bit Debian Lenny (rsyslog 3.18.6).
>>>
>>> try it with ' instead of " (I get those mixed up at least 50% of the
>>> time, rsyslog accepts one and not the other and I can never remember
>>> which one)
>>
>> OK, with ' it started to log again, but still postgres gets to both
>> /var/log/syslog and /var/log/postgresql/postgresql_log.
>>
>> So, not as expected.
>
> And here is why:
>
> Mar 19 07:37:26 sun rsyslogd: error: invalid compare operation ':isequal' - ignoring selector
> Mar 19 07:37:26 sun rsyslogd: error -3003 compare value property - ignoring selector
> Mar 19 07:37:26 sun rsyslogd: the last error occured in /etc/rsyslog.conf, line 58
> Mar 19 07:37:26 sun rsyslogd: warning: selector line without actions will be discarded
>
>
> So, if it's " - it doesn't log anything.
>
> If it's ' - it doesn't like the rule.

when all else fails, look at the documentation ;-)

http://www.rsyslog.com/doc-rsyslog_conf_filter.html

the line should have been

:programname, isequal, 'postgres' ~

David Lang


From david at lang.hm  Fri Mar 19 05:22:13 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 21:22:13 -0700 (PDT)
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <4BA2C875.4060208@wpkg.org>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org>
	<alpine.DEB.2.00.1003181652290.10411@asgard.lang.hm>
	<4BA2BE2D.9020205@wpkg.org>
	<alpine.DEB.2.00.1003181706510.10411@asgard.lang.hm>
	<4BA2C875.4060208@wpkg.org>
Message-ID: <alpine.DEB.2.00.1003182120550.3821@asgard.lang.hm>

On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:

> Am 19.03.2010 01:08, david at lang.hm wrote:
>
>>> where postgres[number] can differ.
>> 
>> hmm, in these cases postgres should be what's in programname
>> 
>> taking a different slant, IIRC you used local0.* to capture the postgres
>> logs into one file, if in the line after that you do
>> 
>> local0.* ~
>> 
>> that should eliminate those logs from going into any files after that.
>
> This one works, but that'll filter out heartbeat, pgpool and others useful, 
> which get to syslog.

I will point out that those entries are getting put into your 
posgresql_log file along with the postgres logs then.

so you may want to change the filter for that line as well (now that I dug 
up the page with the correct syntax rather than going from memory)

David Lang


From david at lang.hm  Fri Mar 19 05:31:50 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 21:31:50 -0700 (PDT)
Subject: [rsyslog] CentOS Issues
In-Reply-To: <a60a2fa91003181914kd27e298se6a8e3913563e4ca@mail.gmail.com>
References: <a60a2fa91003181914kd27e298se6a8e3913563e4ca@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1003182125350.3821@asgard.lang.hm>

On Thu, 18 Mar 2010, joe schmo wrote:

> Hey all,
>
> So I had been using another syslog server for a long time and recently
> decided to make the move over to rsyslog and phplogcon so that we could have
> web searchable logs (I already run RANCID w/ CVS and cvsweb enabled so it
> seemed only fitting to get our syslog server on the webz too)
>
> I went to the forums with these questions and they suggested I ask the
> mailing list. That was actually my second destination anyway because I had
> read several messages from the archive before heading to the forums.
>
> I have a few questions that span a few different versions...first I need to
> apologize in advance if any of these are noob questions, but I have searched
> for them and I can't find any answers that help me.
>
> I am running CentOS 5.3 Final with all latest updates off the main repos.
> rsyslog 2.0.6 running currently w/ phplogmon and mysql.
>
> 1) I have a copy of rsyslog 2.0.6 up and running without any issues (as it
> is the only one I can currently get working...more on that in a sec), other
> than the fact that it can't receive udp messages without manual
> intervention. The only way I can get it to accept udp (network) syslog
> messages is to stop the service, and then run the rsyslog -r514 from
> command-line manually. Is there anyway to get it to use this option by
> default? I need it to start the service with that option enabled. And the
> "just run it manually" answers I have received in other forums aren't
> acceptable for me.
>
> 2) An extension of the above question: when I try to do the "obvious"
> solution to the udp issue and use the imudp module, I realized there IS NO
> imudp.so module in the plugins folder. Where would I even get this module
> for 2.0.6? The most I can find is a copy of imudp.c and its accompanying
> make files, but trying to make only outputs errors associated with missing
> files that belong to the C developers package (which I had installed some
> time ago off an official CentOS repo, no updates available).
>
> 3) Why is it that when I follow all the instructions for compiling and
> installing the newer versions of rsyslog (3.0.0+), I can get all the way
> through the process (I searched and found a way to correct the i686 GCC
> errors) but then it isn't actually installed? Does this have to do with
> CentOS 5.3 support or am I missing a step here? After doing the make
> install, etc, there is no rsyslog.conf file in /etc/ like when I install
> 2.0.6, nor any of the other accompanying directories, except the
> /usr/lib/rsyslog/ folder, which is fully populated with all the correct
> files.
>
> Any guidance would be GREATLY appreciated as I am getting very frustrated
> with the amount of time I have put into trying to get everything working
> correctly. And I thought setting up RANCID and cvsweb was a bit of a chore!
> :)
>
> Thanks in advance!

as you have noticed, rsyslog 2.x is ancient (I am running 5.x in 
production now)

when compiling the new version you should be able to just do make install 
after you have compiled it to install it.

you may need to uninstall the RHEL shipped version, I think that by 
default rsyslog will get installed under /usr/local when compiled from 
source.

installing from source will not create the /etc/rsyslog.conf file for you.

the rsyslog on my laptop (ubuntu with a couple manual tweaks) is:

#  /etc/rsyslog.conf    Configuration file for rsyslog.
#
#                       For more information see
#                       /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
#
#  Default logging rules can be found in /etc/rsyslog.d/50-default.conf
#################
#### MODULES ####
#################
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog   # provides kernel logging support (previously done by 
rklogd)
#$ModLoad immark  # provides --MARK-- message capability
#$KLogPath /var/run/rsyslog/kmsg
# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
$RepeatedMsgReduction on
$RepeatedMsgContainsOriginalMsg on
#
# Include all config files in /etc/rsyslog.d/
#
#$IncludeConfig /etc/rsyslog.d/*.conf
*.* -/var/log/messages

this should be enough to get you started, you will need to uncomment the 
remote reception options and then start rsyslog with rsyslogd -cX where X 
is the major version you are running (3,4,5 depending on which one you 
pick)

David Lang


From sledz at dresearch.de  Fri Mar 19 08:00:39 2010
From: sledz at dresearch.de (Steffen Sledz)
Date: Fri, 19 Mar 2010 08:00:39 +0100
Subject: [rsyslog] patch for architectures without atomic builtins
In-Reply-To: <alpine.DEB.2.00.1003181555170.10411@asgard.lang.hm>
References: <DB561678F08F554EB0D9F21B70E268A505AE8B@exchange.intern.dresearch.de>
	<alpine.DEB.2.00.1003181555170.10411@asgard.lang.hm>
Message-ID: <4BA32117.9030408@dresearch.de>

david at lang.hm wrote:
> the list strips off attachments.
> 
> can you re-send with the patch inline?

Here it is...

diff -Nurd rsyslog-5.4.0.orig/configure.ac rsyslog-5.4.0/configure.ac
--- rsyslog-5.4.0.orig/configure.ac     2010-03-08 13:28:35.000000000 +0000
+++ rsyslog-5.4.0/configure.ac  2010-03-16 11:30:44.000000000 +0000
@@ -87,6 +87,7 @@
 AC_HEADER_TIME
 AC_STRUCT_TM
 AC_C_VOLATILE
+AC_C_TYPEOF

 sa_includes="\
 $ac_includes_default
@@ -129,7 +130,8 @@
 # check for availability of atomic operations
 RS_ATOMIC_OPERATIONS
 RS_ATOMIC_OPERATIONS_64BIT
-
+# fall back to POSIX sems for atomic operations (cpu expensive)
+AC_CHECK_HEADERS([semaphore.h])

 # Additional module directories
 AC_ARG_WITH(moddirs,
diff -Nurd rsyslog-5.4.0.orig/runtime/Makefile.am rsyslog-5.4.0/runtime/Makefile.am
--- rsyslog-5.4.0.orig/runtime/Makefile.am      2010-03-08 13:27:47.000000000 +0000
+++ rsyslog-5.4.0/runtime/Makefile.am   2010-03-15 16:43:12.000000000 +0000
@@ -9,6 +9,7 @@
        rsyslog.h \
        unicode-helper.h \
        atomic.h \
+       atomic-posix-sem.c \
        batch.h \
        syslogd-types.h \
        module-template.h \
diff -Nurd rsyslog-5.4.0.orig/runtime/atomic-posix-sem.c rsyslog-5.4.0/runtime/atomic-posix-sem.c
--- rsyslog-5.4.0.orig/runtime/atomic-posix-sem.c       1970-01-01 00:00:00.000000000 +0000
+++ rsyslog-5.4.0/runtime/atomic-posix-sem.c    2010-03-16 13:22:27.000000000 +0000
@@ -0,0 +1,70 @@
+/* atomic_posix_sem.c: This file supplies an emulation for atomic operations using
+ * POSIX semaphores.
+ *
+ * Copyright 2010 DResearch Digital Media Systems GmbH
+ *
+ * This file is part of the rsyslog runtime library.
+ *
+ * The rsyslog runtime library is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * The rsyslog runtime library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public License
+ * along with the rsyslog runtime library.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * A copy of the GPL can be found in the file "COPYING" in this distribution.
+ * A copy of the LGPL can be found in the file "COPYING.LESSER" in this distribution.
+ */
+
+#include "config.h"
+#ifndef HAVE_ATOMIC_BUILTINS
+#ifdef HAVE_SEMAPHORE_H
+#include <semaphore.h>
+#include <errno.h>
+
+#include "atomic.h"
+#include "rsyslog.h"
+#include "srUtils.h"
+
+sem_t atomicSem;
+
+rsRetVal
+atomicSemInit(void)
+{
+       DEFiRet;
+
+       dbgprintf("init posix semaphore for atomics emulation\n");
+       if(sem_init(&atomicSem, 0, 1) == -1)
+       {
+               char errStr[1024];
+               rs_strerror_r(errno, errStr, sizeof(errStr));
+               dbgprintf("init posix semaphore for atomics emulation failed: %s\n", errStr);
+               iRet = RS_RET_SYS_ERR; /* the right error code ??? */
+       }
+
+       RETiRet;
+}
+
+void
+atomicSemExit(void)
+{
+       dbgprintf("destroy posix semaphore for atomics emulation\n");
+       if(sem_destroy(&atomicSem) == -1)
+       {
+               char errStr[1024];
+               rs_strerror_r(errno, errStr, sizeof(errStr));
+               dbgprintf("destroy posix semaphore for atomics emulation failed: %s\n", errStr);
+       }
+}
+
+#endif /* HAVE_SEMAPHORE_H */
+#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
+
+/* vim:set ai:
+ */
diff -Nurd rsyslog-5.4.0.orig/runtime/atomic.h rsyslog-5.4.0/runtime/atomic.h
--- rsyslog-5.4.0.orig/runtime/atomic.h 2010-03-05 07:20:36.000000000 +0000
+++ rsyslog-5.4.0/runtime/atomic.h      2010-03-16 13:23:57.000000000 +0000
@@ -54,6 +54,122 @@
 #      define ATOMIC_CAS(data, oldVal, newVal) __sync_bool_compare_and_swap(&(data), (oldVal), (newVal));
 #      define ATOMIC_CAS_VAL(data, oldVal, newVal) __sync_val_compare_and_swap(&(data), (oldVal), (newVal));
 #else
+#ifdef HAVE_SEMAPHORE_H
+       /* we use POSIX semaphores instead */
+
+#include "rsyslog.h"
+#include <semaphore.h>
+
+extern sem_t atomicSem;
+rsRetVal atomicSemInit(void);
+void atomicSemExit(void);
+
+#if HAVE_TYPEOF
+#define my_typeof(x) typeof(x)
+#else /* sorry, can't determine types, using 'int' */
+#define my_typeof(x) int
+#endif
+
+#      define ATOMIC_SUB(data, val) \
+({ \
+       my_typeof(data) tmp; \
+       sem_wait(&atomicSem); \
+       tmp = data; \
+       data -= val; \
+       sem_post(&atomicSem); \
+       tmp; \
+})
+
+#      define ATOMIC_ADD(data, val) \
+({ \
+       my_typeof(data) tmp; \
+       sem_wait(&atomicSem); \
+       tmp = data; \
+       data += val; \
+       sem_post(&atomicSem); \
+       tmp; \
+})
+
+#      define ATOMIC_INC_AND_FETCH(data) \
+({ \
+       my_typeof(data) tmp; \
+       sem_wait(&atomicSem); \
+       tmp = data; \
+       data += 1; \
+       sem_post(&atomicSem); \
+       tmp; \
+})
+
+#      define ATOMIC_INC(data) ((void) ATOMIC_INC_AND_FETCH(data))
+
+#      define ATOMIC_DEC_AND_FETCH(data) \
+({ \
+       sem_wait(&atomicSem); \
+       data -= 1; \
+       sem_post(&atomicSem); \
+       data; \
+})
+
+#      define ATOMIC_DEC(data) ((void) ATOMIC_DEC_AND_FETCH(data))
+
+#      define ATOMIC_FETCH_32BIT(data) ((unsigned) ATOMIC_ADD((data), 0xffffffff))
+
+#      define ATOMIC_STORE_1_TO_32BIT(data) \
+({ \
+       my_typeof(data) tmp; \
+       sem_wait(&atomicSem); \
+       tmp = data; \
+       data = 1; \
+       sem_post(&atomicSem); \
+       tmp; \
+})
+
+#      define ATOMIC_STORE_0_TO_INT(data) \
+({ \
+       my_typeof(data) tmp; \
+       sem_wait(&atomicSem); \
+       tmp = data; \
+       data = 0; \
+       sem_post(&atomicSem); \
+       tmp; \
+})
+
+#      define ATOMIC_STORE_1_TO_INT(data) \
+({ \
+       my_typeof(data) tmp; \
+       sem_wait(&atomicSem); \
+       tmp = data; \
+       data = 1; \
+       sem_post(&atomicSem); \
+       tmp; \
+})
+
+#      define ATOMIC_CAS(data, oldVal, newVal) \
+({ \
+       int ret; \
+       sem_wait(&atomicSem); \
+       if(data != oldVal) ret = 0; \
+       else \
+       { \
+               data = newVal; \
+               ret = 1; \
+       } \
+       sem_post(&atomicSem); \
+       ret; \
+})
+
+#      define ATOMIC_CAS_VAL(data, oldVal, newVal) \
+({ \
+       sem_wait(&atomicSem); \
+       if(data == oldVal) \
+       { \
+               data = newVal; \
+       } \
+       sem_post(&atomicSem); \
+       data; \
+})
+
+#else /* not HAVE_SEMAPHORE_H */
        /* note that we gained parctical proof that theoretical problems DO occur
         * if we do not properly address them. See this blog post for details:
         * http://blog.gerhards.net/2009/01/rsyslog-data-race-analysis.html
@@ -68,5 +184,6 @@
 #      define ATOMIC_FETCH_32BIT(data) (data)
 #      define ATOMIC_STORE_1_TO_32BIT(data) (data) = 1
 #endif
+#endif

 #endif /* #ifndef INCLUDED_ATOMIC_H */
diff -Nurd rsyslog-5.4.0.orig/runtime/rsyslog.c rsyslog-5.4.0/runtime/rsyslog.c
--- rsyslog-5.4.0.orig/runtime/rsyslog.c        2010-03-05 07:20:36.000000000 +0000
+++ rsyslog-5.4.0/runtime/rsyslog.c     2010-03-16 08:30:51.000000000 +0000
@@ -81,6 +81,7 @@
 #include "rule.h"
 #include "ruleset.h"
 #include "parser.h"
+#include "atomic.h"

 /* forward definitions */
 static rsRetVal dfltErrLogger(int, uchar *errMsg);
@@ -140,6 +141,12 @@
                CHKiRet(objClassInit(NULL)); /* *THIS* *MUST* always be the first class initilizer being called! */
                CHKiRet(objGetObjInterface(pObjIF)); /* this provides the root pointer for all other queries */

+#ifndef HAVE_ATOMIC_BUILTINS
+#ifdef HAVE_SEMAPHORE_H
+               CHKiRet(atomicSemInit());
+#endif /* HAVE_SEMAPHORE_H */
+#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
+
                /* initialize core classes. We must be very careful with the order of events. Some
                 * classes use others and if we do not initialize them in the right order, we may end
                 * up with an invalid call. The most important thing that can happen is that an error
@@ -216,6 +223,13 @@
                glblClassExit();
                rulesetClassExit();
                ruleClassExit();
+
+#ifndef HAVE_ATOMIC_BUILTINS
+#ifdef HAVE_SEMAPHORE_H
+               atomicSemExit();
+#endif /* HAVE_SEMAPHORE_H */
+#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
+
                objClassExit(); /* *THIS* *MUST/SHOULD?* always be the first class initilizer being called (except debug)! */
        }




From rgerhards at hq.adiscon.com  Fri Mar 19 10:10:42 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 19 Mar 2010 10:10:42 +0100
Subject: [rsyslog] CentOS Issues
References: <a60a2fa91003181914kd27e298se6a8e3913563e4ca@mail.gmail.com>
	<alpine.DEB.2.00.1003182125350.3821@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103AFD@GRFEXC.intern.adiscon.com>

> when compiling the new version you should be able to just do make
> install
> after you have compiled it to install it.

As far as I know, you need to set --libdir and --bindir (or its 64 bit
equivalents) ./configure options on CentOS, but I don't know what the correct
values are for it.

Rainer

> 
> you may need to uninstall the RHEL shipped version, I think that by
> default rsyslog will get installed under /usr/local when compiled from
> source.
> 
> installing from source will not create the /etc/rsyslog.conf file for
> you.
> 
> the rsyslog on my laptop (ubuntu with a couple manual tweaks) is:
> 
> #  /etc/rsyslog.conf    Configuration file for rsyslog.
> #
> #                       For more information see
> #                       /usr/share/doc/rsyslog-
> doc/html/rsyslog_conf.html
> #
> #  Default logging rules can be found in /etc/rsyslog.d/50-default.conf
> #################
> #### MODULES ####
> #################
> $ModLoad imuxsock # provides support for local system logging
> $ModLoad imklog   # provides kernel logging support (previously done by
> rklogd)
> #$ModLoad immark  # provides --MARK-- message capability
> #$KLogPath /var/run/rsyslog/kmsg
> # provides UDP syslog reception
> #$ModLoad imudp
> #$UDPServerRun 514
> # provides TCP syslog reception
> #$ModLoad imtcp
> #$InputTCPServerRun 514
> ###########################
> #### GLOBAL DIRECTIVES ####
> ###########################
> #
> # Use traditional timestamp format.
> # To enable high precision timestamps, comment out the following line.
> #
> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
> #
> # Set the default permissions for all log files.
> #
> $FileOwner syslog
> $FileGroup adm
> $FileCreateMode 0640
> $DirCreateMode 0755
> $Umask 0022
> $PrivDropToUser syslog
> $PrivDropToGroup syslog
> $RepeatedMsgReduction on
> $RepeatedMsgContainsOriginalMsg on
> #
> # Include all config files in /etc/rsyslog.d/
> #
> #$IncludeConfig /etc/rsyslog.d/*.conf
> *.* -/var/log/messages
> 
> this should be enough to get you started, you will need to uncomment
> the
> remote reception options and then start rsyslog with rsyslogd -cX where
> X
> is the major version you are running (3,4,5 depending on which one you
> pick)
> 
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Fri Mar 19 10:13:42 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 19 Mar 2010 10:13:42 +0100
Subject: [rsyslog] CentOS Issues
References: <a60a2fa91003181914kd27e298se6a8e3913563e4ca@mail.gmail.com><alpine.DEB.2.00.1003182125350.3821@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7103AFD@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103AFE@GRFEXC.intern.adiscon.com>

> > when compiling the new version you should be able to just do make
> > install
> > after you have compiled it to install it.
> 
> As far as I know, you need to set --libdir and --bindir (or its 64 bit
> equivalents) ./configure options on CentOS, but I don't know what the
> correct
> values are for it.

I just discovered there is some good info on CentOS in the wiki:

http://wiki.rsyslog.com/index.php/Rsyslog_on_CentOS_success_story

I am not sure if it is too much in-depth, though...

Rainer


From rgerhards at hq.adiscon.com  Fri Mar 19 10:22:39 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 19 Mar 2010 10:22:39 +0100
Subject: [rsyslog] patch for architectures without atomic builtins
References: <DB561678F08F554EB0D9F21B70E268A505AE8B@exchange.intern.dresearch.de><alpine.DEB.2.00.1003181555170.10411@asgard.lang.hm>
	<4BA32117.9030408@dresearch.de>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103AFF@GRFEXC.intern.adiscon.com>

Excellent, many thanks for this very useful contribution! I'll see that I
merge it in as quickly as possible, but it may take until early next week.
I'll probably try to merge it into v4 as well, as this is definitely useful!

Thanks again,
Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Steffen Sledz
> Sent: Friday, March 19, 2010 8:01 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] patch for architectures without atomic builtins
> 
> david at lang.hm wrote:
> > the list strips off attachments.
> >
> > can you re-send with the patch inline?
> 
> Here it is...
> 
> diff -Nurd rsyslog-5.4.0.orig/configure.ac rsyslog-5.4.0/configure.ac
> --- rsyslog-5.4.0.orig/configure.ac     2010-03-08 13:28:35.000000000
> +0000
> +++ rsyslog-5.4.0/configure.ac  2010-03-16 11:30:44.000000000 +0000
> @@ -87,6 +87,7 @@
>  AC_HEADER_TIME
>  AC_STRUCT_TM
>  AC_C_VOLATILE
> +AC_C_TYPEOF
> 
>  sa_includes="\
>  $ac_includes_default
> @@ -129,7 +130,8 @@
>  # check for availability of atomic operations
>  RS_ATOMIC_OPERATIONS
>  RS_ATOMIC_OPERATIONS_64BIT
> -
> +# fall back to POSIX sems for atomic operations (cpu expensive)
> +AC_CHECK_HEADERS([semaphore.h])
> 
>  # Additional module directories
>  AC_ARG_WITH(moddirs,
> diff -Nurd rsyslog-5.4.0.orig/runtime/Makefile.am rsyslog-
> 5.4.0/runtime/Makefile.am
> --- rsyslog-5.4.0.orig/runtime/Makefile.am      2010-03-08
> 13:27:47.000000000 +0000
> +++ rsyslog-5.4.0/runtime/Makefile.am   2010-03-15 16:43:12.000000000
> +0000
> @@ -9,6 +9,7 @@
>         rsyslog.h \
>         unicode-helper.h \
>         atomic.h \
> +       atomic-posix-sem.c \
>         batch.h \
>         syslogd-types.h \
>         module-template.h \
> diff -Nurd rsyslog-5.4.0.orig/runtime/atomic-posix-sem.c rsyslog-
> 5.4.0/runtime/atomic-posix-sem.c
> --- rsyslog-5.4.0.orig/runtime/atomic-posix-sem.c       1970-01-01
> 00:00:00.000000000 +0000
> +++ rsyslog-5.4.0/runtime/atomic-posix-sem.c    2010-03-16
> 13:22:27.000000000 +0000
> @@ -0,0 +1,70 @@
> +/* atomic_posix_sem.c: This file supplies an emulation for atomic
> operations using
> + * POSIX semaphores.
> + *
> + * Copyright 2010 DResearch Digital Media Systems GmbH
> + *
> + * This file is part of the rsyslog runtime library.
> + *
> + * The rsyslog runtime library is free software: you can redistribute
> it and/or modify
> + * it under the terms of the GNU Lesser General Public License as
> published by
> + * the Free Software Foundation, either version 3 of the License, or
> + * (at your option) any later version.
> + *
> + * The rsyslog runtime library is distributed in the hope that it will
> be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> License
> + * along with the rsyslog runtime library.  If not, see
> <http://www.gnu.org/licenses/>.
> + *
> + * A copy of the GPL can be found in the file "COPYING" in this
> distribution.
> + * A copy of the LGPL can be found in the file "COPYING.LESSER" in
> this distribution.
> + */
> +
> +#include "config.h"
> +#ifndef HAVE_ATOMIC_BUILTINS
> +#ifdef HAVE_SEMAPHORE_H
> +#include <semaphore.h>
> +#include <errno.h>
> +
> +#include "atomic.h"
> +#include "rsyslog.h"
> +#include "srUtils.h"
> +
> +sem_t atomicSem;
> +
> +rsRetVal
> +atomicSemInit(void)
> +{
> +       DEFiRet;
> +
> +       dbgprintf("init posix semaphore for atomics emulation\n");
> +       if(sem_init(&atomicSem, 0, 1) == -1)
> +       {
> +               char errStr[1024];
> +               rs_strerror_r(errno, errStr, sizeof(errStr));
> +               dbgprintf("init posix semaphore for atomics emulation
> failed: %s\n", errStr);
> +               iRet = RS_RET_SYS_ERR; /* the right error code ??? */
> +       }
> +
> +       RETiRet;
> +}
> +
> +void
> +atomicSemExit(void)
> +{
> +       dbgprintf("destroy posix semaphore for atomics emulation\n");
> +       if(sem_destroy(&atomicSem) == -1)
> +       {
> +               char errStr[1024];
> +               rs_strerror_r(errno, errStr, sizeof(errStr));
> +               dbgprintf("destroy posix semaphore for atomics
> emulation failed: %s\n", errStr);
> +       }
> +}
> +
> +#endif /* HAVE_SEMAPHORE_H */
> +#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
> +
> +/* vim:set ai:
> + */
> diff -Nurd rsyslog-5.4.0.orig/runtime/atomic.h rsyslog-
> 5.4.0/runtime/atomic.h
> --- rsyslog-5.4.0.orig/runtime/atomic.h 2010-03-05 07:20:36.000000000
> +0000
> +++ rsyslog-5.4.0/runtime/atomic.h      2010-03-16 13:23:57.000000000
> +0000
> @@ -54,6 +54,122 @@
>  #      define ATOMIC_CAS(data, oldVal, newVal)
> __sync_bool_compare_and_swap(&(data), (oldVal), (newVal));
>  #      define ATOMIC_CAS_VAL(data, oldVal, newVal)
> __sync_val_compare_and_swap(&(data), (oldVal), (newVal));
>  #else
> +#ifdef HAVE_SEMAPHORE_H
> +       /* we use POSIX semaphores instead */
> +
> +#include "rsyslog.h"
> +#include <semaphore.h>
> +
> +extern sem_t atomicSem;
> +rsRetVal atomicSemInit(void);
> +void atomicSemExit(void);
> +
> +#if HAVE_TYPEOF
> +#define my_typeof(x) typeof(x)
> +#else /* sorry, can't determine types, using 'int' */
> +#define my_typeof(x) int
> +#endif
> +
> +#      define ATOMIC_SUB(data, val) \
> +({ \
> +       my_typeof(data) tmp; \
> +       sem_wait(&atomicSem); \
> +       tmp = data; \
> +       data -= val; \
> +       sem_post(&atomicSem); \
> +       tmp; \
> +})
> +
> +#      define ATOMIC_ADD(data, val) \
> +({ \
> +       my_typeof(data) tmp; \
> +       sem_wait(&atomicSem); \
> +       tmp = data; \
> +       data += val; \
> +       sem_post(&atomicSem); \
> +       tmp; \
> +})
> +
> +#      define ATOMIC_INC_AND_FETCH(data) \
> +({ \
> +       my_typeof(data) tmp; \
> +       sem_wait(&atomicSem); \
> +       tmp = data; \
> +       data += 1; \
> +       sem_post(&atomicSem); \
> +       tmp; \
> +})
> +
> +#      define ATOMIC_INC(data) ((void) ATOMIC_INC_AND_FETCH(data))
> +
> +#      define ATOMIC_DEC_AND_FETCH(data) \
> +({ \
> +       sem_wait(&atomicSem); \
> +       data -= 1; \
> +       sem_post(&atomicSem); \
> +       data; \
> +})
> +
> +#      define ATOMIC_DEC(data) ((void) ATOMIC_DEC_AND_FETCH(data))
> +
> +#      define ATOMIC_FETCH_32BIT(data) ((unsigned) ATOMIC_ADD((data),
> 0xffffffff))
> +
> +#      define ATOMIC_STORE_1_TO_32BIT(data) \
> +({ \
> +       my_typeof(data) tmp; \
> +       sem_wait(&atomicSem); \
> +       tmp = data; \
> +       data = 1; \
> +       sem_post(&atomicSem); \
> +       tmp; \
> +})
> +
> +#      define ATOMIC_STORE_0_TO_INT(data) \
> +({ \
> +       my_typeof(data) tmp; \
> +       sem_wait(&atomicSem); \
> +       tmp = data; \
> +       data = 0; \
> +       sem_post(&atomicSem); \
> +       tmp; \
> +})
> +
> +#      define ATOMIC_STORE_1_TO_INT(data) \
> +({ \
> +       my_typeof(data) tmp; \
> +       sem_wait(&atomicSem); \
> +       tmp = data; \
> +       data = 1; \
> +       sem_post(&atomicSem); \
> +       tmp; \
> +})
> +
> +#      define ATOMIC_CAS(data, oldVal, newVal) \
> +({ \
> +       int ret; \
> +       sem_wait(&atomicSem); \
> +       if(data != oldVal) ret = 0; \
> +       else \
> +       { \
> +               data = newVal; \
> +               ret = 1; \
> +       } \
> +       sem_post(&atomicSem); \
> +       ret; \
> +})
> +
> +#      define ATOMIC_CAS_VAL(data, oldVal, newVal) \
> +({ \
> +       sem_wait(&atomicSem); \
> +       if(data == oldVal) \
> +       { \
> +               data = newVal; \
> +       } \
> +       sem_post(&atomicSem); \
> +       data; \
> +})
> +
> +#else /* not HAVE_SEMAPHORE_H */
>         /* note that we gained parctical proof that theoretical
> problems DO occur
>          * if we do not properly address them. See this blog post for
> details:
>          * http://blog.gerhards.net/2009/01/rsyslog-data-race-
> analysis.html
> @@ -68,5 +184,6 @@
>  #      define ATOMIC_FETCH_32BIT(data) (data)
>  #      define ATOMIC_STORE_1_TO_32BIT(data) (data) = 1
>  #endif
> +#endif
> 
>  #endif /* #ifndef INCLUDED_ATOMIC_H */
> diff -Nurd rsyslog-5.4.0.orig/runtime/rsyslog.c rsyslog-
> 5.4.0/runtime/rsyslog.c
> --- rsyslog-5.4.0.orig/runtime/rsyslog.c        2010-03-05
> 07:20:36.000000000 +0000
> +++ rsyslog-5.4.0/runtime/rsyslog.c     2010-03-16 08:30:51.000000000
> +0000
> @@ -81,6 +81,7 @@
>  #include "rule.h"
>  #include "ruleset.h"
>  #include "parser.h"
> +#include "atomic.h"
> 
>  /* forward definitions */
>  static rsRetVal dfltErrLogger(int, uchar *errMsg);
> @@ -140,6 +141,12 @@
>                 CHKiRet(objClassInit(NULL)); /* *THIS* *MUST* always be
> the first class initilizer being called! */
>                 CHKiRet(objGetObjInterface(pObjIF)); /* this provides
> the root pointer for all other queries */
> 
> +#ifndef HAVE_ATOMIC_BUILTINS
> +#ifdef HAVE_SEMAPHORE_H
> +               CHKiRet(atomicSemInit());
> +#endif /* HAVE_SEMAPHORE_H */
> +#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
> +
>                 /* initialize core classes. We must be very careful
> with the order of events. Some
>                  * classes use others and if we do not initialize them
> in the right order, we may end
>                  * up with an invalid call. The most important thing
> that can happen is that an error
> @@ -216,6 +223,13 @@
>                 glblClassExit();
>                 rulesetClassExit();
>                 ruleClassExit();
> +
> +#ifndef HAVE_ATOMIC_BUILTINS
> +#ifdef HAVE_SEMAPHORE_H
> +               atomicSemExit();
> +#endif /* HAVE_SEMAPHORE_H */
> +#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
> +
>                 objClassExit(); /* *THIS* *MUST/SHOULD?* always be the
> first class initilizer being called (except debug)! */
>         }
> 
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From theinric at redhat.com  Fri Mar 19 10:47:52 2010
From: theinric at redhat.com (Tomas Heinrich)
Date: Fri, 19 Mar 2010 10:47:52 +0100
Subject: [rsyslog] CentOS Issues
In-Reply-To: <a60a2fa91003181914kd27e298se6a8e3913563e4ca@mail.gmail.com>
References: <a60a2fa91003181914kd27e298se6a8e3913563e4ca@mail.gmail.com>
Message-ID: <4BA34848.8090002@redhat.com>

On 03/19/2010 03:14 AM, joe schmo wrote:

<...>

> 1) I have a copy of rsyslog 2.0.6 up and running without any issues (as it
> is the only one I can currently get working...more on that in a sec), other
> than the fact that it can't receive udp messages without manual
> intervention. The only way I can get it to accept udp (network) syslog
> messages is to stop the service, and then run the rsyslog -r514 from
> command-line manually. Is there anyway to get it to use this option by
> default? I need it to start the service with that option enabled. And the
> "just run it manually" answers I have received in other forums aren't
> acceptable for me.

You can add the command-line options to /etc/sysconfig/rsyslog, e.g.
SYSLOGD_OPTIONS="-m 0 -r512"

> 2) An extension of the above question: when I try to do the "obvious"
> solution to the udp issue and use the imudp module, I realized there IS NO
> imudp.so module in the plugins folder. Where would I even get this module
> for 2.0.6? The most I can find is a copy of imudp.c and its accompanying
> make files, but trying to make only outputs errors associated with missing
> files that belong to the C developers package (which I had installed some
> time ago off an official CentOS repo, no updates available).

You don't need this module (and there isn't any) for v2.

Tomas


From dag at wieers.com  Fri Mar 19 13:50:50 2010
From: dag at wieers.com (Dag Wieers)
Date: Fri, 19 Mar 2010 13:50:50 +0100 (CET)
Subject: [rsyslog] CentOS Issues
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103AFE@GRFEXC.intern.adiscon.com>
References: <a60a2fa91003181914kd27e298se6a8e3913563e4ca@mail.gmail.com><alpine.DEB.2.00.1003182125350.3821@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7103AFD@GRFEXC.intern.adiscon.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103AFE@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.LRH.2.00.1003191341470.2470@pikachu.3ti.be>

On Fri, 19 Mar 2010, Rainer Gerhards wrote:

>>> when compiling the new version you should be able to just do make
>>> install
>>> after you have compiled it to install it.
>>
>> As far as I know, you need to set --libdir and --bindir (or its 64 bit
>> equivalents) ./configure options on CentOS, but I don't know what the
>> correct
>> values are for it.
>
> I just discovered there is some good info on CentOS in the wiki:
>
> http://wiki.rsyslog.com/index.php/Rsyslog_on_CentOS_success_story
>
> I am not sure if it is too much in-depth, though...

Also take notice of the fact that RHEL5.5 Beta includes rsyslog 3.22 
because of previously reported issues:

 	https://bugzilla.redhat.com/show_bug.cgi?id=519192
 	https://bugzilla.redhat.com/show_bug.cgi?id=519201
 	https://bugzilla.redhat.com/show_bug.cgi?id=519203

We are happy that Red Hat support worked with us in resolving our 
issues and are pleased with this outcome.

These updates are also expected in the upcoming CentOS releases.
-- 
--   dag wieers,  dag at wieers.com,  http://dag.wieers.com/   --
[Any errors in spelling, tact or fact are transmission errors]


From mangoo at wpkg.org  Fri Mar 19 14:30:47 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Fri, 19 Mar 2010 14:30:47 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <alpine.DEB.2.00.1003182119540.3821@asgard.lang.hm>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org> <4BA2C8F9.4050607@wpkg.org>
	<alpine.DEB.2.00.1003182119540.3821@asgard.lang.hm>
Message-ID: <4BA37C87.8030306@wpkg.org>

Am 19.03.2010 05:20, david at lang.hm wrote:

>> If it's ' - it doesn't like the rule.
>
> when all else fails, look at the documentation ;-)
>
> http://www.rsyslog.com/doc-rsyslog_conf_filter.html
>
> the line should have been
>
> :programname, isequal, 'postgres' ~

The correct entries should be (" works, ' - doesn't):

:programname, isequal, "postgres" ~
*.*;auth,authpriv.none		-/var/log/syslog


Much thanks for your help!


-- 
Tomasz Chmielewski
http://wpkg.org




From mangoo at wpkg.org  Fri Mar 19 14:31:50 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Fri, 19 Mar 2010 14:31:50 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <alpine.DEB.2.00.1003182120550.3821@asgard.lang.hm>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org>
	<alpine.DEB.2.00.1003181652290.10411@asgard.lang.hm>
	<4BA2BE2D.9020205@wpkg.org>
	<alpine.DEB.2.00.1003181706510.10411@asgard.lang.hm>
	<4BA2C875.4060208@wpkg.org>
	<alpine.DEB.2.00.1003182120550.3821@asgard.lang.hm>
Message-ID: <4BA37CC6.4030203@wpkg.org>

Am 19.03.2010 05:22, david at lang.hm wrote:
> On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:
>
>> Am 19.03.2010 01:08, david at lang.hm wrote:
>>
>>>> where postgres[number] can differ.
>>>
>>> hmm, in these cases postgres should be what's in programname
>>>
>>> taking a different slant, IIRC you used local0.* to capture the postgres
>>> logs into one file, if in the line after that you do
>>>
>>> local0.* ~
>>>
>>> that should eliminate those logs from going into any files after that.
>>
>> This one works, but that'll filter out heartbeat, pgpool and others
>> useful, which get to syslog.
>
> I will point out that those entries are getting put into your
> posgresql_log file along with the postgres logs then.

Yep, but I'm fine with that (they also concern Postgres on this machine, 
but are counted in kilobytes per day, not in several hundred megabytes).


-- 
Tomasz Chmielewski
http://wpkg.org


From sledz at dresearch.de  Fri Mar 19 16:32:02 2010
From: sledz at dresearch.de (Steffen Sledz)
Date: Fri, 19 Mar 2010 16:32:02 +0100
Subject: [rsyslog] patch for architectures without atomic builtins
In-Reply-To: <alpine.DEB.2.00.1003181555170.10411@asgard.lang.hm>
References: <DB561678F08F554EB0D9F21B70E268A505AE8B@exchange.intern.dresearch.de>
	<alpine.DEB.2.00.1003181555170.10411@asgard.lang.hm>
Message-ID: <4BA398F2.4080704@dresearch.de>

david at lang.hm write:
> can you re-send with the patch inline?

I've seen that you use git for rsyslog. I can send our
patch with git-send-email if you have an email address
for it. Or should i use the mailing list address for
this?

Steffen



From david at lang.hm  Fri Mar 19 16:49:37 2010
From: david at lang.hm (david at lang.hm)
Date: Fri, 19 Mar 2010 08:49:37 -0700 (PDT)
Subject: [rsyslog] patch for architectures without atomic builtins
In-Reply-To: <4BA398F2.4080704@dresearch.de>
References: <DB561678F08F554EB0D9F21B70E268A505AE8B@exchange.intern.dresearch.de>
	<alpine.DEB.2.00.1003181555170.10411@asgard.lang.hm>
	<4BA398F2.4080704@dresearch.de>
Message-ID: <alpine.DEB.2.00.1003190847250.3821@asgard.lang.hm>

On Fri, 19 Mar 2010, Steffen Sledz wrote:

> david at lang.hm write:
>> can you re-send with the patch inline?
>
> I've seen that you use git for rsyslog. I can send our
> patch with git-send-email if you have an email address
> for it. Or should i use the mailing list address for
> this?

Rainer is the person who needs to get the patch, I think I saw him respond 
to your other e-mail.

the mailing list address is probably the best for this sort of thing, or 
if you have a git repository that's pubic, you can make a branch there for 
this and send it's address as a pull request.

David Lang


From sledz at dresearch.de  Fri Mar 19 16:56:38 2010
From: sledz at dresearch.de (Steffen Sledz)
Date: Fri, 19 Mar 2010 16:56:38 +0100
Subject: [rsyslog] [PATCH] emulate missing atomic builtins with posix
	semaphores if available
In-Reply-To: <alpine.DEB.2.00.1003190847250.3821@asgard.lang.hm>
References: <alpine.DEB.2.00.1003190847250.3821@asgard.lang.hm>
Message-ID: <1269014198-1533-1-git-send-email-sledz@dresearch.de>

Signed-off-by: Steffen Sledz <sledz at dresearch.de>
Acked-by: Thilo Fromm <t.fromm at dresearch.de>
---
 configure.ac               |    4 +-
 runtime/Makefile.am        |    1 +
 runtime/atomic-posix-sem.c |   70 ++++++++++++++++++++++++++
 runtime/atomic.h           |  117 ++++++++++++++++++++++++++++++++++++++++++++
 runtime/rsyslog.c          |   14 +++++
 5 files changed, 205 insertions(+), 1 deletions(-)
 create mode 100644 runtime/atomic-posix-sem.c

diff --git a/configure.ac b/configure.ac
index c59895d..ca198ea 100644
--- a/configure.ac
+++ b/configure.ac
@@ -87,6 +87,7 @@ AC_TYPE_UINT8_T
 AC_HEADER_TIME
 AC_STRUCT_TM
 AC_C_VOLATILE
+AC_C_TYPEOF
 
 sa_includes="\
 $ac_includes_default
@@ -129,7 +130,8 @@ AC_TRY_COMPILE([
 # check for availability of atomic operations
 RS_ATOMIC_OPERATIONS
 RS_ATOMIC_OPERATIONS_64BIT
-
+# fall back to POSIX sems for atomic operations (cpu expensive)
+AC_CHECK_HEADERS([semaphore.h])
 
 # Additional module directories
 AC_ARG_WITH(moddirs,
diff --git a/runtime/Makefile.am b/runtime/Makefile.am
index 9047c83..2374c57 100644
--- a/runtime/Makefile.am
+++ b/runtime/Makefile.am
@@ -9,6 +9,7 @@ librsyslog_la_SOURCES = \
 	rsyslog.h \
 	unicode-helper.h \
 	atomic.h \
+	atomic-posix-sem.c \
 	batch.h \
 	syslogd-types.h \
 	module-template.h \
diff --git a/runtime/atomic-posix-sem.c b/runtime/atomic-posix-sem.c
new file mode 100644
index 0000000..979fae0
--- /dev/null
+++ b/runtime/atomic-posix-sem.c
@@ -0,0 +1,70 @@
+/* atomic_posix_sem.c: This file supplies an emulation for atomic operations using
+ * POSIX semaphores.
+ *
+ * Copyright 2010 DResearch Digital Media Systems GmbH
+ *
+ * This file is part of the rsyslog runtime library.
+ *
+ * The rsyslog runtime library is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * The rsyslog runtime library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public License
+ * along with the rsyslog runtime library.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * A copy of the GPL can be found in the file "COPYING" in this distribution.
+ * A copy of the LGPL can be found in the file "COPYING.LESSER" in this distribution.
+ */
+
+#include "config.h"
+#ifndef HAVE_ATOMIC_BUILTINS
+#ifdef HAVE_SEMAPHORE_H
+#include <semaphore.h>
+#include <errno.h>
+
+#include "atomic.h"
+#include "rsyslog.h"
+#include "srUtils.h"
+
+sem_t atomicSem;
+
+rsRetVal
+atomicSemInit(void)
+{
+	DEFiRet;
+
+	dbgprintf("init posix semaphore for atomics emulation\n");
+	if(sem_init(&atomicSem, 0, 1) == -1)
+	{
+		char errStr[1024];
+		rs_strerror_r(errno, errStr, sizeof(errStr));
+		dbgprintf("init posix semaphore for atomics emulation failed: %s\n", errStr);
+		iRet = RS_RET_SYS_ERR; /* the right error code ??? */
+	}
+
+	RETiRet;
+}
+
+void
+atomicSemExit(void)
+{
+	dbgprintf("destroy posix semaphore for atomics emulation\n");
+	if(sem_destroy(&atomicSem) == -1)
+	{
+		char errStr[1024];
+		rs_strerror_r(errno, errStr, sizeof(errStr));
+		dbgprintf("destroy posix semaphore for atomics emulation failed: %s\n", errStr);
+	}
+}
+
+#endif /* HAVE_SEMAPHORE_H */
+#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
+
+/* vim:set ai:
+ */
diff --git a/runtime/atomic.h b/runtime/atomic.h
index b507b76..51b14e1 100644
--- a/runtime/atomic.h
+++ b/runtime/atomic.h
@@ -54,6 +54,122 @@
 #	define ATOMIC_CAS(data, oldVal, newVal) __sync_bool_compare_and_swap(&(data), (oldVal), (newVal));
 #	define ATOMIC_CAS_VAL(data, oldVal, newVal) __sync_val_compare_and_swap(&(data), (oldVal), (newVal));
 #else
+#ifdef HAVE_SEMAPHORE_H
+	/* we use POSIX semaphores instead */
+
+#include "rsyslog.h"
+#include <semaphore.h>
+
+extern sem_t atomicSem;
+rsRetVal atomicSemInit(void);
+void atomicSemExit(void);
+
+#if HAVE_TYPEOF
+#define my_typeof(x) typeof(x)
+#else /* sorry, can't determine types, using 'int' */
+#define my_typeof(x) int
+#endif
+
+#	define ATOMIC_SUB(data, val) \
+({ \
+	my_typeof(data) tmp; \
+	sem_wait(&atomicSem); \
+	tmp = data; \
+	data -= val; \
+	sem_post(&atomicSem); \
+	tmp; \
+})
+
+#	define ATOMIC_ADD(data, val) \
+({ \
+	my_typeof(data) tmp; \
+	sem_wait(&atomicSem); \
+	tmp = data; \
+	data += val; \
+	sem_post(&atomicSem); \
+	tmp; \
+})
+
+#	define ATOMIC_INC_AND_FETCH(data) \
+({ \
+	my_typeof(data) tmp; \
+	sem_wait(&atomicSem); \
+	tmp = data; \
+	data += 1; \
+	sem_post(&atomicSem); \
+	tmp; \
+})
+
+#	define ATOMIC_INC(data) ((void) ATOMIC_INC_AND_FETCH(data))
+
+#	define ATOMIC_DEC_AND_FETCH(data) \
+({ \
+	sem_wait(&atomicSem); \
+	data -= 1; \
+	sem_post(&atomicSem); \
+	data; \
+})
+
+#	define ATOMIC_DEC(data) ((void) ATOMIC_DEC_AND_FETCH(data))
+
+#	define ATOMIC_FETCH_32BIT(data) ((unsigned) ATOMIC_ADD((data), 0xffffffff))
+
+#	define ATOMIC_STORE_1_TO_32BIT(data) \
+({ \
+	my_typeof(data) tmp; \
+	sem_wait(&atomicSem); \
+	tmp = data; \
+	data = 1; \
+	sem_post(&atomicSem); \
+	tmp; \
+})
+
+#	define ATOMIC_STORE_0_TO_INT(data) \
+({ \
+	my_typeof(data) tmp; \
+	sem_wait(&atomicSem); \
+	tmp = data; \
+	data = 0; \
+	sem_post(&atomicSem); \
+	tmp; \
+})
+
+#	define ATOMIC_STORE_1_TO_INT(data) \
+({ \
+	my_typeof(data) tmp; \
+	sem_wait(&atomicSem); \
+	tmp = data; \
+	data = 1; \
+	sem_post(&atomicSem); \
+	tmp; \
+})
+
+#	define ATOMIC_CAS(data, oldVal, newVal) \
+({ \
+	int ret; \
+	sem_wait(&atomicSem); \
+	if(data != oldVal) ret = 0; \
+	else \
+	{ \
+		data = newVal; \
+		ret = 1; \
+	} \
+	sem_post(&atomicSem); \
+	ret; \
+})
+
+#	define ATOMIC_CAS_VAL(data, oldVal, newVal) \
+({ \
+	sem_wait(&atomicSem); \
+	if(data == oldVal) \
+	{ \
+		data = newVal; \
+	} \
+	sem_post(&atomicSem); \
+	data; \
+})
+
+#else /* not HAVE_SEMAPHORE_H */
 	/* note that we gained parctical proof that theoretical problems DO occur
 	 * if we do not properly address them. See this blog post for details:
 	 * http://blog.gerhards.net/2009/01/rsyslog-data-race-analysis.html
@@ -68,5 +184,6 @@
 #	define ATOMIC_FETCH_32BIT(data) (data)
 #	define ATOMIC_STORE_1_TO_32BIT(data) (data) = 1
 #endif
+#endif
 
 #endif /* #ifndef INCLUDED_ATOMIC_H */
diff --git a/runtime/rsyslog.c b/runtime/rsyslog.c
index a76ae25..921ad0b 100644
--- a/runtime/rsyslog.c
+++ b/runtime/rsyslog.c
@@ -81,6 +81,7 @@
 #include "rule.h"
 #include "ruleset.h"
 #include "parser.h"
+#include "atomic.h"
 
 /* forward definitions */
 static rsRetVal dfltErrLogger(int, uchar *errMsg);
@@ -140,6 +141,12 @@ rsrtInit(char **ppErrObj, obj_if_t *pObjIF)
 		CHKiRet(objClassInit(NULL)); /* *THIS* *MUST* always be the first class initilizer being called! */
 		CHKiRet(objGetObjInterface(pObjIF)); /* this provides the root pointer for all other queries */
 
+#ifndef HAVE_ATOMIC_BUILTINS
+#ifdef HAVE_SEMAPHORE_H
+		CHKiRet(atomicSemInit());
+#endif /* HAVE_SEMAPHORE_H */
+#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
+
 		/* initialize core classes. We must be very careful with the order of events. Some
 		 * classes use others and if we do not initialize them in the right order, we may end
 		 * up with an invalid call. The most important thing that can happen is that an error
@@ -216,6 +223,13 @@ rsrtExit(void)
 		glblClassExit();
 		rulesetClassExit();
 		ruleClassExit();
+
+#ifndef HAVE_ATOMIC_BUILTINS
+#ifdef HAVE_SEMAPHORE_H
+		atomicSemExit();
+#endif /* HAVE_SEMAPHORE_H */
+#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
+
 		objClassExit(); /* *THIS* *MUST/SHOULD?* always be the first class initilizer being called (except debug)! */
 	}
 
-- 
1.6.4.2



From rgerhards at hq.adiscon.com  Tue Mar 23 11:22:29 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 23 Mar 2010 11:22:29 +0100
Subject: [rsyslog] patch for architectures without atomic builtins
In-Reply-To: <4BA398F2.4080704@dresearch.de>
References: <DB561678F08F554EB0D9F21B70E268A505AE8B@exchange.intern.dresearch.de>
	<alpine.DEB.2.00.1003181555170.10411@asgard.lang.hm>
	<4BA398F2.4080704@dresearch.de>
Message-ID: <1269339749.3320.0.camel@localhost>

thanks again for the contribution, I have just applied it to the master
branch. I will see if I can backport it to v4, what I think would be
useful.

Rainer

On Fri, 2010-03-19 at 16:32 +0100, Steffen Sledz wrote:
> david at lang.hm write:
> > can you re-send with the patch inline?
> 
> I've seen that you use git for rsyslog. I can send our
> patch with git-send-email if you have an email address
> for it. Or should i use the mailing list address for
> this?
> 
> Steffen
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com




From mikolaj at kucharski.name  Thu Mar 25 22:08:43 2010
From: mikolaj at kucharski.name (Mikolaj Kucharski)
Date: Thu, 25 Mar 2010 21:08:43 +0000
Subject: [rsyslog] last message repeated n times from remote hosts
In-Reply-To: <20100210120230.GI3838@x40.openbsd.home.lan>
References: <20100210120230.GI3838@x40.openbsd.home.lan>
Message-ID: <20100325210843.GK3838@x40.openbsd.home.lan>

Any comments?

On Wed, Feb 10, 2010 at 12:02:30PM +0000, Mikolaj Kucharski wrote:
> Hi,
> 
> I have few Linux based machines with sysklogd installed, and I have
> central syslog server based on CentOS 5 with rsyslog-2.0.6-1.el5.
> 
> I have issue with missing hostname when sysklog sends "last message
> repeated N times" over the wire to rsyslog server.
> 
> Let's have a look. I used logger(1) to repeatedly sent one message few
> times followed by one different message. Here is what I see in the log
> file on my central rsyslog server:
> 
> Feb 10 11:39:46 10.101.43.124 root: remote test start
> Feb 10 11:39:54 last message repeated 14 times
> Feb 10 11:39:54 10.101.43.124 root: remote test end
> 
> 
> and here is tcpdump(8) log from the source (10.101.43.124) machine:
> 
> 
> 11:39:46.642297 IP 10.101.43.124.syslog > 10.101.40.116.syslog: SYSLOG user.notice, length: 28
>         0x0000:  4500 0038 0000 4000 4011 d1fb 0a65 2b7c  E..8.. at .@....e+|
>         0x0010:  0a65 2874 0202 0202 0024 68ef 3c31 333e  .e(t.....$h.<13>
>         0x0020:  726f 6f74 3a20 7265 6d6f 7465 2074 6573  root:.remote.tes
>         0x0030:  7420 7374 6172 740a                      t.start.
> 11:39:54.904820 IP 10.101.43.124.syslog > 10.101.40.116.syslog: SYSLOG user.notice, length: 35
>         0x0000:  4500 003f 0000 4000 4011 d1f4 0a65 2b7c  E..?.. at .@....e+|
>         0x0010:  0a65 2874 0202 0202 002b 68f6 3c31 333e  .e(t.....+h.<13>
>         0x0020:  6c61 7374 206d 6573 7361 6765 2072 6570  last.message.rep
>         0x0030:  6561 7465 6420 3134 2074 696d 6573 0a    eated.14.times.
> 11:39:54.904826 IP 10.101.43.124.syslog > 10.101.40.116.syslog: SYSLOG user.notice, length: 26
>         0x0000:  4500 0036 0000 4000 4011 d1fd 0a65 2b7c  E..6.. at .@....e+|
>         0x0010:  0a65 2874 0202 0202 0022 68ed 3c31 333e  .e(t....."h.<13>
>         0x0020:  726f 6f74 3a20 7265 6d6f 7465 2074 6573  root:.remote.tes
>         0x0030:  7420 656e 640a                           t.end.
> 
> 
> I searched the list, and saw a comment which say the fault is on the
> sysklogd end as it never sends hostname in the repeated-n-times packet,
> but from above I cannot see that it ever sends the packet with hostname,
> so I think the issues is on rsyslog side, and not on the sysklogd.
> 
> Could someone shed some light on my issue, as I would like to see all
> the time the source IP or hostname of incomming messages to rsyslog
> daemon.
> 
> Is this missing source hostname/IP a bug of rsyslog?
> 
> Is there any way to workaround that?
> 
> 
> Thanks.
> 
> 
> PS1. I cannot change client machines, I cannot reinstall them with
> different syslog implementation, the only machine where I have
> permission to do modifications is central rsyslog server.
> 
> PS2. I know about DNS and RevDNS and yes, above server doesn't have
> revDNS setup.

-- 
best regards
q#


From pablo at docecosas.com  Fri Mar 26 13:02:41 2010
From: pablo at docecosas.com (Pablo Martinez Schroder)
Date: Fri, 26 Mar 2010 13:02:41 +0100
Subject: [rsyslog] Rsyslog is not using FQDN in all messages
Message-ID: <fe120686aaa393fb1425df1a5fc29530.squirrel@webmail.docecosas.com>

Hi,

I'm using rsyslog-5.4.0 and I having a issue that I cannot understand. I
want to implement a central logging server using stunnel, and I need to
use FQDN always, I don't want to have HOSTNAME trunked to the hostname so
I enable $PreserveFQDN but only some of the messages seems to use the
whole hostname. I've seen this issue in multiple versions of rsyslog, so
I'm quite sure is not caused by rsyslog-5.4.0

I've tested on multiples versions of CentOS and currently I'm testing
rsyslog-5.4.0 on a CentOS 5.4. To to the tests I'm logging all the
messages in a local file, so the interesting part of my /etc/rsyslog.conf
is like this:

$ModLoad immark   # provides --MARK-- message capability
$ModLoad imuxsock # provides support for local system logging (e.g. via
logger command)
$ModLoad imklog   # kernel logging (formerly provided by rklogd)

$PreserveFQDN on

*.* /var/log/everything.log

I start rsyslogd with the "-c 4" option, and if I enable $PreserveFQDN in
the conf messages from kernel and rsyslogd are saved with the proper FQDN
but the rest of the messages are stored as if they were originated from
the hostname, without domain.

# tail /var/log/everything.log
010-03-26T07:51:29.513679-04:00 syslog-test.scrambled.com kernel: imklog
5.4.0, log source = /proc/kmsg started.
2010-03-26T07:51:29.684129-04:00 syslog-test.scrambled.com rsyslogd:
[origin software="rsyslogd" swVersion="5.4.0" x-pid="1977"
x-info="http://www.rsyslog.com"] start
2010-03-26T07:51:29.707163-04:00 syslog-test stunnel:
LOG5[1508:1099114816]: ssyslog connected from 127.0.0.1:35446
2010-03-26T07:51:30.610043-04:00 syslog-test.scrambled.com kernel: Kernel
logging (proc) stopped.
2010-03-26T07:51:30.610519-04:00 syslog-test.scrambled.com rsyslogd:
[origin software="rsyslogd" swVersion="5.4.0" x-pid="1977"
x-info="http://www.rsyslog.com"] exiting on signal 2.

In this case, the only one using "syslog-test" (without the
".scrambled.com" part is stunnel, but all the logs generated from sshd,
pam, cron, etc are generated from "syslog-test" instead of
"syslog-test.scrambled.com".

I only have localhost in the hosts file and if I run hostname I get
"syslog-test.scrambled.com". I even rebooted the server after changing
$PreserveFQDN, just in case it was some weird thing of applications
'caching' the simple hostname (it doesn't make sense, I know). Is there
anything that I'm doing wrong? Is it possible for rsyslog to use the FQDN
for all the logs?

I believe that I need that the FQDN is used locally for it to work with a
remote server (so the HOSTNAME variabvle in the template has the FQDN)

Many thanks in advance.
-- 
Pablo Martinez Schroder




From tbergfeld at hq.adiscon.com  Fri Mar 26 16:04:56 2010
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Fri, 26 Mar 2010 16:04:56 +0100
Subject: [rsyslog] rsyslog 4.6.2 (v4-stable) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103B99@GRFEXC.intern.adiscon.com>

Hi all,

We have just released rsyslog 4.6.2, a member of the v4-stable branch. This
is a bug-fixing release containing some important fixes and also two new
features. It is a recommended update for all users of the v4-stable. See
Changelog for more details.

ChangeLog:

http://www.rsyslog.com/Article449.phtml

Download:

http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-198.phtml


As always, feedback is appreciated.

Best regards,
Tom Bergfeld
--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com


From david at lang.hm  Fri Mar 26 19:19:02 2010
From: david at lang.hm (david at lang.hm)
Date: Fri, 26 Mar 2010 11:19:02 -0700 (PDT)
Subject: [rsyslog] Rsyslog is not using FQDN in all messages
In-Reply-To: <fe120686aaa393fb1425df1a5fc29530.squirrel@webmail.docecosas.com>
References: <fe120686aaa393fb1425df1a5fc29530.squirrel@webmail.docecosas.com>
Message-ID: <alpine.DEB.2.00.1003261116320.12764@asgard.lang.hm>

My first question would be if the logs have the FQDN in them in the first 
place.

you may want to create a log with the format %raw% and see what is 
arriving to your box.

David Lang

On Fri, 26 Mar 2010, Pablo Martinez Schroder wrote:

> Hi,
>
> I'm using rsyslog-5.4.0 and I having a issue that I cannot understand. I
> want to implement a central logging server using stunnel, and I need to
> use FQDN always, I don't want to have HOSTNAME trunked to the hostname so
> I enable $PreserveFQDN but only some of the messages seems to use the
> whole hostname. I've seen this issue in multiple versions of rsyslog, so
> I'm quite sure is not caused by rsyslog-5.4.0
>
> I've tested on multiples versions of CentOS and currently I'm testing
> rsyslog-5.4.0 on a CentOS 5.4. To to the tests I'm logging all the
> messages in a local file, so the interesting part of my /etc/rsyslog.conf
> is like this:
>
> $ModLoad immark   # provides --MARK-- message capability
> $ModLoad imuxsock # provides support for local system logging (e.g. via
> logger command)
> $ModLoad imklog   # kernel logging (formerly provided by rklogd)
>
> $PreserveFQDN on
>
> *.* /var/log/everything.log
>
> I start rsyslogd with the "-c 4" option, and if I enable $PreserveFQDN in
> the conf messages from kernel and rsyslogd are saved with the proper FQDN
> but the rest of the messages are stored as if they were originated from
> the hostname, without domain.
>
> # tail /var/log/everything.log
> 010-03-26T07:51:29.513679-04:00 syslog-test.scrambled.com kernel: imklog
> 5.4.0, log source = /proc/kmsg started.
> 2010-03-26T07:51:29.684129-04:00 syslog-test.scrambled.com rsyslogd:
> [origin software="rsyslogd" swVersion="5.4.0" x-pid="1977"
> x-info="http://www.rsyslog.com"] start
> 2010-03-26T07:51:29.707163-04:00 syslog-test stunnel:
> LOG5[1508:1099114816]: ssyslog connected from 127.0.0.1:35446
> 2010-03-26T07:51:30.610043-04:00 syslog-test.scrambled.com kernel: Kernel
> logging (proc) stopped.
> 2010-03-26T07:51:30.610519-04:00 syslog-test.scrambled.com rsyslogd:
> [origin software="rsyslogd" swVersion="5.4.0" x-pid="1977"
> x-info="http://www.rsyslog.com"] exiting on signal 2.
>
> In this case, the only one using "syslog-test" (without the
> ".scrambled.com" part is stunnel, but all the logs generated from sshd,
> pam, cron, etc are generated from "syslog-test" instead of
> "syslog-test.scrambled.com".
>
> I only have localhost in the hosts file and if I run hostname I get
> "syslog-test.scrambled.com". I even rebooted the server after changing
> $PreserveFQDN, just in case it was some weird thing of applications
> 'caching' the simple hostname (it doesn't make sense, I know). Is there
> anything that I'm doing wrong? Is it possible for rsyslog to use the FQDN
> for all the logs?
>
> I believe that I need that the FQDN is used locally for it to work with a
> remote server (so the HOSTNAME variabvle in the template has the FQDN)
>
> Many thanks in advance.
>


From pablo at docecosas.com  Fri Mar 26 19:28:16 2010
From: pablo at docecosas.com (Pablo Martinez Schroder)
Date: Fri, 26 Mar 2010 19:28:16 +0100
Subject: [rsyslog] Rsyslog is not using FQDN in all messages
In-Reply-To: <alpine.DEB.2.00.1003261116320.12764@asgard.lang.hm>
References: <fe120686aaa393fb1425df1a5fc29530.squirrel@webmail.docecosas.com>
	<alpine.DEB.2.00.1003261116320.12764@asgard.lang.hm>
Message-ID: <bd858f6056203841b7c277706d5f5106.squirrel@webmail.docecosas.com>


> My first question would be if the logs have the FQDN in them in the first
> place.
>
> you may want to create a log with the format %raw% and see what is
> arriving to your box.

I'm currently testing this issue on local, with the configuration below,
and the logs saved are a mix: some of them have a hostname like
syslog-test.scrambled.com (only the logs generated from kernel and
rsyslog) and all the other lines publish syslog-test as hostname.

$PreserveFQDN on

*.* /var/log/everything.log

>> I'm using rsyslog-5.4.0 and I having a issue that I cannot understand. I
>> want to implement a central logging server using stunnel, and I need to
>> use FQDN always, I don't want to have HOSTNAME trunked to the hostname
>> so
>> I enable $PreserveFQDN but only some of the messages seems to use the
>> whole hostname. I've seen this issue in multiple versions of rsyslog, so
>> I'm quite sure is not caused by rsyslog-5.4.0
>>
>> I start rsyslogd with the "-c 4" option, and if I enable $PreserveFQDN
>> in
>> the conf messages from kernel and rsyslogd are saved with the proper
>> FQDN
>> but the rest of the messages are stored as if they were originated from
>> the hostname, without domain.

-- 
Pablo Martinez Schroder



From ralph at crongeyer.com  Sun Mar 28 16:29:20 2010
From: ralph at crongeyer.com (Ralph Crongeyer)
Date: Sun, 28 Mar 2010 10:29:20 -0400
Subject: [rsyslog] High cpu usage - rsyslog 5.4.0.
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>
References: <5fa6e0144d8003c7c72edff17f9f1675@webmail.crongeyer.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>
Message-ID: <4BAF67C0.4000506@crongeyer.com>

List,
I tried 5.4.0 last weekend on two machines and ran it for almost a week. 
During that time I noticed that every day when I got home from work both 
machines server/log collector and my client machines fans were racing 
(both laptops, Compaq Armada M700's) ? I logged in ran top and found 
rsyslog consuming 99% of the cpu on both machines? After I restarted 
rsyslog on both they seemed to run normal. The next day the same thing. 
By this time I was watching more closely. I noticed that it took about 
an 45 to 60 minutes and rsyslog would have consumed 95 to 99% of the cpu 
on both machines?

I tried to reboot both machines but it would still happen. I have 
switched back to 5.3.7 which doesn't have this problem.

I'm running Debian Lenny on both machines.
I compiled rsyslog with this:
./configure --prefix=/usr --sysconfdir=/etc --enable-relp 
--enable-imtemplate --enable-omtemplate --enable-rfc3195 --enable-imdiag 
--enable-mail --enable-zlib --enable-mysql --enable-gnutls 
--enable-imdiag --enable-imfile --enable-omprog --enable-omudpspoof 
--enable-omstdout --enable-diagtools
make
checkinstall -D make install

I used the same commands to make 5.3.7 and 5.4.0.

Is anyone else seeing high cpu loads with 5.4.0?

Thanks,
Ralph

-- 
Reminds me of my expedition into the wilds of Afghanistan. We lost our 
corkscrew and were compelled to live on food and water for several days. - 
WC Fields



From mbiebl at gmail.com  Sun Mar 28 16:49:49 2010
From: mbiebl at gmail.com (Michael Biebl)
Date: Sun, 28 Mar 2010 16:49:49 +0200
Subject: [rsyslog] High cpu usage - rsyslog 5.4.0.
In-Reply-To: <4BAF67C0.4000506@crongeyer.com>
References: <5fa6e0144d8003c7c72edff17f9f1675@webmail.crongeyer.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>
	<4BAF67C0.4000506@crongeyer.com>
Message-ID: <aa76cdac1003280749p4673e404t7e5aefd63e102348@mail.gmail.com>

2010/3/28 Ralph Crongeyer <ralph at crongeyer.com>:
> I'm running Debian Lenny on both machines.

> Is anyone else seeing high cpu loads with 5.4.0?
>

I can confirm his (on Debian unstable) and already notified Rainer
about this particular issue.
Some preliminary testing here seems to indicate it's an issue with pipes.

When I removed
daemon.*;mail.*;\
 news.err;\
 *.=debug;*.=info;\
 *.=notice;*.=warn      |/dev/xconsole

from my rsyslog.conf, I could no longer reproduce the problem.

Would be interesting to know, if you can confirm this.

Cheers,
Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From ralph at crongeyer.com  Sun Mar 28 16:52:54 2010
From: ralph at crongeyer.com (Ralph Crongeyer)
Date: Sun, 28 Mar 2010 10:52:54 -0400
Subject: [rsyslog] High cpu usage - rsyslog 5.4.0.
In-Reply-To: <aa76cdac1003280749p4673e404t7e5aefd63e102348@mail.gmail.com>
References: <5fa6e0144d8003c7c72edff17f9f1675@webmail.crongeyer.com>	<9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>	<4BAF67C0.4000506@crongeyer.com>
	<aa76cdac1003280749p4673e404t7e5aefd63e102348@mail.gmail.com>
Message-ID: <4BAF6D46.9090009@crongeyer.com>

I'll do it right now. Just a second.

Thanks,
Ralph

Michael Biebl wrote:
> 2010/3/28 Ralph Crongeyer <ralph at crongeyer.com>:
>   
>> I'm running Debian Lenny on both machines.
>>     
>
>   
>> Is anyone else seeing high cpu loads with 5.4.0?
>>
>>     
>
> I can confirm his (on Debian unstable) and already notified Rainer
> about this particular issue.
> Some preliminary testing here seems to indicate it's an issue with pipes.
>
> When I removed
> daemon.*;mail.*;\
>  news.err;\
>  *.=debug;*.=info;\
>  *.=notice;*.=warn      |/dev/xconsole
>
> from my rsyslog.conf, I could no longer reproduce the problem.
>
> Would be interesting to know, if you can confirm this.
>
> Cheers,
> Michael
>
>   


-- 
Reminds me of my expedition into the wilds of Afghanistan. We lost our 
corkscrew and were compelled to live on food and water for several days. - 
WC Fields



From ralph at crongeyer.com  Sun Mar 28 17:09:02 2010
From: ralph at crongeyer.com (Ralph Crongeyer)
Date: Sun, 28 Mar 2010 11:09:02 -0400
Subject: [rsyslog] High cpu usage - rsyslog 5.4.0.
In-Reply-To: <aa76cdac1003280749p4673e404t7e5aefd63e102348@mail.gmail.com>
References: <5fa6e0144d8003c7c72edff17f9f1675@webmail.crongeyer.com>	<9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>	<4BAF67C0.4000506@crongeyer.com>
	<aa76cdac1003280749p4673e404t7e5aefd63e102348@mail.gmail.com>
Message-ID: <4BAF710E.8080506@crongeyer.com>

Ok,
I stoped rsyslog apt-get removed the 5.3.7 package, installed the 5.4.0 
package:

root at logs:/opt# aptitude show rsyslog
Package: rsyslog
State: installed
Automatically installed: no
Version: 5.4.0-1
Priority: extra
Section: extra
Maintainer: ralph at crongeyer.com
Uncompressed Size: 3047k
Depends: libc6 (>= 2.7-1), zlib1g (>= 1:1.1.4), lsb-base (>= 3.2-14)
Description: enhanced multi-threaded syslogd

commented out the lines from rsyslog.conf:
#daemon.*;mail.*;\
#       news.err;\
#       *.=debug;*.=info;\
#       *.=notice;*.=warn       |/dev/xconsole


and restarted rsyslog:
root at logs:/opt# /etc/init.d/rsyslog restart
Stopping enhanced syslogd: rsyslogd.
Starting enhanced syslogd: rsyslogd.

I'll get back to you in about an hour.

Thanks,
Ralph

Michael Biebl wrote:
> 2010/3/28 Ralph Crongeyer <ralph at crongeyer.com>:
>   
>> I'm running Debian Lenny on both machines.
>>     
>
>   
>> Is anyone else seeing high cpu loads with 5.4.0?
>>
>>     
>
> I can confirm his (on Debian unstable) and already notified Rainer
> about this particular issue.
> Some preliminary testing here seems to indicate it's an issue with pipes.
>
> When I removed
> daemon.*;mail.*;\
>  news.err;\
>  *.=debug;*.=info;\
>  *.=notice;*.=warn      |/dev/xconsole
>
> from my rsyslog.conf, I could no longer reproduce the problem.
>
> Would be interesting to know, if you can confirm this.
>
> Cheers,
> Michael
>
>   


-- 
Reminds me of my expedition into the wilds of Afghanistan. We lost our 
corkscrew and were compelled to live on food and water for several days. - 
WC Fields



From ralph at crongeyer.com  Sun Mar 28 17:54:21 2010
From: ralph at crongeyer.com (Ralph Crongeyer)
Date: Sun, 28 Mar 2010 11:54:21 -0400
Subject: [rsyslog] rsyslog + mysql + base(snort)
Message-ID: <4BAF7BAD.8020502@crongeyer.com>

All,
I'm would like to try to have a setup like this. Have Snort log to 
rsyslog and then have rsyslog log to a mysql Base schema database.
I know that people use Barnyard and or Barnyard2 for this setup to 
offload the writing to mysql to barnyard so that barnyard could receive 
snort logs and spool them if necessary before writing to mysql should 
mysql not be able to keep up.
It seems to me that rsyslog's spooling capability could eliminate the 
need for barnyard.
How would one go about applying a (for lack of better words) particular 
database schema so that rsyslog could write to the base database?

Does anyone have any thoughts on this?

Thanks,
Ralph

-- 
Reminds me of my expedition into the wilds of Afghanistan. We lost our 
corkscrew and were compelled to live on food and water for several days. - 
WC Fields



From ralph at crongeyer.com  Sun Mar 28 18:44:35 2010
From: ralph at crongeyer.com (Ralph Crongeyer)
Date: Sun, 28 Mar 2010 12:44:35 -0400
Subject: [rsyslog] High cpu usage - rsyslog 5.4.0.
In-Reply-To: <4BAF710E.8080506@crongeyer.com>
References: <5fa6e0144d8003c7c72edff17f9f1675@webmail.crongeyer.com>	<9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>	<4BAF67C0.4000506@crongeyer.com>	<aa76cdac1003280749p4673e404t7e5aefd63e102348@mail.gmail.com>
	<4BAF710E.8080506@crongeyer.com>
Message-ID: <4BAF8773.1020000@crongeyer.com>

It's confirmed!

I've been running for over an hour and a half now and:
load average: 0.06, 0.06, 0.02

Looks good so far.

Let me know if you guys (Michael, Rainer) want me to run in debug mode 
without the lines commented out to see if I can catch the problem for 
more troubleshooting.

Thanks,
Ralph

Ralph Crongeyer wrote:
> Ok,
> I stoped rsyslog apt-get removed the 5.3.7 package, installed the 5.4.0 
> package:
>
> root at logs:/opt# aptitude show rsyslog
> Package: rsyslog
> State: installed
> Automatically installed: no
> Version: 5.4.0-1
> Priority: extra
> Section: extra
> Maintainer: ralph at crongeyer.com
> Uncompressed Size: 3047k
> Depends: libc6 (>= 2.7-1), zlib1g (>= 1:1.1.4), lsb-base (>= 3.2-14)
> Description: enhanced multi-threaded syslogd
>
> commented out the lines from rsyslog.conf:
> #daemon.*;mail.*;\
> #       news.err;\
> #       *.=debug;*.=info;\
> #       *.=notice;*.=warn       |/dev/xconsole
>
>
> and restarted rsyslog:
> root at logs:/opt# /etc/init.d/rsyslog restart
> Stopping enhanced syslogd: rsyslogd.
> Starting enhanced syslogd: rsyslogd.
>
> I'll get back to you in about an hour.
>
> Thanks,
> Ralph
>
> Michael Biebl wrote:
>   
>> 2010/3/28 Ralph Crongeyer <ralph at crongeyer.com>:
>>   
>>     
>>> I'm running Debian Lenny on both machines.
>>>     
>>>       
>>   
>>     
>>> Is anyone else seeing high cpu loads with 5.4.0?
>>>
>>>     
>>>       
>> I can confirm his (on Debian unstable) and already notified Rainer
>> about this particular issue.
>> Some preliminary testing here seems to indicate it's an issue with pipes.
>>
>> When I removed
>> daemon.*;mail.*;\
>>  news.err;\
>>  *.=debug;*.=info;\
>>  *.=notice;*.=warn      |/dev/xconsole
>>
>> from my rsyslog.conf, I could no longer reproduce the problem.
>>
>> Would be interesting to know, if you can confirm this.
>>
>> Cheers,
>> Michael
>>
>>   
>>     
>
>
>   


-- 
Reminds me of my expedition into the wilds of Afghanistan. We lost our 
corkscrew and were compelled to live on food and water for several days. - 
WC Fields



From david at lang.hm  Sun Mar 28 21:10:39 2010
From: david at lang.hm (david at lang.hm)
Date: Sun, 28 Mar 2010 12:10:39 -0700 (PDT)
Subject: [rsyslog] High cpu usage - rsyslog 5.4.0.
In-Reply-To: <4BAF67C0.4000506@crongeyer.com>
References: <5fa6e0144d8003c7c72edff17f9f1675@webmail.crongeyer.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>
	<4BAF67C0.4000506@crongeyer.com>
Message-ID: <alpine.DEB.2.00.1003281209061.10653@asgard.lang.hm>

On Sun, 28 Mar 2010, Ralph Crongeyer wrote:

> List,
> I tried 5.4.0 last weekend on two machines and ran it for almost a week.
> During that time I noticed that every day when I got home from work both
> machines server/log collector and my client machines fans were racing
> (both laptops, Compaq Armada M700's) ? I logged in ran top and found
> rsyslog consuming 99% of the cpu on both machines? After I restarted
> rsyslog on both they seemed to run normal. The next day the same thing.
> By this time I was watching more closely. I noticed that it took about
> an 45 to 60 minutes and rsyslog would have consumed 95 to 99% of the cpu
> on both machines?
>
> I tried to reboot both machines but it would still happen. I have
> switched back to 5.3.7 which doesn't have this problem.
>
> I'm running Debian Lenny on both machines.
> I compiled rsyslog with this:
> ./configure --prefix=/usr --sysconfdir=/etc --enable-relp
> --enable-imtemplate --enable-omtemplate --enable-rfc3195 --enable-imdiag
> --enable-mail --enable-zlib --enable-mysql --enable-gnutls
> --enable-imdiag --enable-imfile --enable-omprog --enable-omudpspoof
> --enable-omstdout --enable-diagtools
> make
> checkinstall -D make install
>
> I used the same commands to make 5.3.7 and 5.4.0.
>
> Is anyone else seeing high cpu loads with 5.4.0?

my ubuntu laptop ends up with hundreds of thousands of messages/sec and 
eats up 100% cpu with an error message about access not being allowed.

other than that I have had no problems with excessive CPU, even under 
heavy loads.

David Lang

> Thanks,
> Ralph
>
>


From david at lang.hm  Sun Mar 28 21:13:19 2010
From: david at lang.hm (david at lang.hm)
Date: Sun, 28 Mar 2010 12:13:19 -0700 (PDT)
Subject: [rsyslog] rsyslog + mysql + base(snort)
In-Reply-To: <4BAF7BAD.8020502@crongeyer.com>
References: <4BAF7BAD.8020502@crongeyer.com>
Message-ID: <alpine.DEB.2.00.1003281211250.10653@asgard.lang.hm>

I was just talking about snort logging friday with this in mind..

My understanding of the snort logs include a significant binary component. 
This would not be trivial to deal with in rsyslog without encoding it 
first.

David Lang

  On Sun, 28 Mar 2010, Ralph Crongeyer wrote:

> All,
> I'm would like to try to have a setup like this. Have Snort log to
> rsyslog and then have rsyslog log to a mysql Base schema database.
> I know that people use Barnyard and or Barnyard2 for this setup to
> offload the writing to mysql to barnyard so that barnyard could receive
> snort logs and spool them if necessary before writing to mysql should
> mysql not be able to keep up.
> It seems to me that rsyslog's spooling capability could eliminate the
> need for barnyard.
> How would one go about applying a (for lack of better words) particular
> database schema so that rsyslog could write to the base database?
>
> Does anyone have any thoughts on this?
>
> Thanks,
> Ralph
>
>


From rgerhards at hq.adiscon.com  Mon Mar 29 07:11:10 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 29 Mar 2010 07:11:10 +0200
Subject: [rsyslog] High cpu usage - rsyslog 5.4.0.
References: <5fa6e0144d8003c7c72edff17f9f1675@webmail.crongeyer.com>	<9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>	<4BAF67C0.4000506@crongeyer.com>	<aa76cdac1003280749p4673e404t7e5aefd63e102348@mail.gmail.com><4BAF710E.8080506@crongeyer.com>
	<4BAF8773.1020000@crongeyer.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103BAA@GRFEXC.intern.adiscon.com>

Hi,

thanks for the information. I have crafted a big patch package for v4 the
last two weeks. I will try to merge it in to v5 this week (I guess this will
be a bit harder than usual). It may fix the issue.

If you have a moment or two, it would be interesting to know if 5.5.2 has the
same problem.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Ralph Crongeyer
> Sent: Sunday, March 28, 2010 6:45 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] High cpu usage - rsyslog 5.4.0.
> 
> It's confirmed!
> 
> I've been running for over an hour and a half now and:
> load average: 0.06, 0.06, 0.02
> 
> Looks good so far.
> 
> Let me know if you guys (Michael, Rainer) want me to run in debug mode
> without the lines commented out to see if I can catch the problem for
> more troubleshooting.
> 
> Thanks,
> Ralph
> 
> Ralph Crongeyer wrote:
> > Ok,
> > I stoped rsyslog apt-get removed the 5.3.7 package, installed the
> 5.4.0
> > package:
> >
> > root at logs:/opt# aptitude show rsyslog
> > Package: rsyslog
> > State: installed
> > Automatically installed: no
> > Version: 5.4.0-1
> > Priority: extra
> > Section: extra
> > Maintainer: ralph at crongeyer.com
> > Uncompressed Size: 3047k
> > Depends: libc6 (>= 2.7-1), zlib1g (>= 1:1.1.4), lsb-base (>= 3.2-14)
> > Description: enhanced multi-threaded syslogd
> >
> > commented out the lines from rsyslog.conf:
> > #daemon.*;mail.*;\
> > #       news.err;\
> > #       *.=debug;*.=info;\
> > #       *.=notice;*.=warn       |/dev/xconsole
> >
> >
> > and restarted rsyslog:
> > root at logs:/opt# /etc/init.d/rsyslog restart
> > Stopping enhanced syslogd: rsyslogd.
> > Starting enhanced syslogd: rsyslogd.
> >
> > I'll get back to you in about an hour.
> >
> > Thanks,
> > Ralph
> >
> > Michael Biebl wrote:
> >
> >> 2010/3/28 Ralph Crongeyer <ralph at crongeyer.com>:
> >>
> >>
> >>> I'm running Debian Lenny on both machines.
> >>>
> >>>
> >>
> >>
> >>> Is anyone else seeing high cpu loads with 5.4.0?
> >>>
> >>>
> >>>
> >> I can confirm his (on Debian unstable) and already notified Rainer
> >> about this particular issue.
> >> Some preliminary testing here seems to indicate it's an issue with
> pipes.
> >>
> >> When I removed
> >> daemon.*;mail.*;\
> >>  news.err;\
> >>  *.=debug;*.=info;\
> >>  *.=notice;*.=warn      |/dev/xconsole
> >>
> >> from my rsyslog.conf, I could no longer reproduce the problem.
> >>
> >> Would be interesting to know, if you can confirm this.
> >>
> >> Cheers,
> >> Michael
> >>
> >>
> >>
> >
> >
> >
> 
> 
> --
> Reminds me of my expedition into the wilds of Afghanistan. We lost our
> corkscrew and were compelled to live on food and water for several
> days. -
> WC Fields
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Mon Mar 29 17:29:37 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 29 Mar 2010 17:29:37 +0200
Subject: [rsyslog] High cpu usage - rsyslog 5.4.0.
References: <5fa6e0144d8003c7c72edff17f9f1675@webmail.crongeyer.com>	<9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>	<4BAF67C0.4000506@crongeyer.com>	<aa76cdac1003280749p4673e404t7e5aefd63e102348@mail.gmail.com><4BAF710E.8080506@crongeyer.com><4BAF8773.1020000@crongeyer.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103BAA@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103BBC@GRFEXC.intern.adiscon.com>

I managed to merge the v4 changes into v5-stable. Can you pull from it's git
head (git.adiscon.com/git/rsyslog.gig) and check if it works? Or do you need
a release tarball.

Note that I need to finally fix an issue in v4 before (the released version
has an interim/work-around fix) before I can take up more work on v5, but
chances are not too bad that the issue is fixed.

The master branch will receive the new patches ASAP, but that merge is a
little bit more complicated.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Monday, March 29, 2010 7:11 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] High cpu usage - rsyslog 5.4.0.
> 
> Hi,
> 
> thanks for the information. I have crafted a big patch package for v4
> the
> last two weeks. I will try to merge it in to v5 this week (I guess this
> will
> be a bit harder than usual). It may fix the issue.
> 
> If you have a moment or two, it would be interesting to know if 5.5.2
> has the
> same problem.
> 
> Rainer
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Ralph Crongeyer
> > Sent: Sunday, March 28, 2010 6:45 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] High cpu usage - rsyslog 5.4.0.
> >
> > It's confirmed!
> >
> > I've been running for over an hour and a half now and:
> > load average: 0.06, 0.06, 0.02
> >
> > Looks good so far.
> >
> > Let me know if you guys (Michael, Rainer) want me to run in debug
> mode
> > without the lines commented out to see if I can catch the problem for
> > more troubleshooting.
> >
> > Thanks,
> > Ralph
> >
> > Ralph Crongeyer wrote:
> > > Ok,
> > > I stoped rsyslog apt-get removed the 5.3.7 package, installed the
> > 5.4.0
> > > package:
> > >
> > > root at logs:/opt# aptitude show rsyslog
> > > Package: rsyslog
> > > State: installed
> > > Automatically installed: no
> > > Version: 5.4.0-1
> > > Priority: extra
> > > Section: extra
> > > Maintainer: ralph at crongeyer.com
> > > Uncompressed Size: 3047k
> > > Depends: libc6 (>= 2.7-1), zlib1g (>= 1:1.1.4), lsb-base (>= 3.2-
> 14)
> > > Description: enhanced multi-threaded syslogd
> > >
> > > commented out the lines from rsyslog.conf:
> > > #daemon.*;mail.*;\
> > > #       news.err;\
> > > #       *.=debug;*.=info;\
> > > #       *.=notice;*.=warn       |/dev/xconsole
> > >
> > >
> > > and restarted rsyslog:
> > > root at logs:/opt# /etc/init.d/rsyslog restart
> > > Stopping enhanced syslogd: rsyslogd.
> > > Starting enhanced syslogd: rsyslogd.
> > >
> > > I'll get back to you in about an hour.
> > >
> > > Thanks,
> > > Ralph
> > >
> > > Michael Biebl wrote:
> > >
> > >> 2010/3/28 Ralph Crongeyer <ralph at crongeyer.com>:
> > >>
> > >>
> > >>> I'm running Debian Lenny on both machines.
> > >>>
> > >>>
> > >>
> > >>
> > >>> Is anyone else seeing high cpu loads with 5.4.0?
> > >>>
> > >>>
> > >>>
> > >> I can confirm his (on Debian unstable) and already notified Rainer
> > >> about this particular issue.
> > >> Some preliminary testing here seems to indicate it's an issue with
> > pipes.
> > >>
> > >> When I removed
> > >> daemon.*;mail.*;\
> > >>  news.err;\
> > >>  *.=debug;*.=info;\
> > >>  *.=notice;*.=warn      |/dev/xconsole
> > >>
> > >> from my rsyslog.conf, I could no longer reproduce the problem.
> > >>
> > >> Would be interesting to know, if you can confirm this.
> > >>
> > >> Cheers,
> > >> Michael
> > >>
> > >>
> > >>
> > >
> > >
> > >
> >
> >
> > --
> > Reminds me of my expedition into the wilds of Afghanistan. We lost
> our
> > corkscrew and were compelled to live on food and water for several
> > days. -
> > WC Fields
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From boonsiong0527 at gmail.com  Tue Mar 30 08:40:57 2010
From: boonsiong0527 at gmail.com (BoOnSiOnG)
Date: Tue, 30 Mar 2010 14:40:57 +0800
Subject: [rsyslog] imfile module
In-Reply-To: <mailman.230.1269930120.2883.rsyslog@lists.adiscon.com>
References: <mailman.230.1269930120.2883.rsyslog@lists.adiscon.com>
Message-ID: <bb92aee11003292340s7fc3ad29q8f143c251a492041@mail.gmail.com>

Hello all,

currently I'm using imfile module to monitor apache Tomcat
catalina.out, I noticed rsyslog will append date time and machine name
to every line, is it possible to configure rsyslog do not alter the
original log?

Mar 29 04:13:08 rsyslogd : INFO: Using <Listener
className="org.apache.jk.config.ApacheConfig"
modJk="PATH_TO_MOD_JK.SO_OR_DLL" />
Mar 29 04:13:08 rsyslogd : Mar 29, 2010 4:13:08 AM
org.apache.coyote.http11.Http11Protocol start
Mar 29 04:13:08 rsyslogd : INFO: Starting Coyote HTTP/1.1 on http-8080
Mar 29 04:13:08 rsyslogd : Mar 29, 2010 4:13:08 AM
org.apache.jk.common.ChannelSocket init
Mar 29 04:13:08 rsyslogd : INFO: JK: ajp13 listening on /0.0.0.0:8009
Mar 29 04:13:08 rsyslogd : Mar 29, 2010 4:13:08 AM
org.apache.jk.server.JkMain start
Mar 29 04:13:08 rsyslogd : INFO: Jk running ID=0 time=0/14 config=null
Mar 29 04:13:08 rsyslogd : Mar 29, 2010 4:13:08 AM
org.apache.catalina.startup.Catalina start

my configuration,
$InputFileName /srv/tomcat/logs/catalina.out
$InputFileTag :
$InputFileStateFile stat-file-tomcat-01
$InputFileSeverity info
$InputFileFacility local3
$InputRunFileMonitor

Thanks. :)


From david at lang.hm  Tue Mar 30 09:03:37 2010
From: david at lang.hm (david at lang.hm)
Date: Tue, 30 Mar 2010 00:03:37 -0700 (PDT)
Subject: [rsyslog] imfile module
In-Reply-To: <bb92aee11003292340s7fc3ad29q8f143c251a492041@mail.gmail.com>
References: <mailman.230.1269930120.2883.rsyslog@lists.adiscon.com>
	<bb92aee11003292340s7fc3ad29q8f143c251a492041@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1003300000170.14525@asgard.lang.hm>

On Tue, 30 Mar 2010, BoOnSiOnG wrote:

> Hello all,
>
> currently I'm using imfile module to monitor apache Tomcat
> catalina.out, I noticed rsyslog will append date time and machine name
> to every line, is it possible to configure rsyslog do not alter the
> original log?

the way to do this would be to change the output format when writing the 
log.

David Lang

> Mar 29 04:13:08 rsyslogd : INFO: Using <Listener
> className="org.apache.jk.config.ApacheConfig"
> modJk="PATH_TO_MOD_JK.SO_OR_DLL" />
> Mar 29 04:13:08 rsyslogd : Mar 29, 2010 4:13:08 AM
> org.apache.coyote.http11.Http11Protocol start
> Mar 29 04:13:08 rsyslogd : INFO: Starting Coyote HTTP/1.1 on http-8080
> Mar 29 04:13:08 rsyslogd : Mar 29, 2010 4:13:08 AM
> org.apache.jk.common.ChannelSocket init
> Mar 29 04:13:08 rsyslogd : INFO: JK: ajp13 listening on /0.0.0.0:8009
> Mar 29 04:13:08 rsyslogd : Mar 29, 2010 4:13:08 AM
> org.apache.jk.server.JkMain start
> Mar 29 04:13:08 rsyslogd : INFO: Jk running ID=0 time=0/14 config=null
> Mar 29 04:13:08 rsyslogd : Mar 29, 2010 4:13:08 AM
> org.apache.catalina.startup.Catalina start
>
> my configuration,
> $InputFileName /srv/tomcat/logs/catalina.out
> $InputFileTag :
> $InputFileStateFile stat-file-tomcat-01
> $InputFileSeverity info
> $InputFileFacility local3
> $InputRunFileMonitor
>
> Thanks. :)
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From mbiebl at gmail.com  Mon Mar  1 00:00:57 2010
From: mbiebl at gmail.com (Michael Biebl)
Date: Mon, 1 Mar 2010 00:00:57 +0100
Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103981@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com>
	<aa76cdac1002251244y2be4e04egc7e0e3eb57494431@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com>
	<aa76cdac1002260355l26269784y8fb8ccd45caeaa12@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com>
	<aa76cdac1002260443p4c7744e2ha1c8c2372c741ebf@mail.gmail.com>
	<aa76cdac1002280515g72c07527mae8ebc08d4ecd404@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103980@GRFEXC.intern.adiscon.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103981@GRFEXC.intern.adiscon.com>
Message-ID: <aa76cdac1002281500g3d50db29rcd20fea1fbf08d48@mail.gmail.com>

2010/2/28 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>> OK, rather than trying to figure out what may be going on here, I'll
>> see that
>> I copy the old omfile code and create a new ompipe. So that will behave
>> exactly like it did before. The new optimizations are not so important
>> for
>> pipes.
>
> I have done this now. I would appreciate if you could checkout the v4-stable
> git head and try with that one. It should now have the exact same behavior as
> in 4.4.2 (the pipe code is now the same!). I tested on Debian and it worked
> like before.

I use $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
in my rsyslog.conf, but the new ompipe module does not seem to respect
that setting but always uses the high-resolution time format.

Other than that, I did not (yet) experience any blank xconsole windows
with v4.6.0-5-gf12a199 (git describe)

Just curious, when I fire up xconsole after boot, I get two log
messages immediately. Where do those log messages come from?

Michael


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From david at lang.hm  Mon Mar  1 03:15:57 2010
From: david at lang.hm (david at lang.hm)
Date: Sun, 28 Feb 2010 18:15:57 -0800 (PST)
Subject: [rsyslog] Feedback requested: inconsistency in config statements
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710397F@GRFEXC.intern.adiscon.com>
References: <001201cab868$d148da7c$100013ac@intern.adiscon.com>
	<alpine.DEB.2.00.1002280426420.15263@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA710397F@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.2.00.1002281346000.15263@asgard.lang.hm>

On Sun, 28 Feb 2010, Rainer Gerhards wrote:

> Hi David,
>
> yes, that makes much more sense to me. Let's call that filter expression tree
> for the time being (the parse tree used for normalization has some similar
> ideas, but a totally different scope and semantics).
>
> While I now see where you are heading, I fail to see how this could
> sufficiently easy be implemented for a real (and complex) configuration.
>
> Let's look at this example conf:
>
> mail.* /Var/log/mail.log
> if programname startswith abc log to /abc
> if programname startswith 'acc' and MSG contains 'error' log to /acc
> # we don't want debug messages from here on
> *.debug ~
> if programname startswith 'acc' or (programname contains 'bde' and (facility
>> 1 and
> facility < 5)) log to @1.1.1.1
> if programname startswith bcd or MSG startswith '123' or MSG contains
> 'error2' log to /bcd
> *.* /var/log/catchall
>
> I simply cannot envision how you would store this as a tree that does not
> look like what we have today.

Ok, Here is what I came up with.

0-1,0-6
   --programname startswith abc log to /abc
   | programname startswith 'acc' and MSG contains 'error' log to /acc
    \programname startswith 'acc' log to @1.1.1.1
   --programname contains 'bde' log to @1.1.1.1
   --MSG startswith '123' log to/bcd
   --MSG contains 'error2' log to /bcd
   --log to /var/log/catchall
0-1,7
   --programname startswith abc log to /abc
    \programname startswith 'acc' and MSG contains 'error' log to /acc
2,0-6
   --log to /var/log/mail.log
   --programname startswith abc log to /abc
   | programname startswith 'acc' and MSG contains 'error' log to /acc
   | programname startswith 'acc' log to @1.1.1.1
    \programname startswith bcd log to /bcd
   --programname contains 'bde' log to @1.1.1.1
   --MSG startswith '123' log to/bcd
   --MSG contains 'error2' log to /bcd
   --log to /var/log/catchall
2,7
   --log to /var/log/mail.log
   --programname startswith abc log to /abc
    \programname startswith 'acc' and MSG contains 'error' log to /acc
3-4,0-6
   --programname startswith abc log to /abc
   | programname startswith 'acc' and MSG contains 'error' log to /acc
   | programname startswith 'acc' log to @1.1.1.1
    \programname startswith bcd log to /bcd
   --programname contains 'bde' log to @1.1.1.1
   --MSG startswith '123' log to/bcd
   --MSG contains 'error2' log to /bcd
   --log to /var/log/catchall
3-4,7
   --programname startswith abc log to /abc
    \programname startswith 'acc' and MSG contains 'error' log to /acc
5-23,0-6
   --programname startswith abc log to /abc
   | programname startswith 'acc' and MSG contains 'error' log to /acc
    \programname startswith 'acc' log to @1.1.1.1
   --programname contains 'bde' log to @1.1.1.1
   --MSG startswith '123' log to/bcd
   --MSG contains 'error2' log to /bcd
   --log to /var/log/catchall
5-23,7
   --programname startswith abc log to /abc
    \programname startswith 'acc' and MSG contains 'error' log to /acc

This looks very ugly and is definantly more complicated, but I think it 
still ends up being a slight (but not drastic) win.

for the initial ruleset you have to check a minimum of 4 tests and a 
maximum of 13 tests (debug messages have 4-5 tests, all others have 12-13)

Assuming the facility/priority lookup can be a table lookup, for the 
tree-based ruleset you have a minimum of 2 tests and a maximum of 7 tests, 
in each case one of the tests is a multi-option comparison which is 
slightly more expensive than a single 'startswith' test, but should be 
cheaper than two 'startswith' tests.

This definantly trades off start time complexity for runtime efficiancy.

David Lang


and here is the long, ugly process I went through to derive this.



first normalize the tests (replace 'or' with two lines, change facility 
and priority to ranges of numbers)

facility = 2 log to /Var/log/mail.log
programname startswith abc log to /abc
programname startswith 'acc' and MSG contains 'error' log to /acc
severity 0..6 and programname startswith 'acc' log to @1.1.1.1
severity 0..6 and programname startswith bcd log to /bcd
facility 2..4 and severity 0..6 and programname contains 'bde' log to @1.1.1.1
severity 0..6 and MSG startswith '123' log to/bcd
severity 0..6 and MSG contains 'error2' log to /bcd
severity 0..6 and log to /var/log/catchall

I then add the full facility/priority ranges


now normalize the facility/severity tests (split overlaps)

we have the following ranges for facility
0-1
2
3-4
5-23

and for severity
0-6
7

to shorten the lines in the e-mail I will letter the remaining 
tests/actions

A) log to /Var/log/mail.log
B) programname startswith abc log to /abc
C) programname startswith 'acc' and MSG contains 'error' log to /acc
D) programname startswith 'acc' log to @1.1.1.1
E) programname startswith bcd log to /bcd
F) programname contains 'bde' log to @1.1.1.1
G) MSG startswith '123' log to/bcd
H) MSG contains 'error2' log to /bcd
I) log to /var/log/catchall

so the tree then becomes (straight translation no reordering)

2,0-6 A
2,7 A
0-1,0-6 B
0-1,7 B
2,0-6 B
2,7 B
3-4,0-6 B
3-4,7 B
5-23,0-6 B
5-23,7 B
0-1,0-6 C
0-1,7 C
2,0-6 C
2,7 C
3-4,0-6 C
3-4,7 C
5-23,0-6 C
5-23,7 C
0-1,0-6 D
2,0-6 D
3-4,0-6 D
5-23,0-6 D
2,0-6 E
3-4,0-6 E
0-1,0-6 F
2,0-6 F
3-4,0-6 F
5-23,0-6 F
0-1,0-6 G
2,0-6 G
3-4,0-6 G
5-23,0-6 G
0-1,0-6 H
2,0-6 H
3-4,0-6 H
5-23,0-6 H
0-1,0-6 I
2,0-6 I
3-4,0-6 I
5-23,0-6 I

now to order them

0-1,0-6 B
0-1,0-6 C
0-1,0-6 D
0-1,0-6 F
0-1,0-6 G
0-1,0-6 H
0-1,0-6 I
0-1,7 B
0-1,7 C
2,0-6 A
2,0-6 B
2,0-6 I
2,0-6 H
2,0-6 G
2,0-6 F
2,0-6 E
2,0-6 D
2,0-6 C
2,7 A
2,7 B
2,7 C
3-4,0-6 B
3-4,0-6 C
3-4,0-6 D
3-4,0-6 E
3-4,0-6 F
3-4,0-6 G
3-4,0-6 H
3-4,0-6 I
3-4,7 B
3-4,7 C
5-23,0-6 B
5-23,0-6 C
5-23,0-6 D
5-23,0-6 F
5-23,0-6 G
5-23,0-6 H
5-23,0-6 I
5-23,7 B
5-23,7 C

I then simplify this to

0-1,0-6 B,C,D,F,G,H,I
0-1,7 B,C
2,0-6 A,B,C,D,E,F,G,H,I
2,7 A,B,C
3-4,0-6 B,C,D,E,F,G,H,I
3-4,7 B,C
5-23,0-6 B,C,D,F,G,H,I
5-23,7 B,C

looking at the lettered tests

A) log to /Var/log/mail.log
B) programname startswith abc log to /abc
C) programname startswith 'acc' and MSG contains 'error' log to /acc
D) programname startswith 'acc' log to @1.1.1.1
E) programname startswith bcd log to /bcd
F) programname contains 'bde' log to @1.1.1.1
G) MSG startswith '123' log to/bcd
H) MSG contains 'error2' log to /bcd
I) log to /var/log/catchall

the only ones that can be combined are B-E

so if you make tests

J) B,C
K) B,C,D
L) B,C,D,E

the results simplify to

0-1,0-6 K,F,G,H,I
0-1,7 J
2,0-6 A,L,F,G,H,I
2,7 A,J
3-4,0-6 L,F,G,H,I
3-4,7 J
5-23,0-6 K,F,G,H,I
5-23,7 J

expanded out to show all tests

0-1,0-6
   --programname startswith abc log to /abc
   | programname startswith 'acc' and MSG contains 'error' log to /acc
    \programname startswith 'acc' log to @1.1.1.1
   --programname contains 'bde' log to @1.1.1.1
   --MSG startswith '123' log to/bcd
   --MSG contains 'error2' log to /bcd
   --log to /var/log/catchall
0-1,7
   --programname startswith abc log to /abc
    \programname startswith 'acc' and MSG contains 'error' log to /acc
2,0-6
   --log to /var/log/mail.log
   --programname startswith abc log to /abc
   | programname startswith 'acc' and MSG contains 'error' log to /acc
   | programname startswith 'acc' log to @1.1.1.1
    \programname startswith bcd log to /bcd
   --programname contains 'bde' log to @1.1.1.1
   --MSG startswith '123' log to/bcd
   --MSG contains 'error2' log to /bcd
   --log to /var/log/catchall
2,7
   --log to /var/log/mail.log
   --programname startswith abc log to /abc
    \programname startswith 'acc' and MSG contains 'error' log to /acc
3-4,0-6
   --programname startswith abc log to /abc
   | programname startswith 'acc' and MSG contains 'error' log to /acc
   | programname startswith 'acc' log to @1.1.1.1
    \programname startswith bcd log to /bcd
   --programname contains 'bde' log to @1.1.1.1
   --MSG startswith '123' log to/bcd
   --MSG contains 'error2' log to /bcd
   --log to /var/log/catchall
3-4,7
   --programname startswith abc log to /abc
    \programname startswith 'acc' and MSG contains 'error' log to /acc
5-23,0-6
   --programname startswith abc log to /abc
   | programname startswith 'acc' and MSG contains 'error' log to /acc
    \programname startswith 'acc' log to @1.1.1.1
   --programname contains 'bde' log to @1.1.1.1
   --MSG startswith '123' log to/bcd
   --MSG contains 'error2' log to /bcd
   --log to /var/log/catchall
5-23,7
   --programname startswith abc log to /abc
    \programname startswith 'acc' and MSG contains 'error' log to /acc

> Rainer
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Sunday, February 28, 2010 2:02 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] Feedback requested: inconsistency in config
>> statements
>>
>> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
>>
>>> Quickly from the mobile: pri based filters require *excatly* one
>> single word (!) lookup currently. You can't beat that.
>>>
>>> I am not totally sure how you think about parse trees. In my view,
>> parsin and filteing are two different things./stages. Maybe we have
>> different concepts on our minds?
>>
>> I am probably using the wrong terminology here.
>>
>> say you have a set of rules that say
>>
>> if programname startswith abc log to /abc
>> if programname startswith acc log to /acc
>> if programname startswith acc log to @1.1.1.1
>> if programname startswith bcd log to /bcd
>>
>> and assuming that these were the only possible programnames for
>> simplicity
>> in the explination.
>>
>> the filtering logic would go somthing like this
>>
>> the rules would compile into a tree
>>
>> -a-b -> /abc
>>    -c -> /acc, at 1.1.1.1
>> -b   -> /bcd
>>
>> receive programname abc
>> I have no facility/Pri filter rules
>> I have no time filter rules
>> I have no system filter rules
>> I have programname filter rules
>>    look at the first character of the programname
>>     it's "a", I have more than one rule that fits that.
>>      look at the second character of the programname
>>       it's a "b", There are no other decisions to make,
>>         invoke the action /abc
>>
>> receive progranmane bcd
>> I have no facility/PRI filter rules
>> I have no time filter rules
>> I have no system filter rules
>> I have programname filter rules
>>    look at the first character of the programname
>>     it's a "b", There are no other decisions to make,
>>       invoke the action /bcd
>>
>> receive programname acc
>> I have no facility/Pri filter rules
>> I have no time filter rules
>> I have no system filter rules
>> I have programname filter rules
>>    look at the first character of the programname
>>     it's "a", I have more than one rule that fits that.
>>      look at the second character of the programname
>>       it's a "c", There are no other decisions to make,
>>         invoke the action /acc and the action @1.1.1.1
>>
>> similarly the facility/pri filtering rules would be either compiled
>> into a
>> tree, or (since it's 256 entries total) into a lookup table with
>> pointers
>> to the list of actions to take for that entry
>>
>> the idea is to spend the time at startup to create the tree that
>> represents the ruleset in order to speed up the processing of each
>> individual message.
>>
>> the real tree would be a bit more complicated than I describe here as
>> it
>> needs to have 'anything else' entries between the known entries (which
>> means that it would not be able to shortcut quite as much as I
>> describe),
>> and have provision for 'do a more complicated thing (like regex) here
>> and
>> if it matches continue'. but except for regex matches, this would make
>> processing the rules pretty much independant of how many rules there
>> were
>> or how complex the ruleset is.
>>
>> there doe snot need to be one single programname filter tree, if you
>> had a
>> rule that said
>> if pri = info and programname startswith def then def
>>
>> the pri tree/table would have an entry for info that would point to a
>> filter tree for programname that just had the check for def in it
>>
>> am I makeing any more sense now?
>>
>> David Lang
>>
>>> rainer
>>>
>>> ----- Urspr?ngliche Nachricht -----
>>> Von: "david at lang.hm" <david at lang.hm>
>>> An: "rsyslog-users" <rsyslog at lists.adiscon.com>
>>> Gesendet: 28.02.10 11:28
>>> Betreff: Re: [rsyslog] Feedback requested: inconsistency in config
>> statements
>>>
>>> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
>>>
>>>>>
>>>>> if the type is depriciated then the destination would be another
>>>>> config_option (which you would then look up)
>>>>>
>>>>> if the type is 'ignore' then it would just be skipped over
>> (possibly
>>>>> with
>>>>> a warning being logged that the config line is being ignored)
>>>>>
>>>>> this would also clean up some of the current cases where a boolean
>>>>> option
>>>>> doesn't honor on/off and true/false.
>>>>
>>>> True/false is NOT a valid boolean option. See
>>>>
>>>>
>> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=18
>> 4c0d87
>>>> 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308
>>>
>>> I was not remembering correctly then, maby it was yes/no vs on/off. I
>> know
>>> I ran into something where what was documented didn't work and I
>> changed
>>> it to another version of a boolean answer and it fixed the problem.
>>>
>>>>> For the second half of the config language (telling rsyslog what to
>> do
>>>>> with the logs it has received) also has several variations in place
>> and
>>>>> is
>>>>> showing issues.
>>>>>
>>>>> However I think that the right answer here is to end up
>> implementing
>>>>> something like the parsing trees and then compile the other config
>>>>> language options into that tree to be consistant (and fast) under
>> the
>>>>> covers, no matter which way the instructions are written (except
>> when
>>>>> you
>>>>> have to use regex matches)
>>>>
>>>> I don't fully agree here with you. For example, the traditional PRI
>> based
>>>> filter is lightning fast. I don't see any way it could be nearly as
>> fast with
>>>> any unified approach. Right now, we have a "unified" filter
>> structure, but it
>>>> contains three filter cases, each one adding potential power at the
>> price of
>>>> decreased speed. I think we need to keep different types of filters
>> in order
>>>> to have some lightning-fast ones. But more of this could be done
>> under the
>>>> hood.
>>>
>>> My guess/expectation is that the tree-based parsing would be about
>> the
>>> same speed as the traditional PRI based filter for choices made based
>> on
>>> PRI, slowing down for other types of conditions only in that more of
>> the
>>> message may need to be scanned (and if there is no selection at a
>> level,
>>> skipping that level should be lightning fast). As a result, a set of
>>> filteres based soley on programname would be as fast as the current
>> PRI
>>> filters.
>>>
>>> David Lang
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From reader at newsguy.com  Mon Mar  1 04:01:34 2010
From: reader at newsguy.com (Harry Putnam)
Date: Sun, 28 Feb 2010 21:01:34 -0600
Subject: [rsyslog] How rsyslog derives hostname
Message-ID: <87hbp0eodt.fsf@newsguy.com>

Running rsyslog-3.22.1

NOTE: Windbag alert.. its a bit long.
(And still more windyness in a PS at the end)

I know newer version are available but 3.22.1 is the newest version my
linux distro (gentoo 2010) offers.

Maybe the mechanism that caused the phenomena I observed has changed
in the later versions.

I'm curious to know how rsyslog derives the hostname.

Before explaining the phenomena I noticed, just be clear that this
problem arose strictly from a blunder on my part.  So, this is not a
complaint about rsyslog.  Just wondering if it points to what could be
a problem.

Due to a blatant blunder on my part, I miss-typed the hosts name
in the /etc/hosts file.  At the 127 line I spelled the hosts name wrong.

    127.0.0.1       logsrv.local.lan   *=>logsvr<=*     localhost

The actual name is `logsrv'
    logsrv
not logsvr.  (I transposed the v and r.)

In this code below from my (no doubt, poorly written)  rsyslog.conf 

[
  This first part contains some fairly standard code.  Mostly
  unchanged from the default /etc/rsyslog.conf that gets installed.
 [ blah ]
]
  [then my own code below]
  [...]

  ## write all localhost output to a debug file
  $template Ldebug,"/var/log/debug.log"
  if \
      $source == 'logsrv'\
  then  -?Ldebug

  ## write all data from remote hosts to:
  ## HOSTNAME/HOSTNAME.log for each client
  $template RDDF,"/var/log/%hostname%/%hostname%.log"
  if \
          $source  != 'logsrv' \
  then    -?RDDF
  [...]

Note: I used the actual name (logsrv) in the code, since I wasn't sure
which of rsyslogs' variables would contain that info instead of
`localhost' or `0.0.0.0' or `127.0.0.1` ... etc.

And understand that this rsyslog.conf was being used in an
experimental environment while I familiarized myself with rsyslog.
 
The idea was to direct all localhost log output to /var/log/debug.log
(in addition to the more normal defaults in rsyslog.conf not shown here).
And log data coming from remote clients to /var/log/dynaDIR/dynaFILE
Where DIR and FILE are both named dynamically after the host sending
the log data.

It worked just like I had hoped.

But I accidentally found what might be seen as a weakness in rsyslog.

I'm not experienced enough to really judge something like this. And
not adept enough at reading source code to see what the mechanism is.

Apparently rsyslog preferred the miss-typed name in /etc/hosts over
the name returned by gethostbyname or the newer getaddrinfo, both
appear to be involved in the source code.

With the mistaken spelling in /etc/hosts, rsyslog (correctly) saw the
mistaken name inside `$source' as not matching `logsrv' in my code, so
wrote to:

  /var/log/logsvr/logsvr.log

Once I got the typo fixed, rsyslog wrote all that same stuff to
/var/log/debug.log as expected.

So it makes me wonder how robust the mechanism for getting the
localhost name is.  rsyslog appears to use gethostbyname and the
newer getaddrinfo.

Further... I'm not sure where cmds like gethostbyname (and similar
ones) get the info either... for all I know they may just grep
/etc/hosts.

I do know that even while I had the mis-spelled name in /etc/hosts, the
`hostname' command still returned the correct name.  So the `hostname'
cmd must rely on something else.

At any rate, rsyslog apparently turns to grepping the /etc/hosts
file at some point, since in this case the log output was directed to a
directory and file named after the TYPO in /etc/hosts.

It took me a few minutes to figure out where the bad name was coming
from since the `hostname' cmd returned the correct name (logsrv).

Some linux distros have a special file that tells the OS what its name
is. 

On my distro (gentoo linux) it's a file at /etc/conf.d/hostname, that
is dedicated to nothing more than giving the host a name.

On Debian its /etc/hostname

Other distros I've used have a few different ways of getting that
information into play, but I haven't used any linux distros that
relied on /etc/hosts.

Is having rsyslog rely on /etc/hosts as the final arbiter about the
host name a good plan?

ps - two asides:
 1) Any suggestions for better code are welcome
 2) Can anyone tell me what the significance of the question mark
    preceding the log file variable name is:

  $template RDDF,"/var/log/%hostname%/%hostname%.log"
  if \
          $source  != 'logsrv' \
  then    -?RDDF
          ------
           here

I know what the dash sinifies but not the question mark. I just
copied what I saw in the examples and it seems to work.
          



From rgerhards at hq.adiscon.com  Mon Mar  1 07:45:48 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 1 Mar 2010 07:45:48 +0100
Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released
References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com><aa76cdac1002251244y2be4e04egc7e0e3eb57494431@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com><aa76cdac1002260355l26269784y8fb8ccd45caeaa12@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com><aa76cdac1002260443p4c7744e2ha1c8c2372c741ebf@mail.gmail.com><aa76cdac1002280515g72c07527mae8ebc08d4ecd404@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103980@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103981@GRFEXC.intern.adiscon.com>
	<aa76cdac1002281500g3d50db29rcd20fea1fbf08d48@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103983@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Monday, March 01, 2010 12:01 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released
> 
> 2010/2/28 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> >> OK, rather than trying to figure out what may be going on here, I'll
> >> see that
> >> I copy the old omfile code and create a new ompipe. So that will
> behave
> >> exactly like it did before. The new optimizations are not so
> important
> >> for
> >> pipes.
> >
> > I have done this now. I would appreciate if you could checkout the
> v4-stable
> > git head and try with that one. It should now have the exact same
> behavior as
> > in 4.4.2 (the pipe code is now the same!). I tested on Debian and it
> worked
> > like before.
> 
> I use $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
> in my rsyslog.conf, but the new ompipe module does not seem to respect
> that setting but always uses the high-resolution time format.

I see - that was a side-effect that pipes used the same output module like
regular pipes. But I have now re-established it, please try the newest git
v4-stable branch.

> 
> Other than that, I did not (yet) experience any blank xconsole windows
> with v4.6.0-5-gf12a199 (git describe)
> 
> Just curious, when I fire up xconsole after boot, I get two log
> messages immediately. Where do those log messages come from?

It seems to stem back to the API usage in the old code. The old code opens
the pipe and tries to write. It disables writing only when the API returns it
would block. The new code used create() instead of open, and that immediately
returned an error if nobody was listening.

It looks like with the old sequence, the OS permits to write a handful of
bytes before it returns EAGAIN. However, I would not take that behavior for
granted: I guess it may be different under different platforms and/or
versions.

Rainer


From mrdemeanour at jackpot.uk.net  Mon Mar  1 08:41:17 2010
From: mrdemeanour at jackpot.uk.net (Mr. Demeanour)
Date: Mon, 01 Mar 2010 07:41:17 +0000
Subject: [rsyslog] How rsyslog derives hostname
In-Reply-To: <87hbp0eodt.fsf@newsguy.com>
References: <87hbp0eodt.fsf@newsguy.com>
Message-ID: <4B8B6F9D.5040803@jackpot.uk.net>

Harry Putnam wrote:
> 
> Further... I'm not sure where cmds like gethostbyname (and similar
> ones) get the info either... for all I know they may just grep
> /etc/hosts.

man resolv.conf (i.e. it's configurable).

-- 
Jack.


From rgerhards at hq.adiscon.com  Mon Mar  1 10:39:27 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 1 Mar 2010 10:39:27 +0100
Subject: [rsyslog] How rsyslog derives hostname
References: <87hbp0eodt.fsf@newsguy.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710398A@GRFEXC.intern.adiscon.com>

I need to re-read this more careful, but for now let me say that rsyslog does
not touch /etc/hosts at all. It exlusively relies on what is returned by the
OS. But the "source" property is not locally generated, it contains whatever
the sender placed into the relevant field.

Please also see this article:

http://www.rsyslog.com/doc-syslog_parsing.html

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Harry Putnam
> Sent: Monday, March 01, 2010 4:02 AM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] How rsyslog derives hostname
> 
> Running rsyslog-3.22.1
> 
> NOTE: Windbag alert.. its a bit long.
> (And still more windyness in a PS at the end)
> 
> I know newer version are available but 3.22.1 is the newest version my
> linux distro (gentoo 2010) offers.
> 
> Maybe the mechanism that caused the phenomena I observed has changed
> in the later versions.
> 
> I'm curious to know how rsyslog derives the hostname.
> 
> Before explaining the phenomena I noticed, just be clear that this
> problem arose strictly from a blunder on my part.  So, this is not a
> complaint about rsyslog.  Just wondering if it points to what could be
> a problem.
> 
> Due to a blatant blunder on my part, I miss-typed the hosts name
> in the /etc/hosts file.  At the 127 line I spelled the hosts name
> wrong.
> 
>     127.0.0.1       logsrv.local.lan   *=>logsvr<=*     localhost
> 
> The actual name is `logsrv'
>     logsrv
> not logsvr.  (I transposed the v and r.)
> 
> In this code below from my (no doubt, poorly written)  rsyslog.conf
> 
> [
>   This first part contains some fairly standard code.  Mostly
>   unchanged from the default /etc/rsyslog.conf that gets installed.
>  [ blah ]
> ]
>   [then my own code below]
>   [...]
> 
>   ## write all localhost output to a debug file
>   $template Ldebug,"/var/log/debug.log"
>   if \
>       $source == 'logsrv'\
>   then  -?Ldebug
> 
>   ## write all data from remote hosts to:
>   ## HOSTNAME/HOSTNAME.log for each client
>   $template RDDF,"/var/log/%hostname%/%hostname%.log"
>   if \
>           $source  != 'logsrv' \
>   then    -?RDDF
>   [...]
> 
> Note: I used the actual name (logsrv) in the code, since I wasn't sure
> which of rsyslogs' variables would contain that info instead of
> `localhost' or `0.0.0.0' or `127.0.0.1` ... etc.
> 
> And understand that this rsyslog.conf was being used in an
> experimental environment while I familiarized myself with rsyslog.
> 
> The idea was to direct all localhost log output to /var/log/debug.log
> (in addition to the more normal defaults in rsyslog.conf not shown
> here).
> And log data coming from remote clients to /var/log/dynaDIR/dynaFILE
> Where DIR and FILE are both named dynamically after the host sending
> the log data.
> 
> It worked just like I had hoped.
> 
> But I accidentally found what might be seen as a weakness in rsyslog.
> 
> I'm not experienced enough to really judge something like this. And
> not adept enough at reading source code to see what the mechanism is.
> 
> Apparently rsyslog preferred the miss-typed name in /etc/hosts over
> the name returned by gethostbyname or the newer getaddrinfo, both
> appear to be involved in the source code.
> 
> With the mistaken spelling in /etc/hosts, rsyslog (correctly) saw the
> mistaken name inside `$source' as not matching `logsrv' in my code, so
> wrote to:
> 
>   /var/log/logsvr/logsvr.log
> 
> Once I got the typo fixed, rsyslog wrote all that same stuff to
> /var/log/debug.log as expected.
> 
> So it makes me wonder how robust the mechanism for getting the
> localhost name is.  rsyslog appears to use gethostbyname and the
> newer getaddrinfo.
> 
> Further... I'm not sure where cmds like gethostbyname (and similar
> ones) get the info either... for all I know they may just grep
> /etc/hosts.
> 
> I do know that even while I had the mis-spelled name in /etc/hosts, the
> `hostname' command still returned the correct name.  So the `hostname'
> cmd must rely on something else.
> 
> At any rate, rsyslog apparently turns to grepping the /etc/hosts
> file at some point, since in this case the log output was directed to a
> directory and file named after the TYPO in /etc/hosts.
> 
> It took me a few minutes to figure out where the bad name was coming
> from since the `hostname' cmd returned the correct name (logsrv).
> 
> Some linux distros have a special file that tells the OS what its name
> is.
> 
> On my distro (gentoo linux) it's a file at /etc/conf.d/hostname, that
> is dedicated to nothing more than giving the host a name.
> 
> On Debian its /etc/hostname
> 
> Other distros I've used have a few different ways of getting that
> information into play, but I haven't used any linux distros that
> relied on /etc/hosts.
> 
> Is having rsyslog rely on /etc/hosts as the final arbiter about the
> host name a good plan?
> 
> ps - two asides:
>  1) Any suggestions for better code are welcome
>  2) Can anyone tell me what the significance of the question mark
>     preceding the log file variable name is:
> 
>   $template RDDF,"/var/log/%hostname%/%hostname%.log"
>   if \
>           $source  != 'logsrv' \
>   then    -?RDDF
>           ------
>            here
> 
> I know what the dash sinifies but not the question mark. I just
> copied what I saw in the examples and it seems to work.
> 
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Mon Mar  1 14:27:57 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 1 Mar 2010 14:27:57 +0100
Subject: [rsyslog] Feedback requested: inconsistency in config statements
References: <001201cab868$d148da7c$100013ac@intern.adiscon.com><alpine.DEB.2.00.1002280426420.15263@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA710397F@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.00.1002281346000.15263@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103992@GRFEXC.intern.adiscon.com>

Hi David,

thanks for the explanation, I now see where you are heading too. I have not
dug into all details, but find the post very interesting and insightful. I'll
archive it so that I have it at hand when I can take the route to actually
implement such an approach. All in all, it still looks rather complex to me.
I guess it would be useful to have a guy with experience in writing compiler
code optimizers at hand when I tackle that beast ;)

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Monday, March 01, 2010 3:16 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Feedback requested: inconsistency in config
> statements
> 
> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
> 
> > Hi David,
> >
> > yes, that makes much more sense to me. Let's call that filter
> expression tree
> > for the time being (the parse tree used for normalization has some
> similar
> > ideas, but a totally different scope and semantics).
> >
> > While I now see where you are heading, I fail to see how this could
> > sufficiently easy be implemented for a real (and complex)
> configuration.
> >
> > Let's look at this example conf:
> >
> > mail.* /Var/log/mail.log
> > if programname startswith abc log to /abc
> > if programname startswith 'acc' and MSG contains 'error' log to /acc
> > # we don't want debug messages from here on
> > *.debug ~
> > if programname startswith 'acc' or (programname contains 'bde' and
> (facility
> >> 1 and
> > facility < 5)) log to @1.1.1.1
> > if programname startswith bcd or MSG startswith '123' or MSG contains
> > 'error2' log to /bcd
> > *.* /var/log/catchall
> >
> > I simply cannot envision how you would store this as a tree that does
> not
> > look like what we have today.
> 
> Ok, Here is what I came up with.
> 
> 0-1,0-6
>    --programname startswith abc log to /abc
>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>     \programname startswith 'acc' log to @1.1.1.1
>    --programname contains 'bde' log to @1.1.1.1
>    --MSG startswith '123' log to/bcd
>    --MSG contains 'error2' log to /bcd
>    --log to /var/log/catchall
> 0-1,7
>    --programname startswith abc log to /abc
>     \programname startswith 'acc' and MSG contains 'error' log to /acc
> 2,0-6
>    --log to /var/log/mail.log
>    --programname startswith abc log to /abc
>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>    | programname startswith 'acc' log to @1.1.1.1
>     \programname startswith bcd log to /bcd
>    --programname contains 'bde' log to @1.1.1.1
>    --MSG startswith '123' log to/bcd
>    --MSG contains 'error2' log to /bcd
>    --log to /var/log/catchall
> 2,7
>    --log to /var/log/mail.log
>    --programname startswith abc log to /abc
>     \programname startswith 'acc' and MSG contains 'error' log to /acc
> 3-4,0-6
>    --programname startswith abc log to /abc
>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>    | programname startswith 'acc' log to @1.1.1.1
>     \programname startswith bcd log to /bcd
>    --programname contains 'bde' log to @1.1.1.1
>    --MSG startswith '123' log to/bcd
>    --MSG contains 'error2' log to /bcd
>    --log to /var/log/catchall
> 3-4,7
>    --programname startswith abc log to /abc
>     \programname startswith 'acc' and MSG contains 'error' log to /acc
> 5-23,0-6
>    --programname startswith abc log to /abc
>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>     \programname startswith 'acc' log to @1.1.1.1
>    --programname contains 'bde' log to @1.1.1.1
>    --MSG startswith '123' log to/bcd
>    --MSG contains 'error2' log to /bcd
>    --log to /var/log/catchall
> 5-23,7
>    --programname startswith abc log to /abc
>     \programname startswith 'acc' and MSG contains 'error' log to /acc
> 
> This looks very ugly and is definantly more complicated, but I think it
> still ends up being a slight (but not drastic) win.
> 
> for the initial ruleset you have to check a minimum of 4 tests and a
> maximum of 13 tests (debug messages have 4-5 tests, all others have 12-
> 13)
> 
> Assuming the facility/priority lookup can be a table lookup, for the
> tree-based ruleset you have a minimum of 2 tests and a maximum of 7
> tests,
> in each case one of the tests is a multi-option comparison which is
> slightly more expensive than a single 'startswith' test, but should be
> cheaper than two 'startswith' tests.
> 
> This definantly trades off start time complexity for runtime
> efficiancy.
> 
> David Lang
> 
> 
> and here is the long, ugly process I went through to derive this.
> 
> 
> 
> first normalize the tests (replace 'or' with two lines, change facility
> and priority to ranges of numbers)
> 
> facility = 2 log to /Var/log/mail.log
> programname startswith abc log to /abc
> programname startswith 'acc' and MSG contains 'error' log to /acc
> severity 0..6 and programname startswith 'acc' log to @1.1.1.1
> severity 0..6 and programname startswith bcd log to /bcd
> facility 2..4 and severity 0..6 and programname contains 'bde' log to
> @1.1.1.1
> severity 0..6 and MSG startswith '123' log to/bcd
> severity 0..6 and MSG contains 'error2' log to /bcd
> severity 0..6 and log to /var/log/catchall
> 
> I then add the full facility/priority ranges
> 
> 
> now normalize the facility/severity tests (split overlaps)
> 
> we have the following ranges for facility
> 0-1
> 2
> 3-4
> 5-23
> 
> and for severity
> 0-6
> 7
> 
> to shorten the lines in the e-mail I will letter the remaining
> tests/actions
> 
> A) log to /Var/log/mail.log
> B) programname startswith abc log to /abc
> C) programname startswith 'acc' and MSG contains 'error' log to /acc
> D) programname startswith 'acc' log to @1.1.1.1
> E) programname startswith bcd log to /bcd
> F) programname contains 'bde' log to @1.1.1.1
> G) MSG startswith '123' log to/bcd
> H) MSG contains 'error2' log to /bcd
> I) log to /var/log/catchall
> 
> so the tree then becomes (straight translation no reordering)
> 
> 2,0-6 A
> 2,7 A
> 0-1,0-6 B
> 0-1,7 B
> 2,0-6 B
> 2,7 B
> 3-4,0-6 B
> 3-4,7 B
> 5-23,0-6 B
> 5-23,7 B
> 0-1,0-6 C
> 0-1,7 C
> 2,0-6 C
> 2,7 C
> 3-4,0-6 C
> 3-4,7 C
> 5-23,0-6 C
> 5-23,7 C
> 0-1,0-6 D
> 2,0-6 D
> 3-4,0-6 D
> 5-23,0-6 D
> 2,0-6 E
> 3-4,0-6 E
> 0-1,0-6 F
> 2,0-6 F
> 3-4,0-6 F
> 5-23,0-6 F
> 0-1,0-6 G
> 2,0-6 G
> 3-4,0-6 G
> 5-23,0-6 G
> 0-1,0-6 H
> 2,0-6 H
> 3-4,0-6 H
> 5-23,0-6 H
> 0-1,0-6 I
> 2,0-6 I
> 3-4,0-6 I
> 5-23,0-6 I
> 
> now to order them
> 
> 0-1,0-6 B
> 0-1,0-6 C
> 0-1,0-6 D
> 0-1,0-6 F
> 0-1,0-6 G
> 0-1,0-6 H
> 0-1,0-6 I
> 0-1,7 B
> 0-1,7 C
> 2,0-6 A
> 2,0-6 B
> 2,0-6 I
> 2,0-6 H
> 2,0-6 G
> 2,0-6 F
> 2,0-6 E
> 2,0-6 D
> 2,0-6 C
> 2,7 A
> 2,7 B
> 2,7 C
> 3-4,0-6 B
> 3-4,0-6 C
> 3-4,0-6 D
> 3-4,0-6 E
> 3-4,0-6 F
> 3-4,0-6 G
> 3-4,0-6 H
> 3-4,0-6 I
> 3-4,7 B
> 3-4,7 C
> 5-23,0-6 B
> 5-23,0-6 C
> 5-23,0-6 D
> 5-23,0-6 F
> 5-23,0-6 G
> 5-23,0-6 H
> 5-23,0-6 I
> 5-23,7 B
> 5-23,7 C
> 
> I then simplify this to
> 
> 0-1,0-6 B,C,D,F,G,H,I
> 0-1,7 B,C
> 2,0-6 A,B,C,D,E,F,G,H,I
> 2,7 A,B,C
> 3-4,0-6 B,C,D,E,F,G,H,I
> 3-4,7 B,C
> 5-23,0-6 B,C,D,F,G,H,I
> 5-23,7 B,C
> 
> looking at the lettered tests
> 
> A) log to /Var/log/mail.log
> B) programname startswith abc log to /abc
> C) programname startswith 'acc' and MSG contains 'error' log to /acc
> D) programname startswith 'acc' log to @1.1.1.1
> E) programname startswith bcd log to /bcd
> F) programname contains 'bde' log to @1.1.1.1
> G) MSG startswith '123' log to/bcd
> H) MSG contains 'error2' log to /bcd
> I) log to /var/log/catchall
> 
> the only ones that can be combined are B-E
> 
> so if you make tests
> 
> J) B,C
> K) B,C,D
> L) B,C,D,E
> 
> the results simplify to
> 
> 0-1,0-6 K,F,G,H,I
> 0-1,7 J
> 2,0-6 A,L,F,G,H,I
> 2,7 A,J
> 3-4,0-6 L,F,G,H,I
> 3-4,7 J
> 5-23,0-6 K,F,G,H,I
> 5-23,7 J
> 
> expanded out to show all tests
> 
> 0-1,0-6
>    --programname startswith abc log to /abc
>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>     \programname startswith 'acc' log to @1.1.1.1
>    --programname contains 'bde' log to @1.1.1.1
>    --MSG startswith '123' log to/bcd
>    --MSG contains 'error2' log to /bcd
>    --log to /var/log/catchall
> 0-1,7
>    --programname startswith abc log to /abc
>     \programname startswith 'acc' and MSG contains 'error' log to /acc
> 2,0-6
>    --log to /var/log/mail.log
>    --programname startswith abc log to /abc
>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>    | programname startswith 'acc' log to @1.1.1.1
>     \programname startswith bcd log to /bcd
>    --programname contains 'bde' log to @1.1.1.1
>    --MSG startswith '123' log to/bcd
>    --MSG contains 'error2' log to /bcd
>    --log to /var/log/catchall
> 2,7
>    --log to /var/log/mail.log
>    --programname startswith abc log to /abc
>     \programname startswith 'acc' and MSG contains 'error' log to /acc
> 3-4,0-6
>    --programname startswith abc log to /abc
>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>    | programname startswith 'acc' log to @1.1.1.1
>     \programname startswith bcd log to /bcd
>    --programname contains 'bde' log to @1.1.1.1
>    --MSG startswith '123' log to/bcd
>    --MSG contains 'error2' log to /bcd
>    --log to /var/log/catchall
> 3-4,7
>    --programname startswith abc log to /abc
>     \programname startswith 'acc' and MSG contains 'error' log to /acc
> 5-23,0-6
>    --programname startswith abc log to /abc
>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>     \programname startswith 'acc' log to @1.1.1.1
>    --programname contains 'bde' log to @1.1.1.1
>    --MSG startswith '123' log to/bcd
>    --MSG contains 'error2' log to /bcd
>    --log to /var/log/catchall
> 5-23,7
>    --programname startswith abc log to /abc
>     \programname startswith 'acc' and MSG contains 'error' log to /acc
> 
> > Rainer
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >> Sent: Sunday, February 28, 2010 2:02 PM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] Feedback requested: inconsistency in config
> >> statements
> >>
> >> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
> >>
> >>> Quickly from the mobile: pri based filters require *excatly* one
> >> single word (!) lookup currently. You can't beat that.
> >>>
> >>> I am not totally sure how you think about parse trees. In my view,
> >> parsin and filteing are two different things./stages. Maybe we have
> >> different concepts on our minds?
> >>
> >> I am probably using the wrong terminology here.
> >>
> >> say you have a set of rules that say
> >>
> >> if programname startswith abc log to /abc
> >> if programname startswith acc log to /acc
> >> if programname startswith acc log to @1.1.1.1
> >> if programname startswith bcd log to /bcd
> >>
> >> and assuming that these were the only possible programnames for
> >> simplicity
> >> in the explination.
> >>
> >> the filtering logic would go somthing like this
> >>
> >> the rules would compile into a tree
> >>
> >> -a-b -> /abc
> >>    -c -> /acc, at 1.1.1.1
> >> -b   -> /bcd
> >>
> >> receive programname abc
> >> I have no facility/Pri filter rules
> >> I have no time filter rules
> >> I have no system filter rules
> >> I have programname filter rules
> >>    look at the first character of the programname
> >>     it's "a", I have more than one rule that fits that.
> >>      look at the second character of the programname
> >>       it's a "b", There are no other decisions to make,
> >>         invoke the action /abc
> >>
> >> receive progranmane bcd
> >> I have no facility/PRI filter rules
> >> I have no time filter rules
> >> I have no system filter rules
> >> I have programname filter rules
> >>    look at the first character of the programname
> >>     it's a "b", There are no other decisions to make,
> >>       invoke the action /bcd
> >>
> >> receive programname acc
> >> I have no facility/Pri filter rules
> >> I have no time filter rules
> >> I have no system filter rules
> >> I have programname filter rules
> >>    look at the first character of the programname
> >>     it's "a", I have more than one rule that fits that.
> >>      look at the second character of the programname
> >>       it's a "c", There are no other decisions to make,
> >>         invoke the action /acc and the action @1.1.1.1
> >>
> >> similarly the facility/pri filtering rules would be either compiled
> >> into a
> >> tree, or (since it's 256 entries total) into a lookup table with
> >> pointers
> >> to the list of actions to take for that entry
> >>
> >> the idea is to spend the time at startup to create the tree that
> >> represents the ruleset in order to speed up the processing of each
> >> individual message.
> >>
> >> the real tree would be a bit more complicated than I describe here
> as
> >> it
> >> needs to have 'anything else' entries between the known entries
> (which
> >> means that it would not be able to shortcut quite as much as I
> >> describe),
> >> and have provision for 'do a more complicated thing (like regex)
> here
> >> and
> >> if it matches continue'. but except for regex matches, this would
> make
> >> processing the rules pretty much independant of how many rules there
> >> were
> >> or how complex the ruleset is.
> >>
> >> there doe snot need to be one single programname filter tree, if you
> >> had a
> >> rule that said
> >> if pri = info and programname startswith def then def
> >>
> >> the pri tree/table would have an entry for info that would point to
> a
> >> filter tree for programname that just had the check for def in it
> >>
> >> am I makeing any more sense now?
> >>
> >> David Lang
> >>
> >>> rainer
> >>>
> >>> ----- Urspr?ngliche Nachricht -----
> >>> Von: "david at lang.hm" <david at lang.hm>
> >>> An: "rsyslog-users" <rsyslog at lists.adiscon.com>
> >>> Gesendet: 28.02.10 11:28
> >>> Betreff: Re: [rsyslog] Feedback requested: inconsistency in config
> >> statements
> >>>
> >>> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
> >>>
> >>>>>
> >>>>> if the type is depriciated then the destination would be another
> >>>>> config_option (which you would then look up)
> >>>>>
> >>>>> if the type is 'ignore' then it would just be skipped over
> >> (possibly
> >>>>> with
> >>>>> a warning being logged that the config line is being ignored)
> >>>>>
> >>>>> this would also clean up some of the current cases where a
> boolean
> >>>>> option
> >>>>> doesn't honor on/off and true/false.
> >>>>
> >>>> True/false is NOT a valid boolean option. See
> >>>>
> >>>>
> >>
> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=18
> >> 4c0d87
> >>>> 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308
> >>>
> >>> I was not remembering correctly then, maby it was yes/no vs on/off.
> I
> >> know
> >>> I ran into something where what was documented didn't work and I
> >> changed
> >>> it to another version of a boolean answer and it fixed the problem.
> >>>
> >>>>> For the second half of the config language (telling rsyslog what
> to
> >> do
> >>>>> with the logs it has received) also has several variations in
> place
> >> and
> >>>>> is
> >>>>> showing issues.
> >>>>>
> >>>>> However I think that the right answer here is to end up
> >> implementing
> >>>>> something like the parsing trees and then compile the other
> config
> >>>>> language options into that tree to be consistant (and fast) under
> >> the
> >>>>> covers, no matter which way the instructions are written (except
> >> when
> >>>>> you
> >>>>> have to use regex matches)
> >>>>
> >>>> I don't fully agree here with you. For example, the traditional
> PRI
> >> based
> >>>> filter is lightning fast. I don't see any way it could be nearly
> as
> >> fast with
> >>>> any unified approach. Right now, we have a "unified" filter
> >> structure, but it
> >>>> contains three filter cases, each one adding potential power at
> the
> >> price of
> >>>> decreased speed. I think we need to keep different types of
> filters
> >> in order
> >>>> to have some lightning-fast ones. But more of this could be done
> >> under the
> >>>> hood.
> >>>
> >>> My guess/expectation is that the tree-based parsing would be about
> >> the
> >>> same speed as the traditional PRI based filter for choices made
> based
> >> on
> >>> PRI, slowing down for other types of conditions only in that more
> of
> >> the
> >>> message may need to be scanned (and if there is no selection at a
> >> level,
> >>> skipping that level should be lightning fast). As a result, a set
> of
> >>> filteres based soley on programname would be as fast as the current
> >> PRI
> >>> filters.
> >>>
> >>> David Lang
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From david at lang.hm  Mon Mar  1 20:39:09 2010
From: david at lang.hm (david at lang.hm)
Date: Mon, 1 Mar 2010 11:39:09 -0800 (PST)
Subject: [rsyslog] Feedback requested: inconsistency in config statements
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103992@GRFEXC.intern.adiscon.com>
References: <001201cab868$d148da7c$100013ac@intern.adiscon.com><alpine.DEB.2.00.1002280426420.15263@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA710397F@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.00.1002281346000.15263@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7103992@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.2.00.1003011130460.5131@asgard.lang.hm>

On Mon, 1 Mar 2010, Rainer Gerhards wrote:

> Hi David,
>
> thanks for the explanation, I now see where you are heading too. I have not
> dug into all details, but find the post very interesting and insightful. I'll
> archive it so that I have it at hand when I can take the route to actually
> implement such an approach. All in all, it still looks rather complex to me.
> I guess it would be useful to have a guy with experience in writing compiler
> code optimizers at hand when I tackle that beast ;)

you may also want to dig into yacc/bison as they are designed to help 
people build parser trees.

I am assuming that this is the type of thing that syslog-ng is doing to be 
able to efficiantly handle a lot of log processing.

I currently have a script that looks for a couple hundred programnames and 
puts the data into ~50 different files depending on what that name is. 
Given that the output side of rsyslog is currently the bottleneck, I would 
not want to try and implement that in rsyslog today, but if rsyslog gained 
the ability to handle this sort of ruleset efficiantly it would greatly 
simplify my life.

David Lang

> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Monday, March 01, 2010 3:16 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] Feedback requested: inconsistency in config
>> statements
>>
>> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
>>
>>> Hi David,
>>>
>>> yes, that makes much more sense to me. Let's call that filter
>> expression tree
>>> for the time being (the parse tree used for normalization has some
>> similar
>>> ideas, but a totally different scope and semantics).
>>>
>>> While I now see where you are heading, I fail to see how this could
>>> sufficiently easy be implemented for a real (and complex)
>> configuration.
>>>
>>> Let's look at this example conf:
>>>
>>> mail.* /Var/log/mail.log
>>> if programname startswith abc log to /abc
>>> if programname startswith 'acc' and MSG contains 'error' log to /acc
>>> # we don't want debug messages from here on
>>> *.debug ~
>>> if programname startswith 'acc' or (programname contains 'bde' and
>> (facility
>>>> 1 and
>>> facility < 5)) log to @1.1.1.1
>>> if programname startswith bcd or MSG startswith '123' or MSG contains
>>> 'error2' log to /bcd
>>> *.* /var/log/catchall
>>>
>>> I simply cannot envision how you would store this as a tree that does
>> not
>>> look like what we have today.
>>
>> Ok, Here is what I came up with.
>>
>> 0-1,0-6
>>    --programname startswith abc log to /abc
>>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>>     \programname startswith 'acc' log to @1.1.1.1
>>    --programname contains 'bde' log to @1.1.1.1
>>    --MSG startswith '123' log to/bcd
>>    --MSG contains 'error2' log to /bcd
>>    --log to /var/log/catchall
>> 0-1,7
>>    --programname startswith abc log to /abc
>>     \programname startswith 'acc' and MSG contains 'error' log to /acc
>> 2,0-6
>>    --log to /var/log/mail.log
>>    --programname startswith abc log to /abc
>>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>>    | programname startswith 'acc' log to @1.1.1.1
>>     \programname startswith bcd log to /bcd
>>    --programname contains 'bde' log to @1.1.1.1
>>    --MSG startswith '123' log to/bcd
>>    --MSG contains 'error2' log to /bcd
>>    --log to /var/log/catchall
>> 2,7
>>    --log to /var/log/mail.log
>>    --programname startswith abc log to /abc
>>     \programname startswith 'acc' and MSG contains 'error' log to /acc
>> 3-4,0-6
>>    --programname startswith abc log to /abc
>>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>>    | programname startswith 'acc' log to @1.1.1.1
>>     \programname startswith bcd log to /bcd
>>    --programname contains 'bde' log to @1.1.1.1
>>    --MSG startswith '123' log to/bcd
>>    --MSG contains 'error2' log to /bcd
>>    --log to /var/log/catchall
>> 3-4,7
>>    --programname startswith abc log to /abc
>>     \programname startswith 'acc' and MSG contains 'error' log to /acc
>> 5-23,0-6
>>    --programname startswith abc log to /abc
>>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>>     \programname startswith 'acc' log to @1.1.1.1
>>    --programname contains 'bde' log to @1.1.1.1
>>    --MSG startswith '123' log to/bcd
>>    --MSG contains 'error2' log to /bcd
>>    --log to /var/log/catchall
>> 5-23,7
>>    --programname startswith abc log to /abc
>>     \programname startswith 'acc' and MSG contains 'error' log to /acc
>>
>> This looks very ugly and is definantly more complicated, but I think it
>> still ends up being a slight (but not drastic) win.
>>
>> for the initial ruleset you have to check a minimum of 4 tests and a
>> maximum of 13 tests (debug messages have 4-5 tests, all others have 12-
>> 13)
>>
>> Assuming the facility/priority lookup can be a table lookup, for the
>> tree-based ruleset you have a minimum of 2 tests and a maximum of 7
>> tests,
>> in each case one of the tests is a multi-option comparison which is
>> slightly more expensive than a single 'startswith' test, but should be
>> cheaper than two 'startswith' tests.
>>
>> This definantly trades off start time complexity for runtime
>> efficiancy.
>>
>> David Lang
>>
>>
>> and here is the long, ugly process I went through to derive this.
>>
>>
>>
>> first normalize the tests (replace 'or' with two lines, change facility
>> and priority to ranges of numbers)
>>
>> facility = 2 log to /Var/log/mail.log
>> programname startswith abc log to /abc
>> programname startswith 'acc' and MSG contains 'error' log to /acc
>> severity 0..6 and programname startswith 'acc' log to @1.1.1.1
>> severity 0..6 and programname startswith bcd log to /bcd
>> facility 2..4 and severity 0..6 and programname contains 'bde' log to
>> @1.1.1.1
>> severity 0..6 and MSG startswith '123' log to/bcd
>> severity 0..6 and MSG contains 'error2' log to /bcd
>> severity 0..6 and log to /var/log/catchall
>>
>> I then add the full facility/priority ranges
>>
>>
>> now normalize the facility/severity tests (split overlaps)
>>
>> we have the following ranges for facility
>> 0-1
>> 2
>> 3-4
>> 5-23
>>
>> and for severity
>> 0-6
>> 7
>>
>> to shorten the lines in the e-mail I will letter the remaining
>> tests/actions
>>
>> A) log to /Var/log/mail.log
>> B) programname startswith abc log to /abc
>> C) programname startswith 'acc' and MSG contains 'error' log to /acc
>> D) programname startswith 'acc' log to @1.1.1.1
>> E) programname startswith bcd log to /bcd
>> F) programname contains 'bde' log to @1.1.1.1
>> G) MSG startswith '123' log to/bcd
>> H) MSG contains 'error2' log to /bcd
>> I) log to /var/log/catchall
>>
>> so the tree then becomes (straight translation no reordering)
>>
>> 2,0-6 A
>> 2,7 A
>> 0-1,0-6 B
>> 0-1,7 B
>> 2,0-6 B
>> 2,7 B
>> 3-4,0-6 B
>> 3-4,7 B
>> 5-23,0-6 B
>> 5-23,7 B
>> 0-1,0-6 C
>> 0-1,7 C
>> 2,0-6 C
>> 2,7 C
>> 3-4,0-6 C
>> 3-4,7 C
>> 5-23,0-6 C
>> 5-23,7 C
>> 0-1,0-6 D
>> 2,0-6 D
>> 3-4,0-6 D
>> 5-23,0-6 D
>> 2,0-6 E
>> 3-4,0-6 E
>> 0-1,0-6 F
>> 2,0-6 F
>> 3-4,0-6 F
>> 5-23,0-6 F
>> 0-1,0-6 G
>> 2,0-6 G
>> 3-4,0-6 G
>> 5-23,0-6 G
>> 0-1,0-6 H
>> 2,0-6 H
>> 3-4,0-6 H
>> 5-23,0-6 H
>> 0-1,0-6 I
>> 2,0-6 I
>> 3-4,0-6 I
>> 5-23,0-6 I
>>
>> now to order them
>>
>> 0-1,0-6 B
>> 0-1,0-6 C
>> 0-1,0-6 D
>> 0-1,0-6 F
>> 0-1,0-6 G
>> 0-1,0-6 H
>> 0-1,0-6 I
>> 0-1,7 B
>> 0-1,7 C
>> 2,0-6 A
>> 2,0-6 B
>> 2,0-6 I
>> 2,0-6 H
>> 2,0-6 G
>> 2,0-6 F
>> 2,0-6 E
>> 2,0-6 D
>> 2,0-6 C
>> 2,7 A
>> 2,7 B
>> 2,7 C
>> 3-4,0-6 B
>> 3-4,0-6 C
>> 3-4,0-6 D
>> 3-4,0-6 E
>> 3-4,0-6 F
>> 3-4,0-6 G
>> 3-4,0-6 H
>> 3-4,0-6 I
>> 3-4,7 B
>> 3-4,7 C
>> 5-23,0-6 B
>> 5-23,0-6 C
>> 5-23,0-6 D
>> 5-23,0-6 F
>> 5-23,0-6 G
>> 5-23,0-6 H
>> 5-23,0-6 I
>> 5-23,7 B
>> 5-23,7 C
>>
>> I then simplify this to
>>
>> 0-1,0-6 B,C,D,F,G,H,I
>> 0-1,7 B,C
>> 2,0-6 A,B,C,D,E,F,G,H,I
>> 2,7 A,B,C
>> 3-4,0-6 B,C,D,E,F,G,H,I
>> 3-4,7 B,C
>> 5-23,0-6 B,C,D,F,G,H,I
>> 5-23,7 B,C
>>
>> looking at the lettered tests
>>
>> A) log to /Var/log/mail.log
>> B) programname startswith abc log to /abc
>> C) programname startswith 'acc' and MSG contains 'error' log to /acc
>> D) programname startswith 'acc' log to @1.1.1.1
>> E) programname startswith bcd log to /bcd
>> F) programname contains 'bde' log to @1.1.1.1
>> G) MSG startswith '123' log to/bcd
>> H) MSG contains 'error2' log to /bcd
>> I) log to /var/log/catchall
>>
>> the only ones that can be combined are B-E
>>
>> so if you make tests
>>
>> J) B,C
>> K) B,C,D
>> L) B,C,D,E
>>
>> the results simplify to
>>
>> 0-1,0-6 K,F,G,H,I
>> 0-1,7 J
>> 2,0-6 A,L,F,G,H,I
>> 2,7 A,J
>> 3-4,0-6 L,F,G,H,I
>> 3-4,7 J
>> 5-23,0-6 K,F,G,H,I
>> 5-23,7 J
>>
>> expanded out to show all tests
>>
>> 0-1,0-6
>>    --programname startswith abc log to /abc
>>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>>     \programname startswith 'acc' log to @1.1.1.1
>>    --programname contains 'bde' log to @1.1.1.1
>>    --MSG startswith '123' log to/bcd
>>    --MSG contains 'error2' log to /bcd
>>    --log to /var/log/catchall
>> 0-1,7
>>    --programname startswith abc log to /abc
>>     \programname startswith 'acc' and MSG contains 'error' log to /acc
>> 2,0-6
>>    --log to /var/log/mail.log
>>    --programname startswith abc log to /abc
>>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>>    | programname startswith 'acc' log to @1.1.1.1
>>     \programname startswith bcd log to /bcd
>>    --programname contains 'bde' log to @1.1.1.1
>>    --MSG startswith '123' log to/bcd
>>    --MSG contains 'error2' log to /bcd
>>    --log to /var/log/catchall
>> 2,7
>>    --log to /var/log/mail.log
>>    --programname startswith abc log to /abc
>>     \programname startswith 'acc' and MSG contains 'error' log to /acc
>> 3-4,0-6
>>    --programname startswith abc log to /abc
>>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>>    | programname startswith 'acc' log to @1.1.1.1
>>     \programname startswith bcd log to /bcd
>>    --programname contains 'bde' log to @1.1.1.1
>>    --MSG startswith '123' log to/bcd
>>    --MSG contains 'error2' log to /bcd
>>    --log to /var/log/catchall
>> 3-4,7
>>    --programname startswith abc log to /abc
>>     \programname startswith 'acc' and MSG contains 'error' log to /acc
>> 5-23,0-6
>>    --programname startswith abc log to /abc
>>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>>     \programname startswith 'acc' log to @1.1.1.1
>>    --programname contains 'bde' log to @1.1.1.1
>>    --MSG startswith '123' log to/bcd
>>    --MSG contains 'error2' log to /bcd
>>    --log to /var/log/catchall
>> 5-23,7
>>    --programname startswith abc log to /abc
>>     \programname startswith 'acc' and MSG contains 'error' log to /acc
>>
>>> Rainer
>>>> -----Original Message-----
>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>>> Sent: Sunday, February 28, 2010 2:02 PM
>>>> To: rsyslog-users
>>>> Subject: Re: [rsyslog] Feedback requested: inconsistency in config
>>>> statements
>>>>
>>>> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
>>>>
>>>>> Quickly from the mobile: pri based filters require *excatly* one
>>>> single word (!) lookup currently. You can't beat that.
>>>>>
>>>>> I am not totally sure how you think about parse trees. In my view,
>>>> parsin and filteing are two different things./stages. Maybe we have
>>>> different concepts on our minds?
>>>>
>>>> I am probably using the wrong terminology here.
>>>>
>>>> say you have a set of rules that say
>>>>
>>>> if programname startswith abc log to /abc
>>>> if programname startswith acc log to /acc
>>>> if programname startswith acc log to @1.1.1.1
>>>> if programname startswith bcd log to /bcd
>>>>
>>>> and assuming that these were the only possible programnames for
>>>> simplicity
>>>> in the explination.
>>>>
>>>> the filtering logic would go somthing like this
>>>>
>>>> the rules would compile into a tree
>>>>
>>>> -a-b -> /abc
>>>>    -c -> /acc, at 1.1.1.1
>>>> -b   -> /bcd
>>>>
>>>> receive programname abc
>>>> I have no facility/Pri filter rules
>>>> I have no time filter rules
>>>> I have no system filter rules
>>>> I have programname filter rules
>>>>    look at the first character of the programname
>>>>     it's "a", I have more than one rule that fits that.
>>>>      look at the second character of the programname
>>>>       it's a "b", There are no other decisions to make,
>>>>         invoke the action /abc
>>>>
>>>> receive progranmane bcd
>>>> I have no facility/PRI filter rules
>>>> I have no time filter rules
>>>> I have no system filter rules
>>>> I have programname filter rules
>>>>    look at the first character of the programname
>>>>     it's a "b", There are no other decisions to make,
>>>>       invoke the action /bcd
>>>>
>>>> receive programname acc
>>>> I have no facility/Pri filter rules
>>>> I have no time filter rules
>>>> I have no system filter rules
>>>> I have programname filter rules
>>>>    look at the first character of the programname
>>>>     it's "a", I have more than one rule that fits that.
>>>>      look at the second character of the programname
>>>>       it's a "c", There are no other decisions to make,
>>>>         invoke the action /acc and the action @1.1.1.1
>>>>
>>>> similarly the facility/pri filtering rules would be either compiled
>>>> into a
>>>> tree, or (since it's 256 entries total) into a lookup table with
>>>> pointers
>>>> to the list of actions to take for that entry
>>>>
>>>> the idea is to spend the time at startup to create the tree that
>>>> represents the ruleset in order to speed up the processing of each
>>>> individual message.
>>>>
>>>> the real tree would be a bit more complicated than I describe here
>> as
>>>> it
>>>> needs to have 'anything else' entries between the known entries
>> (which
>>>> means that it would not be able to shortcut quite as much as I
>>>> describe),
>>>> and have provision for 'do a more complicated thing (like regex)
>> here
>>>> and
>>>> if it matches continue'. but except for regex matches, this would
>> make
>>>> processing the rules pretty much independant of how many rules there
>>>> were
>>>> or how complex the ruleset is.
>>>>
>>>> there doe snot need to be one single programname filter tree, if you
>>>> had a
>>>> rule that said
>>>> if pri = info and programname startswith def then def
>>>>
>>>> the pri tree/table would have an entry for info that would point to
>> a
>>>> filter tree for programname that just had the check for def in it
>>>>
>>>> am I makeing any more sense now?
>>>>
>>>> David Lang
>>>>
>>>>> rainer
>>>>>
>>>>> ----- Urspr?ngliche Nachricht -----
>>>>> Von: "david at lang.hm" <david at lang.hm>
>>>>> An: "rsyslog-users" <rsyslog at lists.adiscon.com>
>>>>> Gesendet: 28.02.10 11:28
>>>>> Betreff: Re: [rsyslog] Feedback requested: inconsistency in config
>>>> statements
>>>>>
>>>>> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
>>>>>
>>>>>>>
>>>>>>> if the type is depriciated then the destination would be another
>>>>>>> config_option (which you would then look up)
>>>>>>>
>>>>>>> if the type is 'ignore' then it would just be skipped over
>>>> (possibly
>>>>>>> with
>>>>>>> a warning being logged that the config line is being ignored)
>>>>>>>
>>>>>>> this would also clean up some of the current cases where a
>> boolean
>>>>>>> option
>>>>>>> doesn't honor on/off and true/false.
>>>>>>
>>>>>> True/false is NOT a valid boolean option. See
>>>>>>
>>>>>>
>>>>
>> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=18
>>>> 4c0d87
>>>>>> 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308
>>>>>
>>>>> I was not remembering correctly then, maby it was yes/no vs on/off.
>> I
>>>> know
>>>>> I ran into something where what was documented didn't work and I
>>>> changed
>>>>> it to another version of a boolean answer and it fixed the problem.
>>>>>
>>>>>>> For the second half of the config language (telling rsyslog what
>> to
>>>> do
>>>>>>> with the logs it has received) also has several variations in
>> place
>>>> and
>>>>>>> is
>>>>>>> showing issues.
>>>>>>>
>>>>>>> However I think that the right answer here is to end up
>>>> implementing
>>>>>>> something like the parsing trees and then compile the other
>> config
>>>>>>> language options into that tree to be consistant (and fast) under
>>>> the
>>>>>>> covers, no matter which way the instructions are written (except
>>>> when
>>>>>>> you
>>>>>>> have to use regex matches)
>>>>>>
>>>>>> I don't fully agree here with you. For example, the traditional
>> PRI
>>>> based
>>>>>> filter is lightning fast. I don't see any way it could be nearly
>> as
>>>> fast with
>>>>>> any unified approach. Right now, we have a "unified" filter
>>>> structure, but it
>>>>>> contains three filter cases, each one adding potential power at
>> the
>>>> price of
>>>>>> decreased speed. I think we need to keep different types of
>> filters
>>>> in order
>>>>>> to have some lightning-fast ones. But more of this could be done
>>>> under the
>>>>>> hood.
>>>>>
>>>>> My guess/expectation is that the tree-based parsing would be about
>>>> the
>>>>> same speed as the traditional PRI based filter for choices made
>> based
>>>> on
>>>>> PRI, slowing down for other types of conditions only in that more
>> of
>>>> the
>>>>> message may need to be scanned (and if there is no selection at a
>>>> level,
>>>>> skipping that level should be lightning fast). As a result, a set
>> of
>>>>> filteres based soley on programname would be as fast as the current
>>>> PRI
>>>>> filters.
>>>>>
>>>>> David Lang
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Mon Mar  1 20:58:51 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 1 Mar 2010 20:58:51 +0100
Subject: [rsyslog] Feedback requested: inconsistency in config statements
References: <001201cab868$d148da7c$100013ac@intern.adiscon.com><alpine.DEB.2.00.1002280426420.15263@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA710397F@GRFEXC.intern.adiscon.com><alpine.DEB.2.00.1002281346000.15263@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA7103992@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.00.1003011130460.5131@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103998@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Monday, March 01, 2010 8:39 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Feedback requested: inconsistency in config
> statements
> 
> On Mon, 1 Mar 2010, Rainer Gerhards wrote:
> 
> > Hi David,
> >
> > thanks for the explanation, I now see where you are heading too. I
> have not
> > dug into all details, but find the post very interesting and
> insightful. I'll
> > archive it so that I have it at hand when I can take the route to
> actually
> > implement such an approach. All in all, it still looks rather complex
> to me.
> > I guess it would be useful to have a guy with experience in writing
> compiler
> > code optimizers at hand when I tackle that beast ;)
> 
> you may also want to dig into yacc/bison as they are designed to help
> people build parser trees.

I know yacc but not bison. Yacc creates decent single-threading parsers out
of a fixed set of rules. I never thought about using them for message
parsing, because of the ample limitations it has (most notably unable to run
on multiple threads!).

But we need to keep in mind that now are talking about the filter
expressions. That is a different beast. The thing to look at probably would
be the gcc code optimizer, but I have to admit that is too large an effort
for me right now. A good (dragon?;)) book on compiler optimization technics
for expression trees would probably the right path to take.

> I am assuming that this is the type of thing that syslog-ng is doing to
> be
> able to efficiantly handle a lot of log processing.

I hardly believe so. I don't see how this could work out well...

> I currently have a script that looks for a couple hundred programnames
> and
> puts the data into ~50 different files depending on what that name is.
> Given that the output side of rsyslog is currently the bottleneck, I
> would
> not want to try and implement that in rsyslog today, but if rsyslog
> gained
> the ability to handle this sort of ruleset efficiantly it would greatly
> simplify my life.

Rather than trying to optimize the whole expression engine, this could
probably done very easy with a normalizing parser. As part of the
normalization, it could also create a meta-property "filename" that than is
used inside the rule engine. I'd assume that the effort to create that is by
far less than optimizing the expression tree. *This* is part of my approach
of log normalizing.

Rianer
> 
> David Lang
> 
> > Rainer
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >> Sent: Monday, March 01, 2010 3:16 AM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] Feedback requested: inconsistency in config
> >> statements
> >>
> >> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
> >>
> >>> Hi David,
> >>>
> >>> yes, that makes much more sense to me. Let's call that filter
> >> expression tree
> >>> for the time being (the parse tree used for normalization has some
> >> similar
> >>> ideas, but a totally different scope and semantics).
> >>>
> >>> While I now see where you are heading, I fail to see how this could
> >>> sufficiently easy be implemented for a real (and complex)
> >> configuration.
> >>>
> >>> Let's look at this example conf:
> >>>
> >>> mail.* /Var/log/mail.log
> >>> if programname startswith abc log to /abc
> >>> if programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>> # we don't want debug messages from here on
> >>> *.debug ~
> >>> if programname startswith 'acc' or (programname contains 'bde' and
> >> (facility
> >>>> 1 and
> >>> facility < 5)) log to @1.1.1.1
> >>> if programname startswith bcd or MSG startswith '123' or MSG
> contains
> >>> 'error2' log to /bcd
> >>> *.* /var/log/catchall
> >>>
> >>> I simply cannot envision how you would store this as a tree that
> does
> >> not
> >>> look like what we have today.
> >>
> >> Ok, Here is what I came up with.
> >>
> >> 0-1,0-6
> >>    --programname startswith abc log to /abc
> >>    | programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>     \programname startswith 'acc' log to @1.1.1.1
> >>    --programname contains 'bde' log to @1.1.1.1
> >>    --MSG startswith '123' log to/bcd
> >>    --MSG contains 'error2' log to /bcd
> >>    --log to /var/log/catchall
> >> 0-1,7
> >>    --programname startswith abc log to /abc
> >>     \programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >> 2,0-6
> >>    --log to /var/log/mail.log
> >>    --programname startswith abc log to /abc
> >>    | programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>    | programname startswith 'acc' log to @1.1.1.1
> >>     \programname startswith bcd log to /bcd
> >>    --programname contains 'bde' log to @1.1.1.1
> >>    --MSG startswith '123' log to/bcd
> >>    --MSG contains 'error2' log to /bcd
> >>    --log to /var/log/catchall
> >> 2,7
> >>    --log to /var/log/mail.log
> >>    --programname startswith abc log to /abc
> >>     \programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >> 3-4,0-6
> >>    --programname startswith abc log to /abc
> >>    | programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>    | programname startswith 'acc' log to @1.1.1.1
> >>     \programname startswith bcd log to /bcd
> >>    --programname contains 'bde' log to @1.1.1.1
> >>    --MSG startswith '123' log to/bcd
> >>    --MSG contains 'error2' log to /bcd
> >>    --log to /var/log/catchall
> >> 3-4,7
> >>    --programname startswith abc log to /abc
> >>     \programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >> 5-23,0-6
> >>    --programname startswith abc log to /abc
> >>    | programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>     \programname startswith 'acc' log to @1.1.1.1
> >>    --programname contains 'bde' log to @1.1.1.1
> >>    --MSG startswith '123' log to/bcd
> >>    --MSG contains 'error2' log to /bcd
> >>    --log to /var/log/catchall
> >> 5-23,7
> >>    --programname startswith abc log to /abc
> >>     \programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>
> >> This looks very ugly and is definantly more complicated, but I think
> it
> >> still ends up being a slight (but not drastic) win.
> >>
> >> for the initial ruleset you have to check a minimum of 4 tests and a
> >> maximum of 13 tests (debug messages have 4-5 tests, all others have
> 12-
> >> 13)
> >>
> >> Assuming the facility/priority lookup can be a table lookup, for the
> >> tree-based ruleset you have a minimum of 2 tests and a maximum of 7
> >> tests,
> >> in each case one of the tests is a multi-option comparison which is
> >> slightly more expensive than a single 'startswith' test, but should
> be
> >> cheaper than two 'startswith' tests.
> >>
> >> This definantly trades off start time complexity for runtime
> >> efficiancy.
> >>
> >> David Lang
> >>
> >>
> >> and here is the long, ugly process I went through to derive this.
> >>
> >>
> >>
> >> first normalize the tests (replace 'or' with two lines, change
> facility
> >> and priority to ranges of numbers)
> >>
> >> facility = 2 log to /Var/log/mail.log
> >> programname startswith abc log to /abc
> >> programname startswith 'acc' and MSG contains 'error' log to /acc
> >> severity 0..6 and programname startswith 'acc' log to @1.1.1.1
> >> severity 0..6 and programname startswith bcd log to /bcd
> >> facility 2..4 and severity 0..6 and programname contains 'bde' log
> to
> >> @1.1.1.1
> >> severity 0..6 and MSG startswith '123' log to/bcd
> >> severity 0..6 and MSG contains 'error2' log to /bcd
> >> severity 0..6 and log to /var/log/catchall
> >>
> >> I then add the full facility/priority ranges
> >>
> >>
> >> now normalize the facility/severity tests (split overlaps)
> >>
> >> we have the following ranges for facility
> >> 0-1
> >> 2
> >> 3-4
> >> 5-23
> >>
> >> and for severity
> >> 0-6
> >> 7
> >>
> >> to shorten the lines in the e-mail I will letter the remaining
> >> tests/actions
> >>
> >> A) log to /Var/log/mail.log
> >> B) programname startswith abc log to /abc
> >> C) programname startswith 'acc' and MSG contains 'error' log to /acc
> >> D) programname startswith 'acc' log to @1.1.1.1
> >> E) programname startswith bcd log to /bcd
> >> F) programname contains 'bde' log to @1.1.1.1
> >> G) MSG startswith '123' log to/bcd
> >> H) MSG contains 'error2' log to /bcd
> >> I) log to /var/log/catchall
> >>
> >> so the tree then becomes (straight translation no reordering)
> >>
> >> 2,0-6 A
> >> 2,7 A
> >> 0-1,0-6 B
> >> 0-1,7 B
> >> 2,0-6 B
> >> 2,7 B
> >> 3-4,0-6 B
> >> 3-4,7 B
> >> 5-23,0-6 B
> >> 5-23,7 B
> >> 0-1,0-6 C
> >> 0-1,7 C
> >> 2,0-6 C
> >> 2,7 C
> >> 3-4,0-6 C
> >> 3-4,7 C
> >> 5-23,0-6 C
> >> 5-23,7 C
> >> 0-1,0-6 D
> >> 2,0-6 D
> >> 3-4,0-6 D
> >> 5-23,0-6 D
> >> 2,0-6 E
> >> 3-4,0-6 E
> >> 0-1,0-6 F
> >> 2,0-6 F
> >> 3-4,0-6 F
> >> 5-23,0-6 F
> >> 0-1,0-6 G
> >> 2,0-6 G
> >> 3-4,0-6 G
> >> 5-23,0-6 G
> >> 0-1,0-6 H
> >> 2,0-6 H
> >> 3-4,0-6 H
> >> 5-23,0-6 H
> >> 0-1,0-6 I
> >> 2,0-6 I
> >> 3-4,0-6 I
> >> 5-23,0-6 I
> >>
> >> now to order them
> >>
> >> 0-1,0-6 B
> >> 0-1,0-6 C
> >> 0-1,0-6 D
> >> 0-1,0-6 F
> >> 0-1,0-6 G
> >> 0-1,0-6 H
> >> 0-1,0-6 I
> >> 0-1,7 B
> >> 0-1,7 C
> >> 2,0-6 A
> >> 2,0-6 B
> >> 2,0-6 I
> >> 2,0-6 H
> >> 2,0-6 G
> >> 2,0-6 F
> >> 2,0-6 E
> >> 2,0-6 D
> >> 2,0-6 C
> >> 2,7 A
> >> 2,7 B
> >> 2,7 C
> >> 3-4,0-6 B
> >> 3-4,0-6 C
> >> 3-4,0-6 D
> >> 3-4,0-6 E
> >> 3-4,0-6 F
> >> 3-4,0-6 G
> >> 3-4,0-6 H
> >> 3-4,0-6 I
> >> 3-4,7 B
> >> 3-4,7 C
> >> 5-23,0-6 B
> >> 5-23,0-6 C
> >> 5-23,0-6 D
> >> 5-23,0-6 F
> >> 5-23,0-6 G
> >> 5-23,0-6 H
> >> 5-23,0-6 I
> >> 5-23,7 B
> >> 5-23,7 C
> >>
> >> I then simplify this to
> >>
> >> 0-1,0-6 B,C,D,F,G,H,I
> >> 0-1,7 B,C
> >> 2,0-6 A,B,C,D,E,F,G,H,I
> >> 2,7 A,B,C
> >> 3-4,0-6 B,C,D,E,F,G,H,I
> >> 3-4,7 B,C
> >> 5-23,0-6 B,C,D,F,G,H,I
> >> 5-23,7 B,C
> >>
> >> looking at the lettered tests
> >>
> >> A) log to /Var/log/mail.log
> >> B) programname startswith abc log to /abc
> >> C) programname startswith 'acc' and MSG contains 'error' log to /acc
> >> D) programname startswith 'acc' log to @1.1.1.1
> >> E) programname startswith bcd log to /bcd
> >> F) programname contains 'bde' log to @1.1.1.1
> >> G) MSG startswith '123' log to/bcd
> >> H) MSG contains 'error2' log to /bcd
> >> I) log to /var/log/catchall
> >>
> >> the only ones that can be combined are B-E
> >>
> >> so if you make tests
> >>
> >> J) B,C
> >> K) B,C,D
> >> L) B,C,D,E
> >>
> >> the results simplify to
> >>
> >> 0-1,0-6 K,F,G,H,I
> >> 0-1,7 J
> >> 2,0-6 A,L,F,G,H,I
> >> 2,7 A,J
> >> 3-4,0-6 L,F,G,H,I
> >> 3-4,7 J
> >> 5-23,0-6 K,F,G,H,I
> >> 5-23,7 J
> >>
> >> expanded out to show all tests
> >>
> >> 0-1,0-6
> >>    --programname startswith abc log to /abc
> >>    | programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>     \programname startswith 'acc' log to @1.1.1.1
> >>    --programname contains 'bde' log to @1.1.1.1
> >>    --MSG startswith '123' log to/bcd
> >>    --MSG contains 'error2' log to /bcd
> >>    --log to /var/log/catchall
> >> 0-1,7
> >>    --programname startswith abc log to /abc
> >>     \programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >> 2,0-6
> >>    --log to /var/log/mail.log
> >>    --programname startswith abc log to /abc
> >>    | programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>    | programname startswith 'acc' log to @1.1.1.1
> >>     \programname startswith bcd log to /bcd
> >>    --programname contains 'bde' log to @1.1.1.1
> >>    --MSG startswith '123' log to/bcd
> >>    --MSG contains 'error2' log to /bcd
> >>    --log to /var/log/catchall
> >> 2,7
> >>    --log to /var/log/mail.log
> >>    --programname startswith abc log to /abc
> >>     \programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >> 3-4,0-6
> >>    --programname startswith abc log to /abc
> >>    | programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>    | programname startswith 'acc' log to @1.1.1.1
> >>     \programname startswith bcd log to /bcd
> >>    --programname contains 'bde' log to @1.1.1.1
> >>    --MSG startswith '123' log to/bcd
> >>    --MSG contains 'error2' log to /bcd
> >>    --log to /var/log/catchall
> >> 3-4,7
> >>    --programname startswith abc log to /abc
> >>     \programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >> 5-23,0-6
> >>    --programname startswith abc log to /abc
> >>    | programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>     \programname startswith 'acc' log to @1.1.1.1
> >>    --programname contains 'bde' log to @1.1.1.1
> >>    --MSG startswith '123' log to/bcd
> >>    --MSG contains 'error2' log to /bcd
> >>    --log to /var/log/catchall
> >> 5-23,7
> >>    --programname startswith abc log to /abc
> >>     \programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>
> >>> Rainer
> >>>> -----Original Message-----
> >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >>>> Sent: Sunday, February 28, 2010 2:02 PM
> >>>> To: rsyslog-users
> >>>> Subject: Re: [rsyslog] Feedback requested: inconsistency in config
> >>>> statements
> >>>>
> >>>> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
> >>>>
> >>>>> Quickly from the mobile: pri based filters require *excatly* one
> >>>> single word (!) lookup currently. You can't beat that.
> >>>>>
> >>>>> I am not totally sure how you think about parse trees. In my
> view,
> >>>> parsin and filteing are two different things./stages. Maybe we
> have
> >>>> different concepts on our minds?
> >>>>
> >>>> I am probably using the wrong terminology here.
> >>>>
> >>>> say you have a set of rules that say
> >>>>
> >>>> if programname startswith abc log to /abc
> >>>> if programname startswith acc log to /acc
> >>>> if programname startswith acc log to @1.1.1.1
> >>>> if programname startswith bcd log to /bcd
> >>>>
> >>>> and assuming that these were the only possible programnames for
> >>>> simplicity
> >>>> in the explination.
> >>>>
> >>>> the filtering logic would go somthing like this
> >>>>
> >>>> the rules would compile into a tree
> >>>>
> >>>> -a-b -> /abc
> >>>>    -c -> /acc, at 1.1.1.1
> >>>> -b   -> /bcd
> >>>>
> >>>> receive programname abc
> >>>> I have no facility/Pri filter rules
> >>>> I have no time filter rules
> >>>> I have no system filter rules
> >>>> I have programname filter rules
> >>>>    look at the first character of the programname
> >>>>     it's "a", I have more than one rule that fits that.
> >>>>      look at the second character of the programname
> >>>>       it's a "b", There are no other decisions to make,
> >>>>         invoke the action /abc
> >>>>
> >>>> receive progranmane bcd
> >>>> I have no facility/PRI filter rules
> >>>> I have no time filter rules
> >>>> I have no system filter rules
> >>>> I have programname filter rules
> >>>>    look at the first character of the programname
> >>>>     it's a "b", There are no other decisions to make,
> >>>>       invoke the action /bcd
> >>>>
> >>>> receive programname acc
> >>>> I have no facility/Pri filter rules
> >>>> I have no time filter rules
> >>>> I have no system filter rules
> >>>> I have programname filter rules
> >>>>    look at the first character of the programname
> >>>>     it's "a", I have more than one rule that fits that.
> >>>>      look at the second character of the programname
> >>>>       it's a "c", There are no other decisions to make,
> >>>>         invoke the action /acc and the action @1.1.1.1
> >>>>
> >>>> similarly the facility/pri filtering rules would be either
> compiled
> >>>> into a
> >>>> tree, or (since it's 256 entries total) into a lookup table with
> >>>> pointers
> >>>> to the list of actions to take for that entry
> >>>>
> >>>> the idea is to spend the time at startup to create the tree that
> >>>> represents the ruleset in order to speed up the processing of each
> >>>> individual message.
> >>>>
> >>>> the real tree would be a bit more complicated than I describe here
> >> as
> >>>> it
> >>>> needs to have 'anything else' entries between the known entries
> >> (which
> >>>> means that it would not be able to shortcut quite as much as I
> >>>> describe),
> >>>> and have provision for 'do a more complicated thing (like regex)
> >> here
> >>>> and
> >>>> if it matches continue'. but except for regex matches, this would
> >> make
> >>>> processing the rules pretty much independant of how many rules
> there
> >>>> were
> >>>> or how complex the ruleset is.
> >>>>
> >>>> there doe snot need to be one single programname filter tree, if
> you
> >>>> had a
> >>>> rule that said
> >>>> if pri = info and programname startswith def then def
> >>>>
> >>>> the pri tree/table would have an entry for info that would point
> to
> >> a
> >>>> filter tree for programname that just had the check for def in it
> >>>>
> >>>> am I makeing any more sense now?
> >>>>
> >>>> David Lang
> >>>>
> >>>>> rainer
> >>>>>
> >>>>> ----- Urspr?ngliche Nachricht -----
> >>>>> Von: "david at lang.hm" <david at lang.hm>
> >>>>> An: "rsyslog-users" <rsyslog at lists.adiscon.com>
> >>>>> Gesendet: 28.02.10 11:28
> >>>>> Betreff: Re: [rsyslog] Feedback requested: inconsistency in
> config
> >>>> statements
> >>>>>
> >>>>> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
> >>>>>
> >>>>>>>
> >>>>>>> if the type is depriciated then the destination would be
> another
> >>>>>>> config_option (which you would then look up)
> >>>>>>>
> >>>>>>> if the type is 'ignore' then it would just be skipped over
> >>>> (possibly
> >>>>>>> with
> >>>>>>> a warning being logged that the config line is being ignored)
> >>>>>>>
> >>>>>>> this would also clean up some of the current cases where a
> >> boolean
> >>>>>>> option
> >>>>>>> doesn't honor on/off and true/false.
> >>>>>>
> >>>>>> True/false is NOT a valid boolean option. See
> >>>>>>
> >>>>>>
> >>>>
> >>
> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=18
> >>>> 4c0d87
> >>>>>> 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308
> >>>>>
> >>>>> I was not remembering correctly then, maby it was yes/no vs
> on/off.
> >> I
> >>>> know
> >>>>> I ran into something where what was documented didn't work and I
> >>>> changed
> >>>>> it to another version of a boolean answer and it fixed the
> problem.
> >>>>>
> >>>>>>> For the second half of the config language (telling rsyslog
> what
> >> to
> >>>> do
> >>>>>>> with the logs it has received) also has several variations in
> >> place
> >>>> and
> >>>>>>> is
> >>>>>>> showing issues.
> >>>>>>>
> >>>>>>> However I think that the right answer here is to end up
> >>>> implementing
> >>>>>>> something like the parsing trees and then compile the other
> >> config
> >>>>>>> language options into that tree to be consistant (and fast)
> under
> >>>> the
> >>>>>>> covers, no matter which way the instructions are written
> (except
> >>>> when
> >>>>>>> you
> >>>>>>> have to use regex matches)
> >>>>>>
> >>>>>> I don't fully agree here with you. For example, the traditional
> >> PRI
> >>>> based
> >>>>>> filter is lightning fast. I don't see any way it could be nearly
> >> as
> >>>> fast with
> >>>>>> any unified approach. Right now, we have a "unified" filter
> >>>> structure, but it
> >>>>>> contains three filter cases, each one adding potential power at
> >> the
> >>>> price of
> >>>>>> decreased speed. I think we need to keep different types of
> >> filters
> >>>> in order
> >>>>>> to have some lightning-fast ones. But more of this could be done
> >>>> under the
> >>>>>> hood.
> >>>>>
> >>>>> My guess/expectation is that the tree-based parsing would be
> about
> >>>> the
> >>>>> same speed as the traditional PRI based filter for choices made
> >> based
> >>>> on
> >>>>> PRI, slowing down for other types of conditions only in that more
> >> of
> >>>> the
> >>>>> message may need to be scanned (and if there is no selection at a
> >>>> level,
> >>>>> skipping that level should be lightning fast). As a result, a set
> >> of
> >>>>> filteres based soley on programname would be as fast as the
> current
> >>>> PRI
> >>>>> filters.
> >>>>>
> >>>>> David Lang
> >>>>> _______________________________________________
> >>>>> rsyslog mailing list
> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>> http://www.rsyslog.com
> >>>>> _______________________________________________
> >>>>> rsyslog mailing list
> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>> http://www.rsyslog.com
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.com
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >>>
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From lanas at securenet.net  Tue Mar  2 02:43:31 2010
From: lanas at securenet.net (lanas)
Date: Mon, 1 Mar 2010 20:43:31 -0500
Subject: [rsyslog] Sending time in remote syslog - test
Message-ID: <20100301204331.1686d17d@mistral.stie>

Hello,

  I did a test using %timereported% in a rsyslog running on a server,
to see if the time of a syslog msg sent from a unit was shown using
the local time of the unit and not the time it was received.

  To easily see the difference I out the time 2 hours back on the
unit.  The unit is running syslogd.  That is, the original syslogd,
not the ng.

  The unit running ran syslogd with the following in its
configuration:

*.*     @10.200.19.162

  The logger utility was used to create log msgs locally on the unit.
These log msgs are then sent to rsyslog on the remote server.

  The workstation was running rsyslog with either:

#$template precise,"%timereported% from: %HOSTNAME% %syslogtag%:%msg%\n"
$template precise,"%TIMESTAMP% from: %HOSTNAME% %syslogtag%:%msg%\n"

  And tests were done with the following commented or not:

$ActionFileDefaultTemplate precise

  (I am not sure if the 'precise' template definition must be used
with $ActionFileDefaultTemplate to take effect)

  The following was then found on the unit after executing the logger
utility:

Feb 27 10:55:34 localhost syslogd 1.4.1#18RR6: restart.
Feb 27 10:55:35 localhost kernel: klogd 1.4.1#18RR6, log source =
                /proc/kmsg started.
Feb 27 10:55:44 localhost root: TEST7

  And these messages were received by rsyslog (after restarting
rsyslog to have its version info included here) :

Feb 27 12:52:32 from: localhost rsyslogd:: [origin software="rsyslogd"
    swVersion="3.22.1" x-pid="6909" x-info="http://www.rsyslog.com"]
    (re)start
Feb 27 12:54:38 from: syslogd 1.4.1#18RR6:: restart.
Feb 27 12:54:39 from: brouter kernel:: klogd 1.4.1#18RR6, log source =
                /proc/kmsg started.
Feb 27 12:54:48 from: brouter root:: TEST7

  The TEST7 msg did now show the unit's time.

  Now, I installed rsyslog at the unit as a drop-in syslog replacement
w/o any configuration change and when rsyslog is used, the local unit
time is received and shown by the server.

  Can we presume that syslogd first generation does not send the time
to a remote syslogging facility ?

Regards,

Al



From rgerhards at hq.adiscon.com  Tue Mar  2 10:03:20 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 2 Mar 2010 10:03:20 +0100
Subject: [rsyslog] Sending time in remote syslog - test
References: <20100301204331.1686d17d@mistral.stie>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71039A1@GRFEXC.intern.adiscon.com>

you can use 

$template raw,"%rawmsg%\n"

to see exactly what the remote box sends. If I recall correctly, sysklogd
sends timestamps, but I may be wrong

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of lanas
> Sent: Tuesday, March 02, 2010 2:44 AM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] Sending time in remote syslog - test
> 
> Hello,
> 
>   I did a test using %timereported% in a rsyslog running on a server,
> to see if the time of a syslog msg sent from a unit was shown using
> the local time of the unit and not the time it was received.
> 
>   To easily see the difference I out the time 2 hours back on the
> unit.  The unit is running syslogd.  That is, the original syslogd,
> not the ng.
> 
>   The unit running ran syslogd with the following in its
> configuration:
> 
> *.*     @10.200.19.162
> 
>   The logger utility was used to create log msgs locally on the unit.
> These log msgs are then sent to rsyslog on the remote server.
> 
>   The workstation was running rsyslog with either:
> 
> #$template precise,"%timereported% from: %HOSTNAME%
> %syslogtag%:%msg%\n"
> $template precise,"%TIMESTAMP% from: %HOSTNAME% %syslogtag%:%msg%\n"
> 
>   And tests were done with the following commented or not:
> 
> $ActionFileDefaultTemplate precise
> 
>   (I am not sure if the 'precise' template definition must be used
> with $ActionFileDefaultTemplate to take effect)
> 
>   The following was then found on the unit after executing the logger
> utility:
> 
> Feb 27 10:55:34 localhost syslogd 1.4.1#18RR6: restart.
> Feb 27 10:55:35 localhost kernel: klogd 1.4.1#18RR6, log source =
>                 /proc/kmsg started.
> Feb 27 10:55:44 localhost root: TEST7
> 
>   And these messages were received by rsyslog (after restarting
> rsyslog to have its version info included here) :
> 
> Feb 27 12:52:32 from: localhost rsyslogd:: [origin software="rsyslogd"
>     swVersion="3.22.1" x-pid="6909" x-info="http://www.rsyslog.com"]
>     (re)start
> Feb 27 12:54:38 from: syslogd 1.4.1#18RR6:: restart.
> Feb 27 12:54:39 from: brouter kernel:: klogd 1.4.1#18RR6, log source =
>                 /proc/kmsg started.
> Feb 27 12:54:48 from: brouter root:: TEST7
> 
>   The TEST7 msg did now show the unit's time.
> 
>   Now, I installed rsyslog at the unit as a drop-in syslog replacement
> w/o any configuration change and when rsyslog is used, the local unit
> time is received and shown by the server.
> 
>   Can we presume that syslogd first generation does not send the time
> to a remote syslogging facility ?
> 
> Regards,
> 
> Al
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From mbiebl at gmail.com  Tue Mar  2 14:11:06 2010
From: mbiebl at gmail.com (Michael Biebl)
Date: Tue, 2 Mar 2010 14:11:06 +0100
Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103983@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com>
	<9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com>
	<aa76cdac1002260355l26269784y8fb8ccd45caeaa12@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com>
	<aa76cdac1002260443p4c7744e2ha1c8c2372c741ebf@mail.gmail.com>
	<aa76cdac1002280515g72c07527mae8ebc08d4ecd404@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103980@GRFEXC.intern.adiscon.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103981@GRFEXC.intern.adiscon.com>
	<aa76cdac1002281500g3d50db29rcd20fea1fbf08d48@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103983@GRFEXC.intern.adiscon.com>
Message-ID: <aa76cdac1003020511n5378077ag313de591d05cb4fa@mail.gmail.com>

2010/3/1 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
>> Sent: Monday, March 01, 2010 12:01 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released
>>
>> I use $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>> in my rsyslog.conf, but the new ompipe module does not seem to respect
>> that setting but always uses the high-resolution time format.
>
> I see - that was a side-effect that pipes used the same output module like
> regular pipes. But I have now re-established it, please try the newest git
> v4-stable branch.

Hi Rainer,

seems current v4-stable works fine now wrt ActionFileDefaultTemplate and ompipe.

Thanks,
Michael


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Tue Mar  2 14:14:08 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 2 Mar 2010 14:14:08 +0100
Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released
References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com><aa76cdac1002260355l26269784y8fb8ccd45caeaa12@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com><aa76cdac1002260443p4c7744e2ha1c8c2372c741ebf@mail.gmail.com><aa76cdac1002280515g72c07527mae8ebc08d4ecd404@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103980@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103981@GRFEXC.intern.adiscon.com><aa76cdac1002281500g3d50db29rcd20fea1fbf08d48@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103983@GRFEXC.intern.adiscon.com>
	<aa76cdac1003020511n5378077ag313de591d05cb4fa@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71039B4@GRFEXC.intern.adiscon.com>

> seems current v4-stable works fine now wrt ActionFileDefaultTemplate
> and ompipe.

Thanks for letting me know. I'll release that as 4.6.1, but wait for a couple
of other things so that they can make a new release together.

Rainer

> 
> Thanks,
> Michael




From david at lang.hm  Wed Mar  3 07:14:30 2010
From: david at lang.hm (david at lang.hm)
Date: Tue, 2 Mar 2010 22:14:30 -0800 (PST)
Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71039B4@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com><aa76cdac1002260355l26269784y8fb8ccd45caeaa12@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com><aa76cdac1002260443p4c7744e2ha1c8c2372c741ebf@mail.gmail.com><aa76cdac1002280515g72c07527mae8ebc08d4ecd404@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103980@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103981@GRFEXC.intern.adiscon.com><aa76cdac1002281500g3d50db29rcd20fea1fbf08d48@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103983@GRFEXC.intern.adiscon.com>
	<aa76cdac1003020511n5378077ag313de591d05cb4fa@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA71039B4@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.2.00.1003022213520.15263@asgard.lang.hm>

slightly off topic, but still related to releases, how close do you think 
we are to a new stable 5.x release?

David Lang

On Tue, 2 Mar 2010, Rainer Gerhards wrote:

>> seems current v4-stable works fine now wrt ActionFileDefaultTemplate
>> and ompipe.
>
> Thanks for letting me know. I'll release that as 4.6.1, but wait for a couple
> of other things so that they can make a new release together.
>
> Rainer
>
>>
>> Thanks,
>> Michael
>
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Wed Mar  3 07:17:11 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 3 Mar 2010 07:17:11 +0100
Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released
References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com><aa76cdac1002260355l26269784y8fb8ccd45caeaa12@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com><aa76cdac1002260443p4c7744e2ha1c8c2372c741ebf@mail.gmail.com><aa76cdac1002280515g72c07527mae8ebc08d4ecd404@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103980@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103981@GRFEXC.intern.adiscon.com><aa76cdac1002281500g3d50db29rcd20fea1fbf08d48@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103983@GRFEXC.intern.adiscon.com><aa76cdac1003020511n5378077ag313de591d05cb4fa@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA71039B4@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.00.1003022213520.15263@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71039B8@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Wednesday, March 03, 2010 7:15 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released
> 
> slightly off topic, but still related to releases, how close do you
> think
> we are to a new stable 5.x release?

very, very close. I originally intended to release it Monday, but an issue
with 4.6.0 came up that, if possible at all, I would like to integrate.
Yesterday, a bug was filed that points into the same direction:

http://bugzilla.adiscon.com/show_bug.cgi?id=182

If I can't find it quickly enough, I'll probably "timeout" next Monday.

Rainer


From reader at newsguy.com  Wed Mar  3 17:43:07 2010
From: reader at newsguy.com (Harry Putnam)
Date: Wed, 03 Mar 2010 10:43:07 -0600
Subject: [rsyslog] How rsyslog derives hostname
References: <87hbp0eodt.fsf@newsguy.com> <4B8B6F9D.5040803@jackpot.uk.net>
Message-ID: <87ocj59x0k.fsf@newsguy.com>

"Mr. Demeanour" <mrdemeanour at jackpot.uk.net> writes:

> Harry Putnam wrote:
>> 
>> Further... I'm not sure where cmds like gethostbyname (and similar
>> ones) get the info either... for all I know they may just grep
>> /etc/hosts.
>
> man resolv.conf (i.e. it's configurable).

I'm not sure how that would return the miss-typed name in /etc/hosts
There is not mention of the hosts file in man resolve.conf what so ever.

If I set a miss-typed name in /etc/hosts

It appears in rsyslog logs   But /etc/resolv.conf only point to
these nameservers

  nameserver 68.87.72.130 <=A comcast nameserver

  nameserver 192.168.0.20 <= My local router which uses the 
                             nameserver above

Neither of these nameservers will be able to resolve my localhosts
name.  It's not a real domain just a home lan.

So something is searching /etc/hosts and giving the info to rsyslog.

However the command `hostname' will return the correct
hostname... not the miss-typed host in /etc/hosts.

Apparently the command `hostname' uses a different mechanism.
I.E scanning /etc/conf.d/hostname... it seems like rsyslog should do
the same to discover the localhosts name.

I see from Gerhards response:
> I need to re-read this more careful, but for now let me say that
> rsyslog does not touch /etc/hosts at all. It exlusively relies on
> what is returned by the OS. But the "source" property is not locally
> generated, it contains whatever the sender placed into the relevant
> field.

That apparently the `sender' (my localhost) is telling rsyslog to use
the miss-typed host name in /etc/hosts.

He says rsyslog exclusively relies on what is returned by the os. Yes
but by what mechanism?

For example: rsyslog could rely on the output of the `hostname' cmd.
But apparently calls something else.

  If it calls resolv.conf... those nameservers will not return
  anything useful.



From rgerhards at hq.adiscon.com  Wed Mar  3 17:49:55 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 3 Mar 2010 17:49:55 +0100
Subject: [rsyslog] How rsyslog derives hostname
References: <87hbp0eodt.fsf@newsguy.com> <4B8B6F9D.5040803@jackpot.uk.net>
	<87ocj59x0k.fsf@newsguy.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71039C8@GRFEXC.intern.adiscon.com>

> He says rsyslog exclusively relies on what is returned by the os. Yes
> but by what mechanism?

gethostname(), what according to the man page is the mechanism to use ;)

http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/net.c;h=ab431f7cddfda4
1b689e6ab4819b5ffe031e52fb;hb=HEAD#l1331

Rainer


From reader at newsguy.com  Wed Mar  3 17:54:09 2010
From: reader at newsguy.com (Harry Putnam)
Date: Wed, 03 Mar 2010 10:54:09 -0600
Subject: [rsyslog] How rsyslog derives hostname
References: <87hbp0eodt.fsf@newsguy.com>
	<9B6E2A8877C38245BFB15CC491A11DA710398A@GRFEXC.intern.adiscon.com>
Message-ID: <87hbox9wi6.fsf@newsguy.com>

"Rainer Gerhards" <rgerhards at hq.adiscon.com> writes:

> I need to re-read this more careful, but for now let me say that rsyslog does
> not touch /etc/hosts at all. It exlusively relies on what is returned by the
> OS. But the "source" property is not locally generated, it contains whatever
> the sender placed into the relevant field.

Good info ... thanks.  So my host is sending the miss-typed host name
to rsyslog by having it in the log line 
   DATE hostname => blah

So I guess the question is how does my host derive the hostname it
sticks into log messages.

So thanks... and I already know the localhost will scan the /etc/hosts
file when resolving the localhost name... so there is the culprit.

Of course the culprit is really me, who miss-typed the host name.

Thanks.



From reader at newsguy.com  Wed Mar  3 18:00:46 2010
From: reader at newsguy.com (Harry Putnam)
Date: Wed, 03 Mar 2010 11:00:46 -0600
Subject: [rsyslog] How rsyslog derives hostname
References: <87hbp0eodt.fsf@newsguy.com> <4B8B6F9D.5040803@jackpot.uk.net>
	<87ocj59x0k.fsf@newsguy.com>
	<9B6E2A8877C38245BFB15CC491A11DA71039C8@GRFEXC.intern.adiscon.com>
Message-ID: <87d3zl9w75.fsf@newsguy.com>

"Rainer Gerhards" <rgerhards at hq.adiscon.com> writes:

>> He says rsyslog exclusively relies on what is returned by the os. Yes
>> but by what mechanism?
>
> gethostname(), what according to the man page is the mechanism to use ;)
>
> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/net.c;h=ab431f7cddfda4
> 1b689e6ab4819b5ffe031e52fb;hb=HEAD#l1331

Looks pretty complicated... gack.

Thanks... Apparently somewhere in the workings of gethostname() the
/etc/hosts file is scanned.

Thanks.




From rgerhards at hq.adiscon.com  Wed Mar  3 18:02:59 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 3 Mar 2010 18:02:59 +0100
Subject: [rsyslog] How rsyslog derives hostname
References: <87hbp0eodt.fsf@newsguy.com><9B6E2A8877C38245BFB15CC491A11DA710398A@GRFEXC.intern.adiscon.com>
	<87hbox9wi6.fsf@newsguy.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71039C9@GRFEXC.intern.adiscon.com>

> > I need to re-read this more careful, but for now let me say that
> rsyslog does
> > not touch /etc/hosts at all. It exlusively relies on what is returned
> by the
> > OS. But the "source" property is not locally generated, it contains
> whatever
> > the sender placed into the relevant field.
> 
> Good info ... thanks.  So my host is sending the miss-typed host name
> to rsyslog by having it in the log line
>    DATE hostname => blah

I still intended to do a write-up in the cookbook, but more important things
got into the way. So here quickly:

You need to be careful when using hostname to build pathes. Bad guys may put
malicous sequences into it. There are property replacer options to solve
that, primarily removing slashes (something along the lines of "secpath" or
so).

If possible, it is better to use fromhost-ip or fromhost, but that is only
the last sender, so not really useful in a relay chain.

Rainer

> 
> So I guess the question is how does my host derive the hostname it
> sticks into log messages.
> 
> So thanks... and I already know the localhost will scan the /etc/hosts
> file when resolving the localhost name... so there is the culprit.
> 
> Of course the culprit is really me, who miss-typed the host name.
> 
> Thanks.
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Wed Mar  3 18:03:33 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 3 Mar 2010 18:03:33 +0100
Subject: [rsyslog] How rsyslog derives hostname
References: <87hbp0eodt.fsf@newsguy.com>
	<4B8B6F9D.5040803@jackpot.uk.net><87ocj59x0k.fsf@newsguy.com><9B6E2A8877C38245BFB15CC491A11DA71039C8@GRFEXC.intern.adiscon.com>
	<87d3zl9w75.fsf@newsguy.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71039CA@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Harry Putnam
> Sent: Wednesday, March 03, 2010 6:01 PM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] How rsyslog derives hostname
> 
> "Rainer Gerhards" <rgerhards at hq.adiscon.com> writes:
> 
> >> He says rsyslog exclusively relies on what is returned by the os.
> Yes
> >> but by what mechanism?
> >
> > gethostname(), what according to the man page is the mechanism to use
> ;)
> >
> >
> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/net.c;h=ab431f7c
> ddfda4
> > 1b689e6ab4819b5ffe031e52fb;hb=HEAD#l1331
> 
> Looks pretty complicated... gack.
> 
> Thanks... Apparently somewhere in the workings of gethostname() the
> /etc/hosts file is scanned.

My (uneducated) guess is that this is distro/version specific.

Raienr


From iamsayan at gmail.com  Wed Mar  3 18:19:52 2010
From: iamsayan at gmail.com (Sayan Chowdhury)
Date: Wed, 3 Mar 2010 22:49:52 +0530
Subject: [rsyslog] How rsyslog derives hostname
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71039CA@GRFEXC.intern.adiscon.com>
References: <87hbp0eodt.fsf@newsguy.com> <4B8B6F9D.5040803@jackpot.uk.net>
	<87ocj59x0k.fsf@newsguy.com>
	<9B6E2A8877C38245BFB15CC491A11DA71039C8@GRFEXC.intern.adiscon.com>
	<87d3zl9w75.fsf@newsguy.com>
	<9B6E2A8877C38245BFB15CC491A11DA71039CA@GRFEXC.intern.adiscon.com>
Message-ID: <87a1ae541003030919j632dac91n75c67717248a21c5@mail.gmail.com>

Hello Harry,
Look at your /etc/host.conf file, it's most probably set up to look at the
/etc/hosts file before the name servers.
Regards,
Sayan

On Wed, Mar 3, 2010 at 10:33 PM, Rainer Gerhards
<rgerhards at hq.adiscon.com>wrote:

> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Harry Putnam
> > Sent: Wednesday, March 03, 2010 6:01 PM
> > To: rsyslog at lists.adiscon.com
> > Subject: Re: [rsyslog] How rsyslog derives hostname
> >
> > "Rainer Gerhards" <rgerhards at hq.adiscon.com> writes:
> >
> > >> He says rsyslog exclusively relies on what is returned by the os.
> > Yes
> > >> but by what mechanism?
> > >
> > > gethostname(), what according to the man page is the mechanism to use
> > ;)
> > >
> > >
> > http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/net.c;h=ab431f7c
> > ddfda4
> > > 1b689e6ab4819b5ffe031e52fb;hb=HEAD#l1331
> >
> > Looks pretty complicated... gack.
> >
> > Thanks... Apparently somewhere in the workings of gethostname() the
> > /etc/hosts file is scanned.
>
> My (uneducated) guess is that this is distro/version specific.
>
> Raienr
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From mrdemeanour at jackpot.uk.net  Wed Mar  3 19:18:16 2010
From: mrdemeanour at jackpot.uk.net (Mr. Demeanour)
Date: Wed, 03 Mar 2010 18:18:16 +0000
Subject: [rsyslog] How rsyslog derives hostname
In-Reply-To: <87ocj59x0k.fsf@newsguy.com>
References: <87hbp0eodt.fsf@newsguy.com> <4B8B6F9D.5040803@jackpot.uk.net>
	<87ocj59x0k.fsf@newsguy.com>
Message-ID: <4B8EA7E8.9020604@jackpot.uk.net>

Harry Putnam wrote:
> "Mr. Demeanour" <mrdemeanour at jackpot.uk.net> writes:
> 
>> Harry Putnam wrote:
>>> Further... I'm not sure where cmds like gethostbyname (and
>>> similar ones) get the info either... for all I know they may just
>>> grep /etc/hosts.
>> man resolv.conf (i.e. it's configurable).
> 
> I'm not sure how that would return the miss-typed name in /etc/hosts 
> There is not mention of the hosts file in man resolve.conf what so
> ever.

Sorry, my response was both too terse and inaccurate. I should have
pointed you instead at /etc/nsswitch.conf, which the name-resolution
library (i.e. gethostbyname()) uses to figure-out how it's supposed to
resolve hostnames into addresses.

There's a line in there that starts with "hosts: ", usually followed on
the same line by "files" and "dns" in that order (and often accompanied
by other bits). That says that to resolve "hosts", you should first look
at "files", then at "dns". In this context, "files" means "/etc/hosts".

So if your nsswitch.conf mentions "files" in the "hosts:" line before
"dns", that is telling the local name-resolution library to look at
/etc/hosts, and use any answer it can get from there in preference to a
real DNS lookup.

I hope that's a bit more helpful!

-- 
Jack.


From tbergfeld at hq.adiscon.com  Thu Mar  4 11:47:36 2010
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Thu, 4 Mar 2010 11:47:36 +0100
Subject: [rsyslog] rsyslog 4.6.1 (v4-stable) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71039DD@GRFEXC.intern.adiscon.com>

Hi all,
We have just released rsyslog 4.6.1.

Rsyslog 4.6.1 is a bug fixing release. Most importantly, it fixes an issue
with the build system that could potentially result in segfaults, especially
on 32 bit machines. Please review the ChangeLog for details.

This is a recommended update for all v4-stable users.

ChangeLog:

http://www.rsyslog.com/Article445.phtml

Download:

http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-196.phtml


As always, feedback is appreciated.

Best regards,
Tom Bergfeld
--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com


From lanas at securenet.net  Thu Mar  4 12:22:13 2010
From: lanas at securenet.net (lanas)
Date: Thu, 4 Mar 2010 06:22:13 -0500
Subject: [rsyslog] Specifying the source address
Message-ID: <20100304062213.5e5c5a3e@mistral.stie>

Hello,

  Is it possible to specify the source address of rsyslog log messages
when logging to a remote host ?

Thanks.


From david at lang.hm  Thu Mar  4 21:13:10 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 4 Mar 2010 12:13:10 -0800 (PST)
Subject: [rsyslog] How rsyslog derives hostname
In-Reply-To: <87a1ae541003030919j632dac91n75c67717248a21c5@mail.gmail.com>
References: <87hbp0eodt.fsf@newsguy.com> <4B8B6F9D.5040803@jackpot.uk.net>
	<87ocj59x0k.fsf@newsguy.com>
	<9B6E2A8877C38245BFB15CC491A11DA71039C8@GRFEXC.intern.adiscon.com>
	<87d3zl9w75.fsf@newsguy.com>
	<9B6E2A8877C38245BFB15CC491A11DA71039CA@GRFEXC.intern.adiscon.com>
	<87a1ae541003030919j632dac91n75c67717248a21c5@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1003041212550.5131@asgard.lang.hm>

or there may be a services-switch file

David Lang

On Wed, 3 Mar 2010, Sayan Chowdhury wrote:

> Hello Harry,
> Look at your /etc/host.conf file, it's most probably set up to look at the
> /etc/hosts file before the name servers.
> Regards,
> Sayan
>
> On Wed, Mar 3, 2010 at 10:33 PM, Rainer Gerhards
> <rgerhards at hq.adiscon.com>wrote:
>
>>> -----Original Message-----
>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>> bounces at lists.adiscon.com] On Behalf Of Harry Putnam
>>> Sent: Wednesday, March 03, 2010 6:01 PM
>>> To: rsyslog at lists.adiscon.com
>>> Subject: Re: [rsyslog] How rsyslog derives hostname
>>>
>>> "Rainer Gerhards" <rgerhards at hq.adiscon.com> writes:
>>>
>>>>> He says rsyslog exclusively relies on what is returned by the os.
>>> Yes
>>>>> but by what mechanism?
>>>>
>>>> gethostname(), what according to the man page is the mechanism to use
>>> ;)
>>>>
>>>>
>>> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/net.c;h=ab431f7c
>>> ddfda4
>>>> 1b689e6ab4819b5ffe031e52fb;hb=HEAD#l1331
>>>
>>> Looks pretty complicated... gack.
>>>
>>> Thanks... Apparently somewhere in the workings of gethostname() the
>>> /etc/hosts file is scanned.
>>
>> My (uneducated) guess is that this is distro/version specific.
>>
>> Raienr
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From reader at newsguy.com  Fri Mar  5 02:39:14 2010
From: reader at newsguy.com (Harry Putnam)
Date: Thu, 04 Mar 2010 19:39:14 -0600
Subject: [rsyslog] How rsyslog derives hostname
References: <87hbp0eodt.fsf@newsguy.com> <4B8B6F9D.5040803@jackpot.uk.net>
	<87ocj59x0k.fsf@newsguy.com>
	<9B6E2A8877C38245BFB15CC491A11DA71039C8@GRFEXC.intern.adiscon.com>
	<87d3zl9w75.fsf@newsguy.com>
	<9B6E2A8877C38245BFB15CC491A11DA71039CA@GRFEXC.intern.adiscon.com>
	<87a1ae541003030919j632dac91n75c67717248a21c5@mail.gmail.com>
Message-ID: <87hbov8s3h.fsf@newsguy.com>

> Hello Harry,
> Look at your /etc/host.conf file, it's most probably set up to look at the
> /etc/hosts file before the name servers.
> Regards,
> Sayan

Yup... that seems to be where its getting set at... thanks.
from /etc/host.conf
[...]
  # This keyword specifies how host lookups are to be performed. It
  # should be followed by one or more lookup methods, separated by
  # commas.  Valid methods are bind, hosts, and nis.
  #
order hosts, bind
[...]



From tbergfeld at hq.adiscon.com  Mon Mar  8 15:29:13 2010
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Mon, 8 Mar 2010 15:29:13 +0100
Subject: [rsyslog] rsyslog 5.4.0 (v5-stable) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103A33@GRFEXC.intern.adiscon.com>

Hi all,
Today, we released rsyslog 5.4.0, a new v5-stable branch. This version begins
a new stable series based on the 5.3.x series, which has been proven rather
well in practice. Most importantly, the previous 5.2.x stable series had some
serious issues. The new 5.4.0 contains fixes for all known problems. The main
new feature is speed: several optimizations were done, including support for
epoll in tcp listeners.

Users of 5.2.x versions are strongly encouraged to upgrade to 5.4.0.


ChangeLog:

http://www.rsyslog.com/Article447.phtml

Download:

http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-197.phtml


As always, feedback is appreciated.

Best regards,
Tom Bergfeld
--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .


From mbiebl at gmail.com  Tue Mar  9 01:25:48 2010
From: mbiebl at gmail.com (Michael Biebl)
Date: Tue, 9 Mar 2010 01:25:48 +0100
Subject: [rsyslog] rsyslog 5.4.0 (v5-stable) released
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103A33@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA7103A33@GRFEXC.intern.adiscon.com>
Message-ID: <aa76cdac1003081625w6dac4ec3raf85bde7491c8338@mail.gmail.com>

Hi everyone!

2010/3/8 Tom Bergfeld <tbergfeld at hq.adiscon.com>:
> Hi all,
> Today, we released rsyslog 5.4.0, a new v5-stable branch. This version begins

So, the help output of rsyslogd has

To run rsyslogd in native mode, use "rsyslogd -c5 <other options>"

Maybe I'm just missing some link to the documentation, but is it
documented somewhere, which features are enabled/available when using
-c5 compared to using -c4 or -c3.
Or put it differently: What's the difference between using -c3 and -c5
for 5.4.0?

Michael


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From pgollucci at p6m7g8.com  Tue Mar  9 02:20:44 2010
From: pgollucci at p6m7g8.com (Philip M. Gollucci)
Date: Tue, 9 Mar 2010 01:20:44 +0000
Subject: [rsyslog] Fwd: cvs commit: ports/sysutils/rsyslog5 Makefile
 distinfo pkg-plist ports/sysutils/rsyslog5/files rsyslogd.in
Message-ID: <4B95A26C.6080301@p6m7g8.com>

fwiw, now in freebsd ports.

-------- Original Message --------
Subject: cvs commit: ports/sysutils/rsyslog5 Makefile distinfo pkg-plist 
         ports/sysutils/rsyslog5/files rsyslogd.in
Date: Mon, 8 Mar 2010 23:15:09 +0000 (UTC)
From: Philip M. Gollucci <pgollucci at FreeBSD.org>
To: ports-committers at FreeBSD.org, cvs-ports at FreeBSD.org, cvs-all at FreeBSD.org

pgollucci    2010-03-08 23:15:09 UTC

   FreeBSD ports repository

   Modified files:
     sysutils/rsyslog5    Makefile distinfo pkg-plist
     sysutils/rsyslog5/files rsyslogd.in
   Log:
   - Update to 5.4.0 [1]
   - Remove -a and old school sockets from rc.d script [myself]

   PR:             ports/144566 [1]
   Submitted by:   Cristiano Rolim Pereira <cristianorolim at hotmail.com> 
(maintainer) [1]

   Feature safe:   yes

   Revision  Changes    Path
   1.13      +1 -1      ports/sysutils/rsyslog5/Makefile
   1.13      +3 -3      ports/sysutils/rsyslog5/distinfo
   1.3       +1 -54     ports/sysutils/rsyslog5/files/rsyslogd.in
   1.8       +1 -0      ports/sysutils/rsyslog5/pkg-plist

http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/Makefile.diff?r1=1.12&r2=1.13&f=h
| --- ports/sysutils/rsyslog5/Makefile	2010/01/28 20:51:59	1.12
| +++ ports/sysutils/rsyslog5/Makefile	2010/03/08 23:15:09	1.13
| @@ -2,11 +2,11 @@
|  # Date created:     29 December 2008
|  # Whom:             Cristiano Rolim Pereira <cristianorolim at hotmail.com>
|  #
| -# $FreeBSD: 
/usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/Makefile,v 1.12 
2010/01/28 20:51:59 glarkin Exp $
| +# $FreeBSD: 
/usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/Makefile,v 1.13 
2010/03/08 23:15:09 pgollucci Exp $
|  #
|
|  PORTNAME=	rsyslog
| -PORTVERSION=	5.3.7
| +PORTVERSION=	5.4.0
|  CATEGORIES=	sysutils
|  MASTER_SITES=	http://download.rsyslog.com/rsyslog/
|  .ifdef MNAME
http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/distinfo.diff?r1=1.12&r2=1.13&f=h
| --- ports/sysutils/rsyslog5/distinfo	2010/01/28 20:51:59	1.12
| +++ ports/sysutils/rsyslog5/distinfo	2010/03/08 23:15:09	1.13
| @@ -1,3 +1,3 @@
| -MD5 (rsyslog-5.3.7.tar.gz) = 1a9b5c76bfea2d0ec28630b8f033d5e2
| -SHA256 (rsyslog-5.3.7.tar.gz) = 
cab7d39d3979e18c23bd0726ffad714c283d1158dea6c611f23eb1718299c029
| -SIZE (rsyslog-5.3.7.tar.gz) = 2109777
| +MD5 (rsyslog-5.4.0.tar.gz) = 291882229d50496f42bd63174076dd37
| +SHA256 (rsyslog-5.4.0.tar.gz) = 
d9cd21d2fcd45fcae65eb0a51927c40315cca02afdc62478abd950febfcf7228
| +SIZE (rsyslog-5.4.0.tar.gz) = 2124201
http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/files/rsyslogd.in.diff?r1=1.2&r2=1.3&f=h
| --- ports/sysutils/rsyslog5/files/rsyslogd.in	2009/12/18 20:44:28	1.2
| +++ ports/sysutils/rsyslog5/files/rsyslogd.in	2010/03/08 23:15:09	1.3
| @@ -1,6 +1,6 @@
|  #!/bin/sh
|  #
| -# $FreeBSD: 
/usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/files/rsyslogd.in,v 
1.2 2009/12/18 20:44:28 miwi Exp $
| +# $FreeBSD: 
/usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/files/rsyslogd.in,v 
1.3 2010/03/08 23:15:09 pgollucci Exp $
|  #
|
|
| @@ -20,59 +20,6 @@ load_rc_config $name
|  pidfile="${rsyslogd_pidfile}"
|  command_args="-i ${pidfile} -f ${rsyslogd_config}"
|  required_files="${rsyslogd_config}"
| -start_precmd="rsyslogd_precmd"
|  extra_commands="reload"
|
| -sockfile="/var/run/rsyslogd.sockets"
| -evalargs="rc_flags=\"\`set_socketlist\` \$rc_flags\""
| -altlog_proglist="named"
| -
| -rsyslogd_precmd()
| -{
| -	local _l _ldir
| -
| -	#	Transitional symlink for old binaries
| -	#
| -	if [ ! -L /dev/log ]; then
| -		ln -sf /var/run/log /dev/log
| -	fi
| -	rm -f /var/run/log
| -
| -	#	Create default list of syslog sockets to watch
| -	#
| -	( umask 022 ; > $sockfile )
| -
| -	#	If running named(8) or ntpd(8) chrooted, added appropriate
| -	#	syslog socket to list of sockets to watch.
| -	#
| -	for _l in $altlog_proglist; do
| -		eval _ldir=\$${_l}_chrootdir
| -		if checkyesno `set_rcvar $_l` && [ -n "$_ldir" ]; then
| -			echo "${_ldir}/var/run/log" >> $sockfile
| -		fi
| -	done
| -
| -	#	If other sockets have been provided, change run_rc_command()'s
| -	#	internal copy of $rsyslogd_flags to force use of specific
| -	#	rsyslogd sockets.
| -	#
| -	if [ -s $sockfile ]; then
| -		echo "/var/run/log" >> $sockfile
| -		eval $evalargs
| -	fi
| -
| -	return 0
| -}
| -
| -set_socketlist()
| -{
| -	local _s _socketargs
| -
| -	_socketargs=
| -	for _s in `cat $sockfile | tr '\n' ' '` ; do
| -		_socketargs="-a $_s $_socketargs"
| -	done
| -	echo $_socketargs
| -}
| -
|  run_rc_command "$1"
http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/pkg-plist.diff?r1=1.7&r2=1.8&f=h
| --- ports/sysutils/rsyslog5/pkg-plist	2010/01/07 04:46:04	1.7
| +++ ports/sysutils/rsyslog5/pkg-plist	2010/03/08 23:15:09	1.8
| @@ -126,6 +126,7 @@ sbin/rsyslogd
|  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_mysql.html
|  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_ng_comparison.html
|  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_packages.html
| +%%PORTDOCS%%%%DOCSDIR%%/rsyslog_pgsql.html
|  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_php_syslog_ng.html
|  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_recording_pri.html
|  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_reliable_forwarding.html


From rgerhards at hq.adiscon.com  Tue Mar  9 07:38:43 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 9 Mar 2010 07:38:43 +0100
Subject: [rsyslog] Fwd: cvs commit: ports/sysutils/rsyslog5 Makefile
	distinfo pkg-plist ports/sysutils/rsyslog5/files rsyslogd.in
References: <4B95A26C.6080301@p6m7g8.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103A45@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Philip M. Gollucci
> Sent: Tuesday, March 09, 2010 2:21 AM
> To: rsyslog-users
> Subject: [rsyslog] Fwd: cvs commit: ports/sysutils/rsyslog5 Makefile
> distinfo pkg-plist ports/sysutils/rsyslog5/files rsyslogd.in
> 
> fwiw, now in freebsd ports.

excellent, this is *very* useful!

Thanks,
Rainer

> 
> -------- Original Message --------
> Subject: cvs commit: ports/sysutils/rsyslog5 Makefile distinfo pkg-
> plist
>          ports/sysutils/rsyslog5/files rsyslogd.in
> Date: Mon, 8 Mar 2010 23:15:09 +0000 (UTC)
> From: Philip M. Gollucci <pgollucci at FreeBSD.org>
> To: ports-committers at FreeBSD.org, cvs-ports at FreeBSD.org, cvs-
> all at FreeBSD.org
> 
> pgollucci    2010-03-08 23:15:09 UTC
> 
>    FreeBSD ports repository
> 
>    Modified files:
>      sysutils/rsyslog5    Makefile distinfo pkg-plist
>      sysutils/rsyslog5/files rsyslogd.in
>    Log:
>    - Update to 5.4.0 [1]
>    - Remove -a and old school sockets from rc.d script [myself]
> 
>    PR:             ports/144566 [1]
>    Submitted by:   Cristiano Rolim Pereira <cristianorolim at hotmail.com>
> (maintainer) [1]
> 
>    Feature safe:   yes
> 
>    Revision  Changes    Path
>    1.13      +1 -1      ports/sysutils/rsyslog5/Makefile
>    1.13      +3 -3      ports/sysutils/rsyslog5/distinfo
>    1.3       +1 -54     ports/sysutils/rsyslog5/files/rsyslogd.in
>    1.8       +1 -0      ports/sysutils/rsyslog5/pkg-plist
> 
> http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/Makefile.diff?r1=1.12
> &r2=1.13&f=h
> | --- ports/sysutils/rsyslog5/Makefile	2010/01/28 20:51:59	1.12
> | +++ ports/sysutils/rsyslog5/Makefile	2010/03/08 23:15:09	1.13
> | @@ -2,11 +2,11 @@
> |  # Date created:     29 December 2008
> |  # Whom:             Cristiano Rolim Pereira
> <cristianorolim at hotmail.com>
> |  #
> | -# $FreeBSD:
> /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/Makefile,v 1.12
> 2010/01/28 20:51:59 glarkin Exp $
> | +# $FreeBSD:
> /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/Makefile,v 1.13
> 2010/03/08 23:15:09 pgollucci Exp $
> |  #
> |
> |  PORTNAME=	rsyslog
> | -PORTVERSION=	5.3.7
> | +PORTVERSION=	5.4.0
> |  CATEGORIES=	sysutils
> |  MASTER_SITES=	http://download.rsyslog.com/rsyslog/
> |  .ifdef MNAME
> http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/distinfo.diff?r1=1.12
> &r2=1.13&f=h
> | --- ports/sysutils/rsyslog5/distinfo	2010/01/28 20:51:59	1.12
> | +++ ports/sysutils/rsyslog5/distinfo	2010/03/08 23:15:09	1.13
> | @@ -1,3 +1,3 @@
> | -MD5 (rsyslog-5.3.7.tar.gz) = 1a9b5c76bfea2d0ec28630b8f033d5e2
> | -SHA256 (rsyslog-5.3.7.tar.gz) =
> cab7d39d3979e18c23bd0726ffad714c283d1158dea6c611f23eb1718299c029
> | -SIZE (rsyslog-5.3.7.tar.gz) = 2109777
> | +MD5 (rsyslog-5.4.0.tar.gz) = 291882229d50496f42bd63174076dd37
> | +SHA256 (rsyslog-5.4.0.tar.gz) =
> d9cd21d2fcd45fcae65eb0a51927c40315cca02afdc62478abd950febfcf7228
> | +SIZE (rsyslog-5.4.0.tar.gz) = 2124201
> http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/files/rsyslogd.in.dif
> f?r1=1.2&r2=1.3&f=h
> | --- ports/sysutils/rsyslog5/files/rsyslogd.in	2009/12/18
> 20:44:28	1.2
> | +++ ports/sysutils/rsyslog5/files/rsyslogd.in	2010/03/08
> 23:15:09	1.3
> | @@ -1,6 +1,6 @@
> |  #!/bin/sh
> |  #
> | -# $FreeBSD:
> /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/files/rsyslogd.i
> n,v
> 1.2 2009/12/18 20:44:28 miwi Exp $
> | +# $FreeBSD:
> /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/files/rsyslogd.i
> n,v
> 1.3 2010/03/08 23:15:09 pgollucci Exp $
> |  #
> |
> |
> | @@ -20,59 +20,6 @@ load_rc_config $name
> |  pidfile="${rsyslogd_pidfile}"
> |  command_args="-i ${pidfile} -f ${rsyslogd_config}"
> |  required_files="${rsyslogd_config}"
> | -start_precmd="rsyslogd_precmd"
> |  extra_commands="reload"
> |
> | -sockfile="/var/run/rsyslogd.sockets"
> | -evalargs="rc_flags=\"\`set_socketlist\` \$rc_flags\""
> | -altlog_proglist="named"
> | -
> | -rsyslogd_precmd()
> | -{
> | -	local _l _ldir
> | -
> | -	#	Transitional symlink for old binaries
> | -	#
> | -	if [ ! -L /dev/log ]; then
> | -		ln -sf /var/run/log /dev/log
> | -	fi
> | -	rm -f /var/run/log
> | -
> | -	#	Create default list of syslog sockets to watch
> | -	#
> | -	( umask 022 ; > $sockfile )
> | -
> | -	#	If running named(8) or ntpd(8) chrooted, added appropriate
> | -	#	syslog socket to list of sockets to watch.
> | -	#
> | -	for _l in $altlog_proglist; do
> | -		eval _ldir=\$${_l}_chrootdir
> | -		if checkyesno `set_rcvar $_l` && [ -n "$_ldir" ]; then
> | -			echo "${_ldir}/var/run/log" >> $sockfile
> | -		fi
> | -	done
> | -
> | -	#	If other sockets have been provided, change
> run_rc_command()'s
> | -	#	internal copy of $rsyslogd_flags to force use of specific
> | -	#	rsyslogd sockets.
> | -	#
> | -	if [ -s $sockfile ]; then
> | -		echo "/var/run/log" >> $sockfile
> | -		eval $evalargs
> | -	fi
> | -
> | -	return 0
> | -}
> | -
> | -set_socketlist()
> | -{
> | -	local _s _socketargs
> | -
> | -	_socketargs=
> | -	for _s in `cat $sockfile | tr '\n' ' '` ; do
> | -		_socketargs="-a $_s $_socketargs"
> | -	done
> | -	echo $_socketargs
> | -}
> | -
> |  run_rc_command "$1"
> http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/pkg-
> plist.diff?r1=1.7&r2=1.8&f=h
> | --- ports/sysutils/rsyslog5/pkg-plist	2010/01/07 04:46:04	1.7
> | +++ ports/sysutils/rsyslog5/pkg-plist	2010/03/08 23:15:09	1.8
> | @@ -126,6 +126,7 @@ sbin/rsyslogd
> |  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_mysql.html
> |  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_ng_comparison.html
> |  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_packages.html
> | +%%PORTDOCS%%%%DOCSDIR%%/rsyslog_pgsql.html
> |  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_php_syslog_ng.html
> |  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_recording_pri.html
> |  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_reliable_forwarding.html
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Tue Mar  9 09:00:01 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 9 Mar 2010 09:00:01 +0100
Subject: [rsyslog] rsyslog 5.4.0 (v5-stable) released
References: <9B6E2A8877C38245BFB15CC491A11DA7103A33@GRFEXC.intern.adiscon.com>
	<aa76cdac1003081625w6dac4ec3raf85bde7491c8338@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103A49@GRFEXC.intern.adiscon.com>

Hi Michael,

The -c option is transitive. So for the diff between -c3 and -c5, you need to
check the compatibility notes for v4 and v5. These are:

http://www.rsyslog.com/doc-v4compatibility.html
http://www.rsyslog.com/doc-v5compatibility.html

I should add to the notes that -c obviously only modifies defaults, and not
program behavior at large. So even with -c3, for example, rsyslog v5 will not
be able to run on zero workers.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Tuesday, March 09, 2010 1:26 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 5.4.0 (v5-stable) released
> 
> Hi everyone!
> 
> 2010/3/8 Tom Bergfeld <tbergfeld at hq.adiscon.com>:
> > Hi all,
> > Today, we released rsyslog 5.4.0, a new v5-stable branch. This
> version begins
> 
> So, the help output of rsyslogd has
> 
> To run rsyslogd in native mode, use "rsyslogd -c5 <other options>"
> 
> Maybe I'm just missing some link to the documentation, but is it
> documented somewhere, which features are enabled/available when using
> -c5 compared to using -c4 or -c3.
> Or put it differently: What's the difference between using -c3 and -c5
> for 5.4.0?
> 
> Michael
> 
> 
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From pgollucci at p6m7g8.com  Thu Mar 11 00:09:51 2010
From: pgollucci at p6m7g8.com (Philip M. Gollucci)
Date: Wed, 10 Mar 2010 23:09:51 +0000
Subject: [rsyslog] rsyslogd 'clients' take out machines if rsyslogd 'server'
	is down
Message-ID: <4B9826BF.7070202@p6m7g8.com>

rsyslog.conf [client version]:
-------------------------------

....

$WorkDirectory /var/spool/rsyslog # location for work (spool) file
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

....
## Local
....

## Remote
*.* :omrelp:cl.tld:2514


If anything on the 'centralized server' fails like mysql is down, or 
rsyslogd is down, or a table is wrong, or insert permissions are wrong, 
or diskio is *really* *really* bad, or the box is 'off' --- about 45 
minutes later. All 50 or so of the clients talking to 'block' and 
suddenly spin in a tight loop and are not killable without -9.

I'm sure its something filling up b/c the timing is nearly to the second 
of the 45 minute mark [given a relatively constant amount of load].

I assume that changing
$ActionResumeRetryCount -1
to
$ActionResumeRetryCount 5

or some other arbirtarily low number in all clients would fix this.  As 
whatever ram buffers are filling up will stop doing that.  If not, maybe 
falling back from RELP to TCP would do the trick [I'd be okay with 
message loss to prevent downtime].

I'm rather more concerned though that it doesn't start queueing to disk 
the spool dir when the ram buffers are full instead of 'crashing' and 
taking the box or all process on it with it.



2542.938673679:800b021c0: rsyslogd 5.3.7 startup, compatibility mode 5, 
module path ''

on FreeBSD 8.0-RELENG-p2








-- 
------------------------------------------------------------------------
1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70  3F8C 75B8 8FFB DB9B 8C1C
Philip M. Gollucci (pgollucci at p6m7g8.com) c: 703.336.9354
VP Apache Infrastructure; Member, Apache Software Foundation
Committer,                        FreeBSD Foundation
Consultant,                       P6M7G8 Inc.
Sr. System Admin,                 Ridecharge Inc.

Work like you don't need the money,
love like you'll never get hurt,
and dance like nobody's watching.


From mangoo at wpkg.org  Thu Mar 18 21:21:06 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Thu, 18 Mar 2010 21:21:06 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
Message-ID: <4BA28B32.6090201@wpkg.org>

I have postgres logging to a separate file through rsyslog with this line:

local0.*                -/var/log/postgresql/postgresql_log


It gets to /var/log/syslog as well, thanks to this line:

*.*;auth,authpriv.none	-/var/log/syslog


As postgres is set to log quite heavily, I would rather not log postgres into /var/log/syslog.

I've read some filtering examples, but I couldn't find one on excluding a given program from a file which is logging everything.

Could anyone point me to a proper example?


Postgres adds such entries through syslog:

Mar 19 04:14:16 sun postgres[388]: [261-1] 2010-03-19 04:14:16 SGT LOG:  duration: 0.037 ms  statement: ROLLBACK


-- 
Tomasz Chmielewski
http://wpkg.org


From sledz at DResearch.DE  Thu Mar 18 21:06:58 2010
From: sledz at DResearch.DE (Sledz, Steffen)
Date: Thu, 18 Mar 2010 21:06:58 +0100
Subject: [rsyslog] patch for architectures without atomic builtins
Message-ID: <DB561678F08F554EB0D9F21B70E268A505AE8B@exchange.intern.dresearch.de>

Hi,

we want to use rsyslogd on an embedded system with an armv5te cpu.
Unfortunately the gcc/libc we have to use for this system do not support
atomic builtins which are vital for rsyslog at the moment. :(

Therefor we created a patch which emulates the atomic builtins
with posix semaphores. This patch should fit to the 5.4.x and
5.5.x versions.

We would be glad if this patch could help to bring rsyslog to
more platforms.

Steffen

From david at lang.hm  Thu Mar 18 23:54:56 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 15:54:56 -0700 (PDT)
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <4BA28B32.6090201@wpkg.org>
References: <4BA28B32.6090201@wpkg.org>
Message-ID: <alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>

On Thu, 18 Mar 2010, Tomasz Chmielewski wrote:

> I have postgres logging to a separate file through rsyslog with this line:
>
> local0.*                -/var/log/postgresql/postgresql_log
>
>
> It gets to /var/log/syslog as well, thanks to this line:
>
> *.*;auth,authpriv.none	-/var/log/syslog
>
>
> As postgres is set to log quite heavily, I would rather not log postgres into /var/log/syslog.
>
> I've read some filtering examples, but I couldn't find one on excluding a given program from a file which is logging everything.
>
> Could anyone point me to a proper example?
>
>
> Postgres adds such entries through syslog:
>
> Mar 19 04:14:16 sun postgres[388]: [261-1] 2010-03-19 04:14:16 SGT LOG:  duration: 0.037 ms  statement: ROLLBACK

you could add

:programname, :isequal, "postgres" ~

ahead of your line to go to /var/log/syslog so that anything with the 
programname of postgres gets dropped first

another thing you could do (if you know that all postgres logs are local0) 
is

local0.* ~

to throw away all logs with facility local0

David Lang


From david at lang.hm  Thu Mar 18 23:55:48 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 15:55:48 -0700 (PDT)
Subject: [rsyslog] patch for architectures without atomic builtins
In-Reply-To: <DB561678F08F554EB0D9F21B70E268A505AE8B@exchange.intern.dresearch.de>
References: <DB561678F08F554EB0D9F21B70E268A505AE8B@exchange.intern.dresearch.de>
Message-ID: <alpine.DEB.2.00.1003181555170.10411@asgard.lang.hm>

On Thu, 18 Mar 2010, Sledz, Steffen wrote:

> Hi,
>
> we want to use rsyslogd on an embedded system with an armv5te cpu.
> Unfortunately the gcc/libc we have to use for this system do not support
> atomic builtins which are vital for rsyslog at the moment. :(
>
> Therefor we created a patch which emulates the atomic builtins
> with posix semaphores. This patch should fit to the 5.4.x and
> 5.5.x versions.
>
> We would be glad if this patch could help to bring rsyslog to
> more platforms.
>
> Steffen

the list strips off attachments.

can you re-send with the patch inline?

David Lang


From mangoo at wpkg.org  Fri Mar 19 00:02:41 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Fri, 19 Mar 2010 00:02:41 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
Message-ID: <4BA2B111.7090604@wpkg.org>

Am 18.03.2010 23:54, david at lang.hm wrote:

>> Postgres adds such entries through syslog:
>>
>> Mar 19 04:14:16 sun postgres[388]: [261-1] 2010-03-19 04:14:16 SGT LOG:  duration: 0.037 ms  statement: ROLLBACK
> 
> you could add
> 
> :programname, :isequal, "postgres" ~
> 
> ahead of your line to go to /var/log/syslog so that anything with the
> programname of postgres gets dropped first

Meaning, like this:

:programname, :isequal, "postgres" ~ *.*;auth,authpriv.none	-/var/log/syslog

?


-- 
Tomasz Chmielewski
http://wpkg.org


From david at lang.hm  Fri Mar 19 00:08:29 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 16:08:29 -0700 (PDT)
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <4BA2B111.7090604@wpkg.org>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
Message-ID: <alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>

On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:

> Am 18.03.2010 23:54, david at lang.hm wrote:
>
>>> Postgres adds such entries through syslog:
>>>
>>> Mar 19 04:14:16 sun postgres[388]: [261-1] 2010-03-19 04:14:16 SGT LOG:  duration: 0.037 ms  statement: ROLLBACK
>>
>> you could add
>>
>> :programname, :isequal, "postgres" ~
>>
>> ahead of your line to go to /var/log/syslog so that anything with the
>> programname of postgres gets dropped first
>
> Meaning, like this:
>
> :programname, :isequal, "postgres" ~ *.*;auth,authpriv.none	-/var/log/syslog

no,

:programname, :isequal, "postgres" ~
*.*;auth,authpriv.none	-/var/log/syslog

each of these is a seperate statement. the ~ as a destination tells 
rsyslog not to proceed further through the rules for this log entry.

David Lang


From mangoo at wpkg.org  Fri Mar 19 00:19:59 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Fri, 19 Mar 2010 00:19:59 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
Message-ID: <4BA2B51F.1060204@wpkg.org>

Am 19.03.2010 00:08, david at lang.hm wrote:

> no,
>
> :programname, :isequal, "postgres" ~
> *.*;auth,authpriv.none -/var/log/syslog
>
> each of these is a seperate statement. the ~ as a destination tells
> rsyslog not to proceed further through the rules for this log entry.

Doesn't work for me.

If I put it like above, nothing gets appended to /var/log/syslog anymore 
(or any other defined log file).

rsyslogd is running as a process.

If it matters, I'm using 64 bit Debian Lenny (rsyslog 3.18.6).


-- 
Tomasz Chmielewski
http://wpkg.org


From david at lang.hm  Fri Mar 19 00:23:22 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 16:23:22 -0700 (PDT)
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <4BA2B51F.1060204@wpkg.org>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
Message-ID: <alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>

On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:

> Am 19.03.2010 00:08, david at lang.hm wrote:
>
>> no,
>> 
>> :programname, :isequal, "postgres" ~
>> *.*;auth,authpriv.none -/var/log/syslog
>> 
>> each of these is a seperate statement. the ~ as a destination tells
>> rsyslog not to proceed further through the rules for this log entry.
>
> Doesn't work for me.
>
> If I put it like above, nothing gets appended to /var/log/syslog anymore (or 
> any other defined log file).
>
> rsyslogd is running as a process.
>
> If it matters, I'm using 64 bit Debian Lenny (rsyslog 3.18.6).

try it with ' instead of " (I get those mixed up at least 50% of the time, 
rsyslog accepts one and not the other and I can never remember which one)

David Lang


From mangoo at wpkg.org  Fri Mar 19 00:40:46 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Fri, 19 Mar 2010 00:40:46 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
Message-ID: <4BA2B9FE.2030008@wpkg.org>

Am 19.03.2010 00:23, david at lang.hm wrote:
> On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:
>
>> Am 19.03.2010 00:08, david at lang.hm wrote:
>>
>>> no,
>>>
>>> :programname, :isequal, "postgres" ~
>>> *.*;auth,authpriv.none -/var/log/syslog
>>>
>>> each of these is a seperate statement. the ~ as a destination tells
>>> rsyslog not to proceed further through the rules for this log entry.
>>
>> Doesn't work for me.
>>
>> If I put it like above, nothing gets appended to /var/log/syslog
>> anymore (or any other defined log file).
>>
>> rsyslogd is running as a process.
>>
>> If it matters, I'm using 64 bit Debian Lenny (rsyslog 3.18.6).
>
> try it with ' instead of " (I get those mixed up at least 50% of the
> time, rsyslog accepts one and not the other and I can never remember
> which one)

OK, with ' it started to log again, but still postgres gets to both 
/var/log/syslog and /var/log/postgresql/postgresql_log.

So, not as expected.

-- 
Tomasz Chmielewski
http://wpkg.org


From david at lang.hm  Fri Mar 19 00:52:45 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 16:52:45 -0700 (PDT)
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <4BA2B9FE.2030008@wpkg.org>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org>
Message-ID: <alpine.DEB.2.00.1003181652290.10411@asgard.lang.hm>

On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:

> Am 19.03.2010 00:23, david at lang.hm wrote:
>> On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:
>> 
>>> Am 19.03.2010 00:08, david at lang.hm wrote:
>>> 
>>>> no,
>>>> 
>>>> :programname, :isequal, "postgres" ~
>>>> *.*;auth,authpriv.none -/var/log/syslog
>>>> 
>>>> each of these is a seperate statement. the ~ as a destination tells
>>>> rsyslog not to proceed further through the rules for this log entry.
>>> 
>>> Doesn't work for me.
>>> 
>>> If I put it like above, nothing gets appended to /var/log/syslog
>>> anymore (or any other defined log file).
>>> 
>>> rsyslogd is running as a process.
>>> 
>>> If it matters, I'm using 64 bit Debian Lenny (rsyslog 3.18.6).
>> 
>> try it with ' instead of " (I get those mixed up at least 50% of the
>> time, rsyslog accepts one and not the other and I can never remember
>> which one)
>
> OK, with ' it started to log again, but still postgres gets to both 
> /var/log/syslog and /var/log/postgresql/postgresql_log.
>
> So, not as expected.

what does the raw log message look like again?

David Lang


From mangoo at wpkg.org  Fri Mar 19 00:58:37 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Fri, 19 Mar 2010 00:58:37 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <alpine.DEB.2.00.1003181652290.10411@asgard.lang.hm>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org>
	<alpine.DEB.2.00.1003181652290.10411@asgard.lang.hm>
Message-ID: <4BA2BE2D.9020205@wpkg.org>

Am 19.03.2010 00:52, david at lang.hm wrote:

>> OK, with ' it started to log again, but still postgres gets to both 
>> /var/log/syslog and /var/log/postgresql/postgresql_log.
>>
>> So, not as expected.
> 
> what does the raw log message look like again?

Like these ones:

Mar 19 07:47:32 db10 postgres[4539]: [87-1] 2010-03-19 07:47:32 SGT LOG:  duration: 0.234 ms  statement: COMMIT
Mar 19 07:47:32 db10 postgres[4539]: [88-1] 2010-03-19 07:47:32 SGT LOG:  duration: 0.042 ms  statement: BEGIN; SET TRANSACTION ISOLATION LEVEL READ 


where postgres[number] can differ.

-- 
Tomasz Chmielewski
http://wpkg.org


From david at lang.hm  Fri Mar 19 01:08:11 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 17:08:11 -0700 (PDT)
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <4BA2BE2D.9020205@wpkg.org>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org>
	<alpine.DEB.2.00.1003181652290.10411@asgard.lang.hm>
	<4BA2BE2D.9020205@wpkg.org>
Message-ID: <alpine.DEB.2.00.1003181706510.10411@asgard.lang.hm>

On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:

> Am 19.03.2010 00:52, david at lang.hm wrote:
>
>>> OK, with ' it started to log again, but still postgres gets to both
>>> /var/log/syslog and /var/log/postgresql/postgresql_log.
>>>
>>> So, not as expected.
>>
>> what does the raw log message look like again?
>
> Like these ones:
>
> Mar 19 07:47:32 db10 postgres[4539]: [87-1] 2010-03-19 07:47:32 SGT LOG:  duration: 0.234 ms  statement: COMMIT
> Mar 19 07:47:32 db10 postgres[4539]: [88-1] 2010-03-19 07:47:32 SGT LOG:  duration: 0.042 ms  statement: BEGIN; SET TRANSACTION ISOLATION LEVEL READ
>
>
> where postgres[number] can differ.

hmm, in these cases postgres should be what's in programname

taking a different slant, IIRC you used local0.* to capture the postgres 
logs into one file, if in the line after that you do

local0.* ~

that should eliminate those logs from going into any files after that.

David Lang


From mangoo at wpkg.org  Fri Mar 19 01:42:29 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Fri, 19 Mar 2010 01:42:29 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <alpine.DEB.2.00.1003181706510.10411@asgard.lang.hm>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org>
	<alpine.DEB.2.00.1003181652290.10411@asgard.lang.hm>
	<4BA2BE2D.9020205@wpkg.org>
	<alpine.DEB.2.00.1003181706510.10411@asgard.lang.hm>
Message-ID: <4BA2C875.4060208@wpkg.org>

Am 19.03.2010 01:08, david at lang.hm wrote:

>> where postgres[number] can differ.
>
> hmm, in these cases postgres should be what's in programname
>
> taking a different slant, IIRC you used local0.* to capture the postgres
> logs into one file, if in the line after that you do
>
> local0.* ~
>
> that should eliminate those logs from going into any files after that.

This one works, but that'll filter out heartbeat, pgpool and others 
useful, which get to syslog.


-- 
Tomasz Chmielewski
http://wpkg.org


From mangoo at wpkg.org  Fri Mar 19 01:44:41 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Fri, 19 Mar 2010 01:44:41 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <4BA2B9FE.2030008@wpkg.org>
References: <4BA28B32.6090201@wpkg.org>	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>	<4BA2B111.7090604@wpkg.org>	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>	<4BA2B51F.1060204@wpkg.org>	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org>
Message-ID: <4BA2C8F9.4050607@wpkg.org>

Am 19.03.2010 00:40, Tomasz Chmielewski wrote:
> Am 19.03.2010 00:23, david at lang.hm wrote:
>> On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:
>>
>>> Am 19.03.2010 00:08, david at lang.hm wrote:
>>>
>>>> no,
>>>>
>>>> :programname, :isequal, "postgres" ~
>>>> *.*;auth,authpriv.none -/var/log/syslog
>>>>
>>>> each of these is a seperate statement. the ~ as a destination tells
>>>> rsyslog not to proceed further through the rules for this log entry.
>>>
>>> Doesn't work for me.
>>>
>>> If I put it like above, nothing gets appended to /var/log/syslog
>>> anymore (or any other defined log file).
>>>
>>> rsyslogd is running as a process.
>>>
>>> If it matters, I'm using 64 bit Debian Lenny (rsyslog 3.18.6).
>>
>> try it with ' instead of " (I get those mixed up at least 50% of the
>> time, rsyslog accepts one and not the other and I can never remember
>> which one)
> 
> OK, with ' it started to log again, but still postgres gets to both
> /var/log/syslog and /var/log/postgresql/postgresql_log.
> 
> So, not as expected.

And here is why:

Mar 19 07:37:26 sun rsyslogd: error: invalid compare operation ':isequal' - ignoring selector
Mar 19 07:37:26 sun rsyslogd: error -3003 compare value property - ignoring selector
Mar 19 07:37:26 sun rsyslogd: the last error occured in /etc/rsyslog.conf, line 58
Mar 19 07:37:26 sun rsyslogd: warning: selector line without actions will be discarded


So, if it's " - it doesn't log anything.

If it's ' - it doesn't like the rule.


-- 
Tomasz Chmielewski
http://wpkg.org



From joeschmo2000 at gmail.com  Fri Mar 19 03:14:00 2010
From: joeschmo2000 at gmail.com (joe schmo)
Date: Thu, 18 Mar 2010 22:14:00 -0400
Subject: [rsyslog] CentOS Issues
Message-ID: <a60a2fa91003181914kd27e298se6a8e3913563e4ca@mail.gmail.com>

Hey all,

So I had been using another syslog server for a long time and recently
decided to make the move over to rsyslog and phplogcon so that we could have
web searchable logs (I already run RANCID w/ CVS and cvsweb enabled so it
seemed only fitting to get our syslog server on the webz too)

I went to the forums with these questions and they suggested I ask the
mailing list. That was actually my second destination anyway because I had
read several messages from the archive before heading to the forums.

I have a few questions that span a few different versions...first I need to
apologize in advance if any of these are noob questions, but I have searched
for them and I can't find any answers that help me.

I am running CentOS 5.3 Final with all latest updates off the main repos.
rsyslog 2.0.6 running currently w/ phplogmon and mysql.

1) I have a copy of rsyslog 2.0.6 up and running without any issues (as it
is the only one I can currently get working...more on that in a sec), other
than the fact that it can't receive udp messages without manual
intervention. The only way I can get it to accept udp (network) syslog
messages is to stop the service, and then run the rsyslog -r514 from
command-line manually. Is there anyway to get it to use this option by
default? I need it to start the service with that option enabled. And the
"just run it manually" answers I have received in other forums aren't
acceptable for me.

2) An extension of the above question: when I try to do the "obvious"
solution to the udp issue and use the imudp module, I realized there IS NO
imudp.so module in the plugins folder. Where would I even get this module
for 2.0.6? The most I can find is a copy of imudp.c and its accompanying
make files, but trying to make only outputs errors associated with missing
files that belong to the C developers package (which I had installed some
time ago off an official CentOS repo, no updates available).

3) Why is it that when I follow all the instructions for compiling and
installing the newer versions of rsyslog (3.0.0+), I can get all the way
through the process (I searched and found a way to correct the i686 GCC
errors) but then it isn't actually installed? Does this have to do with
CentOS 5.3 support or am I missing a step here? After doing the make
install, etc, there is no rsyslog.conf file in /etc/ like when I install
2.0.6, nor any of the other accompanying directories, except the
/usr/lib/rsyslog/ folder, which is fully populated with all the correct
files.

Any guidance would be GREATLY appreciated as I am getting very frustrated
with the amount of time I have put into trying to get everything working
correctly. And I thought setting up RANCID and cvsweb was a bit of a chore!
:)

Thanks in advance!

-Joe


From david at lang.hm  Fri Mar 19 05:20:51 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 21:20:51 -0700 (PDT)
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <4BA2C8F9.4050607@wpkg.org>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org> <4BA2C8F9.4050607@wpkg.org>
Message-ID: <alpine.DEB.2.00.1003182119540.3821@asgard.lang.hm>

On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:

> Am 19.03.2010 00:40, Tomasz Chmielewski wrote:
>> Am 19.03.2010 00:23, david at lang.hm wrote:
>>> On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:
>>>
>>>> Am 19.03.2010 00:08, david at lang.hm wrote:
>>>>
>>>>> no,
>>>>>
>>>>> :programname, :isequal, "postgres" ~
>>>>> *.*;auth,authpriv.none -/var/log/syslog
>>>>>
>>>>> each of these is a seperate statement. the ~ as a destination tells
>>>>> rsyslog not to proceed further through the rules for this log entry.
>>>>
>>>> Doesn't work for me.
>>>>
>>>> If I put it like above, nothing gets appended to /var/log/syslog
>>>> anymore (or any other defined log file).
>>>>
>>>> rsyslogd is running as a process.
>>>>
>>>> If it matters, I'm using 64 bit Debian Lenny (rsyslog 3.18.6).
>>>
>>> try it with ' instead of " (I get those mixed up at least 50% of the
>>> time, rsyslog accepts one and not the other and I can never remember
>>> which one)
>>
>> OK, with ' it started to log again, but still postgres gets to both
>> /var/log/syslog and /var/log/postgresql/postgresql_log.
>>
>> So, not as expected.
>
> And here is why:
>
> Mar 19 07:37:26 sun rsyslogd: error: invalid compare operation ':isequal' - ignoring selector
> Mar 19 07:37:26 sun rsyslogd: error -3003 compare value property - ignoring selector
> Mar 19 07:37:26 sun rsyslogd: the last error occured in /etc/rsyslog.conf, line 58
> Mar 19 07:37:26 sun rsyslogd: warning: selector line without actions will be discarded
>
>
> So, if it's " - it doesn't log anything.
>
> If it's ' - it doesn't like the rule.

when all else fails, look at the documentation ;-)

http://www.rsyslog.com/doc-rsyslog_conf_filter.html

the line should have been

:programname, isequal, 'postgres' ~

David Lang


From david at lang.hm  Fri Mar 19 05:22:13 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 21:22:13 -0700 (PDT)
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <4BA2C875.4060208@wpkg.org>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org>
	<alpine.DEB.2.00.1003181652290.10411@asgard.lang.hm>
	<4BA2BE2D.9020205@wpkg.org>
	<alpine.DEB.2.00.1003181706510.10411@asgard.lang.hm>
	<4BA2C875.4060208@wpkg.org>
Message-ID: <alpine.DEB.2.00.1003182120550.3821@asgard.lang.hm>

On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:

> Am 19.03.2010 01:08, david at lang.hm wrote:
>
>>> where postgres[number] can differ.
>> 
>> hmm, in these cases postgres should be what's in programname
>> 
>> taking a different slant, IIRC you used local0.* to capture the postgres
>> logs into one file, if in the line after that you do
>> 
>> local0.* ~
>> 
>> that should eliminate those logs from going into any files after that.
>
> This one works, but that'll filter out heartbeat, pgpool and others useful, 
> which get to syslog.

I will point out that those entries are getting put into your 
posgresql_log file along with the postgres logs then.

so you may want to change the filter for that line as well (now that I dug 
up the page with the correct syntax rather than going from memory)

David Lang


From david at lang.hm  Fri Mar 19 05:31:50 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 21:31:50 -0700 (PDT)
Subject: [rsyslog] CentOS Issues
In-Reply-To: <a60a2fa91003181914kd27e298se6a8e3913563e4ca@mail.gmail.com>
References: <a60a2fa91003181914kd27e298se6a8e3913563e4ca@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1003182125350.3821@asgard.lang.hm>

On Thu, 18 Mar 2010, joe schmo wrote:

> Hey all,
>
> So I had been using another syslog server for a long time and recently
> decided to make the move over to rsyslog and phplogcon so that we could have
> web searchable logs (I already run RANCID w/ CVS and cvsweb enabled so it
> seemed only fitting to get our syslog server on the webz too)
>
> I went to the forums with these questions and they suggested I ask the
> mailing list. That was actually my second destination anyway because I had
> read several messages from the archive before heading to the forums.
>
> I have a few questions that span a few different versions...first I need to
> apologize in advance if any of these are noob questions, but I have searched
> for them and I can't find any answers that help me.
>
> I am running CentOS 5.3 Final with all latest updates off the main repos.
> rsyslog 2.0.6 running currently w/ phplogmon and mysql.
>
> 1) I have a copy of rsyslog 2.0.6 up and running without any issues (as it
> is the only one I can currently get working...more on that in a sec), other
> than the fact that it can't receive udp messages without manual
> intervention. The only way I can get it to accept udp (network) syslog
> messages is to stop the service, and then run the rsyslog -r514 from
> command-line manually. Is there anyway to get it to use this option by
> default? I need it to start the service with that option enabled. And the
> "just run it manually" answers I have received in other forums aren't
> acceptable for me.
>
> 2) An extension of the above question: when I try to do the "obvious"
> solution to the udp issue and use the imudp module, I realized there IS NO
> imudp.so module in the plugins folder. Where would I even get this module
> for 2.0.6? The most I can find is a copy of imudp.c and its accompanying
> make files, but trying to make only outputs errors associated with missing
> files that belong to the C developers package (which I had installed some
> time ago off an official CentOS repo, no updates available).
>
> 3) Why is it that when I follow all the instructions for compiling and
> installing the newer versions of rsyslog (3.0.0+), I can get all the way
> through the process (I searched and found a way to correct the i686 GCC
> errors) but then it isn't actually installed? Does this have to do with
> CentOS 5.3 support or am I missing a step here? After doing the make
> install, etc, there is no rsyslog.conf file in /etc/ like when I install
> 2.0.6, nor any of the other accompanying directories, except the
> /usr/lib/rsyslog/ folder, which is fully populated with all the correct
> files.
>
> Any guidance would be GREATLY appreciated as I am getting very frustrated
> with the amount of time I have put into trying to get everything working
> correctly. And I thought setting up RANCID and cvsweb was a bit of a chore!
> :)
>
> Thanks in advance!

as you have noticed, rsyslog 2.x is ancient (I am running 5.x in 
production now)

when compiling the new version you should be able to just do make install 
after you have compiled it to install it.

you may need to uninstall the RHEL shipped version, I think that by 
default rsyslog will get installed under /usr/local when compiled from 
source.

installing from source will not create the /etc/rsyslog.conf file for you.

the rsyslog on my laptop (ubuntu with a couple manual tweaks) is:

#  /etc/rsyslog.conf    Configuration file for rsyslog.
#
#                       For more information see
#                       /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
#
#  Default logging rules can be found in /etc/rsyslog.d/50-default.conf
#################
#### MODULES ####
#################
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog   # provides kernel logging support (previously done by 
rklogd)
#$ModLoad immark  # provides --MARK-- message capability
#$KLogPath /var/run/rsyslog/kmsg
# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
$RepeatedMsgReduction on
$RepeatedMsgContainsOriginalMsg on
#
# Include all config files in /etc/rsyslog.d/
#
#$IncludeConfig /etc/rsyslog.d/*.conf
*.* -/var/log/messages

this should be enough to get you started, you will need to uncomment the 
remote reception options and then start rsyslog with rsyslogd -cX where X 
is the major version you are running (3,4,5 depending on which one you 
pick)

David Lang


From sledz at dresearch.de  Fri Mar 19 08:00:39 2010
From: sledz at dresearch.de (Steffen Sledz)
Date: Fri, 19 Mar 2010 08:00:39 +0100
Subject: [rsyslog] patch for architectures without atomic builtins
In-Reply-To: <alpine.DEB.2.00.1003181555170.10411@asgard.lang.hm>
References: <DB561678F08F554EB0D9F21B70E268A505AE8B@exchange.intern.dresearch.de>
	<alpine.DEB.2.00.1003181555170.10411@asgard.lang.hm>
Message-ID: <4BA32117.9030408@dresearch.de>

david at lang.hm wrote:
> the list strips off attachments.
> 
> can you re-send with the patch inline?

Here it is...

diff -Nurd rsyslog-5.4.0.orig/configure.ac rsyslog-5.4.0/configure.ac
--- rsyslog-5.4.0.orig/configure.ac     2010-03-08 13:28:35.000000000 +0000
+++ rsyslog-5.4.0/configure.ac  2010-03-16 11:30:44.000000000 +0000
@@ -87,6 +87,7 @@
 AC_HEADER_TIME
 AC_STRUCT_TM
 AC_C_VOLATILE
+AC_C_TYPEOF

 sa_includes="\
 $ac_includes_default
@@ -129,7 +130,8 @@
 # check for availability of atomic operations
 RS_ATOMIC_OPERATIONS
 RS_ATOMIC_OPERATIONS_64BIT
-
+# fall back to POSIX sems for atomic operations (cpu expensive)
+AC_CHECK_HEADERS([semaphore.h])

 # Additional module directories
 AC_ARG_WITH(moddirs,
diff -Nurd rsyslog-5.4.0.orig/runtime/Makefile.am rsyslog-5.4.0/runtime/Makefile.am
--- rsyslog-5.4.0.orig/runtime/Makefile.am      2010-03-08 13:27:47.000000000 +0000
+++ rsyslog-5.4.0/runtime/Makefile.am   2010-03-15 16:43:12.000000000 +0000
@@ -9,6 +9,7 @@
        rsyslog.h \
        unicode-helper.h \
        atomic.h \
+       atomic-posix-sem.c \
        batch.h \
        syslogd-types.h \
        module-template.h \
diff -Nurd rsyslog-5.4.0.orig/runtime/atomic-posix-sem.c rsyslog-5.4.0/runtime/atomic-posix-sem.c
--- rsyslog-5.4.0.orig/runtime/atomic-posix-sem.c       1970-01-01 00:00:00.000000000 +0000
+++ rsyslog-5.4.0/runtime/atomic-posix-sem.c    2010-03-16 13:22:27.000000000 +0000
@@ -0,0 +1,70 @@
+/* atomic_posix_sem.c: This file supplies an emulation for atomic operations using
+ * POSIX semaphores.
+ *
+ * Copyright 2010 DResearch Digital Media Systems GmbH
+ *
+ * This file is part of the rsyslog runtime library.
+ *
+ * The rsyslog runtime library is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * The rsyslog runtime library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public License
+ * along with the rsyslog runtime library.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * A copy of the GPL can be found in the file "COPYING" in this distribution.
+ * A copy of the LGPL can be found in the file "COPYING.LESSER" in this distribution.
+ */
+
+#include "config.h"
+#ifndef HAVE_ATOMIC_BUILTINS
+#ifdef HAVE_SEMAPHORE_H
+#include <semaphore.h>
+#include <errno.h>
+
+#include "atomic.h"
+#include "rsyslog.h"
+#include "srUtils.h"
+
+sem_t atomicSem;
+
+rsRetVal
+atomicSemInit(void)
+{
+       DEFiRet;
+
+       dbgprintf("init posix semaphore for atomics emulation\n");
+       if(sem_init(&atomicSem, 0, 1) == -1)
+       {
+               char errStr[1024];
+               rs_strerror_r(errno, errStr, sizeof(errStr));
+               dbgprintf("init posix semaphore for atomics emulation failed: %s\n", errStr);
+               iRet = RS_RET_SYS_ERR; /* the right error code ??? */
+       }
+
+       RETiRet;
+}
+
+void
+atomicSemExit(void)
+{
+       dbgprintf("destroy posix semaphore for atomics emulation\n");
+       if(sem_destroy(&atomicSem) == -1)
+       {
+               char errStr[1024];
+               rs_strerror_r(errno, errStr, sizeof(errStr));
+               dbgprintf("destroy posix semaphore for atomics emulation failed: %s\n", errStr);
+       }
+}
+
+#endif /* HAVE_SEMAPHORE_H */
+#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
+
+/* vim:set ai:
+ */
diff -Nurd rsyslog-5.4.0.orig/runtime/atomic.h rsyslog-5.4.0/runtime/atomic.h
--- rsyslog-5.4.0.orig/runtime/atomic.h 2010-03-05 07:20:36.000000000 +0000
+++ rsyslog-5.4.0/runtime/atomic.h      2010-03-16 13:23:57.000000000 +0000
@@ -54,6 +54,122 @@
 #      define ATOMIC_CAS(data, oldVal, newVal) __sync_bool_compare_and_swap(&(data), (oldVal), (newVal));
 #      define ATOMIC_CAS_VAL(data, oldVal, newVal) __sync_val_compare_and_swap(&(data), (oldVal), (newVal));
 #else
+#ifdef HAVE_SEMAPHORE_H
+       /* we use POSIX semaphores instead */
+
+#include "rsyslog.h"
+#include <semaphore.h>
+
+extern sem_t atomicSem;
+rsRetVal atomicSemInit(void);
+void atomicSemExit(void);
+
+#if HAVE_TYPEOF
+#define my_typeof(x) typeof(x)
+#else /* sorry, can't determine types, using 'int' */
+#define my_typeof(x) int
+#endif
+
+#      define ATOMIC_SUB(data, val) \
+({ \
+       my_typeof(data) tmp; \
+       sem_wait(&atomicSem); \
+       tmp = data; \
+       data -= val; \
+       sem_post(&atomicSem); \
+       tmp; \
+})
+
+#      define ATOMIC_ADD(data, val) \
+({ \
+       my_typeof(data) tmp; \
+       sem_wait(&atomicSem); \
+       tmp = data; \
+       data += val; \
+       sem_post(&atomicSem); \
+       tmp; \
+})
+
+#      define ATOMIC_INC_AND_FETCH(data) \
+({ \
+       my_typeof(data) tmp; \
+       sem_wait(&atomicSem); \
+       tmp = data; \
+       data += 1; \
+       sem_post(&atomicSem); \
+       tmp; \
+})
+
+#      define ATOMIC_INC(data) ((void) ATOMIC_INC_AND_FETCH(data))
+
+#      define ATOMIC_DEC_AND_FETCH(data) \
+({ \
+       sem_wait(&atomicSem); \
+       data -= 1; \
+       sem_post(&atomicSem); \
+       data; \
+})
+
+#      define ATOMIC_DEC(data) ((void) ATOMIC_DEC_AND_FETCH(data))
+
+#      define ATOMIC_FETCH_32BIT(data) ((unsigned) ATOMIC_ADD((data), 0xffffffff))
+
+#      define ATOMIC_STORE_1_TO_32BIT(data) \
+({ \
+       my_typeof(data) tmp; \
+       sem_wait(&atomicSem); \
+       tmp = data; \
+       data = 1; \
+       sem_post(&atomicSem); \
+       tmp; \
+})
+
+#      define ATOMIC_STORE_0_TO_INT(data) \
+({ \
+       my_typeof(data) tmp; \
+       sem_wait(&atomicSem); \
+       tmp = data; \
+       data = 0; \
+       sem_post(&atomicSem); \
+       tmp; \
+})
+
+#      define ATOMIC_STORE_1_TO_INT(data) \
+({ \
+       my_typeof(data) tmp; \
+       sem_wait(&atomicSem); \
+       tmp = data; \
+       data = 1; \
+       sem_post(&atomicSem); \
+       tmp; \
+})
+
+#      define ATOMIC_CAS(data, oldVal, newVal) \
+({ \
+       int ret; \
+       sem_wait(&atomicSem); \
+       if(data != oldVal) ret = 0; \
+       else \
+       { \
+               data = newVal; \
+               ret = 1; \
+       } \
+       sem_post(&atomicSem); \
+       ret; \
+})
+
+#      define ATOMIC_CAS_VAL(data, oldVal, newVal) \
+({ \
+       sem_wait(&atomicSem); \
+       if(data == oldVal) \
+       { \
+               data = newVal; \
+       } \
+       sem_post(&atomicSem); \
+       data; \
+})
+
+#else /* not HAVE_SEMAPHORE_H */
        /* note that we gained parctical proof that theoretical problems DO occur
         * if we do not properly address them. See this blog post for details:
         * http://blog.gerhards.net/2009/01/rsyslog-data-race-analysis.html
@@ -68,5 +184,6 @@
 #      define ATOMIC_FETCH_32BIT(data) (data)
 #      define ATOMIC_STORE_1_TO_32BIT(data) (data) = 1
 #endif
+#endif

 #endif /* #ifndef INCLUDED_ATOMIC_H */
diff -Nurd rsyslog-5.4.0.orig/runtime/rsyslog.c rsyslog-5.4.0/runtime/rsyslog.c
--- rsyslog-5.4.0.orig/runtime/rsyslog.c        2010-03-05 07:20:36.000000000 +0000
+++ rsyslog-5.4.0/runtime/rsyslog.c     2010-03-16 08:30:51.000000000 +0000
@@ -81,6 +81,7 @@
 #include "rule.h"
 #include "ruleset.h"
 #include "parser.h"
+#include "atomic.h"

 /* forward definitions */
 static rsRetVal dfltErrLogger(int, uchar *errMsg);
@@ -140,6 +141,12 @@
                CHKiRet(objClassInit(NULL)); /* *THIS* *MUST* always be the first class initilizer being called! */
                CHKiRet(objGetObjInterface(pObjIF)); /* this provides the root pointer for all other queries */

+#ifndef HAVE_ATOMIC_BUILTINS
+#ifdef HAVE_SEMAPHORE_H
+               CHKiRet(atomicSemInit());
+#endif /* HAVE_SEMAPHORE_H */
+#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
+
                /* initialize core classes. We must be very careful with the order of events. Some
                 * classes use others and if we do not initialize them in the right order, we may end
                 * up with an invalid call. The most important thing that can happen is that an error
@@ -216,6 +223,13 @@
                glblClassExit();
                rulesetClassExit();
                ruleClassExit();
+
+#ifndef HAVE_ATOMIC_BUILTINS
+#ifdef HAVE_SEMAPHORE_H
+               atomicSemExit();
+#endif /* HAVE_SEMAPHORE_H */
+#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
+
                objClassExit(); /* *THIS* *MUST/SHOULD?* always be the first class initilizer being called (except debug)! */
        }




From rgerhards at hq.adiscon.com  Fri Mar 19 10:10:42 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 19 Mar 2010 10:10:42 +0100
Subject: [rsyslog] CentOS Issues
References: <a60a2fa91003181914kd27e298se6a8e3913563e4ca@mail.gmail.com>
	<alpine.DEB.2.00.1003182125350.3821@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103AFD@GRFEXC.intern.adiscon.com>

> when compiling the new version you should be able to just do make
> install
> after you have compiled it to install it.

As far as I know, you need to set --libdir and --bindir (or its 64 bit
equivalents) ./configure options on CentOS, but I don't know what the correct
values are for it.

Rainer

> 
> you may need to uninstall the RHEL shipped version, I think that by
> default rsyslog will get installed under /usr/local when compiled from
> source.
> 
> installing from source will not create the /etc/rsyslog.conf file for
> you.
> 
> the rsyslog on my laptop (ubuntu with a couple manual tweaks) is:
> 
> #  /etc/rsyslog.conf    Configuration file for rsyslog.
> #
> #                       For more information see
> #                       /usr/share/doc/rsyslog-
> doc/html/rsyslog_conf.html
> #
> #  Default logging rules can be found in /etc/rsyslog.d/50-default.conf
> #################
> #### MODULES ####
> #################
> $ModLoad imuxsock # provides support for local system logging
> $ModLoad imklog   # provides kernel logging support (previously done by
> rklogd)
> #$ModLoad immark  # provides --MARK-- message capability
> #$KLogPath /var/run/rsyslog/kmsg
> # provides UDP syslog reception
> #$ModLoad imudp
> #$UDPServerRun 514
> # provides TCP syslog reception
> #$ModLoad imtcp
> #$InputTCPServerRun 514
> ###########################
> #### GLOBAL DIRECTIVES ####
> ###########################
> #
> # Use traditional timestamp format.
> # To enable high precision timestamps, comment out the following line.
> #
> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
> #
> # Set the default permissions for all log files.
> #
> $FileOwner syslog
> $FileGroup adm
> $FileCreateMode 0640
> $DirCreateMode 0755
> $Umask 0022
> $PrivDropToUser syslog
> $PrivDropToGroup syslog
> $RepeatedMsgReduction on
> $RepeatedMsgContainsOriginalMsg on
> #
> # Include all config files in /etc/rsyslog.d/
> #
> #$IncludeConfig /etc/rsyslog.d/*.conf
> *.* -/var/log/messages
> 
> this should be enough to get you started, you will need to uncomment
> the
> remote reception options and then start rsyslog with rsyslogd -cX where
> X
> is the major version you are running (3,4,5 depending on which one you
> pick)
> 
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Fri Mar 19 10:13:42 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 19 Mar 2010 10:13:42 +0100
Subject: [rsyslog] CentOS Issues
References: <a60a2fa91003181914kd27e298se6a8e3913563e4ca@mail.gmail.com><alpine.DEB.2.00.1003182125350.3821@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7103AFD@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103AFE@GRFEXC.intern.adiscon.com>

> > when compiling the new version you should be able to just do make
> > install
> > after you have compiled it to install it.
> 
> As far as I know, you need to set --libdir and --bindir (or its 64 bit
> equivalents) ./configure options on CentOS, but I don't know what the
> correct
> values are for it.

I just discovered there is some good info on CentOS in the wiki:

http://wiki.rsyslog.com/index.php/Rsyslog_on_CentOS_success_story

I am not sure if it is too much in-depth, though...

Rainer


From rgerhards at hq.adiscon.com  Fri Mar 19 10:22:39 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 19 Mar 2010 10:22:39 +0100
Subject: [rsyslog] patch for architectures without atomic builtins
References: <DB561678F08F554EB0D9F21B70E268A505AE8B@exchange.intern.dresearch.de><alpine.DEB.2.00.1003181555170.10411@asgard.lang.hm>
	<4BA32117.9030408@dresearch.de>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103AFF@GRFEXC.intern.adiscon.com>

Excellent, many thanks for this very useful contribution! I'll see that I
merge it in as quickly as possible, but it may take until early next week.
I'll probably try to merge it into v4 as well, as this is definitely useful!

Thanks again,
Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Steffen Sledz
> Sent: Friday, March 19, 2010 8:01 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] patch for architectures without atomic builtins
> 
> david at lang.hm wrote:
> > the list strips off attachments.
> >
> > can you re-send with the patch inline?
> 
> Here it is...
> 
> diff -Nurd rsyslog-5.4.0.orig/configure.ac rsyslog-5.4.0/configure.ac
> --- rsyslog-5.4.0.orig/configure.ac     2010-03-08 13:28:35.000000000
> +0000
> +++ rsyslog-5.4.0/configure.ac  2010-03-16 11:30:44.000000000 +0000
> @@ -87,6 +87,7 @@
>  AC_HEADER_TIME
>  AC_STRUCT_TM
>  AC_C_VOLATILE
> +AC_C_TYPEOF
> 
>  sa_includes="\
>  $ac_includes_default
> @@ -129,7 +130,8 @@
>  # check for availability of atomic operations
>  RS_ATOMIC_OPERATIONS
>  RS_ATOMIC_OPERATIONS_64BIT
> -
> +# fall back to POSIX sems for atomic operations (cpu expensive)
> +AC_CHECK_HEADERS([semaphore.h])
> 
>  # Additional module directories
>  AC_ARG_WITH(moddirs,
> diff -Nurd rsyslog-5.4.0.orig/runtime/Makefile.am rsyslog-
> 5.4.0/runtime/Makefile.am
> --- rsyslog-5.4.0.orig/runtime/Makefile.am      2010-03-08
> 13:27:47.000000000 +0000
> +++ rsyslog-5.4.0/runtime/Makefile.am   2010-03-15 16:43:12.000000000
> +0000
> @@ -9,6 +9,7 @@
>         rsyslog.h \
>         unicode-helper.h \
>         atomic.h \
> +       atomic-posix-sem.c \
>         batch.h \
>         syslogd-types.h \
>         module-template.h \
> diff -Nurd rsyslog-5.4.0.orig/runtime/atomic-posix-sem.c rsyslog-
> 5.4.0/runtime/atomic-posix-sem.c
> --- rsyslog-5.4.0.orig/runtime/atomic-posix-sem.c       1970-01-01
> 00:00:00.000000000 +0000
> +++ rsyslog-5.4.0/runtime/atomic-posix-sem.c    2010-03-16
> 13:22:27.000000000 +0000
> @@ -0,0 +1,70 @@
> +/* atomic_posix_sem.c: This file supplies an emulation for atomic
> operations using
> + * POSIX semaphores.
> + *
> + * Copyright 2010 DResearch Digital Media Systems GmbH
> + *
> + * This file is part of the rsyslog runtime library.
> + *
> + * The rsyslog runtime library is free software: you can redistribute
> it and/or modify
> + * it under the terms of the GNU Lesser General Public License as
> published by
> + * the Free Software Foundation, either version 3 of the License, or
> + * (at your option) any later version.
> + *
> + * The rsyslog runtime library is distributed in the hope that it will
> be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> License
> + * along with the rsyslog runtime library.  If not, see
> <http://www.gnu.org/licenses/>.
> + *
> + * A copy of the GPL can be found in the file "COPYING" in this
> distribution.
> + * A copy of the LGPL can be found in the file "COPYING.LESSER" in
> this distribution.
> + */
> +
> +#include "config.h"
> +#ifndef HAVE_ATOMIC_BUILTINS
> +#ifdef HAVE_SEMAPHORE_H
> +#include <semaphore.h>
> +#include <errno.h>
> +
> +#include "atomic.h"
> +#include "rsyslog.h"
> +#include "srUtils.h"
> +
> +sem_t atomicSem;
> +
> +rsRetVal
> +atomicSemInit(void)
> +{
> +       DEFiRet;
> +
> +       dbgprintf("init posix semaphore for atomics emulation\n");
> +       if(sem_init(&atomicSem, 0, 1) == -1)
> +       {
> +               char errStr[1024];
> +               rs_strerror_r(errno, errStr, sizeof(errStr));
> +               dbgprintf("init posix semaphore for atomics emulation
> failed: %s\n", errStr);
> +               iRet = RS_RET_SYS_ERR; /* the right error code ??? */
> +       }
> +
> +       RETiRet;
> +}
> +
> +void
> +atomicSemExit(void)
> +{
> +       dbgprintf("destroy posix semaphore for atomics emulation\n");
> +       if(sem_destroy(&atomicSem) == -1)
> +       {
> +               char errStr[1024];
> +               rs_strerror_r(errno, errStr, sizeof(errStr));
> +               dbgprintf("destroy posix semaphore for atomics
> emulation failed: %s\n", errStr);
> +       }
> +}
> +
> +#endif /* HAVE_SEMAPHORE_H */
> +#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
> +
> +/* vim:set ai:
> + */
> diff -Nurd rsyslog-5.4.0.orig/runtime/atomic.h rsyslog-
> 5.4.0/runtime/atomic.h
> --- rsyslog-5.4.0.orig/runtime/atomic.h 2010-03-05 07:20:36.000000000
> +0000
> +++ rsyslog-5.4.0/runtime/atomic.h      2010-03-16 13:23:57.000000000
> +0000
> @@ -54,6 +54,122 @@
>  #      define ATOMIC_CAS(data, oldVal, newVal)
> __sync_bool_compare_and_swap(&(data), (oldVal), (newVal));
>  #      define ATOMIC_CAS_VAL(data, oldVal, newVal)
> __sync_val_compare_and_swap(&(data), (oldVal), (newVal));
>  #else
> +#ifdef HAVE_SEMAPHORE_H
> +       /* we use POSIX semaphores instead */
> +
> +#include "rsyslog.h"
> +#include <semaphore.h>
> +
> +extern sem_t atomicSem;
> +rsRetVal atomicSemInit(void);
> +void atomicSemExit(void);
> +
> +#if HAVE_TYPEOF
> +#define my_typeof(x) typeof(x)
> +#else /* sorry, can't determine types, using 'int' */
> +#define my_typeof(x) int
> +#endif
> +
> +#      define ATOMIC_SUB(data, val) \
> +({ \
> +       my_typeof(data) tmp; \
> +       sem_wait(&atomicSem); \
> +       tmp = data; \
> +       data -= val; \
> +       sem_post(&atomicSem); \
> +       tmp; \
> +})
> +
> +#      define ATOMIC_ADD(data, val) \
> +({ \
> +       my_typeof(data) tmp; \
> +       sem_wait(&atomicSem); \
> +       tmp = data; \
> +       data += val; \
> +       sem_post(&atomicSem); \
> +       tmp; \
> +})
> +
> +#      define ATOMIC_INC_AND_FETCH(data) \
> +({ \
> +       my_typeof(data) tmp; \
> +       sem_wait(&atomicSem); \
> +       tmp = data; \
> +       data += 1; \
> +       sem_post(&atomicSem); \
> +       tmp; \
> +})
> +
> +#      define ATOMIC_INC(data) ((void) ATOMIC_INC_AND_FETCH(data))
> +
> +#      define ATOMIC_DEC_AND_FETCH(data) \
> +({ \
> +       sem_wait(&atomicSem); \
> +       data -= 1; \
> +       sem_post(&atomicSem); \
> +       data; \
> +})
> +
> +#      define ATOMIC_DEC(data) ((void) ATOMIC_DEC_AND_FETCH(data))
> +
> +#      define ATOMIC_FETCH_32BIT(data) ((unsigned) ATOMIC_ADD((data),
> 0xffffffff))
> +
> +#      define ATOMIC_STORE_1_TO_32BIT(data) \
> +({ \
> +       my_typeof(data) tmp; \
> +       sem_wait(&atomicSem); \
> +       tmp = data; \
> +       data = 1; \
> +       sem_post(&atomicSem); \
> +       tmp; \
> +})
> +
> +#      define ATOMIC_STORE_0_TO_INT(data) \
> +({ \
> +       my_typeof(data) tmp; \
> +       sem_wait(&atomicSem); \
> +       tmp = data; \
> +       data = 0; \
> +       sem_post(&atomicSem); \
> +       tmp; \
> +})
> +
> +#      define ATOMIC_STORE_1_TO_INT(data) \
> +({ \
> +       my_typeof(data) tmp; \
> +       sem_wait(&atomicSem); \
> +       tmp = data; \
> +       data = 1; \
> +       sem_post(&atomicSem); \
> +       tmp; \
> +})
> +
> +#      define ATOMIC_CAS(data, oldVal, newVal) \
> +({ \
> +       int ret; \
> +       sem_wait(&atomicSem); \
> +       if(data != oldVal) ret = 0; \
> +       else \
> +       { \
> +               data = newVal; \
> +               ret = 1; \
> +       } \
> +       sem_post(&atomicSem); \
> +       ret; \
> +})
> +
> +#      define ATOMIC_CAS_VAL(data, oldVal, newVal) \
> +({ \
> +       sem_wait(&atomicSem); \
> +       if(data == oldVal) \
> +       { \
> +               data = newVal; \
> +       } \
> +       sem_post(&atomicSem); \
> +       data; \
> +})
> +
> +#else /* not HAVE_SEMAPHORE_H */
>         /* note that we gained parctical proof that theoretical
> problems DO occur
>          * if we do not properly address them. See this blog post for
> details:
>          * http://blog.gerhards.net/2009/01/rsyslog-data-race-
> analysis.html
> @@ -68,5 +184,6 @@
>  #      define ATOMIC_FETCH_32BIT(data) (data)
>  #      define ATOMIC_STORE_1_TO_32BIT(data) (data) = 1
>  #endif
> +#endif
> 
>  #endif /* #ifndef INCLUDED_ATOMIC_H */
> diff -Nurd rsyslog-5.4.0.orig/runtime/rsyslog.c rsyslog-
> 5.4.0/runtime/rsyslog.c
> --- rsyslog-5.4.0.orig/runtime/rsyslog.c        2010-03-05
> 07:20:36.000000000 +0000
> +++ rsyslog-5.4.0/runtime/rsyslog.c     2010-03-16 08:30:51.000000000
> +0000
> @@ -81,6 +81,7 @@
>  #include "rule.h"
>  #include "ruleset.h"
>  #include "parser.h"
> +#include "atomic.h"
> 
>  /* forward definitions */
>  static rsRetVal dfltErrLogger(int, uchar *errMsg);
> @@ -140,6 +141,12 @@
>                 CHKiRet(objClassInit(NULL)); /* *THIS* *MUST* always be
> the first class initilizer being called! */
>                 CHKiRet(objGetObjInterface(pObjIF)); /* this provides
> the root pointer for all other queries */
> 
> +#ifndef HAVE_ATOMIC_BUILTINS
> +#ifdef HAVE_SEMAPHORE_H
> +               CHKiRet(atomicSemInit());
> +#endif /* HAVE_SEMAPHORE_H */
> +#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
> +
>                 /* initialize core classes. We must be very careful
> with the order of events. Some
>                  * classes use others and if we do not initialize them
> in the right order, we may end
>                  * up with an invalid call. The most important thing
> that can happen is that an error
> @@ -216,6 +223,13 @@
>                 glblClassExit();
>                 rulesetClassExit();
>                 ruleClassExit();
> +
> +#ifndef HAVE_ATOMIC_BUILTINS
> +#ifdef HAVE_SEMAPHORE_H
> +               atomicSemExit();
> +#endif /* HAVE_SEMAPHORE_H */
> +#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
> +
>                 objClassExit(); /* *THIS* *MUST/SHOULD?* always be the
> first class initilizer being called (except debug)! */
>         }
> 
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From theinric at redhat.com  Fri Mar 19 10:47:52 2010
From: theinric at redhat.com (Tomas Heinrich)
Date: Fri, 19 Mar 2010 10:47:52 +0100
Subject: [rsyslog] CentOS Issues
In-Reply-To: <a60a2fa91003181914kd27e298se6a8e3913563e4ca@mail.gmail.com>
References: <a60a2fa91003181914kd27e298se6a8e3913563e4ca@mail.gmail.com>
Message-ID: <4BA34848.8090002@redhat.com>

On 03/19/2010 03:14 AM, joe schmo wrote:

<...>

> 1) I have a copy of rsyslog 2.0.6 up and running without any issues (as it
> is the only one I can currently get working...more on that in a sec), other
> than the fact that it can't receive udp messages without manual
> intervention. The only way I can get it to accept udp (network) syslog
> messages is to stop the service, and then run the rsyslog -r514 from
> command-line manually. Is there anyway to get it to use this option by
> default? I need it to start the service with that option enabled. And the
> "just run it manually" answers I have received in other forums aren't
> acceptable for me.

You can add the command-line options to /etc/sysconfig/rsyslog, e.g.
SYSLOGD_OPTIONS="-m 0 -r512"

> 2) An extension of the above question: when I try to do the "obvious"
> solution to the udp issue and use the imudp module, I realized there IS NO
> imudp.so module in the plugins folder. Where would I even get this module
> for 2.0.6? The most I can find is a copy of imudp.c and its accompanying
> make files, but trying to make only outputs errors associated with missing
> files that belong to the C developers package (which I had installed some
> time ago off an official CentOS repo, no updates available).

You don't need this module (and there isn't any) for v2.

Tomas


From dag at wieers.com  Fri Mar 19 13:50:50 2010
From: dag at wieers.com (Dag Wieers)
Date: Fri, 19 Mar 2010 13:50:50 +0100 (CET)
Subject: [rsyslog] CentOS Issues
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103AFE@GRFEXC.intern.adiscon.com>
References: <a60a2fa91003181914kd27e298se6a8e3913563e4ca@mail.gmail.com><alpine.DEB.2.00.1003182125350.3821@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7103AFD@GRFEXC.intern.adiscon.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103AFE@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.LRH.2.00.1003191341470.2470@pikachu.3ti.be>

On Fri, 19 Mar 2010, Rainer Gerhards wrote:

>>> when compiling the new version you should be able to just do make
>>> install
>>> after you have compiled it to install it.
>>
>> As far as I know, you need to set --libdir and --bindir (or its 64 bit
>> equivalents) ./configure options on CentOS, but I don't know what the
>> correct
>> values are for it.
>
> I just discovered there is some good info on CentOS in the wiki:
>
> http://wiki.rsyslog.com/index.php/Rsyslog_on_CentOS_success_story
>
> I am not sure if it is too much in-depth, though...

Also take notice of the fact that RHEL5.5 Beta includes rsyslog 3.22 
because of previously reported issues:

 	https://bugzilla.redhat.com/show_bug.cgi?id=519192
 	https://bugzilla.redhat.com/show_bug.cgi?id=519201
 	https://bugzilla.redhat.com/show_bug.cgi?id=519203

We are happy that Red Hat support worked with us in resolving our 
issues and are pleased with this outcome.

These updates are also expected in the upcoming CentOS releases.
-- 
--   dag wieers,  dag at wieers.com,  http://dag.wieers.com/   --
[Any errors in spelling, tact or fact are transmission errors]


From mangoo at wpkg.org  Fri Mar 19 14:30:47 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Fri, 19 Mar 2010 14:30:47 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <alpine.DEB.2.00.1003182119540.3821@asgard.lang.hm>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org> <4BA2C8F9.4050607@wpkg.org>
	<alpine.DEB.2.00.1003182119540.3821@asgard.lang.hm>
Message-ID: <4BA37C87.8030306@wpkg.org>

Am 19.03.2010 05:20, david at lang.hm wrote:

>> If it's ' - it doesn't like the rule.
>
> when all else fails, look at the documentation ;-)
>
> http://www.rsyslog.com/doc-rsyslog_conf_filter.html
>
> the line should have been
>
> :programname, isequal, 'postgres' ~

The correct entries should be (" works, ' - doesn't):

:programname, isequal, "postgres" ~
*.*;auth,authpriv.none		-/var/log/syslog


Much thanks for your help!


-- 
Tomasz Chmielewski
http://wpkg.org




From mangoo at wpkg.org  Fri Mar 19 14:31:50 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Fri, 19 Mar 2010 14:31:50 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <alpine.DEB.2.00.1003182120550.3821@asgard.lang.hm>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org>
	<alpine.DEB.2.00.1003181652290.10411@asgard.lang.hm>
	<4BA2BE2D.9020205@wpkg.org>
	<alpine.DEB.2.00.1003181706510.10411@asgard.lang.hm>
	<4BA2C875.4060208@wpkg.org>
	<alpine.DEB.2.00.1003182120550.3821@asgard.lang.hm>
Message-ID: <4BA37CC6.4030203@wpkg.org>

Am 19.03.2010 05:22, david at lang.hm wrote:
> On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:
>
>> Am 19.03.2010 01:08, david at lang.hm wrote:
>>
>>>> where postgres[number] can differ.
>>>
>>> hmm, in these cases postgres should be what's in programname
>>>
>>> taking a different slant, IIRC you used local0.* to capture the postgres
>>> logs into one file, if in the line after that you do
>>>
>>> local0.* ~
>>>
>>> that should eliminate those logs from going into any files after that.
>>
>> This one works, but that'll filter out heartbeat, pgpool and others
>> useful, which get to syslog.
>
> I will point out that those entries are getting put into your
> posgresql_log file along with the postgres logs then.

Yep, but I'm fine with that (they also concern Postgres on this machine, 
but are counted in kilobytes per day, not in several hundred megabytes).


-- 
Tomasz Chmielewski
http://wpkg.org


From sledz at dresearch.de  Fri Mar 19 16:32:02 2010
From: sledz at dresearch.de (Steffen Sledz)
Date: Fri, 19 Mar 2010 16:32:02 +0100
Subject: [rsyslog] patch for architectures without atomic builtins
In-Reply-To: <alpine.DEB.2.00.1003181555170.10411@asgard.lang.hm>
References: <DB561678F08F554EB0D9F21B70E268A505AE8B@exchange.intern.dresearch.de>
	<alpine.DEB.2.00.1003181555170.10411@asgard.lang.hm>
Message-ID: <4BA398F2.4080704@dresearch.de>

david at lang.hm write:
> can you re-send with the patch inline?

I've seen that you use git for rsyslog. I can send our
patch with git-send-email if you have an email address
for it. Or should i use the mailing list address for
this?

Steffen



From david at lang.hm  Fri Mar 19 16:49:37 2010
From: david at lang.hm (david at lang.hm)
Date: Fri, 19 Mar 2010 08:49:37 -0700 (PDT)
Subject: [rsyslog] patch for architectures without atomic builtins
In-Reply-To: <4BA398F2.4080704@dresearch.de>
References: <DB561678F08F554EB0D9F21B70E268A505AE8B@exchange.intern.dresearch.de>
	<alpine.DEB.2.00.1003181555170.10411@asgard.lang.hm>
	<4BA398F2.4080704@dresearch.de>
Message-ID: <alpine.DEB.2.00.1003190847250.3821@asgard.lang.hm>

On Fri, 19 Mar 2010, Steffen Sledz wrote:

> david at lang.hm write:
>> can you re-send with the patch inline?
>
> I've seen that you use git for rsyslog. I can send our
> patch with git-send-email if you have an email address
> for it. Or should i use the mailing list address for
> this?

Rainer is the person who needs to get the patch, I think I saw him respond 
to your other e-mail.

the mailing list address is probably the best for this sort of thing, or 
if you have a git repository that's pubic, you can make a branch there for 
this and send it's address as a pull request.

David Lang


From sledz at dresearch.de  Fri Mar 19 16:56:38 2010
From: sledz at dresearch.de (Steffen Sledz)
Date: Fri, 19 Mar 2010 16:56:38 +0100
Subject: [rsyslog] [PATCH] emulate missing atomic builtins with posix
	semaphores if available
In-Reply-To: <alpine.DEB.2.00.1003190847250.3821@asgard.lang.hm>
References: <alpine.DEB.2.00.1003190847250.3821@asgard.lang.hm>
Message-ID: <1269014198-1533-1-git-send-email-sledz@dresearch.de>

Signed-off-by: Steffen Sledz <sledz at dresearch.de>
Acked-by: Thilo Fromm <t.fromm at dresearch.de>
---
 configure.ac               |    4 +-
 runtime/Makefile.am        |    1 +
 runtime/atomic-posix-sem.c |   70 ++++++++++++++++++++++++++
 runtime/atomic.h           |  117 ++++++++++++++++++++++++++++++++++++++++++++
 runtime/rsyslog.c          |   14 +++++
 5 files changed, 205 insertions(+), 1 deletions(-)
 create mode 100644 runtime/atomic-posix-sem.c

diff --git a/configure.ac b/configure.ac
index c59895d..ca198ea 100644
--- a/configure.ac
+++ b/configure.ac
@@ -87,6 +87,7 @@ AC_TYPE_UINT8_T
 AC_HEADER_TIME
 AC_STRUCT_TM
 AC_C_VOLATILE
+AC_C_TYPEOF
 
 sa_includes="\
 $ac_includes_default
@@ -129,7 +130,8 @@ AC_TRY_COMPILE([
 # check for availability of atomic operations
 RS_ATOMIC_OPERATIONS
 RS_ATOMIC_OPERATIONS_64BIT
-
+# fall back to POSIX sems for atomic operations (cpu expensive)
+AC_CHECK_HEADERS([semaphore.h])
 
 # Additional module directories
 AC_ARG_WITH(moddirs,
diff --git a/runtime/Makefile.am b/runtime/Makefile.am
index 9047c83..2374c57 100644
--- a/runtime/Makefile.am
+++ b/runtime/Makefile.am
@@ -9,6 +9,7 @@ librsyslog_la_SOURCES = \
 	rsyslog.h \
 	unicode-helper.h \
 	atomic.h \
+	atomic-posix-sem.c \
 	batch.h \
 	syslogd-types.h \
 	module-template.h \
diff --git a/runtime/atomic-posix-sem.c b/runtime/atomic-posix-sem.c
new file mode 100644
index 0000000..979fae0
--- /dev/null
+++ b/runtime/atomic-posix-sem.c
@@ -0,0 +1,70 @@
+/* atomic_posix_sem.c: This file supplies an emulation for atomic operations using
+ * POSIX semaphores.
+ *
+ * Copyright 2010 DResearch Digital Media Systems GmbH
+ *
+ * This file is part of the rsyslog runtime library.
+ *
+ * The rsyslog runtime library is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * The rsyslog runtime library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public License
+ * along with the rsyslog runtime library.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * A copy of the GPL can be found in the file "COPYING" in this distribution.
+ * A copy of the LGPL can be found in the file "COPYING.LESSER" in this distribution.
+ */
+
+#include "config.h"
+#ifndef HAVE_ATOMIC_BUILTINS
+#ifdef HAVE_SEMAPHORE_H
+#include <semaphore.h>
+#include <errno.h>
+
+#include "atomic.h"
+#include "rsyslog.h"
+#include "srUtils.h"
+
+sem_t atomicSem;
+
+rsRetVal
+atomicSemInit(void)
+{
+	DEFiRet;
+
+	dbgprintf("init posix semaphore for atomics emulation\n");
+	if(sem_init(&atomicSem, 0, 1) == -1)
+	{
+		char errStr[1024];
+		rs_strerror_r(errno, errStr, sizeof(errStr));
+		dbgprintf("init posix semaphore for atomics emulation failed: %s\n", errStr);
+		iRet = RS_RET_SYS_ERR; /* the right error code ??? */
+	}
+
+	RETiRet;
+}
+
+void
+atomicSemExit(void)
+{
+	dbgprintf("destroy posix semaphore for atomics emulation\n");
+	if(sem_destroy(&atomicSem) == -1)
+	{
+		char errStr[1024];
+		rs_strerror_r(errno, errStr, sizeof(errStr));
+		dbgprintf("destroy posix semaphore for atomics emulation failed: %s\n", errStr);
+	}
+}
+
+#endif /* HAVE_SEMAPHORE_H */
+#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
+
+/* vim:set ai:
+ */
diff --git a/runtime/atomic.h b/runtime/atomic.h
index b507b76..51b14e1 100644
--- a/runtime/atomic.h
+++ b/runtime/atomic.h
@@ -54,6 +54,122 @@
 #	define ATOMIC_CAS(data, oldVal, newVal) __sync_bool_compare_and_swap(&(data), (oldVal), (newVal));
 #	define ATOMIC_CAS_VAL(data, oldVal, newVal) __sync_val_compare_and_swap(&(data), (oldVal), (newVal));
 #else
+#ifdef HAVE_SEMAPHORE_H
+	/* we use POSIX semaphores instead */
+
+#include "rsyslog.h"
+#include <semaphore.h>
+
+extern sem_t atomicSem;
+rsRetVal atomicSemInit(void);
+void atomicSemExit(void);
+
+#if HAVE_TYPEOF
+#define my_typeof(x) typeof(x)
+#else /* sorry, can't determine types, using 'int' */
+#define my_typeof(x) int
+#endif
+
+#	define ATOMIC_SUB(data, val) \
+({ \
+	my_typeof(data) tmp; \
+	sem_wait(&atomicSem); \
+	tmp = data; \
+	data -= val; \
+	sem_post(&atomicSem); \
+	tmp; \
+})
+
+#	define ATOMIC_ADD(data, val) \
+({ \
+	my_typeof(data) tmp; \
+	sem_wait(&atomicSem); \
+	tmp = data; \
+	data += val; \
+	sem_post(&atomicSem); \
+	tmp; \
+})
+
+#	define ATOMIC_INC_AND_FETCH(data) \
+({ \
+	my_typeof(data) tmp; \
+	sem_wait(&atomicSem); \
+	tmp = data; \
+	data += 1; \
+	sem_post(&atomicSem); \
+	tmp; \
+})
+
+#	define ATOMIC_INC(data) ((void) ATOMIC_INC_AND_FETCH(data))
+
+#	define ATOMIC_DEC_AND_FETCH(data) \
+({ \
+	sem_wait(&atomicSem); \
+	data -= 1; \
+	sem_post(&atomicSem); \
+	data; \
+})
+
+#	define ATOMIC_DEC(data) ((void) ATOMIC_DEC_AND_FETCH(data))
+
+#	define ATOMIC_FETCH_32BIT(data) ((unsigned) ATOMIC_ADD((data), 0xffffffff))
+
+#	define ATOMIC_STORE_1_TO_32BIT(data) \
+({ \
+	my_typeof(data) tmp; \
+	sem_wait(&atomicSem); \
+	tmp = data; \
+	data = 1; \
+	sem_post(&atomicSem); \
+	tmp; \
+})
+
+#	define ATOMIC_STORE_0_TO_INT(data) \
+({ \
+	my_typeof(data) tmp; \
+	sem_wait(&atomicSem); \
+	tmp = data; \
+	data = 0; \
+	sem_post(&atomicSem); \
+	tmp; \
+})
+
+#	define ATOMIC_STORE_1_TO_INT(data) \
+({ \
+	my_typeof(data) tmp; \
+	sem_wait(&atomicSem); \
+	tmp = data; \
+	data = 1; \
+	sem_post(&atomicSem); \
+	tmp; \
+})
+
+#	define ATOMIC_CAS(data, oldVal, newVal) \
+({ \
+	int ret; \
+	sem_wait(&atomicSem); \
+	if(data != oldVal) ret = 0; \
+	else \
+	{ \
+		data = newVal; \
+		ret = 1; \
+	} \
+	sem_post(&atomicSem); \
+	ret; \
+})
+
+#	define ATOMIC_CAS_VAL(data, oldVal, newVal) \
+({ \
+	sem_wait(&atomicSem); \
+	if(data == oldVal) \
+	{ \
+		data = newVal; \
+	} \
+	sem_post(&atomicSem); \
+	data; \
+})
+
+#else /* not HAVE_SEMAPHORE_H */
 	/* note that we gained parctical proof that theoretical problems DO occur
 	 * if we do not properly address them. See this blog post for details:
 	 * http://blog.gerhards.net/2009/01/rsyslog-data-race-analysis.html
@@ -68,5 +184,6 @@
 #	define ATOMIC_FETCH_32BIT(data) (data)
 #	define ATOMIC_STORE_1_TO_32BIT(data) (data) = 1
 #endif
+#endif
 
 #endif /* #ifndef INCLUDED_ATOMIC_H */
diff --git a/runtime/rsyslog.c b/runtime/rsyslog.c
index a76ae25..921ad0b 100644
--- a/runtime/rsyslog.c
+++ b/runtime/rsyslog.c
@@ -81,6 +81,7 @@
 #include "rule.h"
 #include "ruleset.h"
 #include "parser.h"
+#include "atomic.h"
 
 /* forward definitions */
 static rsRetVal dfltErrLogger(int, uchar *errMsg);
@@ -140,6 +141,12 @@ rsrtInit(char **ppErrObj, obj_if_t *pObjIF)
 		CHKiRet(objClassInit(NULL)); /* *THIS* *MUST* always be the first class initilizer being called! */
 		CHKiRet(objGetObjInterface(pObjIF)); /* this provides the root pointer for all other queries */
 
+#ifndef HAVE_ATOMIC_BUILTINS
+#ifdef HAVE_SEMAPHORE_H
+		CHKiRet(atomicSemInit());
+#endif /* HAVE_SEMAPHORE_H */
+#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
+
 		/* initialize core classes. We must be very careful with the order of events. Some
 		 * classes use others and if we do not initialize them in the right order, we may end
 		 * up with an invalid call. The most important thing that can happen is that an error
@@ -216,6 +223,13 @@ rsrtExit(void)
 		glblClassExit();
 		rulesetClassExit();
 		ruleClassExit();
+
+#ifndef HAVE_ATOMIC_BUILTINS
+#ifdef HAVE_SEMAPHORE_H
+		atomicSemExit();
+#endif /* HAVE_SEMAPHORE_H */
+#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
+
 		objClassExit(); /* *THIS* *MUST/SHOULD?* always be the first class initilizer being called (except debug)! */
 	}
 
-- 
1.6.4.2



From rgerhards at hq.adiscon.com  Tue Mar 23 11:22:29 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 23 Mar 2010 11:22:29 +0100
Subject: [rsyslog] patch for architectures without atomic builtins
In-Reply-To: <4BA398F2.4080704@dresearch.de>
References: <DB561678F08F554EB0D9F21B70E268A505AE8B@exchange.intern.dresearch.de>
	<alpine.DEB.2.00.1003181555170.10411@asgard.lang.hm>
	<4BA398F2.4080704@dresearch.de>
Message-ID: <1269339749.3320.0.camel@localhost>

thanks again for the contribution, I have just applied it to the master
branch. I will see if I can backport it to v4, what I think would be
useful.

Rainer

On Fri, 2010-03-19 at 16:32 +0100, Steffen Sledz wrote:
> david at lang.hm write:
> > can you re-send with the patch inline?
> 
> I've seen that you use git for rsyslog. I can send our
> patch with git-send-email if you have an email address
> for it. Or should i use the mailing list address for
> this?
> 
> Steffen
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com




From mikolaj at kucharski.name  Thu Mar 25 22:08:43 2010
From: mikolaj at kucharski.name (Mikolaj Kucharski)
Date: Thu, 25 Mar 2010 21:08:43 +0000
Subject: [rsyslog] last message repeated n times from remote hosts
In-Reply-To: <20100210120230.GI3838@x40.openbsd.home.lan>
References: <20100210120230.GI3838@x40.openbsd.home.lan>
Message-ID: <20100325210843.GK3838@x40.openbsd.home.lan>

Any comments?

On Wed, Feb 10, 2010 at 12:02:30PM +0000, Mikolaj Kucharski wrote:
> Hi,
> 
> I have few Linux based machines with sysklogd installed, and I have
> central syslog server based on CentOS 5 with rsyslog-2.0.6-1.el5.
> 
> I have issue with missing hostname when sysklog sends "last message
> repeated N times" over the wire to rsyslog server.
> 
> Let's have a look. I used logger(1) to repeatedly sent one message few
> times followed by one different message. Here is what I see in the log
> file on my central rsyslog server:
> 
> Feb 10 11:39:46 10.101.43.124 root: remote test start
> Feb 10 11:39:54 last message repeated 14 times
> Feb 10 11:39:54 10.101.43.124 root: remote test end
> 
> 
> and here is tcpdump(8) log from the source (10.101.43.124) machine:
> 
> 
> 11:39:46.642297 IP 10.101.43.124.syslog > 10.101.40.116.syslog: SYSLOG user.notice, length: 28
>         0x0000:  4500 0038 0000 4000 4011 d1fb 0a65 2b7c  E..8.. at .@....e+|
>         0x0010:  0a65 2874 0202 0202 0024 68ef 3c31 333e  .e(t.....$h.<13>
>         0x0020:  726f 6f74 3a20 7265 6d6f 7465 2074 6573  root:.remote.tes
>         0x0030:  7420 7374 6172 740a                      t.start.
> 11:39:54.904820 IP 10.101.43.124.syslog > 10.101.40.116.syslog: SYSLOG user.notice, length: 35
>         0x0000:  4500 003f 0000 4000 4011 d1f4 0a65 2b7c  E..?.. at .@....e+|
>         0x0010:  0a65 2874 0202 0202 002b 68f6 3c31 333e  .e(t.....+h.<13>
>         0x0020:  6c61 7374 206d 6573 7361 6765 2072 6570  last.message.rep
>         0x0030:  6561 7465 6420 3134 2074 696d 6573 0a    eated.14.times.
> 11:39:54.904826 IP 10.101.43.124.syslog > 10.101.40.116.syslog: SYSLOG user.notice, length: 26
>         0x0000:  4500 0036 0000 4000 4011 d1fd 0a65 2b7c  E..6.. at .@....e+|
>         0x0010:  0a65 2874 0202 0202 0022 68ed 3c31 333e  .e(t....."h.<13>
>         0x0020:  726f 6f74 3a20 7265 6d6f 7465 2074 6573  root:.remote.tes
>         0x0030:  7420 656e 640a                           t.end.
> 
> 
> I searched the list, and saw a comment which say the fault is on the
> sysklogd end as it never sends hostname in the repeated-n-times packet,
> but from above I cannot see that it ever sends the packet with hostname,
> so I think the issues is on rsyslog side, and not on the sysklogd.
> 
> Could someone shed some light on my issue, as I would like to see all
> the time the source IP or hostname of incomming messages to rsyslog
> daemon.
> 
> Is this missing source hostname/IP a bug of rsyslog?
> 
> Is there any way to workaround that?
> 
> 
> Thanks.
> 
> 
> PS1. I cannot change client machines, I cannot reinstall them with
> different syslog implementation, the only machine where I have
> permission to do modifications is central rsyslog server.
> 
> PS2. I know about DNS and RevDNS and yes, above server doesn't have
> revDNS setup.

-- 
best regards
q#


From pablo at docecosas.com  Fri Mar 26 13:02:41 2010
From: pablo at docecosas.com (Pablo Martinez Schroder)
Date: Fri, 26 Mar 2010 13:02:41 +0100
Subject: [rsyslog] Rsyslog is not using FQDN in all messages
Message-ID: <fe120686aaa393fb1425df1a5fc29530.squirrel@webmail.docecosas.com>

Hi,

I'm using rsyslog-5.4.0 and I having a issue that I cannot understand. I
want to implement a central logging server using stunnel, and I need to
use FQDN always, I don't want to have HOSTNAME trunked to the hostname so
I enable $PreserveFQDN but only some of the messages seems to use the
whole hostname. I've seen this issue in multiple versions of rsyslog, so
I'm quite sure is not caused by rsyslog-5.4.0

I've tested on multiples versions of CentOS and currently I'm testing
rsyslog-5.4.0 on a CentOS 5.4. To to the tests I'm logging all the
messages in a local file, so the interesting part of my /etc/rsyslog.conf
is like this:

$ModLoad immark   # provides --MARK-- message capability
$ModLoad imuxsock # provides support for local system logging (e.g. via
logger command)
$ModLoad imklog   # kernel logging (formerly provided by rklogd)

$PreserveFQDN on

*.* /var/log/everything.log

I start rsyslogd with the "-c 4" option, and if I enable $PreserveFQDN in
the conf messages from kernel and rsyslogd are saved with the proper FQDN
but the rest of the messages are stored as if they were originated from
the hostname, without domain.

# tail /var/log/everything.log
010-03-26T07:51:29.513679-04:00 syslog-test.scrambled.com kernel: imklog
5.4.0, log source = /proc/kmsg started.
2010-03-26T07:51:29.684129-04:00 syslog-test.scrambled.com rsyslogd:
[origin software="rsyslogd" swVersion="5.4.0" x-pid="1977"
x-info="http://www.rsyslog.com"] start
2010-03-26T07:51:29.707163-04:00 syslog-test stunnel:
LOG5[1508:1099114816]: ssyslog connected from 127.0.0.1:35446
2010-03-26T07:51:30.610043-04:00 syslog-test.scrambled.com kernel: Kernel
logging (proc) stopped.
2010-03-26T07:51:30.610519-04:00 syslog-test.scrambled.com rsyslogd:
[origin software="rsyslogd" swVersion="5.4.0" x-pid="1977"
x-info="http://www.rsyslog.com"] exiting on signal 2.

In this case, the only one using "syslog-test" (without the
".scrambled.com" part is stunnel, but all the logs generated from sshd,
pam, cron, etc are generated from "syslog-test" instead of
"syslog-test.scrambled.com".

I only have localhost in the hosts file and if I run hostname I get
"syslog-test.scrambled.com". I even rebooted the server after changing
$PreserveFQDN, just in case it was some weird thing of applications
'caching' the simple hostname (it doesn't make sense, I know). Is there
anything that I'm doing wrong? Is it possible for rsyslog to use the FQDN
for all the logs?

I believe that I need that the FQDN is used locally for it to work with a
remote server (so the HOSTNAME variabvle in the template has the FQDN)

Many thanks in advance.
-- 
Pablo Martinez Schroder




From tbergfeld at hq.adiscon.com  Fri Mar 26 16:04:56 2010
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Fri, 26 Mar 2010 16:04:56 +0100
Subject: [rsyslog] rsyslog 4.6.2 (v4-stable) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103B99@GRFEXC.intern.adiscon.com>

Hi all,

We have just released rsyslog 4.6.2, a member of the v4-stable branch. This
is a bug-fixing release containing some important fixes and also two new
features. It is a recommended update for all users of the v4-stable. See
Changelog for more details.

ChangeLog:

http://www.rsyslog.com/Article449.phtml

Download:

http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-198.phtml


As always, feedback is appreciated.

Best regards,
Tom Bergfeld
--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com


From david at lang.hm  Fri Mar 26 19:19:02 2010
From: david at lang.hm (david at lang.hm)
Date: Fri, 26 Mar 2010 11:19:02 -0700 (PDT)
Subject: [rsyslog] Rsyslog is not using FQDN in all messages
In-Reply-To: <fe120686aaa393fb1425df1a5fc29530.squirrel@webmail.docecosas.com>
References: <fe120686aaa393fb1425df1a5fc29530.squirrel@webmail.docecosas.com>
Message-ID: <alpine.DEB.2.00.1003261116320.12764@asgard.lang.hm>

My first question would be if the logs have the FQDN in them in the first 
place.

you may want to create a log with the format %raw% and see what is 
arriving to your box.

David Lang

On Fri, 26 Mar 2010, Pablo Martinez Schroder wrote:

> Hi,
>
> I'm using rsyslog-5.4.0 and I having a issue that I cannot understand. I
> want to implement a central logging server using stunnel, and I need to
> use FQDN always, I don't want to have HOSTNAME trunked to the hostname so
> I enable $PreserveFQDN but only some of the messages seems to use the
> whole hostname. I've seen this issue in multiple versions of rsyslog, so
> I'm quite sure is not caused by rsyslog-5.4.0
>
> I've tested on multiples versions of CentOS and currently I'm testing
> rsyslog-5.4.0 on a CentOS 5.4. To to the tests I'm logging all the
> messages in a local file, so the interesting part of my /etc/rsyslog.conf
> is like this:
>
> $ModLoad immark   # provides --MARK-- message capability
> $ModLoad imuxsock # provides support for local system logging (e.g. via
> logger command)
> $ModLoad imklog   # kernel logging (formerly provided by rklogd)
>
> $PreserveFQDN on
>
> *.* /var/log/everything.log
>
> I start rsyslogd with the "-c 4" option, and if I enable $PreserveFQDN in
> the conf messages from kernel and rsyslogd are saved with the proper FQDN
> but the rest of the messages are stored as if they were originated from
> the hostname, without domain.
>
> # tail /var/log/everything.log
> 010-03-26T07:51:29.513679-04:00 syslog-test.scrambled.com kernel: imklog
> 5.4.0, log source = /proc/kmsg started.
> 2010-03-26T07:51:29.684129-04:00 syslog-test.scrambled.com rsyslogd:
> [origin software="rsyslogd" swVersion="5.4.0" x-pid="1977"
> x-info="http://www.rsyslog.com"] start
> 2010-03-26T07:51:29.707163-04:00 syslog-test stunnel:
> LOG5[1508:1099114816]: ssyslog connected from 127.0.0.1:35446
> 2010-03-26T07:51:30.610043-04:00 syslog-test.scrambled.com kernel: Kernel
> logging (proc) stopped.
> 2010-03-26T07:51:30.610519-04:00 syslog-test.scrambled.com rsyslogd:
> [origin software="rsyslogd" swVersion="5.4.0" x-pid="1977"
> x-info="http://www.rsyslog.com"] exiting on signal 2.
>
> In this case, the only one using "syslog-test" (without the
> ".scrambled.com" part is stunnel, but all the logs generated from sshd,
> pam, cron, etc are generated from "syslog-test" instead of
> "syslog-test.scrambled.com".
>
> I only have localhost in the hosts file and if I run hostname I get
> "syslog-test.scrambled.com". I even rebooted the server after changing
> $PreserveFQDN, just in case it was some weird thing of applications
> 'caching' the simple hostname (it doesn't make sense, I know). Is there
> anything that I'm doing wrong? Is it possible for rsyslog to use the FQDN
> for all the logs?
>
> I believe that I need that the FQDN is used locally for it to work with a
> remote server (so the HOSTNAME variabvle in the template has the FQDN)
>
> Many thanks in advance.
>


From pablo at docecosas.com  Fri Mar 26 19:28:16 2010
From: pablo at docecosas.com (Pablo Martinez Schroder)
Date: Fri, 26 Mar 2010 19:28:16 +0100
Subject: [rsyslog] Rsyslog is not using FQDN in all messages
In-Reply-To: <alpine.DEB.2.00.1003261116320.12764@asgard.lang.hm>
References: <fe120686aaa393fb1425df1a5fc29530.squirrel@webmail.docecosas.com>
	<alpine.DEB.2.00.1003261116320.12764@asgard.lang.hm>
Message-ID: <bd858f6056203841b7c277706d5f5106.squirrel@webmail.docecosas.com>


> My first question would be if the logs have the FQDN in them in the first
> place.
>
> you may want to create a log with the format %raw% and see what is
> arriving to your box.

I'm currently testing this issue on local, with the configuration below,
and the logs saved are a mix: some of them have a hostname like
syslog-test.scrambled.com (only the logs generated from kernel and
rsyslog) and all the other lines publish syslog-test as hostname.

$PreserveFQDN on

*.* /var/log/everything.log

>> I'm using rsyslog-5.4.0 and I having a issue that I cannot understand. I
>> want to implement a central logging server using stunnel, and I need to
>> use FQDN always, I don't want to have HOSTNAME trunked to the hostname
>> so
>> I enable $PreserveFQDN but only some of the messages seems to use the
>> whole hostname. I've seen this issue in multiple versions of rsyslog, so
>> I'm quite sure is not caused by rsyslog-5.4.0
>>
>> I start rsyslogd with the "-c 4" option, and if I enable $PreserveFQDN
>> in
>> the conf messages from kernel and rsyslogd are saved with the proper
>> FQDN
>> but the rest of the messages are stored as if they were originated from
>> the hostname, without domain.

-- 
Pablo Martinez Schroder



From ralph at crongeyer.com  Sun Mar 28 16:29:20 2010
From: ralph at crongeyer.com (Ralph Crongeyer)
Date: Sun, 28 Mar 2010 10:29:20 -0400
Subject: [rsyslog] High cpu usage - rsyslog 5.4.0.
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>
References: <5fa6e0144d8003c7c72edff17f9f1675@webmail.crongeyer.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>
Message-ID: <4BAF67C0.4000506@crongeyer.com>

List,
I tried 5.4.0 last weekend on two machines and ran it for almost a week. 
During that time I noticed that every day when I got home from work both 
machines server/log collector and my client machines fans were racing 
(both laptops, Compaq Armada M700's) ? I logged in ran top and found 
rsyslog consuming 99% of the cpu on both machines? After I restarted 
rsyslog on both they seemed to run normal. The next day the same thing. 
By this time I was watching more closely. I noticed that it took about 
an 45 to 60 minutes and rsyslog would have consumed 95 to 99% of the cpu 
on both machines?

I tried to reboot both machines but it would still happen. I have 
switched back to 5.3.7 which doesn't have this problem.

I'm running Debian Lenny on both machines.
I compiled rsyslog with this:
./configure --prefix=/usr --sysconfdir=/etc --enable-relp 
--enable-imtemplate --enable-omtemplate --enable-rfc3195 --enable-imdiag 
--enable-mail --enable-zlib --enable-mysql --enable-gnutls 
--enable-imdiag --enable-imfile --enable-omprog --enable-omudpspoof 
--enable-omstdout --enable-diagtools
make
checkinstall -D make install

I used the same commands to make 5.3.7 and 5.4.0.

Is anyone else seeing high cpu loads with 5.4.0?

Thanks,
Ralph

-- 
Reminds me of my expedition into the wilds of Afghanistan. We lost our 
corkscrew and were compelled to live on food and water for several days. - 
WC Fields



From mbiebl at gmail.com  Sun Mar 28 16:49:49 2010
From: mbiebl at gmail.com (Michael Biebl)
Date: Sun, 28 Mar 2010 16:49:49 +0200
Subject: [rsyslog] High cpu usage - rsyslog 5.4.0.
In-Reply-To: <4BAF67C0.4000506@crongeyer.com>
References: <5fa6e0144d8003c7c72edff17f9f1675@webmail.crongeyer.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>
	<4BAF67C0.4000506@crongeyer.com>
Message-ID: <aa76cdac1003280749p4673e404t7e5aefd63e102348@mail.gmail.com>

2010/3/28 Ralph Crongeyer <ralph at crongeyer.com>:
> I'm running Debian Lenny on both machines.

> Is anyone else seeing high cpu loads with 5.4.0?
>

I can confirm his (on Debian unstable) and already notified Rainer
about this particular issue.
Some preliminary testing here seems to indicate it's an issue with pipes.

When I removed
daemon.*;mail.*;\
 news.err;\
 *.=debug;*.=info;\
 *.=notice;*.=warn      |/dev/xconsole

from my rsyslog.conf, I could no longer reproduce the problem.

Would be interesting to know, if you can confirm this.

Cheers,
Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From ralph at crongeyer.com  Sun Mar 28 16:52:54 2010
From: ralph at crongeyer.com (Ralph Crongeyer)
Date: Sun, 28 Mar 2010 10:52:54 -0400
Subject: [rsyslog] High cpu usage - rsyslog 5.4.0.
In-Reply-To: <aa76cdac1003280749p4673e404t7e5aefd63e102348@mail.gmail.com>
References: <5fa6e0144d8003c7c72edff17f9f1675@webmail.crongeyer.com>	<9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>	<4BAF67C0.4000506@crongeyer.com>
	<aa76cdac1003280749p4673e404t7e5aefd63e102348@mail.gmail.com>
Message-ID: <4BAF6D46.9090009@crongeyer.com>

I'll do it right now. Just a second.

Thanks,
Ralph

Michael Biebl wrote:
> 2010/3/28 Ralph Crongeyer <ralph at crongeyer.com>:
>   
>> I'm running Debian Lenny on both machines.
>>     
>
>   
>> Is anyone else seeing high cpu loads with 5.4.0?
>>
>>     
>
> I can confirm his (on Debian unstable) and already notified Rainer
> about this particular issue.
> Some preliminary testing here seems to indicate it's an issue with pipes.
>
> When I removed
> daemon.*;mail.*;\
>  news.err;\
>  *.=debug;*.=info;\
>  *.=notice;*.=warn      |/dev/xconsole
>
> from my rsyslog.conf, I could no longer reproduce the problem.
>
> Would be interesting to know, if you can confirm this.
>
> Cheers,
> Michael
>
>   


-- 
Reminds me of my expedition into the wilds of Afghanistan. We lost our 
corkscrew and were compelled to live on food and water for several days. - 
WC Fields



From ralph at crongeyer.com  Sun Mar 28 17:09:02 2010
From: ralph at crongeyer.com (Ralph Crongeyer)
Date: Sun, 28 Mar 2010 11:09:02 -0400
Subject: [rsyslog] High cpu usage - rsyslog 5.4.0.
In-Reply-To: <aa76cdac1003280749p4673e404t7e5aefd63e102348@mail.gmail.com>
References: <5fa6e0144d8003c7c72edff17f9f1675@webmail.crongeyer.com>	<9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>	<4BAF67C0.4000506@crongeyer.com>
	<aa76cdac1003280749p4673e404t7e5aefd63e102348@mail.gmail.com>
Message-ID: <4BAF710E.8080506@crongeyer.com>

Ok,
I stoped rsyslog apt-get removed the 5.3.7 package, installed the 5.4.0 
package:

root at logs:/opt# aptitude show rsyslog
Package: rsyslog
State: installed
Automatically installed: no
Version: 5.4.0-1
Priority: extra
Section: extra
Maintainer: ralph at crongeyer.com
Uncompressed Size: 3047k
Depends: libc6 (>= 2.7-1), zlib1g (>= 1:1.1.4), lsb-base (>= 3.2-14)
Description: enhanced multi-threaded syslogd

commented out the lines from rsyslog.conf:
#daemon.*;mail.*;\
#       news.err;\
#       *.=debug;*.=info;\
#       *.=notice;*.=warn       |/dev/xconsole


and restarted rsyslog:
root at logs:/opt# /etc/init.d/rsyslog restart
Stopping enhanced syslogd: rsyslogd.
Starting enhanced syslogd: rsyslogd.

I'll get back to you in about an hour.

Thanks,
Ralph

Michael Biebl wrote:
> 2010/3/28 Ralph Crongeyer <ralph at crongeyer.com>:
>   
>> I'm running Debian Lenny on both machines.
>>     
>
>   
>> Is anyone else seeing high cpu loads with 5.4.0?
>>
>>     
>
> I can confirm his (on Debian unstable) and already notified Rainer
> about this particular issue.
> Some preliminary testing here seems to indicate it's an issue with pipes.
>
> When I removed
> daemon.*;mail.*;\
>  news.err;\
>  *.=debug;*.=info;\
>  *.=notice;*.=warn      |/dev/xconsole
>
> from my rsyslog.conf, I could no longer reproduce the problem.
>
> Would be interesting to know, if you can confirm this.
>
> Cheers,
> Michael
>
>   


-- 
Reminds me of my expedition into the wilds of Afghanistan. We lost our 
corkscrew and were compelled to live on food and water for several days. - 
WC Fields



From ralph at crongeyer.com  Sun Mar 28 17:54:21 2010
From: ralph at crongeyer.com (Ralph Crongeyer)
Date: Sun, 28 Mar 2010 11:54:21 -0400
Subject: [rsyslog] rsyslog + mysql + base(snort)
Message-ID: <4BAF7BAD.8020502@crongeyer.com>

All,
I'm would like to try to have a setup like this. Have Snort log to 
rsyslog and then have rsyslog log to a mysql Base schema database.
I know that people use Barnyard and or Barnyard2 for this setup to 
offload the writing to mysql to barnyard so that barnyard could receive 
snort logs and spool them if necessary before writing to mysql should 
mysql not be able to keep up.
It seems to me that rsyslog's spooling capability could eliminate the 
need for barnyard.
How would one go about applying a (for lack of better words) particular 
database schema so that rsyslog could write to the base database?

Does anyone have any thoughts on this?

Thanks,
Ralph

-- 
Reminds me of my expedition into the wilds of Afghanistan. We lost our 
corkscrew and were compelled to live on food and water for several days. - 
WC Fields



From ralph at crongeyer.com  Sun Mar 28 18:44:35 2010
From: ralph at crongeyer.com (Ralph Crongeyer)
Date: Sun, 28 Mar 2010 12:44:35 -0400
Subject: [rsyslog] High cpu usage - rsyslog 5.4.0.
In-Reply-To: <4BAF710E.8080506@crongeyer.com>
References: <5fa6e0144d8003c7c72edff17f9f1675@webmail.crongeyer.com>	<9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>	<4BAF67C0.4000506@crongeyer.com>	<aa76cdac1003280749p4673e404t7e5aefd63e102348@mail.gmail.com>
	<4BAF710E.8080506@crongeyer.com>
Message-ID: <4BAF8773.1020000@crongeyer.com>

It's confirmed!

I've been running for over an hour and a half now and:
load average: 0.06, 0.06, 0.02

Looks good so far.

Let me know if you guys (Michael, Rainer) want me to run in debug mode 
without the lines commented out to see if I can catch the problem for 
more troubleshooting.

Thanks,
Ralph

Ralph Crongeyer wrote:
> Ok,
> I stoped rsyslog apt-get removed the 5.3.7 package, installed the 5.4.0 
> package:
>
> root at logs:/opt# aptitude show rsyslog
> Package: rsyslog
> State: installed
> Automatically installed: no
> Version: 5.4.0-1
> Priority: extra
> Section: extra
> Maintainer: ralph at crongeyer.com
> Uncompressed Size: 3047k
> Depends: libc6 (>= 2.7-1), zlib1g (>= 1:1.1.4), lsb-base (>= 3.2-14)
> Description: enhanced multi-threaded syslogd
>
> commented out the lines from rsyslog.conf:
> #daemon.*;mail.*;\
> #       news.err;\
> #       *.=debug;*.=info;\
> #       *.=notice;*.=warn       |/dev/xconsole
>
>
> and restarted rsyslog:
> root at logs:/opt# /etc/init.d/rsyslog restart
> Stopping enhanced syslogd: rsyslogd.
> Starting enhanced syslogd: rsyslogd.
>
> I'll get back to you in about an hour.
>
> Thanks,
> Ralph
>
> Michael Biebl wrote:
>   
>> 2010/3/28 Ralph Crongeyer <ralph at crongeyer.com>:
>>   
>>     
>>> I'm running Debian Lenny on both machines.
>>>     
>>>       
>>   
>>     
>>> Is anyone else seeing high cpu loads with 5.4.0?
>>>
>>>     
>>>       
>> I can confirm his (on Debian unstable) and already notified Rainer
>> about this particular issue.
>> Some preliminary testing here seems to indicate it's an issue with pipes.
>>
>> When I removed
>> daemon.*;mail.*;\
>>  news.err;\
>>  *.=debug;*.=info;\
>>  *.=notice;*.=warn      |/dev/xconsole
>>
>> from my rsyslog.conf, I could no longer reproduce the problem.
>>
>> Would be interesting to know, if you can confirm this.
>>
>> Cheers,
>> Michael
>>
>>   
>>     
>
>
>   


-- 
Reminds me of my expedition into the wilds of Afghanistan. We lost our 
corkscrew and were compelled to live on food and water for several days. - 
WC Fields



From david at lang.hm  Sun Mar 28 21:10:39 2010
From: david at lang.hm (david at lang.hm)
Date: Sun, 28 Mar 2010 12:10:39 -0700 (PDT)
Subject: [rsyslog] High cpu usage - rsyslog 5.4.0.
In-Reply-To: <4BAF67C0.4000506@crongeyer.com>
References: <5fa6e0144d8003c7c72edff17f9f1675@webmail.crongeyer.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>
	<4BAF67C0.4000506@crongeyer.com>
Message-ID: <alpine.DEB.2.00.1003281209061.10653@asgard.lang.hm>

On Sun, 28 Mar 2010, Ralph Crongeyer wrote:

> List,
> I tried 5.4.0 last weekend on two machines and ran it for almost a week.
> During that time I noticed that every day when I got home from work both
> machines server/log collector and my client machines fans were racing
> (both laptops, Compaq Armada M700's) ? I logged in ran top and found
> rsyslog consuming 99% of the cpu on both machines? After I restarted
> rsyslog on both they seemed to run normal. The next day the same thing.
> By this time I was watching more closely. I noticed that it took about
> an 45 to 60 minutes and rsyslog would have consumed 95 to 99% of the cpu
> on both machines?
>
> I tried to reboot both machines but it would still happen. I have
> switched back to 5.3.7 which doesn't have this problem.
>
> I'm running Debian Lenny on both machines.
> I compiled rsyslog with this:
> ./configure --prefix=/usr --sysconfdir=/etc --enable-relp
> --enable-imtemplate --enable-omtemplate --enable-rfc3195 --enable-imdiag
> --enable-mail --enable-zlib --enable-mysql --enable-gnutls
> --enable-imdiag --enable-imfile --enable-omprog --enable-omudpspoof
> --enable-omstdout --enable-diagtools
> make
> checkinstall -D make install
>
> I used the same commands to make 5.3.7 and 5.4.0.
>
> Is anyone else seeing high cpu loads with 5.4.0?

my ubuntu laptop ends up with hundreds of thousands of messages/sec and 
eats up 100% cpu with an error message about access not being allowed.

other than that I have had no problems with excessive CPU, even under 
heavy loads.

David Lang

> Thanks,
> Ralph
>
>


From david at lang.hm  Sun Mar 28 21:13:19 2010
From: david at lang.hm (david at lang.hm)
Date: Sun, 28 Mar 2010 12:13:19 -0700 (PDT)
Subject: [rsyslog] rsyslog + mysql + base(snort)
In-Reply-To: <4BAF7BAD.8020502@crongeyer.com>
References: <4BAF7BAD.8020502@crongeyer.com>
Message-ID: <alpine.DEB.2.00.1003281211250.10653@asgard.lang.hm>

I was just talking about snort logging friday with this in mind..

My understanding of the snort logs include a significant binary component. 
This would not be trivial to deal with in rsyslog without encoding it 
first.

David Lang

  On Sun, 28 Mar 2010, Ralph Crongeyer wrote:

> All,
> I'm would like to try to have a setup like this. Have Snort log to
> rsyslog and then have rsyslog log to a mysql Base schema database.
> I know that people use Barnyard and or Barnyard2 for this setup to
> offload the writing to mysql to barnyard so that barnyard could receive
> snort logs and spool them if necessary before writing to mysql should
> mysql not be able to keep up.
> It seems to me that rsyslog's spooling capability could eliminate the
> need for barnyard.
> How would one go about applying a (for lack of better words) particular
> database schema so that rsyslog could write to the base database?
>
> Does anyone have any thoughts on this?
>
> Thanks,
> Ralph
>
>


From rgerhards at hq.adiscon.com  Mon Mar 29 07:11:10 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 29 Mar 2010 07:11:10 +0200
Subject: [rsyslog] High cpu usage - rsyslog 5.4.0.
References: <5fa6e0144d8003c7c72edff17f9f1675@webmail.crongeyer.com>	<9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>	<4BAF67C0.4000506@crongeyer.com>	<aa76cdac1003280749p4673e404t7e5aefd63e102348@mail.gmail.com><4BAF710E.8080506@crongeyer.com>
	<4BAF8773.1020000@crongeyer.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103BAA@GRFEXC.intern.adiscon.com>

Hi,

thanks for the information. I have crafted a big patch package for v4 the
last two weeks. I will try to merge it in to v5 this week (I guess this will
be a bit harder than usual). It may fix the issue.

If you have a moment or two, it would be interesting to know if 5.5.2 has the
same problem.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Ralph Crongeyer
> Sent: Sunday, March 28, 2010 6:45 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] High cpu usage - rsyslog 5.4.0.
> 
> It's confirmed!
> 
> I've been running for over an hour and a half now and:
> load average: 0.06, 0.06, 0.02
> 
> Looks good so far.
> 
> Let me know if you guys (Michael, Rainer) want me to run in debug mode
> without the lines commented out to see if I can catch the problem for
> more troubleshooting.
> 
> Thanks,
> Ralph
> 
> Ralph Crongeyer wrote:
> > Ok,
> > I stoped rsyslog apt-get removed the 5.3.7 package, installed the
> 5.4.0
> > package:
> >
> > root at logs:/opt# aptitude show rsyslog
> > Package: rsyslog
> > State: installed
> > Automatically installed: no
> > Version: 5.4.0-1
> > Priority: extra
> > Section: extra
> > Maintainer: ralph at crongeyer.com
> > Uncompressed Size: 3047k
> > Depends: libc6 (>= 2.7-1), zlib1g (>= 1:1.1.4), lsb-base (>= 3.2-14)
> > Description: enhanced multi-threaded syslogd
> >
> > commented out the lines from rsyslog.conf:
> > #daemon.*;mail.*;\
> > #       news.err;\
> > #       *.=debug;*.=info;\
> > #       *.=notice;*.=warn       |/dev/xconsole
> >
> >
> > and restarted rsyslog:
> > root at logs:/opt# /etc/init.d/rsyslog restart
> > Stopping enhanced syslogd: rsyslogd.
> > Starting enhanced syslogd: rsyslogd.
> >
> > I'll get back to you in about an hour.
> >
> > Thanks,
> > Ralph
> >
> > Michael Biebl wrote:
> >
> >> 2010/3/28 Ralph Crongeyer <ralph at crongeyer.com>:
> >>
> >>
> >>> I'm running Debian Lenny on both machines.
> >>>
> >>>
> >>
> >>
> >>> Is anyone else seeing high cpu loads with 5.4.0?
> >>>
> >>>
> >>>
> >> I can confirm his (on Debian unstable) and already notified Rainer
> >> about this particular issue.
> >> Some preliminary testing here seems to indicate it's an issue with
> pipes.
> >>
> >> When I removed
> >> daemon.*;mail.*;\
> >>  news.err;\
> >>  *.=debug;*.=info;\
> >>  *.=notice;*.=warn      |/dev/xconsole
> >>
> >> from my rsyslog.conf, I could no longer reproduce the problem.
> >>
> >> Would be interesting to know, if you can confirm this.
> >>
> >> Cheers,
> >> Michael
> >>
> >>
> >>
> >
> >
> >
> 
> 
> --
> Reminds me of my expedition into the wilds of Afghanistan. We lost our
> corkscrew and were compelled to live on food and water for several
> days. -
> WC Fields
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Mon Mar 29 17:29:37 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 29 Mar 2010 17:29:37 +0200
Subject: [rsyslog] High cpu usage - rsyslog 5.4.0.
References: <5fa6e0144d8003c7c72edff17f9f1675@webmail.crongeyer.com>	<9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>	<4BAF67C0.4000506@crongeyer.com>	<aa76cdac1003280749p4673e404t7e5aefd63e102348@mail.gmail.com><4BAF710E.8080506@crongeyer.com><4BAF8773.1020000@crongeyer.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103BAA@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103BBC@GRFEXC.intern.adiscon.com>

I managed to merge the v4 changes into v5-stable. Can you pull from it's git
head (git.adiscon.com/git/rsyslog.gig) and check if it works? Or do you need
a release tarball.

Note that I need to finally fix an issue in v4 before (the released version
has an interim/work-around fix) before I can take up more work on v5, but
chances are not too bad that the issue is fixed.

The master branch will receive the new patches ASAP, but that merge is a
little bit more complicated.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Monday, March 29, 2010 7:11 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] High cpu usage - rsyslog 5.4.0.
> 
> Hi,
> 
> thanks for the information. I have crafted a big patch package for v4
> the
> last two weeks. I will try to merge it in to v5 this week (I guess this
> will
> be a bit harder than usual). It may fix the issue.
> 
> If you have a moment or two, it would be interesting to know if 5.5.2
> has the
> same problem.
> 
> Rainer
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Ralph Crongeyer
> > Sent: Sunday, March 28, 2010 6:45 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] High cpu usage - rsyslog 5.4.0.
> >
> > It's confirmed!
> >
> > I've been running for over an hour and a half now and:
> > load average: 0.06, 0.06, 0.02
> >
> > Looks good so far.
> >
> > Let me know if you guys (Michael, Rainer) want me to run in debug
> mode
> > without the lines commented out to see if I can catch the problem for
> > more troubleshooting.
> >
> > Thanks,
> > Ralph
> >
> > Ralph Crongeyer wrote:
> > > Ok,
> > > I stoped rsyslog apt-get removed the 5.3.7 package, installed the
> > 5.4.0
> > > package:
> > >
> > > root at logs:/opt# aptitude show rsyslog
> > > Package: rsyslog
> > > State: installed
> > > Automatically installed: no
> > > Version: 5.4.0-1
> > > Priority: extra
> > > Section: extra
> > > Maintainer: ralph at crongeyer.com
> > > Uncompressed Size: 3047k
> > > Depends: libc6 (>= 2.7-1), zlib1g (>= 1:1.1.4), lsb-base (>= 3.2-
> 14)
> > > Description: enhanced multi-threaded syslogd
> > >
> > > commented out the lines from rsyslog.conf:
> > > #daemon.*;mail.*;\
> > > #       news.err;\
> > > #       *.=debug;*.=info;\
> > > #       *.=notice;*.=warn       |/dev/xconsole
> > >
> > >
> > > and restarted rsyslog:
> > > root at logs:/opt# /etc/init.d/rsyslog restart
> > > Stopping enhanced syslogd: rsyslogd.
> > > Starting enhanced syslogd: rsyslogd.
> > >
> > > I'll get back to you in about an hour.
> > >
> > > Thanks,
> > > Ralph
> > >
> > > Michael Biebl wrote:
> > >
> > >> 2010/3/28 Ralph Crongeyer <ralph at crongeyer.com>:
> > >>
> > >>
> > >>> I'm running Debian Lenny on both machines.
> > >>>
> > >>>
> > >>
> > >>
> > >>> Is anyone else seeing high cpu loads with 5.4.0?
> > >>>
> > >>>
> > >>>
> > >> I can confirm his (on Debian unstable) and already notified Rainer
> > >> about this particular issue.
> > >> Some preliminary testing here seems to indicate it's an issue with
> > pipes.
> > >>
> > >> When I removed
> > >> daemon.*;mail.*;\
> > >>  news.err;\
> > >>  *.=debug;*.=info;\
> > >>  *.=notice;*.=warn      |/dev/xconsole
> > >>
> > >> from my rsyslog.conf, I could no longer reproduce the problem.
> > >>
> > >> Would be interesting to know, if you can confirm this.
> > >>
> > >> Cheers,
> > >> Michael
> > >>
> > >>
> > >>
> > >
> > >
> > >
> >
> >
> > --
> > Reminds me of my expedition into the wilds of Afghanistan. We lost
> our
> > corkscrew and were compelled to live on food and water for several
> > days. -
> > WC Fields
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From boonsiong0527 at gmail.com  Tue Mar 30 08:40:57 2010
From: boonsiong0527 at gmail.com (BoOnSiOnG)
Date: Tue, 30 Mar 2010 14:40:57 +0800
Subject: [rsyslog] imfile module
In-Reply-To: <mailman.230.1269930120.2883.rsyslog@lists.adiscon.com>
References: <mailman.230.1269930120.2883.rsyslog@lists.adiscon.com>
Message-ID: <bb92aee11003292340s7fc3ad29q8f143c251a492041@mail.gmail.com>

Hello all,

currently I'm using imfile module to monitor apache Tomcat
catalina.out, I noticed rsyslog will append date time and machine name
to every line, is it possible to configure rsyslog do not alter the
original log?

Mar 29 04:13:08 rsyslogd : INFO: Using <Listener
className="org.apache.jk.config.ApacheConfig"
modJk="PATH_TO_MOD_JK.SO_OR_DLL" />
Mar 29 04:13:08 rsyslogd : Mar 29, 2010 4:13:08 AM
org.apache.coyote.http11.Http11Protocol start
Mar 29 04:13:08 rsyslogd : INFO: Starting Coyote HTTP/1.1 on http-8080
Mar 29 04:13:08 rsyslogd : Mar 29, 2010 4:13:08 AM
org.apache.jk.common.ChannelSocket init
Mar 29 04:13:08 rsyslogd : INFO: JK: ajp13 listening on /0.0.0.0:8009
Mar 29 04:13:08 rsyslogd : Mar 29, 2010 4:13:08 AM
org.apache.jk.server.JkMain start
Mar 29 04:13:08 rsyslogd : INFO: Jk running ID=0 time=0/14 config=null
Mar 29 04:13:08 rsyslogd : Mar 29, 2010 4:13:08 AM
org.apache.catalina.startup.Catalina start

my configuration,
$InputFileName /srv/tomcat/logs/catalina.out
$InputFileTag :
$InputFileStateFile stat-file-tomcat-01
$InputFileSeverity info
$InputFileFacility local3
$InputRunFileMonitor

Thanks. :)


From david at lang.hm  Tue Mar 30 09:03:37 2010
From: david at lang.hm (david at lang.hm)
Date: Tue, 30 Mar 2010 00:03:37 -0700 (PDT)
Subject: [rsyslog] imfile module
In-Reply-To: <bb92aee11003292340s7fc3ad29q8f143c251a492041@mail.gmail.com>
References: <mailman.230.1269930120.2883.rsyslog@lists.adiscon.com>
	<bb92aee11003292340s7fc3ad29q8f143c251a492041@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1003300000170.14525@asgard.lang.hm>

On Tue, 30 Mar 2010, BoOnSiOnG wrote:

> Hello all,
>
> currently I'm using imfile module to monitor apache Tomcat
> catalina.out, I noticed rsyslog will append date time and machine name
> to every line, is it possible to configure rsyslog do not alter the
> original log?

the way to do this would be to change the output format when writing the 
log.

David Lang

> Mar 29 04:13:08 rsyslogd : INFO: Using <Listener
> className="org.apache.jk.config.ApacheConfig"
> modJk="PATH_TO_MOD_JK.SO_OR_DLL" />
> Mar 29 04:13:08 rsyslogd : Mar 29, 2010 4:13:08 AM
> org.apache.coyote.http11.Http11Protocol start
> Mar 29 04:13:08 rsyslogd : INFO: Starting Coyote HTTP/1.1 on http-8080
> Mar 29 04:13:08 rsyslogd : Mar 29, 2010 4:13:08 AM
> org.apache.jk.common.ChannelSocket init
> Mar 29 04:13:08 rsyslogd : INFO: JK: ajp13 listening on /0.0.0.0:8009
> Mar 29 04:13:08 rsyslogd : Mar 29, 2010 4:13:08 AM
> org.apache.jk.server.JkMain start
> Mar 29 04:13:08 rsyslogd : INFO: Jk running ID=0 time=0/14 config=null
> Mar 29 04:13:08 rsyslogd : Mar 29, 2010 4:13:08 AM
> org.apache.catalina.startup.Catalina start
>
> my configuration,
> $InputFileName /srv/tomcat/logs/catalina.out
> $InputFileTag :
> $InputFileStateFile stat-file-tomcat-01
> $InputFileSeverity info
> $InputFileFacility local3
> $InputRunFileMonitor
>
> Thanks. :)
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From mbiebl at gmail.com  Mon Mar  1 00:00:57 2010
From: mbiebl at gmail.com (Michael Biebl)
Date: Mon, 1 Mar 2010 00:00:57 +0100
Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103981@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com>
	<aa76cdac1002251244y2be4e04egc7e0e3eb57494431@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com>
	<aa76cdac1002260355l26269784y8fb8ccd45caeaa12@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com>
	<aa76cdac1002260443p4c7744e2ha1c8c2372c741ebf@mail.gmail.com>
	<aa76cdac1002280515g72c07527mae8ebc08d4ecd404@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103980@GRFEXC.intern.adiscon.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103981@GRFEXC.intern.adiscon.com>
Message-ID: <aa76cdac1002281500g3d50db29rcd20fea1fbf08d48@mail.gmail.com>

2010/2/28 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>> OK, rather than trying to figure out what may be going on here, I'll
>> see that
>> I copy the old omfile code and create a new ompipe. So that will behave
>> exactly like it did before. The new optimizations are not so important
>> for
>> pipes.
>
> I have done this now. I would appreciate if you could checkout the v4-stable
> git head and try with that one. It should now have the exact same behavior as
> in 4.4.2 (the pipe code is now the same!). I tested on Debian and it worked
> like before.

I use $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
in my rsyslog.conf, but the new ompipe module does not seem to respect
that setting but always uses the high-resolution time format.

Other than that, I did not (yet) experience any blank xconsole windows
with v4.6.0-5-gf12a199 (git describe)

Just curious, when I fire up xconsole after boot, I get two log
messages immediately. Where do those log messages come from?

Michael


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From david at lang.hm  Mon Mar  1 03:15:57 2010
From: david at lang.hm (david at lang.hm)
Date: Sun, 28 Feb 2010 18:15:57 -0800 (PST)
Subject: [rsyslog] Feedback requested: inconsistency in config statements
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710397F@GRFEXC.intern.adiscon.com>
References: <001201cab868$d148da7c$100013ac@intern.adiscon.com>
	<alpine.DEB.2.00.1002280426420.15263@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA710397F@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.2.00.1002281346000.15263@asgard.lang.hm>

On Sun, 28 Feb 2010, Rainer Gerhards wrote:

> Hi David,
>
> yes, that makes much more sense to me. Let's call that filter expression tree
> for the time being (the parse tree used for normalization has some similar
> ideas, but a totally different scope and semantics).
>
> While I now see where you are heading, I fail to see how this could
> sufficiently easy be implemented for a real (and complex) configuration.
>
> Let's look at this example conf:
>
> mail.* /Var/log/mail.log
> if programname startswith abc log to /abc
> if programname startswith 'acc' and MSG contains 'error' log to /acc
> # we don't want debug messages from here on
> *.debug ~
> if programname startswith 'acc' or (programname contains 'bde' and (facility
>> 1 and
> facility < 5)) log to @1.1.1.1
> if programname startswith bcd or MSG startswith '123' or MSG contains
> 'error2' log to /bcd
> *.* /var/log/catchall
>
> I simply cannot envision how you would store this as a tree that does not
> look like what we have today.

Ok, Here is what I came up with.

0-1,0-6
   --programname startswith abc log to /abc
   | programname startswith 'acc' and MSG contains 'error' log to /acc
    \programname startswith 'acc' log to @1.1.1.1
   --programname contains 'bde' log to @1.1.1.1
   --MSG startswith '123' log to/bcd
   --MSG contains 'error2' log to /bcd
   --log to /var/log/catchall
0-1,7
   --programname startswith abc log to /abc
    \programname startswith 'acc' and MSG contains 'error' log to /acc
2,0-6
   --log to /var/log/mail.log
   --programname startswith abc log to /abc
   | programname startswith 'acc' and MSG contains 'error' log to /acc
   | programname startswith 'acc' log to @1.1.1.1
    \programname startswith bcd log to /bcd
   --programname contains 'bde' log to @1.1.1.1
   --MSG startswith '123' log to/bcd
   --MSG contains 'error2' log to /bcd
   --log to /var/log/catchall
2,7
   --log to /var/log/mail.log
   --programname startswith abc log to /abc
    \programname startswith 'acc' and MSG contains 'error' log to /acc
3-4,0-6
   --programname startswith abc log to /abc
   | programname startswith 'acc' and MSG contains 'error' log to /acc
   | programname startswith 'acc' log to @1.1.1.1
    \programname startswith bcd log to /bcd
   --programname contains 'bde' log to @1.1.1.1
   --MSG startswith '123' log to/bcd
   --MSG contains 'error2' log to /bcd
   --log to /var/log/catchall
3-4,7
   --programname startswith abc log to /abc
    \programname startswith 'acc' and MSG contains 'error' log to /acc
5-23,0-6
   --programname startswith abc log to /abc
   | programname startswith 'acc' and MSG contains 'error' log to /acc
    \programname startswith 'acc' log to @1.1.1.1
   --programname contains 'bde' log to @1.1.1.1
   --MSG startswith '123' log to/bcd
   --MSG contains 'error2' log to /bcd
   --log to /var/log/catchall
5-23,7
   --programname startswith abc log to /abc
    \programname startswith 'acc' and MSG contains 'error' log to /acc

This looks very ugly and is definantly more complicated, but I think it 
still ends up being a slight (but not drastic) win.

for the initial ruleset you have to check a minimum of 4 tests and a 
maximum of 13 tests (debug messages have 4-5 tests, all others have 12-13)

Assuming the facility/priority lookup can be a table lookup, for the 
tree-based ruleset you have a minimum of 2 tests and a maximum of 7 tests, 
in each case one of the tests is a multi-option comparison which is 
slightly more expensive than a single 'startswith' test, but should be 
cheaper than two 'startswith' tests.

This definantly trades off start time complexity for runtime efficiancy.

David Lang


and here is the long, ugly process I went through to derive this.



first normalize the tests (replace 'or' with two lines, change facility 
and priority to ranges of numbers)

facility = 2 log to /Var/log/mail.log
programname startswith abc log to /abc
programname startswith 'acc' and MSG contains 'error' log to /acc
severity 0..6 and programname startswith 'acc' log to @1.1.1.1
severity 0..6 and programname startswith bcd log to /bcd
facility 2..4 and severity 0..6 and programname contains 'bde' log to @1.1.1.1
severity 0..6 and MSG startswith '123' log to/bcd
severity 0..6 and MSG contains 'error2' log to /bcd
severity 0..6 and log to /var/log/catchall

I then add the full facility/priority ranges


now normalize the facility/severity tests (split overlaps)

we have the following ranges for facility
0-1
2
3-4
5-23

and for severity
0-6
7

to shorten the lines in the e-mail I will letter the remaining 
tests/actions

A) log to /Var/log/mail.log
B) programname startswith abc log to /abc
C) programname startswith 'acc' and MSG contains 'error' log to /acc
D) programname startswith 'acc' log to @1.1.1.1
E) programname startswith bcd log to /bcd
F) programname contains 'bde' log to @1.1.1.1
G) MSG startswith '123' log to/bcd
H) MSG contains 'error2' log to /bcd
I) log to /var/log/catchall

so the tree then becomes (straight translation no reordering)

2,0-6 A
2,7 A
0-1,0-6 B
0-1,7 B
2,0-6 B
2,7 B
3-4,0-6 B
3-4,7 B
5-23,0-6 B
5-23,7 B
0-1,0-6 C
0-1,7 C
2,0-6 C
2,7 C
3-4,0-6 C
3-4,7 C
5-23,0-6 C
5-23,7 C
0-1,0-6 D
2,0-6 D
3-4,0-6 D
5-23,0-6 D
2,0-6 E
3-4,0-6 E
0-1,0-6 F
2,0-6 F
3-4,0-6 F
5-23,0-6 F
0-1,0-6 G
2,0-6 G
3-4,0-6 G
5-23,0-6 G
0-1,0-6 H
2,0-6 H
3-4,0-6 H
5-23,0-6 H
0-1,0-6 I
2,0-6 I
3-4,0-6 I
5-23,0-6 I

now to order them

0-1,0-6 B
0-1,0-6 C
0-1,0-6 D
0-1,0-6 F
0-1,0-6 G
0-1,0-6 H
0-1,0-6 I
0-1,7 B
0-1,7 C
2,0-6 A
2,0-6 B
2,0-6 I
2,0-6 H
2,0-6 G
2,0-6 F
2,0-6 E
2,0-6 D
2,0-6 C
2,7 A
2,7 B
2,7 C
3-4,0-6 B
3-4,0-6 C
3-4,0-6 D
3-4,0-6 E
3-4,0-6 F
3-4,0-6 G
3-4,0-6 H
3-4,0-6 I
3-4,7 B
3-4,7 C
5-23,0-6 B
5-23,0-6 C
5-23,0-6 D
5-23,0-6 F
5-23,0-6 G
5-23,0-6 H
5-23,0-6 I
5-23,7 B
5-23,7 C

I then simplify this to

0-1,0-6 B,C,D,F,G,H,I
0-1,7 B,C
2,0-6 A,B,C,D,E,F,G,H,I
2,7 A,B,C
3-4,0-6 B,C,D,E,F,G,H,I
3-4,7 B,C
5-23,0-6 B,C,D,F,G,H,I
5-23,7 B,C

looking at the lettered tests

A) log to /Var/log/mail.log
B) programname startswith abc log to /abc
C) programname startswith 'acc' and MSG contains 'error' log to /acc
D) programname startswith 'acc' log to @1.1.1.1
E) programname startswith bcd log to /bcd
F) programname contains 'bde' log to @1.1.1.1
G) MSG startswith '123' log to/bcd
H) MSG contains 'error2' log to /bcd
I) log to /var/log/catchall

the only ones that can be combined are B-E

so if you make tests

J) B,C
K) B,C,D
L) B,C,D,E

the results simplify to

0-1,0-6 K,F,G,H,I
0-1,7 J
2,0-6 A,L,F,G,H,I
2,7 A,J
3-4,0-6 L,F,G,H,I
3-4,7 J
5-23,0-6 K,F,G,H,I
5-23,7 J

expanded out to show all tests

0-1,0-6
   --programname startswith abc log to /abc
   | programname startswith 'acc' and MSG contains 'error' log to /acc
    \programname startswith 'acc' log to @1.1.1.1
   --programname contains 'bde' log to @1.1.1.1
   --MSG startswith '123' log to/bcd
   --MSG contains 'error2' log to /bcd
   --log to /var/log/catchall
0-1,7
   --programname startswith abc log to /abc
    \programname startswith 'acc' and MSG contains 'error' log to /acc
2,0-6
   --log to /var/log/mail.log
   --programname startswith abc log to /abc
   | programname startswith 'acc' and MSG contains 'error' log to /acc
   | programname startswith 'acc' log to @1.1.1.1
    \programname startswith bcd log to /bcd
   --programname contains 'bde' log to @1.1.1.1
   --MSG startswith '123' log to/bcd
   --MSG contains 'error2' log to /bcd
   --log to /var/log/catchall
2,7
   --log to /var/log/mail.log
   --programname startswith abc log to /abc
    \programname startswith 'acc' and MSG contains 'error' log to /acc
3-4,0-6
   --programname startswith abc log to /abc
   | programname startswith 'acc' and MSG contains 'error' log to /acc
   | programname startswith 'acc' log to @1.1.1.1
    \programname startswith bcd log to /bcd
   --programname contains 'bde' log to @1.1.1.1
   --MSG startswith '123' log to/bcd
   --MSG contains 'error2' log to /bcd
   --log to /var/log/catchall
3-4,7
   --programname startswith abc log to /abc
    \programname startswith 'acc' and MSG contains 'error' log to /acc
5-23,0-6
   --programname startswith abc log to /abc
   | programname startswith 'acc' and MSG contains 'error' log to /acc
    \programname startswith 'acc' log to @1.1.1.1
   --programname contains 'bde' log to @1.1.1.1
   --MSG startswith '123' log to/bcd
   --MSG contains 'error2' log to /bcd
   --log to /var/log/catchall
5-23,7
   --programname startswith abc log to /abc
    \programname startswith 'acc' and MSG contains 'error' log to /acc

> Rainer
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Sunday, February 28, 2010 2:02 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] Feedback requested: inconsistency in config
>> statements
>>
>> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
>>
>>> Quickly from the mobile: pri based filters require *excatly* one
>> single word (!) lookup currently. You can't beat that.
>>>
>>> I am not totally sure how you think about parse trees. In my view,
>> parsin and filteing are two different things./stages. Maybe we have
>> different concepts on our minds?
>>
>> I am probably using the wrong terminology here.
>>
>> say you have a set of rules that say
>>
>> if programname startswith abc log to /abc
>> if programname startswith acc log to /acc
>> if programname startswith acc log to @1.1.1.1
>> if programname startswith bcd log to /bcd
>>
>> and assuming that these were the only possible programnames for
>> simplicity
>> in the explination.
>>
>> the filtering logic would go somthing like this
>>
>> the rules would compile into a tree
>>
>> -a-b -> /abc
>>    -c -> /acc, at 1.1.1.1
>> -b   -> /bcd
>>
>> receive programname abc
>> I have no facility/Pri filter rules
>> I have no time filter rules
>> I have no system filter rules
>> I have programname filter rules
>>    look at the first character of the programname
>>     it's "a", I have more than one rule that fits that.
>>      look at the second character of the programname
>>       it's a "b", There are no other decisions to make,
>>         invoke the action /abc
>>
>> receive progranmane bcd
>> I have no facility/PRI filter rules
>> I have no time filter rules
>> I have no system filter rules
>> I have programname filter rules
>>    look at the first character of the programname
>>     it's a "b", There are no other decisions to make,
>>       invoke the action /bcd
>>
>> receive programname acc
>> I have no facility/Pri filter rules
>> I have no time filter rules
>> I have no system filter rules
>> I have programname filter rules
>>    look at the first character of the programname
>>     it's "a", I have more than one rule that fits that.
>>      look at the second character of the programname
>>       it's a "c", There are no other decisions to make,
>>         invoke the action /acc and the action @1.1.1.1
>>
>> similarly the facility/pri filtering rules would be either compiled
>> into a
>> tree, or (since it's 256 entries total) into a lookup table with
>> pointers
>> to the list of actions to take for that entry
>>
>> the idea is to spend the time at startup to create the tree that
>> represents the ruleset in order to speed up the processing of each
>> individual message.
>>
>> the real tree would be a bit more complicated than I describe here as
>> it
>> needs to have 'anything else' entries between the known entries (which
>> means that it would not be able to shortcut quite as much as I
>> describe),
>> and have provision for 'do a more complicated thing (like regex) here
>> and
>> if it matches continue'. but except for regex matches, this would make
>> processing the rules pretty much independant of how many rules there
>> were
>> or how complex the ruleset is.
>>
>> there doe snot need to be one single programname filter tree, if you
>> had a
>> rule that said
>> if pri = info and programname startswith def then def
>>
>> the pri tree/table would have an entry for info that would point to a
>> filter tree for programname that just had the check for def in it
>>
>> am I makeing any more sense now?
>>
>> David Lang
>>
>>> rainer
>>>
>>> ----- Urspr?ngliche Nachricht -----
>>> Von: "david at lang.hm" <david at lang.hm>
>>> An: "rsyslog-users" <rsyslog at lists.adiscon.com>
>>> Gesendet: 28.02.10 11:28
>>> Betreff: Re: [rsyslog] Feedback requested: inconsistency in config
>> statements
>>>
>>> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
>>>
>>>>>
>>>>> if the type is depriciated then the destination would be another
>>>>> config_option (which you would then look up)
>>>>>
>>>>> if the type is 'ignore' then it would just be skipped over
>> (possibly
>>>>> with
>>>>> a warning being logged that the config line is being ignored)
>>>>>
>>>>> this would also clean up some of the current cases where a boolean
>>>>> option
>>>>> doesn't honor on/off and true/false.
>>>>
>>>> True/false is NOT a valid boolean option. See
>>>>
>>>>
>> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=18
>> 4c0d87
>>>> 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308
>>>
>>> I was not remembering correctly then, maby it was yes/no vs on/off. I
>> know
>>> I ran into something where what was documented didn't work and I
>> changed
>>> it to another version of a boolean answer and it fixed the problem.
>>>
>>>>> For the second half of the config language (telling rsyslog what to
>> do
>>>>> with the logs it has received) also has several variations in place
>> and
>>>>> is
>>>>> showing issues.
>>>>>
>>>>> However I think that the right answer here is to end up
>> implementing
>>>>> something like the parsing trees and then compile the other config
>>>>> language options into that tree to be consistant (and fast) under
>> the
>>>>> covers, no matter which way the instructions are written (except
>> when
>>>>> you
>>>>> have to use regex matches)
>>>>
>>>> I don't fully agree here with you. For example, the traditional PRI
>> based
>>>> filter is lightning fast. I don't see any way it could be nearly as
>> fast with
>>>> any unified approach. Right now, we have a "unified" filter
>> structure, but it
>>>> contains three filter cases, each one adding potential power at the
>> price of
>>>> decreased speed. I think we need to keep different types of filters
>> in order
>>>> to have some lightning-fast ones. But more of this could be done
>> under the
>>>> hood.
>>>
>>> My guess/expectation is that the tree-based parsing would be about
>> the
>>> same speed as the traditional PRI based filter for choices made based
>> on
>>> PRI, slowing down for other types of conditions only in that more of
>> the
>>> message may need to be scanned (and if there is no selection at a
>> level,
>>> skipping that level should be lightning fast). As a result, a set of
>>> filteres based soley on programname would be as fast as the current
>> PRI
>>> filters.
>>>
>>> David Lang
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From reader at newsguy.com  Mon Mar  1 04:01:34 2010
From: reader at newsguy.com (Harry Putnam)
Date: Sun, 28 Feb 2010 21:01:34 -0600
Subject: [rsyslog] How rsyslog derives hostname
Message-ID: <87hbp0eodt.fsf@newsguy.com>

Running rsyslog-3.22.1

NOTE: Windbag alert.. its a bit long.
(And still more windyness in a PS at the end)

I know newer version are available but 3.22.1 is the newest version my
linux distro (gentoo 2010) offers.

Maybe the mechanism that caused the phenomena I observed has changed
in the later versions.

I'm curious to know how rsyslog derives the hostname.

Before explaining the phenomena I noticed, just be clear that this
problem arose strictly from a blunder on my part.  So, this is not a
complaint about rsyslog.  Just wondering if it points to what could be
a problem.

Due to a blatant blunder on my part, I miss-typed the hosts name
in the /etc/hosts file.  At the 127 line I spelled the hosts name wrong.

    127.0.0.1       logsrv.local.lan   *=>logsvr<=*     localhost

The actual name is `logsrv'
    logsrv
not logsvr.  (I transposed the v and r.)

In this code below from my (no doubt, poorly written)  rsyslog.conf 

[
  This first part contains some fairly standard code.  Mostly
  unchanged from the default /etc/rsyslog.conf that gets installed.
 [ blah ]
]
  [then my own code below]
  [...]

  ## write all localhost output to a debug file
  $template Ldebug,"/var/log/debug.log"
  if \
      $source == 'logsrv'\
  then  -?Ldebug

  ## write all data from remote hosts to:
  ## HOSTNAME/HOSTNAME.log for each client
  $template RDDF,"/var/log/%hostname%/%hostname%.log"
  if \
          $source  != 'logsrv' \
  then    -?RDDF
  [...]

Note: I used the actual name (logsrv) in the code, since I wasn't sure
which of rsyslogs' variables would contain that info instead of
`localhost' or `0.0.0.0' or `127.0.0.1` ... etc.

And understand that this rsyslog.conf was being used in an
experimental environment while I familiarized myself with rsyslog.
 
The idea was to direct all localhost log output to /var/log/debug.log
(in addition to the more normal defaults in rsyslog.conf not shown here).
And log data coming from remote clients to /var/log/dynaDIR/dynaFILE
Where DIR and FILE are both named dynamically after the host sending
the log data.

It worked just like I had hoped.

But I accidentally found what might be seen as a weakness in rsyslog.

I'm not experienced enough to really judge something like this. And
not adept enough at reading source code to see what the mechanism is.

Apparently rsyslog preferred the miss-typed name in /etc/hosts over
the name returned by gethostbyname or the newer getaddrinfo, both
appear to be involved in the source code.

With the mistaken spelling in /etc/hosts, rsyslog (correctly) saw the
mistaken name inside `$source' as not matching `logsrv' in my code, so
wrote to:

  /var/log/logsvr/logsvr.log

Once I got the typo fixed, rsyslog wrote all that same stuff to
/var/log/debug.log as expected.

So it makes me wonder how robust the mechanism for getting the
localhost name is.  rsyslog appears to use gethostbyname and the
newer getaddrinfo.

Further... I'm not sure where cmds like gethostbyname (and similar
ones) get the info either... for all I know they may just grep
/etc/hosts.

I do know that even while I had the mis-spelled name in /etc/hosts, the
`hostname' command still returned the correct name.  So the `hostname'
cmd must rely on something else.

At any rate, rsyslog apparently turns to grepping the /etc/hosts
file at some point, since in this case the log output was directed to a
directory and file named after the TYPO in /etc/hosts.

It took me a few minutes to figure out where the bad name was coming
from since the `hostname' cmd returned the correct name (logsrv).

Some linux distros have a special file that tells the OS what its name
is. 

On my distro (gentoo linux) it's a file at /etc/conf.d/hostname, that
is dedicated to nothing more than giving the host a name.

On Debian its /etc/hostname

Other distros I've used have a few different ways of getting that
information into play, but I haven't used any linux distros that
relied on /etc/hosts.

Is having rsyslog rely on /etc/hosts as the final arbiter about the
host name a good plan?

ps - two asides:
 1) Any suggestions for better code are welcome
 2) Can anyone tell me what the significance of the question mark
    preceding the log file variable name is:

  $template RDDF,"/var/log/%hostname%/%hostname%.log"
  if \
          $source  != 'logsrv' \
  then    -?RDDF
          ------
           here

I know what the dash sinifies but not the question mark. I just
copied what I saw in the examples and it seems to work.
          



From rgerhards at hq.adiscon.com  Mon Mar  1 07:45:48 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 1 Mar 2010 07:45:48 +0100
Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released
References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com><aa76cdac1002251244y2be4e04egc7e0e3eb57494431@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com><aa76cdac1002260355l26269784y8fb8ccd45caeaa12@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com><aa76cdac1002260443p4c7744e2ha1c8c2372c741ebf@mail.gmail.com><aa76cdac1002280515g72c07527mae8ebc08d4ecd404@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103980@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103981@GRFEXC.intern.adiscon.com>
	<aa76cdac1002281500g3d50db29rcd20fea1fbf08d48@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103983@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Monday, March 01, 2010 12:01 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released
> 
> 2010/2/28 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> >> OK, rather than trying to figure out what may be going on here, I'll
> >> see that
> >> I copy the old omfile code and create a new ompipe. So that will
> behave
> >> exactly like it did before. The new optimizations are not so
> important
> >> for
> >> pipes.
> >
> > I have done this now. I would appreciate if you could checkout the
> v4-stable
> > git head and try with that one. It should now have the exact same
> behavior as
> > in 4.4.2 (the pipe code is now the same!). I tested on Debian and it
> worked
> > like before.
> 
> I use $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
> in my rsyslog.conf, but the new ompipe module does not seem to respect
> that setting but always uses the high-resolution time format.

I see - that was a side-effect that pipes used the same output module like
regular pipes. But I have now re-established it, please try the newest git
v4-stable branch.

> 
> Other than that, I did not (yet) experience any blank xconsole windows
> with v4.6.0-5-gf12a199 (git describe)
> 
> Just curious, when I fire up xconsole after boot, I get two log
> messages immediately. Where do those log messages come from?

It seems to stem back to the API usage in the old code. The old code opens
the pipe and tries to write. It disables writing only when the API returns it
would block. The new code used create() instead of open, and that immediately
returned an error if nobody was listening.

It looks like with the old sequence, the OS permits to write a handful of
bytes before it returns EAGAIN. However, I would not take that behavior for
granted: I guess it may be different under different platforms and/or
versions.

Rainer


From mrdemeanour at jackpot.uk.net  Mon Mar  1 08:41:17 2010
From: mrdemeanour at jackpot.uk.net (Mr. Demeanour)
Date: Mon, 01 Mar 2010 07:41:17 +0000
Subject: [rsyslog] How rsyslog derives hostname
In-Reply-To: <87hbp0eodt.fsf@newsguy.com>
References: <87hbp0eodt.fsf@newsguy.com>
Message-ID: <4B8B6F9D.5040803@jackpot.uk.net>

Harry Putnam wrote:
> 
> Further... I'm not sure where cmds like gethostbyname (and similar
> ones) get the info either... for all I know they may just grep
> /etc/hosts.

man resolv.conf (i.e. it's configurable).

-- 
Jack.


From rgerhards at hq.adiscon.com  Mon Mar  1 10:39:27 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 1 Mar 2010 10:39:27 +0100
Subject: [rsyslog] How rsyslog derives hostname
References: <87hbp0eodt.fsf@newsguy.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710398A@GRFEXC.intern.adiscon.com>

I need to re-read this more careful, but for now let me say that rsyslog does
not touch /etc/hosts at all. It exlusively relies on what is returned by the
OS. But the "source" property is not locally generated, it contains whatever
the sender placed into the relevant field.

Please also see this article:

http://www.rsyslog.com/doc-syslog_parsing.html

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Harry Putnam
> Sent: Monday, March 01, 2010 4:02 AM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] How rsyslog derives hostname
> 
> Running rsyslog-3.22.1
> 
> NOTE: Windbag alert.. its a bit long.
> (And still more windyness in a PS at the end)
> 
> I know newer version are available but 3.22.1 is the newest version my
> linux distro (gentoo 2010) offers.
> 
> Maybe the mechanism that caused the phenomena I observed has changed
> in the later versions.
> 
> I'm curious to know how rsyslog derives the hostname.
> 
> Before explaining the phenomena I noticed, just be clear that this
> problem arose strictly from a blunder on my part.  So, this is not a
> complaint about rsyslog.  Just wondering if it points to what could be
> a problem.
> 
> Due to a blatant blunder on my part, I miss-typed the hosts name
> in the /etc/hosts file.  At the 127 line I spelled the hosts name
> wrong.
> 
>     127.0.0.1       logsrv.local.lan   *=>logsvr<=*     localhost
> 
> The actual name is `logsrv'
>     logsrv
> not logsvr.  (I transposed the v and r.)
> 
> In this code below from my (no doubt, poorly written)  rsyslog.conf
> 
> [
>   This first part contains some fairly standard code.  Mostly
>   unchanged from the default /etc/rsyslog.conf that gets installed.
>  [ blah ]
> ]
>   [then my own code below]
>   [...]
> 
>   ## write all localhost output to a debug file
>   $template Ldebug,"/var/log/debug.log"
>   if \
>       $source == 'logsrv'\
>   then  -?Ldebug
> 
>   ## write all data from remote hosts to:
>   ## HOSTNAME/HOSTNAME.log for each client
>   $template RDDF,"/var/log/%hostname%/%hostname%.log"
>   if \
>           $source  != 'logsrv' \
>   then    -?RDDF
>   [...]
> 
> Note: I used the actual name (logsrv) in the code, since I wasn't sure
> which of rsyslogs' variables would contain that info instead of
> `localhost' or `0.0.0.0' or `127.0.0.1` ... etc.
> 
> And understand that this rsyslog.conf was being used in an
> experimental environment while I familiarized myself with rsyslog.
> 
> The idea was to direct all localhost log output to /var/log/debug.log
> (in addition to the more normal defaults in rsyslog.conf not shown
> here).
> And log data coming from remote clients to /var/log/dynaDIR/dynaFILE
> Where DIR and FILE are both named dynamically after the host sending
> the log data.
> 
> It worked just like I had hoped.
> 
> But I accidentally found what might be seen as a weakness in rsyslog.
> 
> I'm not experienced enough to really judge something like this. And
> not adept enough at reading source code to see what the mechanism is.
> 
> Apparently rsyslog preferred the miss-typed name in /etc/hosts over
> the name returned by gethostbyname or the newer getaddrinfo, both
> appear to be involved in the source code.
> 
> With the mistaken spelling in /etc/hosts, rsyslog (correctly) saw the
> mistaken name inside `$source' as not matching `logsrv' in my code, so
> wrote to:
> 
>   /var/log/logsvr/logsvr.log
> 
> Once I got the typo fixed, rsyslog wrote all that same stuff to
> /var/log/debug.log as expected.
> 
> So it makes me wonder how robust the mechanism for getting the
> localhost name is.  rsyslog appears to use gethostbyname and the
> newer getaddrinfo.
> 
> Further... I'm not sure where cmds like gethostbyname (and similar
> ones) get the info either... for all I know they may just grep
> /etc/hosts.
> 
> I do know that even while I had the mis-spelled name in /etc/hosts, the
> `hostname' command still returned the correct name.  So the `hostname'
> cmd must rely on something else.
> 
> At any rate, rsyslog apparently turns to grepping the /etc/hosts
> file at some point, since in this case the log output was directed to a
> directory and file named after the TYPO in /etc/hosts.
> 
> It took me a few minutes to figure out where the bad name was coming
> from since the `hostname' cmd returned the correct name (logsrv).
> 
> Some linux distros have a special file that tells the OS what its name
> is.
> 
> On my distro (gentoo linux) it's a file at /etc/conf.d/hostname, that
> is dedicated to nothing more than giving the host a name.
> 
> On Debian its /etc/hostname
> 
> Other distros I've used have a few different ways of getting that
> information into play, but I haven't used any linux distros that
> relied on /etc/hosts.
> 
> Is having rsyslog rely on /etc/hosts as the final arbiter about the
> host name a good plan?
> 
> ps - two asides:
>  1) Any suggestions for better code are welcome
>  2) Can anyone tell me what the significance of the question mark
>     preceding the log file variable name is:
> 
>   $template RDDF,"/var/log/%hostname%/%hostname%.log"
>   if \
>           $source  != 'logsrv' \
>   then    -?RDDF
>           ------
>            here
> 
> I know what the dash sinifies but not the question mark. I just
> copied what I saw in the examples and it seems to work.
> 
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Mon Mar  1 14:27:57 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 1 Mar 2010 14:27:57 +0100
Subject: [rsyslog] Feedback requested: inconsistency in config statements
References: <001201cab868$d148da7c$100013ac@intern.adiscon.com><alpine.DEB.2.00.1002280426420.15263@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA710397F@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.00.1002281346000.15263@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103992@GRFEXC.intern.adiscon.com>

Hi David,

thanks for the explanation, I now see where you are heading too. I have not
dug into all details, but find the post very interesting and insightful. I'll
archive it so that I have it at hand when I can take the route to actually
implement such an approach. All in all, it still looks rather complex to me.
I guess it would be useful to have a guy with experience in writing compiler
code optimizers at hand when I tackle that beast ;)

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Monday, March 01, 2010 3:16 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Feedback requested: inconsistency in config
> statements
> 
> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
> 
> > Hi David,
> >
> > yes, that makes much more sense to me. Let's call that filter
> expression tree
> > for the time being (the parse tree used for normalization has some
> similar
> > ideas, but a totally different scope and semantics).
> >
> > While I now see where you are heading, I fail to see how this could
> > sufficiently easy be implemented for a real (and complex)
> configuration.
> >
> > Let's look at this example conf:
> >
> > mail.* /Var/log/mail.log
> > if programname startswith abc log to /abc
> > if programname startswith 'acc' and MSG contains 'error' log to /acc
> > # we don't want debug messages from here on
> > *.debug ~
> > if programname startswith 'acc' or (programname contains 'bde' and
> (facility
> >> 1 and
> > facility < 5)) log to @1.1.1.1
> > if programname startswith bcd or MSG startswith '123' or MSG contains
> > 'error2' log to /bcd
> > *.* /var/log/catchall
> >
> > I simply cannot envision how you would store this as a tree that does
> not
> > look like what we have today.
> 
> Ok, Here is what I came up with.
> 
> 0-1,0-6
>    --programname startswith abc log to /abc
>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>     \programname startswith 'acc' log to @1.1.1.1
>    --programname contains 'bde' log to @1.1.1.1
>    --MSG startswith '123' log to/bcd
>    --MSG contains 'error2' log to /bcd
>    --log to /var/log/catchall
> 0-1,7
>    --programname startswith abc log to /abc
>     \programname startswith 'acc' and MSG contains 'error' log to /acc
> 2,0-6
>    --log to /var/log/mail.log
>    --programname startswith abc log to /abc
>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>    | programname startswith 'acc' log to @1.1.1.1
>     \programname startswith bcd log to /bcd
>    --programname contains 'bde' log to @1.1.1.1
>    --MSG startswith '123' log to/bcd
>    --MSG contains 'error2' log to /bcd
>    --log to /var/log/catchall
> 2,7
>    --log to /var/log/mail.log
>    --programname startswith abc log to /abc
>     \programname startswith 'acc' and MSG contains 'error' log to /acc
> 3-4,0-6
>    --programname startswith abc log to /abc
>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>    | programname startswith 'acc' log to @1.1.1.1
>     \programname startswith bcd log to /bcd
>    --programname contains 'bde' log to @1.1.1.1
>    --MSG startswith '123' log to/bcd
>    --MSG contains 'error2' log to /bcd
>    --log to /var/log/catchall
> 3-4,7
>    --programname startswith abc log to /abc
>     \programname startswith 'acc' and MSG contains 'error' log to /acc
> 5-23,0-6
>    --programname startswith abc log to /abc
>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>     \programname startswith 'acc' log to @1.1.1.1
>    --programname contains 'bde' log to @1.1.1.1
>    --MSG startswith '123' log to/bcd
>    --MSG contains 'error2' log to /bcd
>    --log to /var/log/catchall
> 5-23,7
>    --programname startswith abc log to /abc
>     \programname startswith 'acc' and MSG contains 'error' log to /acc
> 
> This looks very ugly and is definantly more complicated, but I think it
> still ends up being a slight (but not drastic) win.
> 
> for the initial ruleset you have to check a minimum of 4 tests and a
> maximum of 13 tests (debug messages have 4-5 tests, all others have 12-
> 13)
> 
> Assuming the facility/priority lookup can be a table lookup, for the
> tree-based ruleset you have a minimum of 2 tests and a maximum of 7
> tests,
> in each case one of the tests is a multi-option comparison which is
> slightly more expensive than a single 'startswith' test, but should be
> cheaper than two 'startswith' tests.
> 
> This definantly trades off start time complexity for runtime
> efficiancy.
> 
> David Lang
> 
> 
> and here is the long, ugly process I went through to derive this.
> 
> 
> 
> first normalize the tests (replace 'or' with two lines, change facility
> and priority to ranges of numbers)
> 
> facility = 2 log to /Var/log/mail.log
> programname startswith abc log to /abc
> programname startswith 'acc' and MSG contains 'error' log to /acc
> severity 0..6 and programname startswith 'acc' log to @1.1.1.1
> severity 0..6 and programname startswith bcd log to /bcd
> facility 2..4 and severity 0..6 and programname contains 'bde' log to
> @1.1.1.1
> severity 0..6 and MSG startswith '123' log to/bcd
> severity 0..6 and MSG contains 'error2' log to /bcd
> severity 0..6 and log to /var/log/catchall
> 
> I then add the full facility/priority ranges
> 
> 
> now normalize the facility/severity tests (split overlaps)
> 
> we have the following ranges for facility
> 0-1
> 2
> 3-4
> 5-23
> 
> and for severity
> 0-6
> 7
> 
> to shorten the lines in the e-mail I will letter the remaining
> tests/actions
> 
> A) log to /Var/log/mail.log
> B) programname startswith abc log to /abc
> C) programname startswith 'acc' and MSG contains 'error' log to /acc
> D) programname startswith 'acc' log to @1.1.1.1
> E) programname startswith bcd log to /bcd
> F) programname contains 'bde' log to @1.1.1.1
> G) MSG startswith '123' log to/bcd
> H) MSG contains 'error2' log to /bcd
> I) log to /var/log/catchall
> 
> so the tree then becomes (straight translation no reordering)
> 
> 2,0-6 A
> 2,7 A
> 0-1,0-6 B
> 0-1,7 B
> 2,0-6 B
> 2,7 B
> 3-4,0-6 B
> 3-4,7 B
> 5-23,0-6 B
> 5-23,7 B
> 0-1,0-6 C
> 0-1,7 C
> 2,0-6 C
> 2,7 C
> 3-4,0-6 C
> 3-4,7 C
> 5-23,0-6 C
> 5-23,7 C
> 0-1,0-6 D
> 2,0-6 D
> 3-4,0-6 D
> 5-23,0-6 D
> 2,0-6 E
> 3-4,0-6 E
> 0-1,0-6 F
> 2,0-6 F
> 3-4,0-6 F
> 5-23,0-6 F
> 0-1,0-6 G
> 2,0-6 G
> 3-4,0-6 G
> 5-23,0-6 G
> 0-1,0-6 H
> 2,0-6 H
> 3-4,0-6 H
> 5-23,0-6 H
> 0-1,0-6 I
> 2,0-6 I
> 3-4,0-6 I
> 5-23,0-6 I
> 
> now to order them
> 
> 0-1,0-6 B
> 0-1,0-6 C
> 0-1,0-6 D
> 0-1,0-6 F
> 0-1,0-6 G
> 0-1,0-6 H
> 0-1,0-6 I
> 0-1,7 B
> 0-1,7 C
> 2,0-6 A
> 2,0-6 B
> 2,0-6 I
> 2,0-6 H
> 2,0-6 G
> 2,0-6 F
> 2,0-6 E
> 2,0-6 D
> 2,0-6 C
> 2,7 A
> 2,7 B
> 2,7 C
> 3-4,0-6 B
> 3-4,0-6 C
> 3-4,0-6 D
> 3-4,0-6 E
> 3-4,0-6 F
> 3-4,0-6 G
> 3-4,0-6 H
> 3-4,0-6 I
> 3-4,7 B
> 3-4,7 C
> 5-23,0-6 B
> 5-23,0-6 C
> 5-23,0-6 D
> 5-23,0-6 F
> 5-23,0-6 G
> 5-23,0-6 H
> 5-23,0-6 I
> 5-23,7 B
> 5-23,7 C
> 
> I then simplify this to
> 
> 0-1,0-6 B,C,D,F,G,H,I
> 0-1,7 B,C
> 2,0-6 A,B,C,D,E,F,G,H,I
> 2,7 A,B,C
> 3-4,0-6 B,C,D,E,F,G,H,I
> 3-4,7 B,C
> 5-23,0-6 B,C,D,F,G,H,I
> 5-23,7 B,C
> 
> looking at the lettered tests
> 
> A) log to /Var/log/mail.log
> B) programname startswith abc log to /abc
> C) programname startswith 'acc' and MSG contains 'error' log to /acc
> D) programname startswith 'acc' log to @1.1.1.1
> E) programname startswith bcd log to /bcd
> F) programname contains 'bde' log to @1.1.1.1
> G) MSG startswith '123' log to/bcd
> H) MSG contains 'error2' log to /bcd
> I) log to /var/log/catchall
> 
> the only ones that can be combined are B-E
> 
> so if you make tests
> 
> J) B,C
> K) B,C,D
> L) B,C,D,E
> 
> the results simplify to
> 
> 0-1,0-6 K,F,G,H,I
> 0-1,7 J
> 2,0-6 A,L,F,G,H,I
> 2,7 A,J
> 3-4,0-6 L,F,G,H,I
> 3-4,7 J
> 5-23,0-6 K,F,G,H,I
> 5-23,7 J
> 
> expanded out to show all tests
> 
> 0-1,0-6
>    --programname startswith abc log to /abc
>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>     \programname startswith 'acc' log to @1.1.1.1
>    --programname contains 'bde' log to @1.1.1.1
>    --MSG startswith '123' log to/bcd
>    --MSG contains 'error2' log to /bcd
>    --log to /var/log/catchall
> 0-1,7
>    --programname startswith abc log to /abc
>     \programname startswith 'acc' and MSG contains 'error' log to /acc
> 2,0-6
>    --log to /var/log/mail.log
>    --programname startswith abc log to /abc
>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>    | programname startswith 'acc' log to @1.1.1.1
>     \programname startswith bcd log to /bcd
>    --programname contains 'bde' log to @1.1.1.1
>    --MSG startswith '123' log to/bcd
>    --MSG contains 'error2' log to /bcd
>    --log to /var/log/catchall
> 2,7
>    --log to /var/log/mail.log
>    --programname startswith abc log to /abc
>     \programname startswith 'acc' and MSG contains 'error' log to /acc
> 3-4,0-6
>    --programname startswith abc log to /abc
>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>    | programname startswith 'acc' log to @1.1.1.1
>     \programname startswith bcd log to /bcd
>    --programname contains 'bde' log to @1.1.1.1
>    --MSG startswith '123' log to/bcd
>    --MSG contains 'error2' log to /bcd
>    --log to /var/log/catchall
> 3-4,7
>    --programname startswith abc log to /abc
>     \programname startswith 'acc' and MSG contains 'error' log to /acc
> 5-23,0-6
>    --programname startswith abc log to /abc
>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>     \programname startswith 'acc' log to @1.1.1.1
>    --programname contains 'bde' log to @1.1.1.1
>    --MSG startswith '123' log to/bcd
>    --MSG contains 'error2' log to /bcd
>    --log to /var/log/catchall
> 5-23,7
>    --programname startswith abc log to /abc
>     \programname startswith 'acc' and MSG contains 'error' log to /acc
> 
> > Rainer
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >> Sent: Sunday, February 28, 2010 2:02 PM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] Feedback requested: inconsistency in config
> >> statements
> >>
> >> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
> >>
> >>> Quickly from the mobile: pri based filters require *excatly* one
> >> single word (!) lookup currently. You can't beat that.
> >>>
> >>> I am not totally sure how you think about parse trees. In my view,
> >> parsin and filteing are two different things./stages. Maybe we have
> >> different concepts on our minds?
> >>
> >> I am probably using the wrong terminology here.
> >>
> >> say you have a set of rules that say
> >>
> >> if programname startswith abc log to /abc
> >> if programname startswith acc log to /acc
> >> if programname startswith acc log to @1.1.1.1
> >> if programname startswith bcd log to /bcd
> >>
> >> and assuming that these were the only possible programnames for
> >> simplicity
> >> in the explination.
> >>
> >> the filtering logic would go somthing like this
> >>
> >> the rules would compile into a tree
> >>
> >> -a-b -> /abc
> >>    -c -> /acc, at 1.1.1.1
> >> -b   -> /bcd
> >>
> >> receive programname abc
> >> I have no facility/Pri filter rules
> >> I have no time filter rules
> >> I have no system filter rules
> >> I have programname filter rules
> >>    look at the first character of the programname
> >>     it's "a", I have more than one rule that fits that.
> >>      look at the second character of the programname
> >>       it's a "b", There are no other decisions to make,
> >>         invoke the action /abc
> >>
> >> receive progranmane bcd
> >> I have no facility/PRI filter rules
> >> I have no time filter rules
> >> I have no system filter rules
> >> I have programname filter rules
> >>    look at the first character of the programname
> >>     it's a "b", There are no other decisions to make,
> >>       invoke the action /bcd
> >>
> >> receive programname acc
> >> I have no facility/Pri filter rules
> >> I have no time filter rules
> >> I have no system filter rules
> >> I have programname filter rules
> >>    look at the first character of the programname
> >>     it's "a", I have more than one rule that fits that.
> >>      look at the second character of the programname
> >>       it's a "c", There are no other decisions to make,
> >>         invoke the action /acc and the action @1.1.1.1
> >>
> >> similarly the facility/pri filtering rules would be either compiled
> >> into a
> >> tree, or (since it's 256 entries total) into a lookup table with
> >> pointers
> >> to the list of actions to take for that entry
> >>
> >> the idea is to spend the time at startup to create the tree that
> >> represents the ruleset in order to speed up the processing of each
> >> individual message.
> >>
> >> the real tree would be a bit more complicated than I describe here
> as
> >> it
> >> needs to have 'anything else' entries between the known entries
> (which
> >> means that it would not be able to shortcut quite as much as I
> >> describe),
> >> and have provision for 'do a more complicated thing (like regex)
> here
> >> and
> >> if it matches continue'. but except for regex matches, this would
> make
> >> processing the rules pretty much independant of how many rules there
> >> were
> >> or how complex the ruleset is.
> >>
> >> there doe snot need to be one single programname filter tree, if you
> >> had a
> >> rule that said
> >> if pri = info and programname startswith def then def
> >>
> >> the pri tree/table would have an entry for info that would point to
> a
> >> filter tree for programname that just had the check for def in it
> >>
> >> am I makeing any more sense now?
> >>
> >> David Lang
> >>
> >>> rainer
> >>>
> >>> ----- Urspr?ngliche Nachricht -----
> >>> Von: "david at lang.hm" <david at lang.hm>
> >>> An: "rsyslog-users" <rsyslog at lists.adiscon.com>
> >>> Gesendet: 28.02.10 11:28
> >>> Betreff: Re: [rsyslog] Feedback requested: inconsistency in config
> >> statements
> >>>
> >>> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
> >>>
> >>>>>
> >>>>> if the type is depriciated then the destination would be another
> >>>>> config_option (which you would then look up)
> >>>>>
> >>>>> if the type is 'ignore' then it would just be skipped over
> >> (possibly
> >>>>> with
> >>>>> a warning being logged that the config line is being ignored)
> >>>>>
> >>>>> this would also clean up some of the current cases where a
> boolean
> >>>>> option
> >>>>> doesn't honor on/off and true/false.
> >>>>
> >>>> True/false is NOT a valid boolean option. See
> >>>>
> >>>>
> >>
> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=18
> >> 4c0d87
> >>>> 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308
> >>>
> >>> I was not remembering correctly then, maby it was yes/no vs on/off.
> I
> >> know
> >>> I ran into something where what was documented didn't work and I
> >> changed
> >>> it to another version of a boolean answer and it fixed the problem.
> >>>
> >>>>> For the second half of the config language (telling rsyslog what
> to
> >> do
> >>>>> with the logs it has received) also has several variations in
> place
> >> and
> >>>>> is
> >>>>> showing issues.
> >>>>>
> >>>>> However I think that the right answer here is to end up
> >> implementing
> >>>>> something like the parsing trees and then compile the other
> config
> >>>>> language options into that tree to be consistant (and fast) under
> >> the
> >>>>> covers, no matter which way the instructions are written (except
> >> when
> >>>>> you
> >>>>> have to use regex matches)
> >>>>
> >>>> I don't fully agree here with you. For example, the traditional
> PRI
> >> based
> >>>> filter is lightning fast. I don't see any way it could be nearly
> as
> >> fast with
> >>>> any unified approach. Right now, we have a "unified" filter
> >> structure, but it
> >>>> contains three filter cases, each one adding potential power at
> the
> >> price of
> >>>> decreased speed. I think we need to keep different types of
> filters
> >> in order
> >>>> to have some lightning-fast ones. But more of this could be done
> >> under the
> >>>> hood.
> >>>
> >>> My guess/expectation is that the tree-based parsing would be about
> >> the
> >>> same speed as the traditional PRI based filter for choices made
> based
> >> on
> >>> PRI, slowing down for other types of conditions only in that more
> of
> >> the
> >>> message may need to be scanned (and if there is no selection at a
> >> level,
> >>> skipping that level should be lightning fast). As a result, a set
> of
> >>> filteres based soley on programname would be as fast as the current
> >> PRI
> >>> filters.
> >>>
> >>> David Lang
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From david at lang.hm  Mon Mar  1 20:39:09 2010
From: david at lang.hm (david at lang.hm)
Date: Mon, 1 Mar 2010 11:39:09 -0800 (PST)
Subject: [rsyslog] Feedback requested: inconsistency in config statements
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103992@GRFEXC.intern.adiscon.com>
References: <001201cab868$d148da7c$100013ac@intern.adiscon.com><alpine.DEB.2.00.1002280426420.15263@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA710397F@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.00.1002281346000.15263@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7103992@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.2.00.1003011130460.5131@asgard.lang.hm>

On Mon, 1 Mar 2010, Rainer Gerhards wrote:

> Hi David,
>
> thanks for the explanation, I now see where you are heading too. I have not
> dug into all details, but find the post very interesting and insightful. I'll
> archive it so that I have it at hand when I can take the route to actually
> implement such an approach. All in all, it still looks rather complex to me.
> I guess it would be useful to have a guy with experience in writing compiler
> code optimizers at hand when I tackle that beast ;)

you may also want to dig into yacc/bison as they are designed to help 
people build parser trees.

I am assuming that this is the type of thing that syslog-ng is doing to be 
able to efficiantly handle a lot of log processing.

I currently have a script that looks for a couple hundred programnames and 
puts the data into ~50 different files depending on what that name is. 
Given that the output side of rsyslog is currently the bottleneck, I would 
not want to try and implement that in rsyslog today, but if rsyslog gained 
the ability to handle this sort of ruleset efficiantly it would greatly 
simplify my life.

David Lang

> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Monday, March 01, 2010 3:16 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] Feedback requested: inconsistency in config
>> statements
>>
>> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
>>
>>> Hi David,
>>>
>>> yes, that makes much more sense to me. Let's call that filter
>> expression tree
>>> for the time being (the parse tree used for normalization has some
>> similar
>>> ideas, but a totally different scope and semantics).
>>>
>>> While I now see where you are heading, I fail to see how this could
>>> sufficiently easy be implemented for a real (and complex)
>> configuration.
>>>
>>> Let's look at this example conf:
>>>
>>> mail.* /Var/log/mail.log
>>> if programname startswith abc log to /abc
>>> if programname startswith 'acc' and MSG contains 'error' log to /acc
>>> # we don't want debug messages from here on
>>> *.debug ~
>>> if programname startswith 'acc' or (programname contains 'bde' and
>> (facility
>>>> 1 and
>>> facility < 5)) log to @1.1.1.1
>>> if programname startswith bcd or MSG startswith '123' or MSG contains
>>> 'error2' log to /bcd
>>> *.* /var/log/catchall
>>>
>>> I simply cannot envision how you would store this as a tree that does
>> not
>>> look like what we have today.
>>
>> Ok, Here is what I came up with.
>>
>> 0-1,0-6
>>    --programname startswith abc log to /abc
>>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>>     \programname startswith 'acc' log to @1.1.1.1
>>    --programname contains 'bde' log to @1.1.1.1
>>    --MSG startswith '123' log to/bcd
>>    --MSG contains 'error2' log to /bcd
>>    --log to /var/log/catchall
>> 0-1,7
>>    --programname startswith abc log to /abc
>>     \programname startswith 'acc' and MSG contains 'error' log to /acc
>> 2,0-6
>>    --log to /var/log/mail.log
>>    --programname startswith abc log to /abc
>>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>>    | programname startswith 'acc' log to @1.1.1.1
>>     \programname startswith bcd log to /bcd
>>    --programname contains 'bde' log to @1.1.1.1
>>    --MSG startswith '123' log to/bcd
>>    --MSG contains 'error2' log to /bcd
>>    --log to /var/log/catchall
>> 2,7
>>    --log to /var/log/mail.log
>>    --programname startswith abc log to /abc
>>     \programname startswith 'acc' and MSG contains 'error' log to /acc
>> 3-4,0-6
>>    --programname startswith abc log to /abc
>>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>>    | programname startswith 'acc' log to @1.1.1.1
>>     \programname startswith bcd log to /bcd
>>    --programname contains 'bde' log to @1.1.1.1
>>    --MSG startswith '123' log to/bcd
>>    --MSG contains 'error2' log to /bcd
>>    --log to /var/log/catchall
>> 3-4,7
>>    --programname startswith abc log to /abc
>>     \programname startswith 'acc' and MSG contains 'error' log to /acc
>> 5-23,0-6
>>    --programname startswith abc log to /abc
>>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>>     \programname startswith 'acc' log to @1.1.1.1
>>    --programname contains 'bde' log to @1.1.1.1
>>    --MSG startswith '123' log to/bcd
>>    --MSG contains 'error2' log to /bcd
>>    --log to /var/log/catchall
>> 5-23,7
>>    --programname startswith abc log to /abc
>>     \programname startswith 'acc' and MSG contains 'error' log to /acc
>>
>> This looks very ugly and is definantly more complicated, but I think it
>> still ends up being a slight (but not drastic) win.
>>
>> for the initial ruleset you have to check a minimum of 4 tests and a
>> maximum of 13 tests (debug messages have 4-5 tests, all others have 12-
>> 13)
>>
>> Assuming the facility/priority lookup can be a table lookup, for the
>> tree-based ruleset you have a minimum of 2 tests and a maximum of 7
>> tests,
>> in each case one of the tests is a multi-option comparison which is
>> slightly more expensive than a single 'startswith' test, but should be
>> cheaper than two 'startswith' tests.
>>
>> This definantly trades off start time complexity for runtime
>> efficiancy.
>>
>> David Lang
>>
>>
>> and here is the long, ugly process I went through to derive this.
>>
>>
>>
>> first normalize the tests (replace 'or' with two lines, change facility
>> and priority to ranges of numbers)
>>
>> facility = 2 log to /Var/log/mail.log
>> programname startswith abc log to /abc
>> programname startswith 'acc' and MSG contains 'error' log to /acc
>> severity 0..6 and programname startswith 'acc' log to @1.1.1.1
>> severity 0..6 and programname startswith bcd log to /bcd
>> facility 2..4 and severity 0..6 and programname contains 'bde' log to
>> @1.1.1.1
>> severity 0..6 and MSG startswith '123' log to/bcd
>> severity 0..6 and MSG contains 'error2' log to /bcd
>> severity 0..6 and log to /var/log/catchall
>>
>> I then add the full facility/priority ranges
>>
>>
>> now normalize the facility/severity tests (split overlaps)
>>
>> we have the following ranges for facility
>> 0-1
>> 2
>> 3-4
>> 5-23
>>
>> and for severity
>> 0-6
>> 7
>>
>> to shorten the lines in the e-mail I will letter the remaining
>> tests/actions
>>
>> A) log to /Var/log/mail.log
>> B) programname startswith abc log to /abc
>> C) programname startswith 'acc' and MSG contains 'error' log to /acc
>> D) programname startswith 'acc' log to @1.1.1.1
>> E) programname startswith bcd log to /bcd
>> F) programname contains 'bde' log to @1.1.1.1
>> G) MSG startswith '123' log to/bcd
>> H) MSG contains 'error2' log to /bcd
>> I) log to /var/log/catchall
>>
>> so the tree then becomes (straight translation no reordering)
>>
>> 2,0-6 A
>> 2,7 A
>> 0-1,0-6 B
>> 0-1,7 B
>> 2,0-6 B
>> 2,7 B
>> 3-4,0-6 B
>> 3-4,7 B
>> 5-23,0-6 B
>> 5-23,7 B
>> 0-1,0-6 C
>> 0-1,7 C
>> 2,0-6 C
>> 2,7 C
>> 3-4,0-6 C
>> 3-4,7 C
>> 5-23,0-6 C
>> 5-23,7 C
>> 0-1,0-6 D
>> 2,0-6 D
>> 3-4,0-6 D
>> 5-23,0-6 D
>> 2,0-6 E
>> 3-4,0-6 E
>> 0-1,0-6 F
>> 2,0-6 F
>> 3-4,0-6 F
>> 5-23,0-6 F
>> 0-1,0-6 G
>> 2,0-6 G
>> 3-4,0-6 G
>> 5-23,0-6 G
>> 0-1,0-6 H
>> 2,0-6 H
>> 3-4,0-6 H
>> 5-23,0-6 H
>> 0-1,0-6 I
>> 2,0-6 I
>> 3-4,0-6 I
>> 5-23,0-6 I
>>
>> now to order them
>>
>> 0-1,0-6 B
>> 0-1,0-6 C
>> 0-1,0-6 D
>> 0-1,0-6 F
>> 0-1,0-6 G
>> 0-1,0-6 H
>> 0-1,0-6 I
>> 0-1,7 B
>> 0-1,7 C
>> 2,0-6 A
>> 2,0-6 B
>> 2,0-6 I
>> 2,0-6 H
>> 2,0-6 G
>> 2,0-6 F
>> 2,0-6 E
>> 2,0-6 D
>> 2,0-6 C
>> 2,7 A
>> 2,7 B
>> 2,7 C
>> 3-4,0-6 B
>> 3-4,0-6 C
>> 3-4,0-6 D
>> 3-4,0-6 E
>> 3-4,0-6 F
>> 3-4,0-6 G
>> 3-4,0-6 H
>> 3-4,0-6 I
>> 3-4,7 B
>> 3-4,7 C
>> 5-23,0-6 B
>> 5-23,0-6 C
>> 5-23,0-6 D
>> 5-23,0-6 F
>> 5-23,0-6 G
>> 5-23,0-6 H
>> 5-23,0-6 I
>> 5-23,7 B
>> 5-23,7 C
>>
>> I then simplify this to
>>
>> 0-1,0-6 B,C,D,F,G,H,I
>> 0-1,7 B,C
>> 2,0-6 A,B,C,D,E,F,G,H,I
>> 2,7 A,B,C
>> 3-4,0-6 B,C,D,E,F,G,H,I
>> 3-4,7 B,C
>> 5-23,0-6 B,C,D,F,G,H,I
>> 5-23,7 B,C
>>
>> looking at the lettered tests
>>
>> A) log to /Var/log/mail.log
>> B) programname startswith abc log to /abc
>> C) programname startswith 'acc' and MSG contains 'error' log to /acc
>> D) programname startswith 'acc' log to @1.1.1.1
>> E) programname startswith bcd log to /bcd
>> F) programname contains 'bde' log to @1.1.1.1
>> G) MSG startswith '123' log to/bcd
>> H) MSG contains 'error2' log to /bcd
>> I) log to /var/log/catchall
>>
>> the only ones that can be combined are B-E
>>
>> so if you make tests
>>
>> J) B,C
>> K) B,C,D
>> L) B,C,D,E
>>
>> the results simplify to
>>
>> 0-1,0-6 K,F,G,H,I
>> 0-1,7 J
>> 2,0-6 A,L,F,G,H,I
>> 2,7 A,J
>> 3-4,0-6 L,F,G,H,I
>> 3-4,7 J
>> 5-23,0-6 K,F,G,H,I
>> 5-23,7 J
>>
>> expanded out to show all tests
>>
>> 0-1,0-6
>>    --programname startswith abc log to /abc
>>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>>     \programname startswith 'acc' log to @1.1.1.1
>>    --programname contains 'bde' log to @1.1.1.1
>>    --MSG startswith '123' log to/bcd
>>    --MSG contains 'error2' log to /bcd
>>    --log to /var/log/catchall
>> 0-1,7
>>    --programname startswith abc log to /abc
>>     \programname startswith 'acc' and MSG contains 'error' log to /acc
>> 2,0-6
>>    --log to /var/log/mail.log
>>    --programname startswith abc log to /abc
>>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>>    | programname startswith 'acc' log to @1.1.1.1
>>     \programname startswith bcd log to /bcd
>>    --programname contains 'bde' log to @1.1.1.1
>>    --MSG startswith '123' log to/bcd
>>    --MSG contains 'error2' log to /bcd
>>    --log to /var/log/catchall
>> 2,7
>>    --log to /var/log/mail.log
>>    --programname startswith abc log to /abc
>>     \programname startswith 'acc' and MSG contains 'error' log to /acc
>> 3-4,0-6
>>    --programname startswith abc log to /abc
>>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>>    | programname startswith 'acc' log to @1.1.1.1
>>     \programname startswith bcd log to /bcd
>>    --programname contains 'bde' log to @1.1.1.1
>>    --MSG startswith '123' log to/bcd
>>    --MSG contains 'error2' log to /bcd
>>    --log to /var/log/catchall
>> 3-4,7
>>    --programname startswith abc log to /abc
>>     \programname startswith 'acc' and MSG contains 'error' log to /acc
>> 5-23,0-6
>>    --programname startswith abc log to /abc
>>    | programname startswith 'acc' and MSG contains 'error' log to /acc
>>     \programname startswith 'acc' log to @1.1.1.1
>>    --programname contains 'bde' log to @1.1.1.1
>>    --MSG startswith '123' log to/bcd
>>    --MSG contains 'error2' log to /bcd
>>    --log to /var/log/catchall
>> 5-23,7
>>    --programname startswith abc log to /abc
>>     \programname startswith 'acc' and MSG contains 'error' log to /acc
>>
>>> Rainer
>>>> -----Original Message-----
>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>>> Sent: Sunday, February 28, 2010 2:02 PM
>>>> To: rsyslog-users
>>>> Subject: Re: [rsyslog] Feedback requested: inconsistency in config
>>>> statements
>>>>
>>>> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
>>>>
>>>>> Quickly from the mobile: pri based filters require *excatly* one
>>>> single word (!) lookup currently. You can't beat that.
>>>>>
>>>>> I am not totally sure how you think about parse trees. In my view,
>>>> parsin and filteing are two different things./stages. Maybe we have
>>>> different concepts on our minds?
>>>>
>>>> I am probably using the wrong terminology here.
>>>>
>>>> say you have a set of rules that say
>>>>
>>>> if programname startswith abc log to /abc
>>>> if programname startswith acc log to /acc
>>>> if programname startswith acc log to @1.1.1.1
>>>> if programname startswith bcd log to /bcd
>>>>
>>>> and assuming that these were the only possible programnames for
>>>> simplicity
>>>> in the explination.
>>>>
>>>> the filtering logic would go somthing like this
>>>>
>>>> the rules would compile into a tree
>>>>
>>>> -a-b -> /abc
>>>>    -c -> /acc, at 1.1.1.1
>>>> -b   -> /bcd
>>>>
>>>> receive programname abc
>>>> I have no facility/Pri filter rules
>>>> I have no time filter rules
>>>> I have no system filter rules
>>>> I have programname filter rules
>>>>    look at the first character of the programname
>>>>     it's "a", I have more than one rule that fits that.
>>>>      look at the second character of the programname
>>>>       it's a "b", There are no other decisions to make,
>>>>         invoke the action /abc
>>>>
>>>> receive progranmane bcd
>>>> I have no facility/PRI filter rules
>>>> I have no time filter rules
>>>> I have no system filter rules
>>>> I have programname filter rules
>>>>    look at the first character of the programname
>>>>     it's a "b", There are no other decisions to make,
>>>>       invoke the action /bcd
>>>>
>>>> receive programname acc
>>>> I have no facility/Pri filter rules
>>>> I have no time filter rules
>>>> I have no system filter rules
>>>> I have programname filter rules
>>>>    look at the first character of the programname
>>>>     it's "a", I have more than one rule that fits that.
>>>>      look at the second character of the programname
>>>>       it's a "c", There are no other decisions to make,
>>>>         invoke the action /acc and the action @1.1.1.1
>>>>
>>>> similarly the facility/pri filtering rules would be either compiled
>>>> into a
>>>> tree, or (since it's 256 entries total) into a lookup table with
>>>> pointers
>>>> to the list of actions to take for that entry
>>>>
>>>> the idea is to spend the time at startup to create the tree that
>>>> represents the ruleset in order to speed up the processing of each
>>>> individual message.
>>>>
>>>> the real tree would be a bit more complicated than I describe here
>> as
>>>> it
>>>> needs to have 'anything else' entries between the known entries
>> (which
>>>> means that it would not be able to shortcut quite as much as I
>>>> describe),
>>>> and have provision for 'do a more complicated thing (like regex)
>> here
>>>> and
>>>> if it matches continue'. but except for regex matches, this would
>> make
>>>> processing the rules pretty much independant of how many rules there
>>>> were
>>>> or how complex the ruleset is.
>>>>
>>>> there doe snot need to be one single programname filter tree, if you
>>>> had a
>>>> rule that said
>>>> if pri = info and programname startswith def then def
>>>>
>>>> the pri tree/table would have an entry for info that would point to
>> a
>>>> filter tree for programname that just had the check for def in it
>>>>
>>>> am I makeing any more sense now?
>>>>
>>>> David Lang
>>>>
>>>>> rainer
>>>>>
>>>>> ----- Urspr?ngliche Nachricht -----
>>>>> Von: "david at lang.hm" <david at lang.hm>
>>>>> An: "rsyslog-users" <rsyslog at lists.adiscon.com>
>>>>> Gesendet: 28.02.10 11:28
>>>>> Betreff: Re: [rsyslog] Feedback requested: inconsistency in config
>>>> statements
>>>>>
>>>>> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
>>>>>
>>>>>>>
>>>>>>> if the type is depriciated then the destination would be another
>>>>>>> config_option (which you would then look up)
>>>>>>>
>>>>>>> if the type is 'ignore' then it would just be skipped over
>>>> (possibly
>>>>>>> with
>>>>>>> a warning being logged that the config line is being ignored)
>>>>>>>
>>>>>>> this would also clean up some of the current cases where a
>> boolean
>>>>>>> option
>>>>>>> doesn't honor on/off and true/false.
>>>>>>
>>>>>> True/false is NOT a valid boolean option. See
>>>>>>
>>>>>>
>>>>
>> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=18
>>>> 4c0d87
>>>>>> 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308
>>>>>
>>>>> I was not remembering correctly then, maby it was yes/no vs on/off.
>> I
>>>> know
>>>>> I ran into something where what was documented didn't work and I
>>>> changed
>>>>> it to another version of a boolean answer and it fixed the problem.
>>>>>
>>>>>>> For the second half of the config language (telling rsyslog what
>> to
>>>> do
>>>>>>> with the logs it has received) also has several variations in
>> place
>>>> and
>>>>>>> is
>>>>>>> showing issues.
>>>>>>>
>>>>>>> However I think that the right answer here is to end up
>>>> implementing
>>>>>>> something like the parsing trees and then compile the other
>> config
>>>>>>> language options into that tree to be consistant (and fast) under
>>>> the
>>>>>>> covers, no matter which way the instructions are written (except
>>>> when
>>>>>>> you
>>>>>>> have to use regex matches)
>>>>>>
>>>>>> I don't fully agree here with you. For example, the traditional
>> PRI
>>>> based
>>>>>> filter is lightning fast. I don't see any way it could be nearly
>> as
>>>> fast with
>>>>>> any unified approach. Right now, we have a "unified" filter
>>>> structure, but it
>>>>>> contains three filter cases, each one adding potential power at
>> the
>>>> price of
>>>>>> decreased speed. I think we need to keep different types of
>> filters
>>>> in order
>>>>>> to have some lightning-fast ones. But more of this could be done
>>>> under the
>>>>>> hood.
>>>>>
>>>>> My guess/expectation is that the tree-based parsing would be about
>>>> the
>>>>> same speed as the traditional PRI based filter for choices made
>> based
>>>> on
>>>>> PRI, slowing down for other types of conditions only in that more
>> of
>>>> the
>>>>> message may need to be scanned (and if there is no selection at a
>>>> level,
>>>>> skipping that level should be lightning fast). As a result, a set
>> of
>>>>> filteres based soley on programname would be as fast as the current
>>>> PRI
>>>>> filters.
>>>>>
>>>>> David Lang
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Mon Mar  1 20:58:51 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 1 Mar 2010 20:58:51 +0100
Subject: [rsyslog] Feedback requested: inconsistency in config statements
References: <001201cab868$d148da7c$100013ac@intern.adiscon.com><alpine.DEB.2.00.1002280426420.15263@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA710397F@GRFEXC.intern.adiscon.com><alpine.DEB.2.00.1002281346000.15263@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA7103992@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.00.1003011130460.5131@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103998@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Monday, March 01, 2010 8:39 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Feedback requested: inconsistency in config
> statements
> 
> On Mon, 1 Mar 2010, Rainer Gerhards wrote:
> 
> > Hi David,
> >
> > thanks for the explanation, I now see where you are heading too. I
> have not
> > dug into all details, but find the post very interesting and
> insightful. I'll
> > archive it so that I have it at hand when I can take the route to
> actually
> > implement such an approach. All in all, it still looks rather complex
> to me.
> > I guess it would be useful to have a guy with experience in writing
> compiler
> > code optimizers at hand when I tackle that beast ;)
> 
> you may also want to dig into yacc/bison as they are designed to help
> people build parser trees.

I know yacc but not bison. Yacc creates decent single-threading parsers out
of a fixed set of rules. I never thought about using them for message
parsing, because of the ample limitations it has (most notably unable to run
on multiple threads!).

But we need to keep in mind that now are talking about the filter
expressions. That is a different beast. The thing to look at probably would
be the gcc code optimizer, but I have to admit that is too large an effort
for me right now. A good (dragon?;)) book on compiler optimization technics
for expression trees would probably the right path to take.

> I am assuming that this is the type of thing that syslog-ng is doing to
> be
> able to efficiantly handle a lot of log processing.

I hardly believe so. I don't see how this could work out well...

> I currently have a script that looks for a couple hundred programnames
> and
> puts the data into ~50 different files depending on what that name is.
> Given that the output side of rsyslog is currently the bottleneck, I
> would
> not want to try and implement that in rsyslog today, but if rsyslog
> gained
> the ability to handle this sort of ruleset efficiantly it would greatly
> simplify my life.

Rather than trying to optimize the whole expression engine, this could
probably done very easy with a normalizing parser. As part of the
normalization, it could also create a meta-property "filename" that than is
used inside the rule engine. I'd assume that the effort to create that is by
far less than optimizing the expression tree. *This* is part of my approach
of log normalizing.

Rianer
> 
> David Lang
> 
> > Rainer
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >> Sent: Monday, March 01, 2010 3:16 AM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] Feedback requested: inconsistency in config
> >> statements
> >>
> >> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
> >>
> >>> Hi David,
> >>>
> >>> yes, that makes much more sense to me. Let's call that filter
> >> expression tree
> >>> for the time being (the parse tree used for normalization has some
> >> similar
> >>> ideas, but a totally different scope and semantics).
> >>>
> >>> While I now see where you are heading, I fail to see how this could
> >>> sufficiently easy be implemented for a real (and complex)
> >> configuration.
> >>>
> >>> Let's look at this example conf:
> >>>
> >>> mail.* /Var/log/mail.log
> >>> if programname startswith abc log to /abc
> >>> if programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>> # we don't want debug messages from here on
> >>> *.debug ~
> >>> if programname startswith 'acc' or (programname contains 'bde' and
> >> (facility
> >>>> 1 and
> >>> facility < 5)) log to @1.1.1.1
> >>> if programname startswith bcd or MSG startswith '123' or MSG
> contains
> >>> 'error2' log to /bcd
> >>> *.* /var/log/catchall
> >>>
> >>> I simply cannot envision how you would store this as a tree that
> does
> >> not
> >>> look like what we have today.
> >>
> >> Ok, Here is what I came up with.
> >>
> >> 0-1,0-6
> >>    --programname startswith abc log to /abc
> >>    | programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>     \programname startswith 'acc' log to @1.1.1.1
> >>    --programname contains 'bde' log to @1.1.1.1
> >>    --MSG startswith '123' log to/bcd
> >>    --MSG contains 'error2' log to /bcd
> >>    --log to /var/log/catchall
> >> 0-1,7
> >>    --programname startswith abc log to /abc
> >>     \programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >> 2,0-6
> >>    --log to /var/log/mail.log
> >>    --programname startswith abc log to /abc
> >>    | programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>    | programname startswith 'acc' log to @1.1.1.1
> >>     \programname startswith bcd log to /bcd
> >>    --programname contains 'bde' log to @1.1.1.1
> >>    --MSG startswith '123' log to/bcd
> >>    --MSG contains 'error2' log to /bcd
> >>    --log to /var/log/catchall
> >> 2,7
> >>    --log to /var/log/mail.log
> >>    --programname startswith abc log to /abc
> >>     \programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >> 3-4,0-6
> >>    --programname startswith abc log to /abc
> >>    | programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>    | programname startswith 'acc' log to @1.1.1.1
> >>     \programname startswith bcd log to /bcd
> >>    --programname contains 'bde' log to @1.1.1.1
> >>    --MSG startswith '123' log to/bcd
> >>    --MSG contains 'error2' log to /bcd
> >>    --log to /var/log/catchall
> >> 3-4,7
> >>    --programname startswith abc log to /abc
> >>     \programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >> 5-23,0-6
> >>    --programname startswith abc log to /abc
> >>    | programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>     \programname startswith 'acc' log to @1.1.1.1
> >>    --programname contains 'bde' log to @1.1.1.1
> >>    --MSG startswith '123' log to/bcd
> >>    --MSG contains 'error2' log to /bcd
> >>    --log to /var/log/catchall
> >> 5-23,7
> >>    --programname startswith abc log to /abc
> >>     \programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>
> >> This looks very ugly and is definantly more complicated, but I think
> it
> >> still ends up being a slight (but not drastic) win.
> >>
> >> for the initial ruleset you have to check a minimum of 4 tests and a
> >> maximum of 13 tests (debug messages have 4-5 tests, all others have
> 12-
> >> 13)
> >>
> >> Assuming the facility/priority lookup can be a table lookup, for the
> >> tree-based ruleset you have a minimum of 2 tests and a maximum of 7
> >> tests,
> >> in each case one of the tests is a multi-option comparison which is
> >> slightly more expensive than a single 'startswith' test, but should
> be
> >> cheaper than two 'startswith' tests.
> >>
> >> This definantly trades off start time complexity for runtime
> >> efficiancy.
> >>
> >> David Lang
> >>
> >>
> >> and here is the long, ugly process I went through to derive this.
> >>
> >>
> >>
> >> first normalize the tests (replace 'or' with two lines, change
> facility
> >> and priority to ranges of numbers)
> >>
> >> facility = 2 log to /Var/log/mail.log
> >> programname startswith abc log to /abc
> >> programname startswith 'acc' and MSG contains 'error' log to /acc
> >> severity 0..6 and programname startswith 'acc' log to @1.1.1.1
> >> severity 0..6 and programname startswith bcd log to /bcd
> >> facility 2..4 and severity 0..6 and programname contains 'bde' log
> to
> >> @1.1.1.1
> >> severity 0..6 and MSG startswith '123' log to/bcd
> >> severity 0..6 and MSG contains 'error2' log to /bcd
> >> severity 0..6 and log to /var/log/catchall
> >>
> >> I then add the full facility/priority ranges
> >>
> >>
> >> now normalize the facility/severity tests (split overlaps)
> >>
> >> we have the following ranges for facility
> >> 0-1
> >> 2
> >> 3-4
> >> 5-23
> >>
> >> and for severity
> >> 0-6
> >> 7
> >>
> >> to shorten the lines in the e-mail I will letter the remaining
> >> tests/actions
> >>
> >> A) log to /Var/log/mail.log
> >> B) programname startswith abc log to /abc
> >> C) programname startswith 'acc' and MSG contains 'error' log to /acc
> >> D) programname startswith 'acc' log to @1.1.1.1
> >> E) programname startswith bcd log to /bcd
> >> F) programname contains 'bde' log to @1.1.1.1
> >> G) MSG startswith '123' log to/bcd
> >> H) MSG contains 'error2' log to /bcd
> >> I) log to /var/log/catchall
> >>
> >> so the tree then becomes (straight translation no reordering)
> >>
> >> 2,0-6 A
> >> 2,7 A
> >> 0-1,0-6 B
> >> 0-1,7 B
> >> 2,0-6 B
> >> 2,7 B
> >> 3-4,0-6 B
> >> 3-4,7 B
> >> 5-23,0-6 B
> >> 5-23,7 B
> >> 0-1,0-6 C
> >> 0-1,7 C
> >> 2,0-6 C
> >> 2,7 C
> >> 3-4,0-6 C
> >> 3-4,7 C
> >> 5-23,0-6 C
> >> 5-23,7 C
> >> 0-1,0-6 D
> >> 2,0-6 D
> >> 3-4,0-6 D
> >> 5-23,0-6 D
> >> 2,0-6 E
> >> 3-4,0-6 E
> >> 0-1,0-6 F
> >> 2,0-6 F
> >> 3-4,0-6 F
> >> 5-23,0-6 F
> >> 0-1,0-6 G
> >> 2,0-6 G
> >> 3-4,0-6 G
> >> 5-23,0-6 G
> >> 0-1,0-6 H
> >> 2,0-6 H
> >> 3-4,0-6 H
> >> 5-23,0-6 H
> >> 0-1,0-6 I
> >> 2,0-6 I
> >> 3-4,0-6 I
> >> 5-23,0-6 I
> >>
> >> now to order them
> >>
> >> 0-1,0-6 B
> >> 0-1,0-6 C
> >> 0-1,0-6 D
> >> 0-1,0-6 F
> >> 0-1,0-6 G
> >> 0-1,0-6 H
> >> 0-1,0-6 I
> >> 0-1,7 B
> >> 0-1,7 C
> >> 2,0-6 A
> >> 2,0-6 B
> >> 2,0-6 I
> >> 2,0-6 H
> >> 2,0-6 G
> >> 2,0-6 F
> >> 2,0-6 E
> >> 2,0-6 D
> >> 2,0-6 C
> >> 2,7 A
> >> 2,7 B
> >> 2,7 C
> >> 3-4,0-6 B
> >> 3-4,0-6 C
> >> 3-4,0-6 D
> >> 3-4,0-6 E
> >> 3-4,0-6 F
> >> 3-4,0-6 G
> >> 3-4,0-6 H
> >> 3-4,0-6 I
> >> 3-4,7 B
> >> 3-4,7 C
> >> 5-23,0-6 B
> >> 5-23,0-6 C
> >> 5-23,0-6 D
> >> 5-23,0-6 F
> >> 5-23,0-6 G
> >> 5-23,0-6 H
> >> 5-23,0-6 I
> >> 5-23,7 B
> >> 5-23,7 C
> >>
> >> I then simplify this to
> >>
> >> 0-1,0-6 B,C,D,F,G,H,I
> >> 0-1,7 B,C
> >> 2,0-6 A,B,C,D,E,F,G,H,I
> >> 2,7 A,B,C
> >> 3-4,0-6 B,C,D,E,F,G,H,I
> >> 3-4,7 B,C
> >> 5-23,0-6 B,C,D,F,G,H,I
> >> 5-23,7 B,C
> >>
> >> looking at the lettered tests
> >>
> >> A) log to /Var/log/mail.log
> >> B) programname startswith abc log to /abc
> >> C) programname startswith 'acc' and MSG contains 'error' log to /acc
> >> D) programname startswith 'acc' log to @1.1.1.1
> >> E) programname startswith bcd log to /bcd
> >> F) programname contains 'bde' log to @1.1.1.1
> >> G) MSG startswith '123' log to/bcd
> >> H) MSG contains 'error2' log to /bcd
> >> I) log to /var/log/catchall
> >>
> >> the only ones that can be combined are B-E
> >>
> >> so if you make tests
> >>
> >> J) B,C
> >> K) B,C,D
> >> L) B,C,D,E
> >>
> >> the results simplify to
> >>
> >> 0-1,0-6 K,F,G,H,I
> >> 0-1,7 J
> >> 2,0-6 A,L,F,G,H,I
> >> 2,7 A,J
> >> 3-4,0-6 L,F,G,H,I
> >> 3-4,7 J
> >> 5-23,0-6 K,F,G,H,I
> >> 5-23,7 J
> >>
> >> expanded out to show all tests
> >>
> >> 0-1,0-6
> >>    --programname startswith abc log to /abc
> >>    | programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>     \programname startswith 'acc' log to @1.1.1.1
> >>    --programname contains 'bde' log to @1.1.1.1
> >>    --MSG startswith '123' log to/bcd
> >>    --MSG contains 'error2' log to /bcd
> >>    --log to /var/log/catchall
> >> 0-1,7
> >>    --programname startswith abc log to /abc
> >>     \programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >> 2,0-6
> >>    --log to /var/log/mail.log
> >>    --programname startswith abc log to /abc
> >>    | programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>    | programname startswith 'acc' log to @1.1.1.1
> >>     \programname startswith bcd log to /bcd
> >>    --programname contains 'bde' log to @1.1.1.1
> >>    --MSG startswith '123' log to/bcd
> >>    --MSG contains 'error2' log to /bcd
> >>    --log to /var/log/catchall
> >> 2,7
> >>    --log to /var/log/mail.log
> >>    --programname startswith abc log to /abc
> >>     \programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >> 3-4,0-6
> >>    --programname startswith abc log to /abc
> >>    | programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>    | programname startswith 'acc' log to @1.1.1.1
> >>     \programname startswith bcd log to /bcd
> >>    --programname contains 'bde' log to @1.1.1.1
> >>    --MSG startswith '123' log to/bcd
> >>    --MSG contains 'error2' log to /bcd
> >>    --log to /var/log/catchall
> >> 3-4,7
> >>    --programname startswith abc log to /abc
> >>     \programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >> 5-23,0-6
> >>    --programname startswith abc log to /abc
> >>    | programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>     \programname startswith 'acc' log to @1.1.1.1
> >>    --programname contains 'bde' log to @1.1.1.1
> >>    --MSG startswith '123' log to/bcd
> >>    --MSG contains 'error2' log to /bcd
> >>    --log to /var/log/catchall
> >> 5-23,7
> >>    --programname startswith abc log to /abc
> >>     \programname startswith 'acc' and MSG contains 'error' log to
> /acc
> >>
> >>> Rainer
> >>>> -----Original Message-----
> >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >>>> Sent: Sunday, February 28, 2010 2:02 PM
> >>>> To: rsyslog-users
> >>>> Subject: Re: [rsyslog] Feedback requested: inconsistency in config
> >>>> statements
> >>>>
> >>>> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
> >>>>
> >>>>> Quickly from the mobile: pri based filters require *excatly* one
> >>>> single word (!) lookup currently. You can't beat that.
> >>>>>
> >>>>> I am not totally sure how you think about parse trees. In my
> view,
> >>>> parsin and filteing are two different things./stages. Maybe we
> have
> >>>> different concepts on our minds?
> >>>>
> >>>> I am probably using the wrong terminology here.
> >>>>
> >>>> say you have a set of rules that say
> >>>>
> >>>> if programname startswith abc log to /abc
> >>>> if programname startswith acc log to /acc
> >>>> if programname startswith acc log to @1.1.1.1
> >>>> if programname startswith bcd log to /bcd
> >>>>
> >>>> and assuming that these were the only possible programnames for
> >>>> simplicity
> >>>> in the explination.
> >>>>
> >>>> the filtering logic would go somthing like this
> >>>>
> >>>> the rules would compile into a tree
> >>>>
> >>>> -a-b -> /abc
> >>>>    -c -> /acc, at 1.1.1.1
> >>>> -b   -> /bcd
> >>>>
> >>>> receive programname abc
> >>>> I have no facility/Pri filter rules
> >>>> I have no time filter rules
> >>>> I have no system filter rules
> >>>> I have programname filter rules
> >>>>    look at the first character of the programname
> >>>>     it's "a", I have more than one rule that fits that.
> >>>>      look at the second character of the programname
> >>>>       it's a "b", There are no other decisions to make,
> >>>>         invoke the action /abc
> >>>>
> >>>> receive progranmane bcd
> >>>> I have no facility/PRI filter rules
> >>>> I have no time filter rules
> >>>> I have no system filter rules
> >>>> I have programname filter rules
> >>>>    look at the first character of the programname
> >>>>     it's a "b", There are no other decisions to make,
> >>>>       invoke the action /bcd
> >>>>
> >>>> receive programname acc
> >>>> I have no facility/Pri filter rules
> >>>> I have no time filter rules
> >>>> I have no system filter rules
> >>>> I have programname filter rules
> >>>>    look at the first character of the programname
> >>>>     it's "a", I have more than one rule that fits that.
> >>>>      look at the second character of the programname
> >>>>       it's a "c", There are no other decisions to make,
> >>>>         invoke the action /acc and the action @1.1.1.1
> >>>>
> >>>> similarly the facility/pri filtering rules would be either
> compiled
> >>>> into a
> >>>> tree, or (since it's 256 entries total) into a lookup table with
> >>>> pointers
> >>>> to the list of actions to take for that entry
> >>>>
> >>>> the idea is to spend the time at startup to create the tree that
> >>>> represents the ruleset in order to speed up the processing of each
> >>>> individual message.
> >>>>
> >>>> the real tree would be a bit more complicated than I describe here
> >> as
> >>>> it
> >>>> needs to have 'anything else' entries between the known entries
> >> (which
> >>>> means that it would not be able to shortcut quite as much as I
> >>>> describe),
> >>>> and have provision for 'do a more complicated thing (like regex)
> >> here
> >>>> and
> >>>> if it matches continue'. but except for regex matches, this would
> >> make
> >>>> processing the rules pretty much independant of how many rules
> there
> >>>> were
> >>>> or how complex the ruleset is.
> >>>>
> >>>> there doe snot need to be one single programname filter tree, if
> you
> >>>> had a
> >>>> rule that said
> >>>> if pri = info and programname startswith def then def
> >>>>
> >>>> the pri tree/table would have an entry for info that would point
> to
> >> a
> >>>> filter tree for programname that just had the check for def in it
> >>>>
> >>>> am I makeing any more sense now?
> >>>>
> >>>> David Lang
> >>>>
> >>>>> rainer
> >>>>>
> >>>>> ----- Urspr?ngliche Nachricht -----
> >>>>> Von: "david at lang.hm" <david at lang.hm>
> >>>>> An: "rsyslog-users" <rsyslog at lists.adiscon.com>
> >>>>> Gesendet: 28.02.10 11:28
> >>>>> Betreff: Re: [rsyslog] Feedback requested: inconsistency in
> config
> >>>> statements
> >>>>>
> >>>>> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
> >>>>>
> >>>>>>>
> >>>>>>> if the type is depriciated then the destination would be
> another
> >>>>>>> config_option (which you would then look up)
> >>>>>>>
> >>>>>>> if the type is 'ignore' then it would just be skipped over
> >>>> (possibly
> >>>>>>> with
> >>>>>>> a warning being logged that the config line is being ignored)
> >>>>>>>
> >>>>>>> this would also clean up some of the current cases where a
> >> boolean
> >>>>>>> option
> >>>>>>> doesn't honor on/off and true/false.
> >>>>>>
> >>>>>> True/false is NOT a valid boolean option. See
> >>>>>>
> >>>>>>
> >>>>
> >>
> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=18
> >>>> 4c0d87
> >>>>>> 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308
> >>>>>
> >>>>> I was not remembering correctly then, maby it was yes/no vs
> on/off.
> >> I
> >>>> know
> >>>>> I ran into something where what was documented didn't work and I
> >>>> changed
> >>>>> it to another version of a boolean answer and it fixed the
> problem.
> >>>>>
> >>>>>>> For the second half of the config language (telling rsyslog
> what
> >> to
> >>>> do
> >>>>>>> with the logs it has received) also has several variations in
> >> place
> >>>> and
> >>>>>>> is
> >>>>>>> showing issues.
> >>>>>>>
> >>>>>>> However I think that the right answer here is to end up
> >>>> implementing
> >>>>>>> something like the parsing trees and then compile the other
> >> config
> >>>>>>> language options into that tree to be consistant (and fast)
> under
> >>>> the
> >>>>>>> covers, no matter which way the instructions are written
> (except
> >>>> when
> >>>>>>> you
> >>>>>>> have to use regex matches)
> >>>>>>
> >>>>>> I don't fully agree here with you. For example, the traditional
> >> PRI
> >>>> based
> >>>>>> filter is lightning fast. I don't see any way it could be nearly
> >> as
> >>>> fast with
> >>>>>> any unified approach. Right now, we have a "unified" filter
> >>>> structure, but it
> >>>>>> contains three filter cases, each one adding potential power at
> >> the
> >>>> price of
> >>>>>> decreased speed. I think we need to keep different types of
> >> filters
> >>>> in order
> >>>>>> to have some lightning-fast ones. But more of this could be done
> >>>> under the
> >>>>>> hood.
> >>>>>
> >>>>> My guess/expectation is that the tree-based parsing would be
> about
> >>>> the
> >>>>> same speed as the traditional PRI based filter for choices made
> >> based
> >>>> on
> >>>>> PRI, slowing down for other types of conditions only in that more
> >> of
> >>>> the
> >>>>> message may need to be scanned (and if there is no selection at a
> >>>> level,
> >>>>> skipping that level should be lightning fast). As a result, a set
> >> of
> >>>>> filteres based soley on programname would be as fast as the
> current
> >>>> PRI
> >>>>> filters.
> >>>>>
> >>>>> David Lang
> >>>>> _______________________________________________
> >>>>> rsyslog mailing list
> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>> http://www.rsyslog.com
> >>>>> _______________________________________________
> >>>>> rsyslog mailing list
> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>> http://www.rsyslog.com
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.com
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >>>
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From lanas at securenet.net  Tue Mar  2 02:43:31 2010
From: lanas at securenet.net (lanas)
Date: Mon, 1 Mar 2010 20:43:31 -0500
Subject: [rsyslog] Sending time in remote syslog - test
Message-ID: <20100301204331.1686d17d@mistral.stie>

Hello,

  I did a test using %timereported% in a rsyslog running on a server,
to see if the time of a syslog msg sent from a unit was shown using
the local time of the unit and not the time it was received.

  To easily see the difference I out the time 2 hours back on the
unit.  The unit is running syslogd.  That is, the original syslogd,
not the ng.

  The unit running ran syslogd with the following in its
configuration:

*.*     @10.200.19.162

  The logger utility was used to create log msgs locally on the unit.
These log msgs are then sent to rsyslog on the remote server.

  The workstation was running rsyslog with either:

#$template precise,"%timereported% from: %HOSTNAME% %syslogtag%:%msg%\n"
$template precise,"%TIMESTAMP% from: %HOSTNAME% %syslogtag%:%msg%\n"

  And tests were done with the following commented or not:

$ActionFileDefaultTemplate precise

  (I am not sure if the 'precise' template definition must be used
with $ActionFileDefaultTemplate to take effect)

  The following was then found on the unit after executing the logger
utility:

Feb 27 10:55:34 localhost syslogd 1.4.1#18RR6: restart.
Feb 27 10:55:35 localhost kernel: klogd 1.4.1#18RR6, log source =
                /proc/kmsg started.
Feb 27 10:55:44 localhost root: TEST7

  And these messages were received by rsyslog (after restarting
rsyslog to have its version info included here) :

Feb 27 12:52:32 from: localhost rsyslogd:: [origin software="rsyslogd"
    swVersion="3.22.1" x-pid="6909" x-info="http://www.rsyslog.com"]
    (re)start
Feb 27 12:54:38 from: syslogd 1.4.1#18RR6:: restart.
Feb 27 12:54:39 from: brouter kernel:: klogd 1.4.1#18RR6, log source =
                /proc/kmsg started.
Feb 27 12:54:48 from: brouter root:: TEST7

  The TEST7 msg did now show the unit's time.

  Now, I installed rsyslog at the unit as a drop-in syslog replacement
w/o any configuration change and when rsyslog is used, the local unit
time is received and shown by the server.

  Can we presume that syslogd first generation does not send the time
to a remote syslogging facility ?

Regards,

Al



From rgerhards at hq.adiscon.com  Tue Mar  2 10:03:20 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 2 Mar 2010 10:03:20 +0100
Subject: [rsyslog] Sending time in remote syslog - test
References: <20100301204331.1686d17d@mistral.stie>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71039A1@GRFEXC.intern.adiscon.com>

you can use 

$template raw,"%rawmsg%\n"

to see exactly what the remote box sends. If I recall correctly, sysklogd
sends timestamps, but I may be wrong

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of lanas
> Sent: Tuesday, March 02, 2010 2:44 AM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] Sending time in remote syslog - test
> 
> Hello,
> 
>   I did a test using %timereported% in a rsyslog running on a server,
> to see if the time of a syslog msg sent from a unit was shown using
> the local time of the unit and not the time it was received.
> 
>   To easily see the difference I out the time 2 hours back on the
> unit.  The unit is running syslogd.  That is, the original syslogd,
> not the ng.
> 
>   The unit running ran syslogd with the following in its
> configuration:
> 
> *.*     @10.200.19.162
> 
>   The logger utility was used to create log msgs locally on the unit.
> These log msgs are then sent to rsyslog on the remote server.
> 
>   The workstation was running rsyslog with either:
> 
> #$template precise,"%timereported% from: %HOSTNAME%
> %syslogtag%:%msg%\n"
> $template precise,"%TIMESTAMP% from: %HOSTNAME% %syslogtag%:%msg%\n"
> 
>   And tests were done with the following commented or not:
> 
> $ActionFileDefaultTemplate precise
> 
>   (I am not sure if the 'precise' template definition must be used
> with $ActionFileDefaultTemplate to take effect)
> 
>   The following was then found on the unit after executing the logger
> utility:
> 
> Feb 27 10:55:34 localhost syslogd 1.4.1#18RR6: restart.
> Feb 27 10:55:35 localhost kernel: klogd 1.4.1#18RR6, log source =
>                 /proc/kmsg started.
> Feb 27 10:55:44 localhost root: TEST7
> 
>   And these messages were received by rsyslog (after restarting
> rsyslog to have its version info included here) :
> 
> Feb 27 12:52:32 from: localhost rsyslogd:: [origin software="rsyslogd"
>     swVersion="3.22.1" x-pid="6909" x-info="http://www.rsyslog.com"]
>     (re)start
> Feb 27 12:54:38 from: syslogd 1.4.1#18RR6:: restart.
> Feb 27 12:54:39 from: brouter kernel:: klogd 1.4.1#18RR6, log source =
>                 /proc/kmsg started.
> Feb 27 12:54:48 from: brouter root:: TEST7
> 
>   The TEST7 msg did now show the unit's time.
> 
>   Now, I installed rsyslog at the unit as a drop-in syslog replacement
> w/o any configuration change and when rsyslog is used, the local unit
> time is received and shown by the server.
> 
>   Can we presume that syslogd first generation does not send the time
> to a remote syslogging facility ?
> 
> Regards,
> 
> Al
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From mbiebl at gmail.com  Tue Mar  2 14:11:06 2010
From: mbiebl at gmail.com (Michael Biebl)
Date: Tue, 2 Mar 2010 14:11:06 +0100
Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103983@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com>
	<9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com>
	<aa76cdac1002260355l26269784y8fb8ccd45caeaa12@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com>
	<aa76cdac1002260443p4c7744e2ha1c8c2372c741ebf@mail.gmail.com>
	<aa76cdac1002280515g72c07527mae8ebc08d4ecd404@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103980@GRFEXC.intern.adiscon.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103981@GRFEXC.intern.adiscon.com>
	<aa76cdac1002281500g3d50db29rcd20fea1fbf08d48@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103983@GRFEXC.intern.adiscon.com>
Message-ID: <aa76cdac1003020511n5378077ag313de591d05cb4fa@mail.gmail.com>

2010/3/1 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
>> Sent: Monday, March 01, 2010 12:01 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released
>>
>> I use $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>> in my rsyslog.conf, but the new ompipe module does not seem to respect
>> that setting but always uses the high-resolution time format.
>
> I see - that was a side-effect that pipes used the same output module like
> regular pipes. But I have now re-established it, please try the newest git
> v4-stable branch.

Hi Rainer,

seems current v4-stable works fine now wrt ActionFileDefaultTemplate and ompipe.

Thanks,
Michael


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Tue Mar  2 14:14:08 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 2 Mar 2010 14:14:08 +0100
Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released
References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com><aa76cdac1002260355l26269784y8fb8ccd45caeaa12@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com><aa76cdac1002260443p4c7744e2ha1c8c2372c741ebf@mail.gmail.com><aa76cdac1002280515g72c07527mae8ebc08d4ecd404@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103980@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103981@GRFEXC.intern.adiscon.com><aa76cdac1002281500g3d50db29rcd20fea1fbf08d48@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103983@GRFEXC.intern.adiscon.com>
	<aa76cdac1003020511n5378077ag313de591d05cb4fa@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71039B4@GRFEXC.intern.adiscon.com>

> seems current v4-stable works fine now wrt ActionFileDefaultTemplate
> and ompipe.

Thanks for letting me know. I'll release that as 4.6.1, but wait for a couple
of other things so that they can make a new release together.

Rainer

> 
> Thanks,
> Michael




From david at lang.hm  Wed Mar  3 07:14:30 2010
From: david at lang.hm (david at lang.hm)
Date: Tue, 2 Mar 2010 22:14:30 -0800 (PST)
Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71039B4@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com><aa76cdac1002260355l26269784y8fb8ccd45caeaa12@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com><aa76cdac1002260443p4c7744e2ha1c8c2372c741ebf@mail.gmail.com><aa76cdac1002280515g72c07527mae8ebc08d4ecd404@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103980@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103981@GRFEXC.intern.adiscon.com><aa76cdac1002281500g3d50db29rcd20fea1fbf08d48@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103983@GRFEXC.intern.adiscon.com>
	<aa76cdac1003020511n5378077ag313de591d05cb4fa@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA71039B4@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.2.00.1003022213520.15263@asgard.lang.hm>

slightly off topic, but still related to releases, how close do you think 
we are to a new stable 5.x release?

David Lang

On Tue, 2 Mar 2010, Rainer Gerhards wrote:

>> seems current v4-stable works fine now wrt ActionFileDefaultTemplate
>> and ompipe.
>
> Thanks for letting me know. I'll release that as 4.6.1, but wait for a couple
> of other things so that they can make a new release together.
>
> Rainer
>
>>
>> Thanks,
>> Michael
>
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Wed Mar  3 07:17:11 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 3 Mar 2010 07:17:11 +0100
Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released
References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com><aa76cdac1002260355l26269784y8fb8ccd45caeaa12@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com><aa76cdac1002260443p4c7744e2ha1c8c2372c741ebf@mail.gmail.com><aa76cdac1002280515g72c07527mae8ebc08d4ecd404@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103980@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103981@GRFEXC.intern.adiscon.com><aa76cdac1002281500g3d50db29rcd20fea1fbf08d48@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103983@GRFEXC.intern.adiscon.com><aa76cdac1003020511n5378077ag313de591d05cb4fa@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA71039B4@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.00.1003022213520.15263@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71039B8@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Wednesday, March 03, 2010 7:15 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released
> 
> slightly off topic, but still related to releases, how close do you
> think
> we are to a new stable 5.x release?

very, very close. I originally intended to release it Monday, but an issue
with 4.6.0 came up that, if possible at all, I would like to integrate.
Yesterday, a bug was filed that points into the same direction:

http://bugzilla.adiscon.com/show_bug.cgi?id=182

If I can't find it quickly enough, I'll probably "timeout" next Monday.

Rainer


From reader at newsguy.com  Wed Mar  3 17:43:07 2010
From: reader at newsguy.com (Harry Putnam)
Date: Wed, 03 Mar 2010 10:43:07 -0600
Subject: [rsyslog] How rsyslog derives hostname
References: <87hbp0eodt.fsf@newsguy.com> <4B8B6F9D.5040803@jackpot.uk.net>
Message-ID: <87ocj59x0k.fsf@newsguy.com>

"Mr. Demeanour" <mrdemeanour at jackpot.uk.net> writes:

> Harry Putnam wrote:
>> 
>> Further... I'm not sure where cmds like gethostbyname (and similar
>> ones) get the info either... for all I know they may just grep
>> /etc/hosts.
>
> man resolv.conf (i.e. it's configurable).

I'm not sure how that would return the miss-typed name in /etc/hosts
There is not mention of the hosts file in man resolve.conf what so ever.

If I set a miss-typed name in /etc/hosts

It appears in rsyslog logs   But /etc/resolv.conf only point to
these nameservers

  nameserver 68.87.72.130 <=A comcast nameserver

  nameserver 192.168.0.20 <= My local router which uses the 
                             nameserver above

Neither of these nameservers will be able to resolve my localhosts
name.  It's not a real domain just a home lan.

So something is searching /etc/hosts and giving the info to rsyslog.

However the command `hostname' will return the correct
hostname... not the miss-typed host in /etc/hosts.

Apparently the command `hostname' uses a different mechanism.
I.E scanning /etc/conf.d/hostname... it seems like rsyslog should do
the same to discover the localhosts name.

I see from Gerhards response:
> I need to re-read this more careful, but for now let me say that
> rsyslog does not touch /etc/hosts at all. It exlusively relies on
> what is returned by the OS. But the "source" property is not locally
> generated, it contains whatever the sender placed into the relevant
> field.

That apparently the `sender' (my localhost) is telling rsyslog to use
the miss-typed host name in /etc/hosts.

He says rsyslog exclusively relies on what is returned by the os. Yes
but by what mechanism?

For example: rsyslog could rely on the output of the `hostname' cmd.
But apparently calls something else.

  If it calls resolv.conf... those nameservers will not return
  anything useful.



From rgerhards at hq.adiscon.com  Wed Mar  3 17:49:55 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 3 Mar 2010 17:49:55 +0100
Subject: [rsyslog] How rsyslog derives hostname
References: <87hbp0eodt.fsf@newsguy.com> <4B8B6F9D.5040803@jackpot.uk.net>
	<87ocj59x0k.fsf@newsguy.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71039C8@GRFEXC.intern.adiscon.com>

> He says rsyslog exclusively relies on what is returned by the os. Yes
> but by what mechanism?

gethostname(), what according to the man page is the mechanism to use ;)

http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/net.c;h=ab431f7cddfda4
1b689e6ab4819b5ffe031e52fb;hb=HEAD#l1331

Rainer


From reader at newsguy.com  Wed Mar  3 17:54:09 2010
From: reader at newsguy.com (Harry Putnam)
Date: Wed, 03 Mar 2010 10:54:09 -0600
Subject: [rsyslog] How rsyslog derives hostname
References: <87hbp0eodt.fsf@newsguy.com>
	<9B6E2A8877C38245BFB15CC491A11DA710398A@GRFEXC.intern.adiscon.com>
Message-ID: <87hbox9wi6.fsf@newsguy.com>

"Rainer Gerhards" <rgerhards at hq.adiscon.com> writes:

> I need to re-read this more careful, but for now let me say that rsyslog does
> not touch /etc/hosts at all. It exlusively relies on what is returned by the
> OS. But the "source" property is not locally generated, it contains whatever
> the sender placed into the relevant field.

Good info ... thanks.  So my host is sending the miss-typed host name
to rsyslog by having it in the log line 
   DATE hostname => blah

So I guess the question is how does my host derive the hostname it
sticks into log messages.

So thanks... and I already know the localhost will scan the /etc/hosts
file when resolving the localhost name... so there is the culprit.

Of course the culprit is really me, who miss-typed the host name.

Thanks.



From reader at newsguy.com  Wed Mar  3 18:00:46 2010
From: reader at newsguy.com (Harry Putnam)
Date: Wed, 03 Mar 2010 11:00:46 -0600
Subject: [rsyslog] How rsyslog derives hostname
References: <87hbp0eodt.fsf@newsguy.com> <4B8B6F9D.5040803@jackpot.uk.net>
	<87ocj59x0k.fsf@newsguy.com>
	<9B6E2A8877C38245BFB15CC491A11DA71039C8@GRFEXC.intern.adiscon.com>
Message-ID: <87d3zl9w75.fsf@newsguy.com>

"Rainer Gerhards" <rgerhards at hq.adiscon.com> writes:

>> He says rsyslog exclusively relies on what is returned by the os. Yes
>> but by what mechanism?
>
> gethostname(), what according to the man page is the mechanism to use ;)
>
> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/net.c;h=ab431f7cddfda4
> 1b689e6ab4819b5ffe031e52fb;hb=HEAD#l1331

Looks pretty complicated... gack.

Thanks... Apparently somewhere in the workings of gethostname() the
/etc/hosts file is scanned.

Thanks.




From rgerhards at hq.adiscon.com  Wed Mar  3 18:02:59 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 3 Mar 2010 18:02:59 +0100
Subject: [rsyslog] How rsyslog derives hostname
References: <87hbp0eodt.fsf@newsguy.com><9B6E2A8877C38245BFB15CC491A11DA710398A@GRFEXC.intern.adiscon.com>
	<87hbox9wi6.fsf@newsguy.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71039C9@GRFEXC.intern.adiscon.com>

> > I need to re-read this more careful, but for now let me say that
> rsyslog does
> > not touch /etc/hosts at all. It exlusively relies on what is returned
> by the
> > OS. But the "source" property is not locally generated, it contains
> whatever
> > the sender placed into the relevant field.
> 
> Good info ... thanks.  So my host is sending the miss-typed host name
> to rsyslog by having it in the log line
>    DATE hostname => blah

I still intended to do a write-up in the cookbook, but more important things
got into the way. So here quickly:

You need to be careful when using hostname to build pathes. Bad guys may put
malicous sequences into it. There are property replacer options to solve
that, primarily removing slashes (something along the lines of "secpath" or
so).

If possible, it is better to use fromhost-ip or fromhost, but that is only
the last sender, so not really useful in a relay chain.

Rainer

> 
> So I guess the question is how does my host derive the hostname it
> sticks into log messages.
> 
> So thanks... and I already know the localhost will scan the /etc/hosts
> file when resolving the localhost name... so there is the culprit.
> 
> Of course the culprit is really me, who miss-typed the host name.
> 
> Thanks.
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Wed Mar  3 18:03:33 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 3 Mar 2010 18:03:33 +0100
Subject: [rsyslog] How rsyslog derives hostname
References: <87hbp0eodt.fsf@newsguy.com>
	<4B8B6F9D.5040803@jackpot.uk.net><87ocj59x0k.fsf@newsguy.com><9B6E2A8877C38245BFB15CC491A11DA71039C8@GRFEXC.intern.adiscon.com>
	<87d3zl9w75.fsf@newsguy.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71039CA@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Harry Putnam
> Sent: Wednesday, March 03, 2010 6:01 PM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] How rsyslog derives hostname
> 
> "Rainer Gerhards" <rgerhards at hq.adiscon.com> writes:
> 
> >> He says rsyslog exclusively relies on what is returned by the os.
> Yes
> >> but by what mechanism?
> >
> > gethostname(), what according to the man page is the mechanism to use
> ;)
> >
> >
> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/net.c;h=ab431f7c
> ddfda4
> > 1b689e6ab4819b5ffe031e52fb;hb=HEAD#l1331
> 
> Looks pretty complicated... gack.
> 
> Thanks... Apparently somewhere in the workings of gethostname() the
> /etc/hosts file is scanned.

My (uneducated) guess is that this is distro/version specific.

Raienr


From iamsayan at gmail.com  Wed Mar  3 18:19:52 2010
From: iamsayan at gmail.com (Sayan Chowdhury)
Date: Wed, 3 Mar 2010 22:49:52 +0530
Subject: [rsyslog] How rsyslog derives hostname
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71039CA@GRFEXC.intern.adiscon.com>
References: <87hbp0eodt.fsf@newsguy.com> <4B8B6F9D.5040803@jackpot.uk.net>
	<87ocj59x0k.fsf@newsguy.com>
	<9B6E2A8877C38245BFB15CC491A11DA71039C8@GRFEXC.intern.adiscon.com>
	<87d3zl9w75.fsf@newsguy.com>
	<9B6E2A8877C38245BFB15CC491A11DA71039CA@GRFEXC.intern.adiscon.com>
Message-ID: <87a1ae541003030919j632dac91n75c67717248a21c5@mail.gmail.com>

Hello Harry,
Look at your /etc/host.conf file, it's most probably set up to look at the
/etc/hosts file before the name servers.
Regards,
Sayan

On Wed, Mar 3, 2010 at 10:33 PM, Rainer Gerhards
<rgerhards at hq.adiscon.com>wrote:

> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Harry Putnam
> > Sent: Wednesday, March 03, 2010 6:01 PM
> > To: rsyslog at lists.adiscon.com
> > Subject: Re: [rsyslog] How rsyslog derives hostname
> >
> > "Rainer Gerhards" <rgerhards at hq.adiscon.com> writes:
> >
> > >> He says rsyslog exclusively relies on what is returned by the os.
> > Yes
> > >> but by what mechanism?
> > >
> > > gethostname(), what according to the man page is the mechanism to use
> > ;)
> > >
> > >
> > http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/net.c;h=ab431f7c
> > ddfda4
> > > 1b689e6ab4819b5ffe031e52fb;hb=HEAD#l1331
> >
> > Looks pretty complicated... gack.
> >
> > Thanks... Apparently somewhere in the workings of gethostname() the
> > /etc/hosts file is scanned.
>
> My (uneducated) guess is that this is distro/version specific.
>
> Raienr
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From mrdemeanour at jackpot.uk.net  Wed Mar  3 19:18:16 2010
From: mrdemeanour at jackpot.uk.net (Mr. Demeanour)
Date: Wed, 03 Mar 2010 18:18:16 +0000
Subject: [rsyslog] How rsyslog derives hostname
In-Reply-To: <87ocj59x0k.fsf@newsguy.com>
References: <87hbp0eodt.fsf@newsguy.com> <4B8B6F9D.5040803@jackpot.uk.net>
	<87ocj59x0k.fsf@newsguy.com>
Message-ID: <4B8EA7E8.9020604@jackpot.uk.net>

Harry Putnam wrote:
> "Mr. Demeanour" <mrdemeanour at jackpot.uk.net> writes:
> 
>> Harry Putnam wrote:
>>> Further... I'm not sure where cmds like gethostbyname (and
>>> similar ones) get the info either... for all I know they may just
>>> grep /etc/hosts.
>> man resolv.conf (i.e. it's configurable).
> 
> I'm not sure how that would return the miss-typed name in /etc/hosts 
> There is not mention of the hosts file in man resolve.conf what so
> ever.

Sorry, my response was both too terse and inaccurate. I should have
pointed you instead at /etc/nsswitch.conf, which the name-resolution
library (i.e. gethostbyname()) uses to figure-out how it's supposed to
resolve hostnames into addresses.

There's a line in there that starts with "hosts: ", usually followed on
the same line by "files" and "dns" in that order (and often accompanied
by other bits). That says that to resolve "hosts", you should first look
at "files", then at "dns". In this context, "files" means "/etc/hosts".

So if your nsswitch.conf mentions "files" in the "hosts:" line before
"dns", that is telling the local name-resolution library to look at
/etc/hosts, and use any answer it can get from there in preference to a
real DNS lookup.

I hope that's a bit more helpful!

-- 
Jack.


From tbergfeld at hq.adiscon.com  Thu Mar  4 11:47:36 2010
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Thu, 4 Mar 2010 11:47:36 +0100
Subject: [rsyslog] rsyslog 4.6.1 (v4-stable) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71039DD@GRFEXC.intern.adiscon.com>

Hi all,
We have just released rsyslog 4.6.1.

Rsyslog 4.6.1 is a bug fixing release. Most importantly, it fixes an issue
with the build system that could potentially result in segfaults, especially
on 32 bit machines. Please review the ChangeLog for details.

This is a recommended update for all v4-stable users.

ChangeLog:

http://www.rsyslog.com/Article445.phtml

Download:

http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-196.phtml


As always, feedback is appreciated.

Best regards,
Tom Bergfeld
--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com


From lanas at securenet.net  Thu Mar  4 12:22:13 2010
From: lanas at securenet.net (lanas)
Date: Thu, 4 Mar 2010 06:22:13 -0500
Subject: [rsyslog] Specifying the source address
Message-ID: <20100304062213.5e5c5a3e@mistral.stie>

Hello,

  Is it possible to specify the source address of rsyslog log messages
when logging to a remote host ?

Thanks.


From david at lang.hm  Thu Mar  4 21:13:10 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 4 Mar 2010 12:13:10 -0800 (PST)
Subject: [rsyslog] How rsyslog derives hostname
In-Reply-To: <87a1ae541003030919j632dac91n75c67717248a21c5@mail.gmail.com>
References: <87hbp0eodt.fsf@newsguy.com> <4B8B6F9D.5040803@jackpot.uk.net>
	<87ocj59x0k.fsf@newsguy.com>
	<9B6E2A8877C38245BFB15CC491A11DA71039C8@GRFEXC.intern.adiscon.com>
	<87d3zl9w75.fsf@newsguy.com>
	<9B6E2A8877C38245BFB15CC491A11DA71039CA@GRFEXC.intern.adiscon.com>
	<87a1ae541003030919j632dac91n75c67717248a21c5@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1003041212550.5131@asgard.lang.hm>

or there may be a services-switch file

David Lang

On Wed, 3 Mar 2010, Sayan Chowdhury wrote:

> Hello Harry,
> Look at your /etc/host.conf file, it's most probably set up to look at the
> /etc/hosts file before the name servers.
> Regards,
> Sayan
>
> On Wed, Mar 3, 2010 at 10:33 PM, Rainer Gerhards
> <rgerhards at hq.adiscon.com>wrote:
>
>>> -----Original Message-----
>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>> bounces at lists.adiscon.com] On Behalf Of Harry Putnam
>>> Sent: Wednesday, March 03, 2010 6:01 PM
>>> To: rsyslog at lists.adiscon.com
>>> Subject: Re: [rsyslog] How rsyslog derives hostname
>>>
>>> "Rainer Gerhards" <rgerhards at hq.adiscon.com> writes:
>>>
>>>>> He says rsyslog exclusively relies on what is returned by the os.
>>> Yes
>>>>> but by what mechanism?
>>>>
>>>> gethostname(), what according to the man page is the mechanism to use
>>> ;)
>>>>
>>>>
>>> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/net.c;h=ab431f7c
>>> ddfda4
>>>> 1b689e6ab4819b5ffe031e52fb;hb=HEAD#l1331
>>>
>>> Looks pretty complicated... gack.
>>>
>>> Thanks... Apparently somewhere in the workings of gethostname() the
>>> /etc/hosts file is scanned.
>>
>> My (uneducated) guess is that this is distro/version specific.
>>
>> Raienr
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From reader at newsguy.com  Fri Mar  5 02:39:14 2010
From: reader at newsguy.com (Harry Putnam)
Date: Thu, 04 Mar 2010 19:39:14 -0600
Subject: [rsyslog] How rsyslog derives hostname
References: <87hbp0eodt.fsf@newsguy.com> <4B8B6F9D.5040803@jackpot.uk.net>
	<87ocj59x0k.fsf@newsguy.com>
	<9B6E2A8877C38245BFB15CC491A11DA71039C8@GRFEXC.intern.adiscon.com>
	<87d3zl9w75.fsf@newsguy.com>
	<9B6E2A8877C38245BFB15CC491A11DA71039CA@GRFEXC.intern.adiscon.com>
	<87a1ae541003030919j632dac91n75c67717248a21c5@mail.gmail.com>
Message-ID: <87hbov8s3h.fsf@newsguy.com>

> Hello Harry,
> Look at your /etc/host.conf file, it's most probably set up to look at the
> /etc/hosts file before the name servers.
> Regards,
> Sayan

Yup... that seems to be where its getting set at... thanks.
from /etc/host.conf
[...]
  # This keyword specifies how host lookups are to be performed. It
  # should be followed by one or more lookup methods, separated by
  # commas.  Valid methods are bind, hosts, and nis.
  #
order hosts, bind
[...]



From tbergfeld at hq.adiscon.com  Mon Mar  8 15:29:13 2010
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Mon, 8 Mar 2010 15:29:13 +0100
Subject: [rsyslog] rsyslog 5.4.0 (v5-stable) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103A33@GRFEXC.intern.adiscon.com>

Hi all,
Today, we released rsyslog 5.4.0, a new v5-stable branch. This version begins
a new stable series based on the 5.3.x series, which has been proven rather
well in practice. Most importantly, the previous 5.2.x stable series had some
serious issues. The new 5.4.0 contains fixes for all known problems. The main
new feature is speed: several optimizations were done, including support for
epoll in tcp listeners.

Users of 5.2.x versions are strongly encouraged to upgrade to 5.4.0.


ChangeLog:

http://www.rsyslog.com/Article447.phtml

Download:

http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-197.phtml


As always, feedback is appreciated.

Best regards,
Tom Bergfeld
--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .


From mbiebl at gmail.com  Tue Mar  9 01:25:48 2010
From: mbiebl at gmail.com (Michael Biebl)
Date: Tue, 9 Mar 2010 01:25:48 +0100
Subject: [rsyslog] rsyslog 5.4.0 (v5-stable) released
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103A33@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA7103A33@GRFEXC.intern.adiscon.com>
Message-ID: <aa76cdac1003081625w6dac4ec3raf85bde7491c8338@mail.gmail.com>

Hi everyone!

2010/3/8 Tom Bergfeld <tbergfeld at hq.adiscon.com>:
> Hi all,
> Today, we released rsyslog 5.4.0, a new v5-stable branch. This version begins

So, the help output of rsyslogd has

To run rsyslogd in native mode, use "rsyslogd -c5 <other options>"

Maybe I'm just missing some link to the documentation, but is it
documented somewhere, which features are enabled/available when using
-c5 compared to using -c4 or -c3.
Or put it differently: What's the difference between using -c3 and -c5
for 5.4.0?

Michael


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From pgollucci at p6m7g8.com  Tue Mar  9 02:20:44 2010
From: pgollucci at p6m7g8.com (Philip M. Gollucci)
Date: Tue, 9 Mar 2010 01:20:44 +0000
Subject: [rsyslog] Fwd: cvs commit: ports/sysutils/rsyslog5 Makefile
 distinfo pkg-plist ports/sysutils/rsyslog5/files rsyslogd.in
Message-ID: <4B95A26C.6080301@p6m7g8.com>

fwiw, now in freebsd ports.

-------- Original Message --------
Subject: cvs commit: ports/sysutils/rsyslog5 Makefile distinfo pkg-plist 
         ports/sysutils/rsyslog5/files rsyslogd.in
Date: Mon, 8 Mar 2010 23:15:09 +0000 (UTC)
From: Philip M. Gollucci <pgollucci at FreeBSD.org>
To: ports-committers at FreeBSD.org, cvs-ports at FreeBSD.org, cvs-all at FreeBSD.org

pgollucci    2010-03-08 23:15:09 UTC

   FreeBSD ports repository

   Modified files:
     sysutils/rsyslog5    Makefile distinfo pkg-plist
     sysutils/rsyslog5/files rsyslogd.in
   Log:
   - Update to 5.4.0 [1]
   - Remove -a and old school sockets from rc.d script [myself]

   PR:             ports/144566 [1]
   Submitted by:   Cristiano Rolim Pereira <cristianorolim at hotmail.com> 
(maintainer) [1]

   Feature safe:   yes

   Revision  Changes    Path
   1.13      +1 -1      ports/sysutils/rsyslog5/Makefile
   1.13      +3 -3      ports/sysutils/rsyslog5/distinfo
   1.3       +1 -54     ports/sysutils/rsyslog5/files/rsyslogd.in
   1.8       +1 -0      ports/sysutils/rsyslog5/pkg-plist

http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/Makefile.diff?r1=1.12&r2=1.13&f=h
| --- ports/sysutils/rsyslog5/Makefile	2010/01/28 20:51:59	1.12
| +++ ports/sysutils/rsyslog5/Makefile	2010/03/08 23:15:09	1.13
| @@ -2,11 +2,11 @@
|  # Date created:     29 December 2008
|  # Whom:             Cristiano Rolim Pereira <cristianorolim at hotmail.com>
|  #
| -# $FreeBSD: 
/usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/Makefile,v 1.12 
2010/01/28 20:51:59 glarkin Exp $
| +# $FreeBSD: 
/usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/Makefile,v 1.13 
2010/03/08 23:15:09 pgollucci Exp $
|  #
|
|  PORTNAME=	rsyslog
| -PORTVERSION=	5.3.7
| +PORTVERSION=	5.4.0
|  CATEGORIES=	sysutils
|  MASTER_SITES=	http://download.rsyslog.com/rsyslog/
|  .ifdef MNAME
http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/distinfo.diff?r1=1.12&r2=1.13&f=h
| --- ports/sysutils/rsyslog5/distinfo	2010/01/28 20:51:59	1.12
| +++ ports/sysutils/rsyslog5/distinfo	2010/03/08 23:15:09	1.13
| @@ -1,3 +1,3 @@
| -MD5 (rsyslog-5.3.7.tar.gz) = 1a9b5c76bfea2d0ec28630b8f033d5e2
| -SHA256 (rsyslog-5.3.7.tar.gz) = 
cab7d39d3979e18c23bd0726ffad714c283d1158dea6c611f23eb1718299c029
| -SIZE (rsyslog-5.3.7.tar.gz) = 2109777
| +MD5 (rsyslog-5.4.0.tar.gz) = 291882229d50496f42bd63174076dd37
| +SHA256 (rsyslog-5.4.0.tar.gz) = 
d9cd21d2fcd45fcae65eb0a51927c40315cca02afdc62478abd950febfcf7228
| +SIZE (rsyslog-5.4.0.tar.gz) = 2124201
http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/files/rsyslogd.in.diff?r1=1.2&r2=1.3&f=h
| --- ports/sysutils/rsyslog5/files/rsyslogd.in	2009/12/18 20:44:28	1.2
| +++ ports/sysutils/rsyslog5/files/rsyslogd.in	2010/03/08 23:15:09	1.3
| @@ -1,6 +1,6 @@
|  #!/bin/sh
|  #
| -# $FreeBSD: 
/usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/files/rsyslogd.in,v 
1.2 2009/12/18 20:44:28 miwi Exp $
| +# $FreeBSD: 
/usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/files/rsyslogd.in,v 
1.3 2010/03/08 23:15:09 pgollucci Exp $
|  #
|
|
| @@ -20,59 +20,6 @@ load_rc_config $name
|  pidfile="${rsyslogd_pidfile}"
|  command_args="-i ${pidfile} -f ${rsyslogd_config}"
|  required_files="${rsyslogd_config}"
| -start_precmd="rsyslogd_precmd"
|  extra_commands="reload"
|
| -sockfile="/var/run/rsyslogd.sockets"
| -evalargs="rc_flags=\"\`set_socketlist\` \$rc_flags\""
| -altlog_proglist="named"
| -
| -rsyslogd_precmd()
| -{
| -	local _l _ldir
| -
| -	#	Transitional symlink for old binaries
| -	#
| -	if [ ! -L /dev/log ]; then
| -		ln -sf /var/run/log /dev/log
| -	fi
| -	rm -f /var/run/log
| -
| -	#	Create default list of syslog sockets to watch
| -	#
| -	( umask 022 ; > $sockfile )
| -
| -	#	If running named(8) or ntpd(8) chrooted, added appropriate
| -	#	syslog socket to list of sockets to watch.
| -	#
| -	for _l in $altlog_proglist; do
| -		eval _ldir=\$${_l}_chrootdir
| -		if checkyesno `set_rcvar $_l` && [ -n "$_ldir" ]; then
| -			echo "${_ldir}/var/run/log" >> $sockfile
| -		fi
| -	done
| -
| -	#	If other sockets have been provided, change run_rc_command()'s
| -	#	internal copy of $rsyslogd_flags to force use of specific
| -	#	rsyslogd sockets.
| -	#
| -	if [ -s $sockfile ]; then
| -		echo "/var/run/log" >> $sockfile
| -		eval $evalargs
| -	fi
| -
| -	return 0
| -}
| -
| -set_socketlist()
| -{
| -	local _s _socketargs
| -
| -	_socketargs=
| -	for _s in `cat $sockfile | tr '\n' ' '` ; do
| -		_socketargs="-a $_s $_socketargs"
| -	done
| -	echo $_socketargs
| -}
| -
|  run_rc_command "$1"
http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/pkg-plist.diff?r1=1.7&r2=1.8&f=h
| --- ports/sysutils/rsyslog5/pkg-plist	2010/01/07 04:46:04	1.7
| +++ ports/sysutils/rsyslog5/pkg-plist	2010/03/08 23:15:09	1.8
| @@ -126,6 +126,7 @@ sbin/rsyslogd
|  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_mysql.html
|  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_ng_comparison.html
|  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_packages.html
| +%%PORTDOCS%%%%DOCSDIR%%/rsyslog_pgsql.html
|  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_php_syslog_ng.html
|  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_recording_pri.html
|  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_reliable_forwarding.html


From rgerhards at hq.adiscon.com  Tue Mar  9 07:38:43 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 9 Mar 2010 07:38:43 +0100
Subject: [rsyslog] Fwd: cvs commit: ports/sysutils/rsyslog5 Makefile
	distinfo pkg-plist ports/sysutils/rsyslog5/files rsyslogd.in
References: <4B95A26C.6080301@p6m7g8.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103A45@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Philip M. Gollucci
> Sent: Tuesday, March 09, 2010 2:21 AM
> To: rsyslog-users
> Subject: [rsyslog] Fwd: cvs commit: ports/sysutils/rsyslog5 Makefile
> distinfo pkg-plist ports/sysutils/rsyslog5/files rsyslogd.in
> 
> fwiw, now in freebsd ports.

excellent, this is *very* useful!

Thanks,
Rainer

> 
> -------- Original Message --------
> Subject: cvs commit: ports/sysutils/rsyslog5 Makefile distinfo pkg-
> plist
>          ports/sysutils/rsyslog5/files rsyslogd.in
> Date: Mon, 8 Mar 2010 23:15:09 +0000 (UTC)
> From: Philip M. Gollucci <pgollucci at FreeBSD.org>
> To: ports-committers at FreeBSD.org, cvs-ports at FreeBSD.org, cvs-
> all at FreeBSD.org
> 
> pgollucci    2010-03-08 23:15:09 UTC
> 
>    FreeBSD ports repository
> 
>    Modified files:
>      sysutils/rsyslog5    Makefile distinfo pkg-plist
>      sysutils/rsyslog5/files rsyslogd.in
>    Log:
>    - Update to 5.4.0 [1]
>    - Remove -a and old school sockets from rc.d script [myself]
> 
>    PR:             ports/144566 [1]
>    Submitted by:   Cristiano Rolim Pereira <cristianorolim at hotmail.com>
> (maintainer) [1]
> 
>    Feature safe:   yes
> 
>    Revision  Changes    Path
>    1.13      +1 -1      ports/sysutils/rsyslog5/Makefile
>    1.13      +3 -3      ports/sysutils/rsyslog5/distinfo
>    1.3       +1 -54     ports/sysutils/rsyslog5/files/rsyslogd.in
>    1.8       +1 -0      ports/sysutils/rsyslog5/pkg-plist
> 
> http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/Makefile.diff?r1=1.12
> &r2=1.13&f=h
> | --- ports/sysutils/rsyslog5/Makefile	2010/01/28 20:51:59	1.12
> | +++ ports/sysutils/rsyslog5/Makefile	2010/03/08 23:15:09	1.13
> | @@ -2,11 +2,11 @@
> |  # Date created:     29 December 2008
> |  # Whom:             Cristiano Rolim Pereira
> <cristianorolim at hotmail.com>
> |  #
> | -# $FreeBSD:
> /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/Makefile,v 1.12
> 2010/01/28 20:51:59 glarkin Exp $
> | +# $FreeBSD:
> /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/Makefile,v 1.13
> 2010/03/08 23:15:09 pgollucci Exp $
> |  #
> |
> |  PORTNAME=	rsyslog
> | -PORTVERSION=	5.3.7
> | +PORTVERSION=	5.4.0
> |  CATEGORIES=	sysutils
> |  MASTER_SITES=	http://download.rsyslog.com/rsyslog/
> |  .ifdef MNAME
> http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/distinfo.diff?r1=1.12
> &r2=1.13&f=h
> | --- ports/sysutils/rsyslog5/distinfo	2010/01/28 20:51:59	1.12
> | +++ ports/sysutils/rsyslog5/distinfo	2010/03/08 23:15:09	1.13
> | @@ -1,3 +1,3 @@
> | -MD5 (rsyslog-5.3.7.tar.gz) = 1a9b5c76bfea2d0ec28630b8f033d5e2
> | -SHA256 (rsyslog-5.3.7.tar.gz) =
> cab7d39d3979e18c23bd0726ffad714c283d1158dea6c611f23eb1718299c029
> | -SIZE (rsyslog-5.3.7.tar.gz) = 2109777
> | +MD5 (rsyslog-5.4.0.tar.gz) = 291882229d50496f42bd63174076dd37
> | +SHA256 (rsyslog-5.4.0.tar.gz) =
> d9cd21d2fcd45fcae65eb0a51927c40315cca02afdc62478abd950febfcf7228
> | +SIZE (rsyslog-5.4.0.tar.gz) = 2124201
> http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/files/rsyslogd.in.dif
> f?r1=1.2&r2=1.3&f=h
> | --- ports/sysutils/rsyslog5/files/rsyslogd.in	2009/12/18
> 20:44:28	1.2
> | +++ ports/sysutils/rsyslog5/files/rsyslogd.in	2010/03/08
> 23:15:09	1.3
> | @@ -1,6 +1,6 @@
> |  #!/bin/sh
> |  #
> | -# $FreeBSD:
> /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/files/rsyslogd.i
> n,v
> 1.2 2009/12/18 20:44:28 miwi Exp $
> | +# $FreeBSD:
> /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/files/rsyslogd.i
> n,v
> 1.3 2010/03/08 23:15:09 pgollucci Exp $
> |  #
> |
> |
> | @@ -20,59 +20,6 @@ load_rc_config $name
> |  pidfile="${rsyslogd_pidfile}"
> |  command_args="-i ${pidfile} -f ${rsyslogd_config}"
> |  required_files="${rsyslogd_config}"
> | -start_precmd="rsyslogd_precmd"
> |  extra_commands="reload"
> |
> | -sockfile="/var/run/rsyslogd.sockets"
> | -evalargs="rc_flags=\"\`set_socketlist\` \$rc_flags\""
> | -altlog_proglist="named"
> | -
> | -rsyslogd_precmd()
> | -{
> | -	local _l _ldir
> | -
> | -	#	Transitional symlink for old binaries
> | -	#
> | -	if [ ! -L /dev/log ]; then
> | -		ln -sf /var/run/log /dev/log
> | -	fi
> | -	rm -f /var/run/log
> | -
> | -	#	Create default list of syslog sockets to watch
> | -	#
> | -	( umask 022 ; > $sockfile )
> | -
> | -	#	If running named(8) or ntpd(8) chrooted, added appropriate
> | -	#	syslog socket to list of sockets to watch.
> | -	#
> | -	for _l in $altlog_proglist; do
> | -		eval _ldir=\$${_l}_chrootdir
> | -		if checkyesno `set_rcvar $_l` && [ -n "$_ldir" ]; then
> | -			echo "${_ldir}/var/run/log" >> $sockfile
> | -		fi
> | -	done
> | -
> | -	#	If other sockets have been provided, change
> run_rc_command()'s
> | -	#	internal copy of $rsyslogd_flags to force use of specific
> | -	#	rsyslogd sockets.
> | -	#
> | -	if [ -s $sockfile ]; then
> | -		echo "/var/run/log" >> $sockfile
> | -		eval $evalargs
> | -	fi
> | -
> | -	return 0
> | -}
> | -
> | -set_socketlist()
> | -{
> | -	local _s _socketargs
> | -
> | -	_socketargs=
> | -	for _s in `cat $sockfile | tr '\n' ' '` ; do
> | -		_socketargs="-a $_s $_socketargs"
> | -	done
> | -	echo $_socketargs
> | -}
> | -
> |  run_rc_command "$1"
> http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/pkg-
> plist.diff?r1=1.7&r2=1.8&f=h
> | --- ports/sysutils/rsyslog5/pkg-plist	2010/01/07 04:46:04	1.7
> | +++ ports/sysutils/rsyslog5/pkg-plist	2010/03/08 23:15:09	1.8
> | @@ -126,6 +126,7 @@ sbin/rsyslogd
> |  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_mysql.html
> |  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_ng_comparison.html
> |  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_packages.html
> | +%%PORTDOCS%%%%DOCSDIR%%/rsyslog_pgsql.html
> |  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_php_syslog_ng.html
> |  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_recording_pri.html
> |  %%PORTDOCS%%%%DOCSDIR%%/rsyslog_reliable_forwarding.html
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Tue Mar  9 09:00:01 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 9 Mar 2010 09:00:01 +0100
Subject: [rsyslog] rsyslog 5.4.0 (v5-stable) released
References: <9B6E2A8877C38245BFB15CC491A11DA7103A33@GRFEXC.intern.adiscon.com>
	<aa76cdac1003081625w6dac4ec3raf85bde7491c8338@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103A49@GRFEXC.intern.adiscon.com>

Hi Michael,

The -c option is transitive. So for the diff between -c3 and -c5, you need to
check the compatibility notes for v4 and v5. These are:

http://www.rsyslog.com/doc-v4compatibility.html
http://www.rsyslog.com/doc-v5compatibility.html

I should add to the notes that -c obviously only modifies defaults, and not
program behavior at large. So even with -c3, for example, rsyslog v5 will not
be able to run on zero workers.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Tuesday, March 09, 2010 1:26 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 5.4.0 (v5-stable) released
> 
> Hi everyone!
> 
> 2010/3/8 Tom Bergfeld <tbergfeld at hq.adiscon.com>:
> > Hi all,
> > Today, we released rsyslog 5.4.0, a new v5-stable branch. This
> version begins
> 
> So, the help output of rsyslogd has
> 
> To run rsyslogd in native mode, use "rsyslogd -c5 <other options>"
> 
> Maybe I'm just missing some link to the documentation, but is it
> documented somewhere, which features are enabled/available when using
> -c5 compared to using -c4 or -c3.
> Or put it differently: What's the difference between using -c3 and -c5
> for 5.4.0?
> 
> Michael
> 
> 
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From pgollucci at p6m7g8.com  Thu Mar 11 00:09:51 2010
From: pgollucci at p6m7g8.com (Philip M. Gollucci)
Date: Wed, 10 Mar 2010 23:09:51 +0000
Subject: [rsyslog] rsyslogd 'clients' take out machines if rsyslogd 'server'
	is down
Message-ID: <4B9826BF.7070202@p6m7g8.com>

rsyslog.conf [client version]:
-------------------------------

....

$WorkDirectory /var/spool/rsyslog # location for work (spool) file
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

....
## Local
....

## Remote
*.* :omrelp:cl.tld:2514


If anything on the 'centralized server' fails like mysql is down, or 
rsyslogd is down, or a table is wrong, or insert permissions are wrong, 
or diskio is *really* *really* bad, or the box is 'off' --- about 45 
minutes later. All 50 or so of the clients talking to 'block' and 
suddenly spin in a tight loop and are not killable without -9.

I'm sure its something filling up b/c the timing is nearly to the second 
of the 45 minute mark [given a relatively constant amount of load].

I assume that changing
$ActionResumeRetryCount -1
to
$ActionResumeRetryCount 5

or some other arbirtarily low number in all clients would fix this.  As 
whatever ram buffers are filling up will stop doing that.  If not, maybe 
falling back from RELP to TCP would do the trick [I'd be okay with 
message loss to prevent downtime].

I'm rather more concerned though that it doesn't start queueing to disk 
the spool dir when the ram buffers are full instead of 'crashing' and 
taking the box or all process on it with it.



2542.938673679:800b021c0: rsyslogd 5.3.7 startup, compatibility mode 5, 
module path ''

on FreeBSD 8.0-RELENG-p2








-- 
------------------------------------------------------------------------
1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70  3F8C 75B8 8FFB DB9B 8C1C
Philip M. Gollucci (pgollucci at p6m7g8.com) c: 703.336.9354
VP Apache Infrastructure; Member, Apache Software Foundation
Committer,                        FreeBSD Foundation
Consultant,                       P6M7G8 Inc.
Sr. System Admin,                 Ridecharge Inc.

Work like you don't need the money,
love like you'll never get hurt,
and dance like nobody's watching.


From mangoo at wpkg.org  Thu Mar 18 21:21:06 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Thu, 18 Mar 2010 21:21:06 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
Message-ID: <4BA28B32.6090201@wpkg.org>

I have postgres logging to a separate file through rsyslog with this line:

local0.*                -/var/log/postgresql/postgresql_log


It gets to /var/log/syslog as well, thanks to this line:

*.*;auth,authpriv.none	-/var/log/syslog


As postgres is set to log quite heavily, I would rather not log postgres into /var/log/syslog.

I've read some filtering examples, but I couldn't find one on excluding a given program from a file which is logging everything.

Could anyone point me to a proper example?


Postgres adds such entries through syslog:

Mar 19 04:14:16 sun postgres[388]: [261-1] 2010-03-19 04:14:16 SGT LOG:  duration: 0.037 ms  statement: ROLLBACK


-- 
Tomasz Chmielewski
http://wpkg.org


From sledz at DResearch.DE  Thu Mar 18 21:06:58 2010
From: sledz at DResearch.DE (Sledz, Steffen)
Date: Thu, 18 Mar 2010 21:06:58 +0100
Subject: [rsyslog] patch for architectures without atomic builtins
Message-ID: <DB561678F08F554EB0D9F21B70E268A505AE8B@exchange.intern.dresearch.de>

Hi,

we want to use rsyslogd on an embedded system with an armv5te cpu.
Unfortunately the gcc/libc we have to use for this system do not support
atomic builtins which are vital for rsyslog at the moment. :(

Therefor we created a patch which emulates the atomic builtins
with posix semaphores. This patch should fit to the 5.4.x and
5.5.x versions.

We would be glad if this patch could help to bring rsyslog to
more platforms.

Steffen

From david at lang.hm  Thu Mar 18 23:54:56 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 15:54:56 -0700 (PDT)
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <4BA28B32.6090201@wpkg.org>
References: <4BA28B32.6090201@wpkg.org>
Message-ID: <alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>

On Thu, 18 Mar 2010, Tomasz Chmielewski wrote:

> I have postgres logging to a separate file through rsyslog with this line:
>
> local0.*                -/var/log/postgresql/postgresql_log
>
>
> It gets to /var/log/syslog as well, thanks to this line:
>
> *.*;auth,authpriv.none	-/var/log/syslog
>
>
> As postgres is set to log quite heavily, I would rather not log postgres into /var/log/syslog.
>
> I've read some filtering examples, but I couldn't find one on excluding a given program from a file which is logging everything.
>
> Could anyone point me to a proper example?
>
>
> Postgres adds such entries through syslog:
>
> Mar 19 04:14:16 sun postgres[388]: [261-1] 2010-03-19 04:14:16 SGT LOG:  duration: 0.037 ms  statement: ROLLBACK

you could add

:programname, :isequal, "postgres" ~

ahead of your line to go to /var/log/syslog so that anything with the 
programname of postgres gets dropped first

another thing you could do (if you know that all postgres logs are local0) 
is

local0.* ~

to throw away all logs with facility local0

David Lang


From david at lang.hm  Thu Mar 18 23:55:48 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 15:55:48 -0700 (PDT)
Subject: [rsyslog] patch for architectures without atomic builtins
In-Reply-To: <DB561678F08F554EB0D9F21B70E268A505AE8B@exchange.intern.dresearch.de>
References: <DB561678F08F554EB0D9F21B70E268A505AE8B@exchange.intern.dresearch.de>
Message-ID: <alpine.DEB.2.00.1003181555170.10411@asgard.lang.hm>

On Thu, 18 Mar 2010, Sledz, Steffen wrote:

> Hi,
>
> we want to use rsyslogd on an embedded system with an armv5te cpu.
> Unfortunately the gcc/libc we have to use for this system do not support
> atomic builtins which are vital for rsyslog at the moment. :(
>
> Therefor we created a patch which emulates the atomic builtins
> with posix semaphores. This patch should fit to the 5.4.x and
> 5.5.x versions.
>
> We would be glad if this patch could help to bring rsyslog to
> more platforms.
>
> Steffen

the list strips off attachments.

can you re-send with the patch inline?

David Lang


From mangoo at wpkg.org  Fri Mar 19 00:02:41 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Fri, 19 Mar 2010 00:02:41 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
Message-ID: <4BA2B111.7090604@wpkg.org>

Am 18.03.2010 23:54, david at lang.hm wrote:

>> Postgres adds such entries through syslog:
>>
>> Mar 19 04:14:16 sun postgres[388]: [261-1] 2010-03-19 04:14:16 SGT LOG:  duration: 0.037 ms  statement: ROLLBACK
> 
> you could add
> 
> :programname, :isequal, "postgres" ~
> 
> ahead of your line to go to /var/log/syslog so that anything with the
> programname of postgres gets dropped first

Meaning, like this:

:programname, :isequal, "postgres" ~ *.*;auth,authpriv.none	-/var/log/syslog

?


-- 
Tomasz Chmielewski
http://wpkg.org


From david at lang.hm  Fri Mar 19 00:08:29 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 16:08:29 -0700 (PDT)
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <4BA2B111.7090604@wpkg.org>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
Message-ID: <alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>

On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:

> Am 18.03.2010 23:54, david at lang.hm wrote:
>
>>> Postgres adds such entries through syslog:
>>>
>>> Mar 19 04:14:16 sun postgres[388]: [261-1] 2010-03-19 04:14:16 SGT LOG:  duration: 0.037 ms  statement: ROLLBACK
>>
>> you could add
>>
>> :programname, :isequal, "postgres" ~
>>
>> ahead of your line to go to /var/log/syslog so that anything with the
>> programname of postgres gets dropped first
>
> Meaning, like this:
>
> :programname, :isequal, "postgres" ~ *.*;auth,authpriv.none	-/var/log/syslog

no,

:programname, :isequal, "postgres" ~
*.*;auth,authpriv.none	-/var/log/syslog

each of these is a seperate statement. the ~ as a destination tells 
rsyslog not to proceed further through the rules for this log entry.

David Lang


From mangoo at wpkg.org  Fri Mar 19 00:19:59 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Fri, 19 Mar 2010 00:19:59 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
Message-ID: <4BA2B51F.1060204@wpkg.org>

Am 19.03.2010 00:08, david at lang.hm wrote:

> no,
>
> :programname, :isequal, "postgres" ~
> *.*;auth,authpriv.none -/var/log/syslog
>
> each of these is a seperate statement. the ~ as a destination tells
> rsyslog not to proceed further through the rules for this log entry.

Doesn't work for me.

If I put it like above, nothing gets appended to /var/log/syslog anymore 
(or any other defined log file).

rsyslogd is running as a process.

If it matters, I'm using 64 bit Debian Lenny (rsyslog 3.18.6).


-- 
Tomasz Chmielewski
http://wpkg.org


From david at lang.hm  Fri Mar 19 00:23:22 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 16:23:22 -0700 (PDT)
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <4BA2B51F.1060204@wpkg.org>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
Message-ID: <alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>

On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:

> Am 19.03.2010 00:08, david at lang.hm wrote:
>
>> no,
>> 
>> :programname, :isequal, "postgres" ~
>> *.*;auth,authpriv.none -/var/log/syslog
>> 
>> each of these is a seperate statement. the ~ as a destination tells
>> rsyslog not to proceed further through the rules for this log entry.
>
> Doesn't work for me.
>
> If I put it like above, nothing gets appended to /var/log/syslog anymore (or 
> any other defined log file).
>
> rsyslogd is running as a process.
>
> If it matters, I'm using 64 bit Debian Lenny (rsyslog 3.18.6).

try it with ' instead of " (I get those mixed up at least 50% of the time, 
rsyslog accepts one and not the other and I can never remember which one)

David Lang


From mangoo at wpkg.org  Fri Mar 19 00:40:46 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Fri, 19 Mar 2010 00:40:46 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
Message-ID: <4BA2B9FE.2030008@wpkg.org>

Am 19.03.2010 00:23, david at lang.hm wrote:
> On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:
>
>> Am 19.03.2010 00:08, david at lang.hm wrote:
>>
>>> no,
>>>
>>> :programname, :isequal, "postgres" ~
>>> *.*;auth,authpriv.none -/var/log/syslog
>>>
>>> each of these is a seperate statement. the ~ as a destination tells
>>> rsyslog not to proceed further through the rules for this log entry.
>>
>> Doesn't work for me.
>>
>> If I put it like above, nothing gets appended to /var/log/syslog
>> anymore (or any other defined log file).
>>
>> rsyslogd is running as a process.
>>
>> If it matters, I'm using 64 bit Debian Lenny (rsyslog 3.18.6).
>
> try it with ' instead of " (I get those mixed up at least 50% of the
> time, rsyslog accepts one and not the other and I can never remember
> which one)

OK, with ' it started to log again, but still postgres gets to both 
/var/log/syslog and /var/log/postgresql/postgresql_log.

So, not as expected.

-- 
Tomasz Chmielewski
http://wpkg.org


From david at lang.hm  Fri Mar 19 00:52:45 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 16:52:45 -0700 (PDT)
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <4BA2B9FE.2030008@wpkg.org>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org>
Message-ID: <alpine.DEB.2.00.1003181652290.10411@asgard.lang.hm>

On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:

> Am 19.03.2010 00:23, david at lang.hm wrote:
>> On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:
>> 
>>> Am 19.03.2010 00:08, david at lang.hm wrote:
>>> 
>>>> no,
>>>> 
>>>> :programname, :isequal, "postgres" ~
>>>> *.*;auth,authpriv.none -/var/log/syslog
>>>> 
>>>> each of these is a seperate statement. the ~ as a destination tells
>>>> rsyslog not to proceed further through the rules for this log entry.
>>> 
>>> Doesn't work for me.
>>> 
>>> If I put it like above, nothing gets appended to /var/log/syslog
>>> anymore (or any other defined log file).
>>> 
>>> rsyslogd is running as a process.
>>> 
>>> If it matters, I'm using 64 bit Debian Lenny (rsyslog 3.18.6).
>> 
>> try it with ' instead of " (I get those mixed up at least 50% of the
>> time, rsyslog accepts one and not the other and I can never remember
>> which one)
>
> OK, with ' it started to log again, but still postgres gets to both 
> /var/log/syslog and /var/log/postgresql/postgresql_log.
>
> So, not as expected.

what does the raw log message look like again?

David Lang


From mangoo at wpkg.org  Fri Mar 19 00:58:37 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Fri, 19 Mar 2010 00:58:37 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <alpine.DEB.2.00.1003181652290.10411@asgard.lang.hm>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org>
	<alpine.DEB.2.00.1003181652290.10411@asgard.lang.hm>
Message-ID: <4BA2BE2D.9020205@wpkg.org>

Am 19.03.2010 00:52, david at lang.hm wrote:

>> OK, with ' it started to log again, but still postgres gets to both 
>> /var/log/syslog and /var/log/postgresql/postgresql_log.
>>
>> So, not as expected.
> 
> what does the raw log message look like again?

Like these ones:

Mar 19 07:47:32 db10 postgres[4539]: [87-1] 2010-03-19 07:47:32 SGT LOG:  duration: 0.234 ms  statement: COMMIT
Mar 19 07:47:32 db10 postgres[4539]: [88-1] 2010-03-19 07:47:32 SGT LOG:  duration: 0.042 ms  statement: BEGIN; SET TRANSACTION ISOLATION LEVEL READ 


where postgres[number] can differ.

-- 
Tomasz Chmielewski
http://wpkg.org


From david at lang.hm  Fri Mar 19 01:08:11 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 17:08:11 -0700 (PDT)
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <4BA2BE2D.9020205@wpkg.org>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org>
	<alpine.DEB.2.00.1003181652290.10411@asgard.lang.hm>
	<4BA2BE2D.9020205@wpkg.org>
Message-ID: <alpine.DEB.2.00.1003181706510.10411@asgard.lang.hm>

On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:

> Am 19.03.2010 00:52, david at lang.hm wrote:
>
>>> OK, with ' it started to log again, but still postgres gets to both
>>> /var/log/syslog and /var/log/postgresql/postgresql_log.
>>>
>>> So, not as expected.
>>
>> what does the raw log message look like again?
>
> Like these ones:
>
> Mar 19 07:47:32 db10 postgres[4539]: [87-1] 2010-03-19 07:47:32 SGT LOG:  duration: 0.234 ms  statement: COMMIT
> Mar 19 07:47:32 db10 postgres[4539]: [88-1] 2010-03-19 07:47:32 SGT LOG:  duration: 0.042 ms  statement: BEGIN; SET TRANSACTION ISOLATION LEVEL READ
>
>
> where postgres[number] can differ.

hmm, in these cases postgres should be what's in programname

taking a different slant, IIRC you used local0.* to capture the postgres 
logs into one file, if in the line after that you do

local0.* ~

that should eliminate those logs from going into any files after that.

David Lang


From mangoo at wpkg.org  Fri Mar 19 01:42:29 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Fri, 19 Mar 2010 01:42:29 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <alpine.DEB.2.00.1003181706510.10411@asgard.lang.hm>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org>
	<alpine.DEB.2.00.1003181652290.10411@asgard.lang.hm>
	<4BA2BE2D.9020205@wpkg.org>
	<alpine.DEB.2.00.1003181706510.10411@asgard.lang.hm>
Message-ID: <4BA2C875.4060208@wpkg.org>

Am 19.03.2010 01:08, david at lang.hm wrote:

>> where postgres[number] can differ.
>
> hmm, in these cases postgres should be what's in programname
>
> taking a different slant, IIRC you used local0.* to capture the postgres
> logs into one file, if in the line after that you do
>
> local0.* ~
>
> that should eliminate those logs from going into any files after that.

This one works, but that'll filter out heartbeat, pgpool and others 
useful, which get to syslog.


-- 
Tomasz Chmielewski
http://wpkg.org


From mangoo at wpkg.org  Fri Mar 19 01:44:41 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Fri, 19 Mar 2010 01:44:41 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <4BA2B9FE.2030008@wpkg.org>
References: <4BA28B32.6090201@wpkg.org>	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>	<4BA2B111.7090604@wpkg.org>	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>	<4BA2B51F.1060204@wpkg.org>	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org>
Message-ID: <4BA2C8F9.4050607@wpkg.org>

Am 19.03.2010 00:40, Tomasz Chmielewski wrote:
> Am 19.03.2010 00:23, david at lang.hm wrote:
>> On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:
>>
>>> Am 19.03.2010 00:08, david at lang.hm wrote:
>>>
>>>> no,
>>>>
>>>> :programname, :isequal, "postgres" ~
>>>> *.*;auth,authpriv.none -/var/log/syslog
>>>>
>>>> each of these is a seperate statement. the ~ as a destination tells
>>>> rsyslog not to proceed further through the rules for this log entry.
>>>
>>> Doesn't work for me.
>>>
>>> If I put it like above, nothing gets appended to /var/log/syslog
>>> anymore (or any other defined log file).
>>>
>>> rsyslogd is running as a process.
>>>
>>> If it matters, I'm using 64 bit Debian Lenny (rsyslog 3.18.6).
>>
>> try it with ' instead of " (I get those mixed up at least 50% of the
>> time, rsyslog accepts one and not the other and I can never remember
>> which one)
> 
> OK, with ' it started to log again, but still postgres gets to both
> /var/log/syslog and /var/log/postgresql/postgresql_log.
> 
> So, not as expected.

And here is why:

Mar 19 07:37:26 sun rsyslogd: error: invalid compare operation ':isequal' - ignoring selector
Mar 19 07:37:26 sun rsyslogd: error -3003 compare value property - ignoring selector
Mar 19 07:37:26 sun rsyslogd: the last error occured in /etc/rsyslog.conf, line 58
Mar 19 07:37:26 sun rsyslogd: warning: selector line without actions will be discarded


So, if it's " - it doesn't log anything.

If it's ' - it doesn't like the rule.


-- 
Tomasz Chmielewski
http://wpkg.org



From joeschmo2000 at gmail.com  Fri Mar 19 03:14:00 2010
From: joeschmo2000 at gmail.com (joe schmo)
Date: Thu, 18 Mar 2010 22:14:00 -0400
Subject: [rsyslog] CentOS Issues
Message-ID: <a60a2fa91003181914kd27e298se6a8e3913563e4ca@mail.gmail.com>

Hey all,

So I had been using another syslog server for a long time and recently
decided to make the move over to rsyslog and phplogcon so that we could have
web searchable logs (I already run RANCID w/ CVS and cvsweb enabled so it
seemed only fitting to get our syslog server on the webz too)

I went to the forums with these questions and they suggested I ask the
mailing list. That was actually my second destination anyway because I had
read several messages from the archive before heading to the forums.

I have a few questions that span a few different versions...first I need to
apologize in advance if any of these are noob questions, but I have searched
for them and I can't find any answers that help me.

I am running CentOS 5.3 Final with all latest updates off the main repos.
rsyslog 2.0.6 running currently w/ phplogmon and mysql.

1) I have a copy of rsyslog 2.0.6 up and running without any issues (as it
is the only one I can currently get working...more on that in a sec), other
than the fact that it can't receive udp messages without manual
intervention. The only way I can get it to accept udp (network) syslog
messages is to stop the service, and then run the rsyslog -r514 from
command-line manually. Is there anyway to get it to use this option by
default? I need it to start the service with that option enabled. And the
"just run it manually" answers I have received in other forums aren't
acceptable for me.

2) An extension of the above question: when I try to do the "obvious"
solution to the udp issue and use the imudp module, I realized there IS NO
imudp.so module in the plugins folder. Where would I even get this module
for 2.0.6? The most I can find is a copy of imudp.c and its accompanying
make files, but trying to make only outputs errors associated with missing
files that belong to the C developers package (which I had installed some
time ago off an official CentOS repo, no updates available).

3) Why is it that when I follow all the instructions for compiling and
installing the newer versions of rsyslog (3.0.0+), I can get all the way
through the process (I searched and found a way to correct the i686 GCC
errors) but then it isn't actually installed? Does this have to do with
CentOS 5.3 support or am I missing a step here? After doing the make
install, etc, there is no rsyslog.conf file in /etc/ like when I install
2.0.6, nor any of the other accompanying directories, except the
/usr/lib/rsyslog/ folder, which is fully populated with all the correct
files.

Any guidance would be GREATLY appreciated as I am getting very frustrated
with the amount of time I have put into trying to get everything working
correctly. And I thought setting up RANCID and cvsweb was a bit of a chore!
:)

Thanks in advance!

-Joe


From david at lang.hm  Fri Mar 19 05:20:51 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 21:20:51 -0700 (PDT)
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <4BA2C8F9.4050607@wpkg.org>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org> <4BA2C8F9.4050607@wpkg.org>
Message-ID: <alpine.DEB.2.00.1003182119540.3821@asgard.lang.hm>

On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:

> Am 19.03.2010 00:40, Tomasz Chmielewski wrote:
>> Am 19.03.2010 00:23, david at lang.hm wrote:
>>> On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:
>>>
>>>> Am 19.03.2010 00:08, david at lang.hm wrote:
>>>>
>>>>> no,
>>>>>
>>>>> :programname, :isequal, "postgres" ~
>>>>> *.*;auth,authpriv.none -/var/log/syslog
>>>>>
>>>>> each of these is a seperate statement. the ~ as a destination tells
>>>>> rsyslog not to proceed further through the rules for this log entry.
>>>>
>>>> Doesn't work for me.
>>>>
>>>> If I put it like above, nothing gets appended to /var/log/syslog
>>>> anymore (or any other defined log file).
>>>>
>>>> rsyslogd is running as a process.
>>>>
>>>> If it matters, I'm using 64 bit Debian Lenny (rsyslog 3.18.6).
>>>
>>> try it with ' instead of " (I get those mixed up at least 50% of the
>>> time, rsyslog accepts one and not the other and I can never remember
>>> which one)
>>
>> OK, with ' it started to log again, but still postgres gets to both
>> /var/log/syslog and /var/log/postgresql/postgresql_log.
>>
>> So, not as expected.
>
> And here is why:
>
> Mar 19 07:37:26 sun rsyslogd: error: invalid compare operation ':isequal' - ignoring selector
> Mar 19 07:37:26 sun rsyslogd: error -3003 compare value property - ignoring selector
> Mar 19 07:37:26 sun rsyslogd: the last error occured in /etc/rsyslog.conf, line 58
> Mar 19 07:37:26 sun rsyslogd: warning: selector line without actions will be discarded
>
>
> So, if it's " - it doesn't log anything.
>
> If it's ' - it doesn't like the rule.

when all else fails, look at the documentation ;-)

http://www.rsyslog.com/doc-rsyslog_conf_filter.html

the line should have been

:programname, isequal, 'postgres' ~

David Lang


From david at lang.hm  Fri Mar 19 05:22:13 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 21:22:13 -0700 (PDT)
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <4BA2C875.4060208@wpkg.org>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org>
	<alpine.DEB.2.00.1003181652290.10411@asgard.lang.hm>
	<4BA2BE2D.9020205@wpkg.org>
	<alpine.DEB.2.00.1003181706510.10411@asgard.lang.hm>
	<4BA2C875.4060208@wpkg.org>
Message-ID: <alpine.DEB.2.00.1003182120550.3821@asgard.lang.hm>

On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:

> Am 19.03.2010 01:08, david at lang.hm wrote:
>
>>> where postgres[number] can differ.
>> 
>> hmm, in these cases postgres should be what's in programname
>> 
>> taking a different slant, IIRC you used local0.* to capture the postgres
>> logs into one file, if in the line after that you do
>> 
>> local0.* ~
>> 
>> that should eliminate those logs from going into any files after that.
>
> This one works, but that'll filter out heartbeat, pgpool and others useful, 
> which get to syslog.

I will point out that those entries are getting put into your 
posgresql_log file along with the postgres logs then.

so you may want to change the filter for that line as well (now that I dug 
up the page with the correct syntax rather than going from memory)

David Lang


From david at lang.hm  Fri Mar 19 05:31:50 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 18 Mar 2010 21:31:50 -0700 (PDT)
Subject: [rsyslog] CentOS Issues
In-Reply-To: <a60a2fa91003181914kd27e298se6a8e3913563e4ca@mail.gmail.com>
References: <a60a2fa91003181914kd27e298se6a8e3913563e4ca@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1003182125350.3821@asgard.lang.hm>

On Thu, 18 Mar 2010, joe schmo wrote:

> Hey all,
>
> So I had been using another syslog server for a long time and recently
> decided to make the move over to rsyslog and phplogcon so that we could have
> web searchable logs (I already run RANCID w/ CVS and cvsweb enabled so it
> seemed only fitting to get our syslog server on the webz too)
>
> I went to the forums with these questions and they suggested I ask the
> mailing list. That was actually my second destination anyway because I had
> read several messages from the archive before heading to the forums.
>
> I have a few questions that span a few different versions...first I need to
> apologize in advance if any of these are noob questions, but I have searched
> for them and I can't find any answers that help me.
>
> I am running CentOS 5.3 Final with all latest updates off the main repos.
> rsyslog 2.0.6 running currently w/ phplogmon and mysql.
>
> 1) I have a copy of rsyslog 2.0.6 up and running without any issues (as it
> is the only one I can currently get working...more on that in a sec), other
> than the fact that it can't receive udp messages without manual
> intervention. The only way I can get it to accept udp (network) syslog
> messages is to stop the service, and then run the rsyslog -r514 from
> command-line manually. Is there anyway to get it to use this option by
> default? I need it to start the service with that option enabled. And the
> "just run it manually" answers I have received in other forums aren't
> acceptable for me.
>
> 2) An extension of the above question: when I try to do the "obvious"
> solution to the udp issue and use the imudp module, I realized there IS NO
> imudp.so module in the plugins folder. Where would I even get this module
> for 2.0.6? The most I can find is a copy of imudp.c and its accompanying
> make files, but trying to make only outputs errors associated with missing
> files that belong to the C developers package (which I had installed some
> time ago off an official CentOS repo, no updates available).
>
> 3) Why is it that when I follow all the instructions for compiling and
> installing the newer versions of rsyslog (3.0.0+), I can get all the way
> through the process (I searched and found a way to correct the i686 GCC
> errors) but then it isn't actually installed? Does this have to do with
> CentOS 5.3 support or am I missing a step here? After doing the make
> install, etc, there is no rsyslog.conf file in /etc/ like when I install
> 2.0.6, nor any of the other accompanying directories, except the
> /usr/lib/rsyslog/ folder, which is fully populated with all the correct
> files.
>
> Any guidance would be GREATLY appreciated as I am getting very frustrated
> with the amount of time I have put into trying to get everything working
> correctly. And I thought setting up RANCID and cvsweb was a bit of a chore!
> :)
>
> Thanks in advance!

as you have noticed, rsyslog 2.x is ancient (I am running 5.x in 
production now)

when compiling the new version you should be able to just do make install 
after you have compiled it to install it.

you may need to uninstall the RHEL shipped version, I think that by 
default rsyslog will get installed under /usr/local when compiled from 
source.

installing from source will not create the /etc/rsyslog.conf file for you.

the rsyslog on my laptop (ubuntu with a couple manual tweaks) is:

#  /etc/rsyslog.conf    Configuration file for rsyslog.
#
#                       For more information see
#                       /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
#
#  Default logging rules can be found in /etc/rsyslog.d/50-default.conf
#################
#### MODULES ####
#################
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog   # provides kernel logging support (previously done by 
rklogd)
#$ModLoad immark  # provides --MARK-- message capability
#$KLogPath /var/run/rsyslog/kmsg
# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
$RepeatedMsgReduction on
$RepeatedMsgContainsOriginalMsg on
#
# Include all config files in /etc/rsyslog.d/
#
#$IncludeConfig /etc/rsyslog.d/*.conf
*.* -/var/log/messages

this should be enough to get you started, you will need to uncomment the 
remote reception options and then start rsyslog with rsyslogd -cX where X 
is the major version you are running (3,4,5 depending on which one you 
pick)

David Lang


From sledz at dresearch.de  Fri Mar 19 08:00:39 2010
From: sledz at dresearch.de (Steffen Sledz)
Date: Fri, 19 Mar 2010 08:00:39 +0100
Subject: [rsyslog] patch for architectures without atomic builtins
In-Reply-To: <alpine.DEB.2.00.1003181555170.10411@asgard.lang.hm>
References: <DB561678F08F554EB0D9F21B70E268A505AE8B@exchange.intern.dresearch.de>
	<alpine.DEB.2.00.1003181555170.10411@asgard.lang.hm>
Message-ID: <4BA32117.9030408@dresearch.de>

david at lang.hm wrote:
> the list strips off attachments.
> 
> can you re-send with the patch inline?

Here it is...

diff -Nurd rsyslog-5.4.0.orig/configure.ac rsyslog-5.4.0/configure.ac
--- rsyslog-5.4.0.orig/configure.ac     2010-03-08 13:28:35.000000000 +0000
+++ rsyslog-5.4.0/configure.ac  2010-03-16 11:30:44.000000000 +0000
@@ -87,6 +87,7 @@
 AC_HEADER_TIME
 AC_STRUCT_TM
 AC_C_VOLATILE
+AC_C_TYPEOF

 sa_includes="\
 $ac_includes_default
@@ -129,7 +130,8 @@
 # check for availability of atomic operations
 RS_ATOMIC_OPERATIONS
 RS_ATOMIC_OPERATIONS_64BIT
-
+# fall back to POSIX sems for atomic operations (cpu expensive)
+AC_CHECK_HEADERS([semaphore.h])

 # Additional module directories
 AC_ARG_WITH(moddirs,
diff -Nurd rsyslog-5.4.0.orig/runtime/Makefile.am rsyslog-5.4.0/runtime/Makefile.am
--- rsyslog-5.4.0.orig/runtime/Makefile.am      2010-03-08 13:27:47.000000000 +0000
+++ rsyslog-5.4.0/runtime/Makefile.am   2010-03-15 16:43:12.000000000 +0000
@@ -9,6 +9,7 @@
        rsyslog.h \
        unicode-helper.h \
        atomic.h \
+       atomic-posix-sem.c \
        batch.h \
        syslogd-types.h \
        module-template.h \
diff -Nurd rsyslog-5.4.0.orig/runtime/atomic-posix-sem.c rsyslog-5.4.0/runtime/atomic-posix-sem.c
--- rsyslog-5.4.0.orig/runtime/atomic-posix-sem.c       1970-01-01 00:00:00.000000000 +0000
+++ rsyslog-5.4.0/runtime/atomic-posix-sem.c    2010-03-16 13:22:27.000000000 +0000
@@ -0,0 +1,70 @@
+/* atomic_posix_sem.c: This file supplies an emulation for atomic operations using
+ * POSIX semaphores.
+ *
+ * Copyright 2010 DResearch Digital Media Systems GmbH
+ *
+ * This file is part of the rsyslog runtime library.
+ *
+ * The rsyslog runtime library is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * The rsyslog runtime library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public License
+ * along with the rsyslog runtime library.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * A copy of the GPL can be found in the file "COPYING" in this distribution.
+ * A copy of the LGPL can be found in the file "COPYING.LESSER" in this distribution.
+ */
+
+#include "config.h"
+#ifndef HAVE_ATOMIC_BUILTINS
+#ifdef HAVE_SEMAPHORE_H
+#include <semaphore.h>
+#include <errno.h>
+
+#include "atomic.h"
+#include "rsyslog.h"
+#include "srUtils.h"
+
+sem_t atomicSem;
+
+rsRetVal
+atomicSemInit(void)
+{
+       DEFiRet;
+
+       dbgprintf("init posix semaphore for atomics emulation\n");
+       if(sem_init(&atomicSem, 0, 1) == -1)
+       {
+               char errStr[1024];
+               rs_strerror_r(errno, errStr, sizeof(errStr));
+               dbgprintf("init posix semaphore for atomics emulation failed: %s\n", errStr);
+               iRet = RS_RET_SYS_ERR; /* the right error code ??? */
+       }
+
+       RETiRet;
+}
+
+void
+atomicSemExit(void)
+{
+       dbgprintf("destroy posix semaphore for atomics emulation\n");
+       if(sem_destroy(&atomicSem) == -1)
+       {
+               char errStr[1024];
+               rs_strerror_r(errno, errStr, sizeof(errStr));
+               dbgprintf("destroy posix semaphore for atomics emulation failed: %s\n", errStr);
+       }
+}
+
+#endif /* HAVE_SEMAPHORE_H */
+#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
+
+/* vim:set ai:
+ */
diff -Nurd rsyslog-5.4.0.orig/runtime/atomic.h rsyslog-5.4.0/runtime/atomic.h
--- rsyslog-5.4.0.orig/runtime/atomic.h 2010-03-05 07:20:36.000000000 +0000
+++ rsyslog-5.4.0/runtime/atomic.h      2010-03-16 13:23:57.000000000 +0000
@@ -54,6 +54,122 @@
 #      define ATOMIC_CAS(data, oldVal, newVal) __sync_bool_compare_and_swap(&(data), (oldVal), (newVal));
 #      define ATOMIC_CAS_VAL(data, oldVal, newVal) __sync_val_compare_and_swap(&(data), (oldVal), (newVal));
 #else
+#ifdef HAVE_SEMAPHORE_H
+       /* we use POSIX semaphores instead */
+
+#include "rsyslog.h"
+#include <semaphore.h>
+
+extern sem_t atomicSem;
+rsRetVal atomicSemInit(void);
+void atomicSemExit(void);
+
+#if HAVE_TYPEOF
+#define my_typeof(x) typeof(x)
+#else /* sorry, can't determine types, using 'int' */
+#define my_typeof(x) int
+#endif
+
+#      define ATOMIC_SUB(data, val) \
+({ \
+       my_typeof(data) tmp; \
+       sem_wait(&atomicSem); \
+       tmp = data; \
+       data -= val; \
+       sem_post(&atomicSem); \
+       tmp; \
+})
+
+#      define ATOMIC_ADD(data, val) \
+({ \
+       my_typeof(data) tmp; \
+       sem_wait(&atomicSem); \
+       tmp = data; \
+       data += val; \
+       sem_post(&atomicSem); \
+       tmp; \
+})
+
+#      define ATOMIC_INC_AND_FETCH(data) \
+({ \
+       my_typeof(data) tmp; \
+       sem_wait(&atomicSem); \
+       tmp = data; \
+       data += 1; \
+       sem_post(&atomicSem); \
+       tmp; \
+})
+
+#      define ATOMIC_INC(data) ((void) ATOMIC_INC_AND_FETCH(data))
+
+#      define ATOMIC_DEC_AND_FETCH(data) \
+({ \
+       sem_wait(&atomicSem); \
+       data -= 1; \
+       sem_post(&atomicSem); \
+       data; \
+})
+
+#      define ATOMIC_DEC(data) ((void) ATOMIC_DEC_AND_FETCH(data))
+
+#      define ATOMIC_FETCH_32BIT(data) ((unsigned) ATOMIC_ADD((data), 0xffffffff))
+
+#      define ATOMIC_STORE_1_TO_32BIT(data) \
+({ \
+       my_typeof(data) tmp; \
+       sem_wait(&atomicSem); \
+       tmp = data; \
+       data = 1; \
+       sem_post(&atomicSem); \
+       tmp; \
+})
+
+#      define ATOMIC_STORE_0_TO_INT(data) \
+({ \
+       my_typeof(data) tmp; \
+       sem_wait(&atomicSem); \
+       tmp = data; \
+       data = 0; \
+       sem_post(&atomicSem); \
+       tmp; \
+})
+
+#      define ATOMIC_STORE_1_TO_INT(data) \
+({ \
+       my_typeof(data) tmp; \
+       sem_wait(&atomicSem); \
+       tmp = data; \
+       data = 1; \
+       sem_post(&atomicSem); \
+       tmp; \
+})
+
+#      define ATOMIC_CAS(data, oldVal, newVal) \
+({ \
+       int ret; \
+       sem_wait(&atomicSem); \
+       if(data != oldVal) ret = 0; \
+       else \
+       { \
+               data = newVal; \
+               ret = 1; \
+       } \
+       sem_post(&atomicSem); \
+       ret; \
+})
+
+#      define ATOMIC_CAS_VAL(data, oldVal, newVal) \
+({ \
+       sem_wait(&atomicSem); \
+       if(data == oldVal) \
+       { \
+               data = newVal; \
+       } \
+       sem_post(&atomicSem); \
+       data; \
+})
+
+#else /* not HAVE_SEMAPHORE_H */
        /* note that we gained parctical proof that theoretical problems DO occur
         * if we do not properly address them. See this blog post for details:
         * http://blog.gerhards.net/2009/01/rsyslog-data-race-analysis.html
@@ -68,5 +184,6 @@
 #      define ATOMIC_FETCH_32BIT(data) (data)
 #      define ATOMIC_STORE_1_TO_32BIT(data) (data) = 1
 #endif
+#endif

 #endif /* #ifndef INCLUDED_ATOMIC_H */
diff -Nurd rsyslog-5.4.0.orig/runtime/rsyslog.c rsyslog-5.4.0/runtime/rsyslog.c
--- rsyslog-5.4.0.orig/runtime/rsyslog.c        2010-03-05 07:20:36.000000000 +0000
+++ rsyslog-5.4.0/runtime/rsyslog.c     2010-03-16 08:30:51.000000000 +0000
@@ -81,6 +81,7 @@
 #include "rule.h"
 #include "ruleset.h"
 #include "parser.h"
+#include "atomic.h"

 /* forward definitions */
 static rsRetVal dfltErrLogger(int, uchar *errMsg);
@@ -140,6 +141,12 @@
                CHKiRet(objClassInit(NULL)); /* *THIS* *MUST* always be the first class initilizer being called! */
                CHKiRet(objGetObjInterface(pObjIF)); /* this provides the root pointer for all other queries */

+#ifndef HAVE_ATOMIC_BUILTINS
+#ifdef HAVE_SEMAPHORE_H
+               CHKiRet(atomicSemInit());
+#endif /* HAVE_SEMAPHORE_H */
+#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
+
                /* initialize core classes. We must be very careful with the order of events. Some
                 * classes use others and if we do not initialize them in the right order, we may end
                 * up with an invalid call. The most important thing that can happen is that an error
@@ -216,6 +223,13 @@
                glblClassExit();
                rulesetClassExit();
                ruleClassExit();
+
+#ifndef HAVE_ATOMIC_BUILTINS
+#ifdef HAVE_SEMAPHORE_H
+               atomicSemExit();
+#endif /* HAVE_SEMAPHORE_H */
+#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
+
                objClassExit(); /* *THIS* *MUST/SHOULD?* always be the first class initilizer being called (except debug)! */
        }




From rgerhards at hq.adiscon.com  Fri Mar 19 10:10:42 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 19 Mar 2010 10:10:42 +0100
Subject: [rsyslog] CentOS Issues
References: <a60a2fa91003181914kd27e298se6a8e3913563e4ca@mail.gmail.com>
	<alpine.DEB.2.00.1003182125350.3821@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103AFD@GRFEXC.intern.adiscon.com>

> when compiling the new version you should be able to just do make
> install
> after you have compiled it to install it.

As far as I know, you need to set --libdir and --bindir (or its 64 bit
equivalents) ./configure options on CentOS, but I don't know what the correct
values are for it.

Rainer

> 
> you may need to uninstall the RHEL shipped version, I think that by
> default rsyslog will get installed under /usr/local when compiled from
> source.
> 
> installing from source will not create the /etc/rsyslog.conf file for
> you.
> 
> the rsyslog on my laptop (ubuntu with a couple manual tweaks) is:
> 
> #  /etc/rsyslog.conf    Configuration file for rsyslog.
> #
> #                       For more information see
> #                       /usr/share/doc/rsyslog-
> doc/html/rsyslog_conf.html
> #
> #  Default logging rules can be found in /etc/rsyslog.d/50-default.conf
> #################
> #### MODULES ####
> #################
> $ModLoad imuxsock # provides support for local system logging
> $ModLoad imklog   # provides kernel logging support (previously done by
> rklogd)
> #$ModLoad immark  # provides --MARK-- message capability
> #$KLogPath /var/run/rsyslog/kmsg
> # provides UDP syslog reception
> #$ModLoad imudp
> #$UDPServerRun 514
> # provides TCP syslog reception
> #$ModLoad imtcp
> #$InputTCPServerRun 514
> ###########################
> #### GLOBAL DIRECTIVES ####
> ###########################
> #
> # Use traditional timestamp format.
> # To enable high precision timestamps, comment out the following line.
> #
> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
> #
> # Set the default permissions for all log files.
> #
> $FileOwner syslog
> $FileGroup adm
> $FileCreateMode 0640
> $DirCreateMode 0755
> $Umask 0022
> $PrivDropToUser syslog
> $PrivDropToGroup syslog
> $RepeatedMsgReduction on
> $RepeatedMsgContainsOriginalMsg on
> #
> # Include all config files in /etc/rsyslog.d/
> #
> #$IncludeConfig /etc/rsyslog.d/*.conf
> *.* -/var/log/messages
> 
> this should be enough to get you started, you will need to uncomment
> the
> remote reception options and then start rsyslog with rsyslogd -cX where
> X
> is the major version you are running (3,4,5 depending on which one you
> pick)
> 
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Fri Mar 19 10:13:42 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 19 Mar 2010 10:13:42 +0100
Subject: [rsyslog] CentOS Issues
References: <a60a2fa91003181914kd27e298se6a8e3913563e4ca@mail.gmail.com><alpine.DEB.2.00.1003182125350.3821@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7103AFD@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103AFE@GRFEXC.intern.adiscon.com>

> > when compiling the new version you should be able to just do make
> > install
> > after you have compiled it to install it.
> 
> As far as I know, you need to set --libdir and --bindir (or its 64 bit
> equivalents) ./configure options on CentOS, but I don't know what the
> correct
> values are for it.

I just discovered there is some good info on CentOS in the wiki:

http://wiki.rsyslog.com/index.php/Rsyslog_on_CentOS_success_story

I am not sure if it is too much in-depth, though...

Rainer


From rgerhards at hq.adiscon.com  Fri Mar 19 10:22:39 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 19 Mar 2010 10:22:39 +0100
Subject: [rsyslog] patch for architectures without atomic builtins
References: <DB561678F08F554EB0D9F21B70E268A505AE8B@exchange.intern.dresearch.de><alpine.DEB.2.00.1003181555170.10411@asgard.lang.hm>
	<4BA32117.9030408@dresearch.de>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103AFF@GRFEXC.intern.adiscon.com>

Excellent, many thanks for this very useful contribution! I'll see that I
merge it in as quickly as possible, but it may take until early next week.
I'll probably try to merge it into v4 as well, as this is definitely useful!

Thanks again,
Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Steffen Sledz
> Sent: Friday, March 19, 2010 8:01 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] patch for architectures without atomic builtins
> 
> david at lang.hm wrote:
> > the list strips off attachments.
> >
> > can you re-send with the patch inline?
> 
> Here it is...
> 
> diff -Nurd rsyslog-5.4.0.orig/configure.ac rsyslog-5.4.0/configure.ac
> --- rsyslog-5.4.0.orig/configure.ac     2010-03-08 13:28:35.000000000
> +0000
> +++ rsyslog-5.4.0/configure.ac  2010-03-16 11:30:44.000000000 +0000
> @@ -87,6 +87,7 @@
>  AC_HEADER_TIME
>  AC_STRUCT_TM
>  AC_C_VOLATILE
> +AC_C_TYPEOF
> 
>  sa_includes="\
>  $ac_includes_default
> @@ -129,7 +130,8 @@
>  # check for availability of atomic operations
>  RS_ATOMIC_OPERATIONS
>  RS_ATOMIC_OPERATIONS_64BIT
> -
> +# fall back to POSIX sems for atomic operations (cpu expensive)
> +AC_CHECK_HEADERS([semaphore.h])
> 
>  # Additional module directories
>  AC_ARG_WITH(moddirs,
> diff -Nurd rsyslog-5.4.0.orig/runtime/Makefile.am rsyslog-
> 5.4.0/runtime/Makefile.am
> --- rsyslog-5.4.0.orig/runtime/Makefile.am      2010-03-08
> 13:27:47.000000000 +0000
> +++ rsyslog-5.4.0/runtime/Makefile.am   2010-03-15 16:43:12.000000000
> +0000
> @@ -9,6 +9,7 @@
>         rsyslog.h \
>         unicode-helper.h \
>         atomic.h \
> +       atomic-posix-sem.c \
>         batch.h \
>         syslogd-types.h \
>         module-template.h \
> diff -Nurd rsyslog-5.4.0.orig/runtime/atomic-posix-sem.c rsyslog-
> 5.4.0/runtime/atomic-posix-sem.c
> --- rsyslog-5.4.0.orig/runtime/atomic-posix-sem.c       1970-01-01
> 00:00:00.000000000 +0000
> +++ rsyslog-5.4.0/runtime/atomic-posix-sem.c    2010-03-16
> 13:22:27.000000000 +0000
> @@ -0,0 +1,70 @@
> +/* atomic_posix_sem.c: This file supplies an emulation for atomic
> operations using
> + * POSIX semaphores.
> + *
> + * Copyright 2010 DResearch Digital Media Systems GmbH
> + *
> + * This file is part of the rsyslog runtime library.
> + *
> + * The rsyslog runtime library is free software: you can redistribute
> it and/or modify
> + * it under the terms of the GNU Lesser General Public License as
> published by
> + * the Free Software Foundation, either version 3 of the License, or
> + * (at your option) any later version.
> + *
> + * The rsyslog runtime library is distributed in the hope that it will
> be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> License
> + * along with the rsyslog runtime library.  If not, see
> <http://www.gnu.org/licenses/>.
> + *
> + * A copy of the GPL can be found in the file "COPYING" in this
> distribution.
> + * A copy of the LGPL can be found in the file "COPYING.LESSER" in
> this distribution.
> + */
> +
> +#include "config.h"
> +#ifndef HAVE_ATOMIC_BUILTINS
> +#ifdef HAVE_SEMAPHORE_H
> +#include <semaphore.h>
> +#include <errno.h>
> +
> +#include "atomic.h"
> +#include "rsyslog.h"
> +#include "srUtils.h"
> +
> +sem_t atomicSem;
> +
> +rsRetVal
> +atomicSemInit(void)
> +{
> +       DEFiRet;
> +
> +       dbgprintf("init posix semaphore for atomics emulation\n");
> +       if(sem_init(&atomicSem, 0, 1) == -1)
> +       {
> +               char errStr[1024];
> +               rs_strerror_r(errno, errStr, sizeof(errStr));
> +               dbgprintf("init posix semaphore for atomics emulation
> failed: %s\n", errStr);
> +               iRet = RS_RET_SYS_ERR; /* the right error code ??? */
> +       }
> +
> +       RETiRet;
> +}
> +
> +void
> +atomicSemExit(void)
> +{
> +       dbgprintf("destroy posix semaphore for atomics emulation\n");
> +       if(sem_destroy(&atomicSem) == -1)
> +       {
> +               char errStr[1024];
> +               rs_strerror_r(errno, errStr, sizeof(errStr));
> +               dbgprintf("destroy posix semaphore for atomics
> emulation failed: %s\n", errStr);
> +       }
> +}
> +
> +#endif /* HAVE_SEMAPHORE_H */
> +#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
> +
> +/* vim:set ai:
> + */
> diff -Nurd rsyslog-5.4.0.orig/runtime/atomic.h rsyslog-
> 5.4.0/runtime/atomic.h
> --- rsyslog-5.4.0.orig/runtime/atomic.h 2010-03-05 07:20:36.000000000
> +0000
> +++ rsyslog-5.4.0/runtime/atomic.h      2010-03-16 13:23:57.000000000
> +0000
> @@ -54,6 +54,122 @@
>  #      define ATOMIC_CAS(data, oldVal, newVal)
> __sync_bool_compare_and_swap(&(data), (oldVal), (newVal));
>  #      define ATOMIC_CAS_VAL(data, oldVal, newVal)
> __sync_val_compare_and_swap(&(data), (oldVal), (newVal));
>  #else
> +#ifdef HAVE_SEMAPHORE_H
> +       /* we use POSIX semaphores instead */
> +
> +#include "rsyslog.h"
> +#include <semaphore.h>
> +
> +extern sem_t atomicSem;
> +rsRetVal atomicSemInit(void);
> +void atomicSemExit(void);
> +
> +#if HAVE_TYPEOF
> +#define my_typeof(x) typeof(x)
> +#else /* sorry, can't determine types, using 'int' */
> +#define my_typeof(x) int
> +#endif
> +
> +#      define ATOMIC_SUB(data, val) \
> +({ \
> +       my_typeof(data) tmp; \
> +       sem_wait(&atomicSem); \
> +       tmp = data; \
> +       data -= val; \
> +       sem_post(&atomicSem); \
> +       tmp; \
> +})
> +
> +#      define ATOMIC_ADD(data, val) \
> +({ \
> +       my_typeof(data) tmp; \
> +       sem_wait(&atomicSem); \
> +       tmp = data; \
> +       data += val; \
> +       sem_post(&atomicSem); \
> +       tmp; \
> +})
> +
> +#      define ATOMIC_INC_AND_FETCH(data) \
> +({ \
> +       my_typeof(data) tmp; \
> +       sem_wait(&atomicSem); \
> +       tmp = data; \
> +       data += 1; \
> +       sem_post(&atomicSem); \
> +       tmp; \
> +})
> +
> +#      define ATOMIC_INC(data) ((void) ATOMIC_INC_AND_FETCH(data))
> +
> +#      define ATOMIC_DEC_AND_FETCH(data) \
> +({ \
> +       sem_wait(&atomicSem); \
> +       data -= 1; \
> +       sem_post(&atomicSem); \
> +       data; \
> +})
> +
> +#      define ATOMIC_DEC(data) ((void) ATOMIC_DEC_AND_FETCH(data))
> +
> +#      define ATOMIC_FETCH_32BIT(data) ((unsigned) ATOMIC_ADD((data),
> 0xffffffff))
> +
> +#      define ATOMIC_STORE_1_TO_32BIT(data) \
> +({ \
> +       my_typeof(data) tmp; \
> +       sem_wait(&atomicSem); \
> +       tmp = data; \
> +       data = 1; \
> +       sem_post(&atomicSem); \
> +       tmp; \
> +})
> +
> +#      define ATOMIC_STORE_0_TO_INT(data) \
> +({ \
> +       my_typeof(data) tmp; \
> +       sem_wait(&atomicSem); \
> +       tmp = data; \
> +       data = 0; \
> +       sem_post(&atomicSem); \
> +       tmp; \
> +})
> +
> +#      define ATOMIC_STORE_1_TO_INT(data) \
> +({ \
> +       my_typeof(data) tmp; \
> +       sem_wait(&atomicSem); \
> +       tmp = data; \
> +       data = 1; \
> +       sem_post(&atomicSem); \
> +       tmp; \
> +})
> +
> +#      define ATOMIC_CAS(data, oldVal, newVal) \
> +({ \
> +       int ret; \
> +       sem_wait(&atomicSem); \
> +       if(data != oldVal) ret = 0; \
> +       else \
> +       { \
> +               data = newVal; \
> +               ret = 1; \
> +       } \
> +       sem_post(&atomicSem); \
> +       ret; \
> +})
> +
> +#      define ATOMIC_CAS_VAL(data, oldVal, newVal) \
> +({ \
> +       sem_wait(&atomicSem); \
> +       if(data == oldVal) \
> +       { \
> +               data = newVal; \
> +       } \
> +       sem_post(&atomicSem); \
> +       data; \
> +})
> +
> +#else /* not HAVE_SEMAPHORE_H */
>         /* note that we gained parctical proof that theoretical
> problems DO occur
>          * if we do not properly address them. See this blog post for
> details:
>          * http://blog.gerhards.net/2009/01/rsyslog-data-race-
> analysis.html
> @@ -68,5 +184,6 @@
>  #      define ATOMIC_FETCH_32BIT(data) (data)
>  #      define ATOMIC_STORE_1_TO_32BIT(data) (data) = 1
>  #endif
> +#endif
> 
>  #endif /* #ifndef INCLUDED_ATOMIC_H */
> diff -Nurd rsyslog-5.4.0.orig/runtime/rsyslog.c rsyslog-
> 5.4.0/runtime/rsyslog.c
> --- rsyslog-5.4.0.orig/runtime/rsyslog.c        2010-03-05
> 07:20:36.000000000 +0000
> +++ rsyslog-5.4.0/runtime/rsyslog.c     2010-03-16 08:30:51.000000000
> +0000
> @@ -81,6 +81,7 @@
>  #include "rule.h"
>  #include "ruleset.h"
>  #include "parser.h"
> +#include "atomic.h"
> 
>  /* forward definitions */
>  static rsRetVal dfltErrLogger(int, uchar *errMsg);
> @@ -140,6 +141,12 @@
>                 CHKiRet(objClassInit(NULL)); /* *THIS* *MUST* always be
> the first class initilizer being called! */
>                 CHKiRet(objGetObjInterface(pObjIF)); /* this provides
> the root pointer for all other queries */
> 
> +#ifndef HAVE_ATOMIC_BUILTINS
> +#ifdef HAVE_SEMAPHORE_H
> +               CHKiRet(atomicSemInit());
> +#endif /* HAVE_SEMAPHORE_H */
> +#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
> +
>                 /* initialize core classes. We must be very careful
> with the order of events. Some
>                  * classes use others and if we do not initialize them
> in the right order, we may end
>                  * up with an invalid call. The most important thing
> that can happen is that an error
> @@ -216,6 +223,13 @@
>                 glblClassExit();
>                 rulesetClassExit();
>                 ruleClassExit();
> +
> +#ifndef HAVE_ATOMIC_BUILTINS
> +#ifdef HAVE_SEMAPHORE_H
> +               atomicSemExit();
> +#endif /* HAVE_SEMAPHORE_H */
> +#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
> +
>                 objClassExit(); /* *THIS* *MUST/SHOULD?* always be the
> first class initilizer being called (except debug)! */
>         }
> 
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From theinric at redhat.com  Fri Mar 19 10:47:52 2010
From: theinric at redhat.com (Tomas Heinrich)
Date: Fri, 19 Mar 2010 10:47:52 +0100
Subject: [rsyslog] CentOS Issues
In-Reply-To: <a60a2fa91003181914kd27e298se6a8e3913563e4ca@mail.gmail.com>
References: <a60a2fa91003181914kd27e298se6a8e3913563e4ca@mail.gmail.com>
Message-ID: <4BA34848.8090002@redhat.com>

On 03/19/2010 03:14 AM, joe schmo wrote:

<...>

> 1) I have a copy of rsyslog 2.0.6 up and running without any issues (as it
> is the only one I can currently get working...more on that in a sec), other
> than the fact that it can't receive udp messages without manual
> intervention. The only way I can get it to accept udp (network) syslog
> messages is to stop the service, and then run the rsyslog -r514 from
> command-line manually. Is there anyway to get it to use this option by
> default? I need it to start the service with that option enabled. And the
> "just run it manually" answers I have received in other forums aren't
> acceptable for me.

You can add the command-line options to /etc/sysconfig/rsyslog, e.g.
SYSLOGD_OPTIONS="-m 0 -r512"

> 2) An extension of the above question: when I try to do the "obvious"
> solution to the udp issue and use the imudp module, I realized there IS NO
> imudp.so module in the plugins folder. Where would I even get this module
> for 2.0.6? The most I can find is a copy of imudp.c and its accompanying
> make files, but trying to make only outputs errors associated with missing
> files that belong to the C developers package (which I had installed some
> time ago off an official CentOS repo, no updates available).

You don't need this module (and there isn't any) for v2.

Tomas


From dag at wieers.com  Fri Mar 19 13:50:50 2010
From: dag at wieers.com (Dag Wieers)
Date: Fri, 19 Mar 2010 13:50:50 +0100 (CET)
Subject: [rsyslog] CentOS Issues
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103AFE@GRFEXC.intern.adiscon.com>
References: <a60a2fa91003181914kd27e298se6a8e3913563e4ca@mail.gmail.com><alpine.DEB.2.00.1003182125350.3821@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7103AFD@GRFEXC.intern.adiscon.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103AFE@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.LRH.2.00.1003191341470.2470@pikachu.3ti.be>

On Fri, 19 Mar 2010, Rainer Gerhards wrote:

>>> when compiling the new version you should be able to just do make
>>> install
>>> after you have compiled it to install it.
>>
>> As far as I know, you need to set --libdir and --bindir (or its 64 bit
>> equivalents) ./configure options on CentOS, but I don't know what the
>> correct
>> values are for it.
>
> I just discovered there is some good info on CentOS in the wiki:
>
> http://wiki.rsyslog.com/index.php/Rsyslog_on_CentOS_success_story
>
> I am not sure if it is too much in-depth, though...

Also take notice of the fact that RHEL5.5 Beta includes rsyslog 3.22 
because of previously reported issues:

 	https://bugzilla.redhat.com/show_bug.cgi?id=519192
 	https://bugzilla.redhat.com/show_bug.cgi?id=519201
 	https://bugzilla.redhat.com/show_bug.cgi?id=519203

We are happy that Red Hat support worked with us in resolving our 
issues and are pleased with this outcome.

These updates are also expected in the upcoming CentOS releases.
-- 
--   dag wieers,  dag at wieers.com,  http://dag.wieers.com/   --
[Any errors in spelling, tact or fact are transmission errors]


From mangoo at wpkg.org  Fri Mar 19 14:30:47 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Fri, 19 Mar 2010 14:30:47 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <alpine.DEB.2.00.1003182119540.3821@asgard.lang.hm>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org> <4BA2C8F9.4050607@wpkg.org>
	<alpine.DEB.2.00.1003182119540.3821@asgard.lang.hm>
Message-ID: <4BA37C87.8030306@wpkg.org>

Am 19.03.2010 05:20, david at lang.hm wrote:

>> If it's ' - it doesn't like the rule.
>
> when all else fails, look at the documentation ;-)
>
> http://www.rsyslog.com/doc-rsyslog_conf_filter.html
>
> the line should have been
>
> :programname, isequal, 'postgres' ~

The correct entries should be (" works, ' - doesn't):

:programname, isequal, "postgres" ~
*.*;auth,authpriv.none		-/var/log/syslog


Much thanks for your help!


-- 
Tomasz Chmielewski
http://wpkg.org




From mangoo at wpkg.org  Fri Mar 19 14:31:50 2010
From: mangoo at wpkg.org (Tomasz Chmielewski)
Date: Fri, 19 Mar 2010 14:31:50 +0100
Subject: [rsyslog] log *.*, except logs by "postgres"
In-Reply-To: <alpine.DEB.2.00.1003182120550.3821@asgard.lang.hm>
References: <4BA28B32.6090201@wpkg.org>
	<alpine.DEB.2.00.1003181552590.10411@asgard.lang.hm>
	<4BA2B111.7090604@wpkg.org>
	<alpine.DEB.2.00.1003181607370.10411@asgard.lang.hm>
	<4BA2B51F.1060204@wpkg.org>
	<alpine.DEB.2.00.1003181622440.10411@asgard.lang.hm>
	<4BA2B9FE.2030008@wpkg.org>
	<alpine.DEB.2.00.1003181652290.10411@asgard.lang.hm>
	<4BA2BE2D.9020205@wpkg.org>
	<alpine.DEB.2.00.1003181706510.10411@asgard.lang.hm>
	<4BA2C875.4060208@wpkg.org>
	<alpine.DEB.2.00.1003182120550.3821@asgard.lang.hm>
Message-ID: <4BA37CC6.4030203@wpkg.org>

Am 19.03.2010 05:22, david at lang.hm wrote:
> On Fri, 19 Mar 2010, Tomasz Chmielewski wrote:
>
>> Am 19.03.2010 01:08, david at lang.hm wrote:
>>
>>>> where postgres[number] can differ.
>>>
>>> hmm, in these cases postgres should be what's in programname
>>>
>>> taking a different slant, IIRC you used local0.* to capture the postgres
>>> logs into one file, if in the line after that you do
>>>
>>> local0.* ~
>>>
>>> that should eliminate those logs from going into any files after that.
>>
>> This one works, but that'll filter out heartbeat, pgpool and others
>> useful, which get to syslog.
>
> I will point out that those entries are getting put into your
> posgresql_log file along with the postgres logs then.

Yep, but I'm fine with that (they also concern Postgres on this machine, 
but are counted in kilobytes per day, not in several hundred megabytes).


-- 
Tomasz Chmielewski
http://wpkg.org


From sledz at dresearch.de  Fri Mar 19 16:32:02 2010
From: sledz at dresearch.de (Steffen Sledz)
Date: Fri, 19 Mar 2010 16:32:02 +0100
Subject: [rsyslog] patch for architectures without atomic builtins
In-Reply-To: <alpine.DEB.2.00.1003181555170.10411@asgard.lang.hm>
References: <DB561678F08F554EB0D9F21B70E268A505AE8B@exchange.intern.dresearch.de>
	<alpine.DEB.2.00.1003181555170.10411@asgard.lang.hm>
Message-ID: <4BA398F2.4080704@dresearch.de>

david at lang.hm write:
> can you re-send with the patch inline?

I've seen that you use git for rsyslog. I can send our
patch with git-send-email if you have an email address
for it. Or should i use the mailing list address for
this?

Steffen



From david at lang.hm  Fri Mar 19 16:49:37 2010
From: david at lang.hm (david at lang.hm)
Date: Fri, 19 Mar 2010 08:49:37 -0700 (PDT)
Subject: [rsyslog] patch for architectures without atomic builtins
In-Reply-To: <4BA398F2.4080704@dresearch.de>
References: <DB561678F08F554EB0D9F21B70E268A505AE8B@exchange.intern.dresearch.de>
	<alpine.DEB.2.00.1003181555170.10411@asgard.lang.hm>
	<4BA398F2.4080704@dresearch.de>
Message-ID: <alpine.DEB.2.00.1003190847250.3821@asgard.lang.hm>

On Fri, 19 Mar 2010, Steffen Sledz wrote:

> david at lang.hm write:
>> can you re-send with the patch inline?
>
> I've seen that you use git for rsyslog. I can send our
> patch with git-send-email if you have an email address
> for it. Or should i use the mailing list address for
> this?

Rainer is the person who needs to get the patch, I think I saw him respond 
to your other e-mail.

the mailing list address is probably the best for this sort of thing, or 
if you have a git repository that's pubic, you can make a branch there for 
this and send it's address as a pull request.

David Lang


From sledz at dresearch.de  Fri Mar 19 16:56:38 2010
From: sledz at dresearch.de (Steffen Sledz)
Date: Fri, 19 Mar 2010 16:56:38 +0100
Subject: [rsyslog] [PATCH] emulate missing atomic builtins with posix
	semaphores if available
In-Reply-To: <alpine.DEB.2.00.1003190847250.3821@asgard.lang.hm>
References: <alpine.DEB.2.00.1003190847250.3821@asgard.lang.hm>
Message-ID: <1269014198-1533-1-git-send-email-sledz@dresearch.de>

Signed-off-by: Steffen Sledz <sledz at dresearch.de>
Acked-by: Thilo Fromm <t.fromm at dresearch.de>
---
 configure.ac               |    4 +-
 runtime/Makefile.am        |    1 +
 runtime/atomic-posix-sem.c |   70 ++++++++++++++++++++++++++
 runtime/atomic.h           |  117 ++++++++++++++++++++++++++++++++++++++++++++
 runtime/rsyslog.c          |   14 +++++
 5 files changed, 205 insertions(+), 1 deletions(-)
 create mode 100644 runtime/atomic-posix-sem.c

diff --git a/configure.ac b/configure.ac
index c59895d..ca198ea 100644
--- a/configure.ac
+++ b/configure.ac
@@ -87,6 +87,7 @@ AC_TYPE_UINT8_T
 AC_HEADER_TIME
 AC_STRUCT_TM
 AC_C_VOLATILE
+AC_C_TYPEOF
 
 sa_includes="\
 $ac_includes_default
@@ -129,7 +130,8 @@ AC_TRY_COMPILE([
 # check for availability of atomic operations
 RS_ATOMIC_OPERATIONS
 RS_ATOMIC_OPERATIONS_64BIT
-
+# fall back to POSIX sems for atomic operations (cpu expensive)
+AC_CHECK_HEADERS([semaphore.h])
 
 # Additional module directories
 AC_ARG_WITH(moddirs,
diff --git a/runtime/Makefile.am b/runtime/Makefile.am
index 9047c83..2374c57 100644
--- a/runtime/Makefile.am
+++ b/runtime/Makefile.am
@@ -9,6 +9,7 @@ librsyslog_la_SOURCES = \
 	rsyslog.h \
 	unicode-helper.h \
 	atomic.h \
+	atomic-posix-sem.c \
 	batch.h \
 	syslogd-types.h \
 	module-template.h \
diff --git a/runtime/atomic-posix-sem.c b/runtime/atomic-posix-sem.c
new file mode 100644
index 0000000..979fae0
--- /dev/null
+++ b/runtime/atomic-posix-sem.c
@@ -0,0 +1,70 @@
+/* atomic_posix_sem.c: This file supplies an emulation for atomic operations using
+ * POSIX semaphores.
+ *
+ * Copyright 2010 DResearch Digital Media Systems GmbH
+ *
+ * This file is part of the rsyslog runtime library.
+ *
+ * The rsyslog runtime library is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * The rsyslog runtime library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public License
+ * along with the rsyslog runtime library.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * A copy of the GPL can be found in the file "COPYING" in this distribution.
+ * A copy of the LGPL can be found in the file "COPYING.LESSER" in this distribution.
+ */
+
+#include "config.h"
+#ifndef HAVE_ATOMIC_BUILTINS
+#ifdef HAVE_SEMAPHORE_H
+#include <semaphore.h>
+#include <errno.h>
+
+#include "atomic.h"
+#include "rsyslog.h"
+#include "srUtils.h"
+
+sem_t atomicSem;
+
+rsRetVal
+atomicSemInit(void)
+{
+	DEFiRet;
+
+	dbgprintf("init posix semaphore for atomics emulation\n");
+	if(sem_init(&atomicSem, 0, 1) == -1)
+	{
+		char errStr[1024];
+		rs_strerror_r(errno, errStr, sizeof(errStr));
+		dbgprintf("init posix semaphore for atomics emulation failed: %s\n", errStr);
+		iRet = RS_RET_SYS_ERR; /* the right error code ??? */
+	}
+
+	RETiRet;
+}
+
+void
+atomicSemExit(void)
+{
+	dbgprintf("destroy posix semaphore for atomics emulation\n");
+	if(sem_destroy(&atomicSem) == -1)
+	{
+		char errStr[1024];
+		rs_strerror_r(errno, errStr, sizeof(errStr));
+		dbgprintf("destroy posix semaphore for atomics emulation failed: %s\n", errStr);
+	}
+}
+
+#endif /* HAVE_SEMAPHORE_H */
+#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
+
+/* vim:set ai:
+ */
diff --git a/runtime/atomic.h b/runtime/atomic.h
index b507b76..51b14e1 100644
--- a/runtime/atomic.h
+++ b/runtime/atomic.h
@@ -54,6 +54,122 @@
 #	define ATOMIC_CAS(data, oldVal, newVal) __sync_bool_compare_and_swap(&(data), (oldVal), (newVal));
 #	define ATOMIC_CAS_VAL(data, oldVal, newVal) __sync_val_compare_and_swap(&(data), (oldVal), (newVal));
 #else
+#ifdef HAVE_SEMAPHORE_H
+	/* we use POSIX semaphores instead */
+
+#include "rsyslog.h"
+#include <semaphore.h>
+
+extern sem_t atomicSem;
+rsRetVal atomicSemInit(void);
+void atomicSemExit(void);
+
+#if HAVE_TYPEOF
+#define my_typeof(x) typeof(x)
+#else /* sorry, can't determine types, using 'int' */
+#define my_typeof(x) int
+#endif
+
+#	define ATOMIC_SUB(data, val) \
+({ \
+	my_typeof(data) tmp; \
+	sem_wait(&atomicSem); \
+	tmp = data; \
+	data -= val; \
+	sem_post(&atomicSem); \
+	tmp; \
+})
+
+#	define ATOMIC_ADD(data, val) \
+({ \
+	my_typeof(data) tmp; \
+	sem_wait(&atomicSem); \
+	tmp = data; \
+	data += val; \
+	sem_post(&atomicSem); \
+	tmp; \
+})
+
+#	define ATOMIC_INC_AND_FETCH(data) \
+({ \
+	my_typeof(data) tmp; \
+	sem_wait(&atomicSem); \
+	tmp = data; \
+	data += 1; \
+	sem_post(&atomicSem); \
+	tmp; \
+})
+
+#	define ATOMIC_INC(data) ((void) ATOMIC_INC_AND_FETCH(data))
+
+#	define ATOMIC_DEC_AND_FETCH(data) \
+({ \
+	sem_wait(&atomicSem); \
+	data -= 1; \
+	sem_post(&atomicSem); \
+	data; \
+})
+
+#	define ATOMIC_DEC(data) ((void) ATOMIC_DEC_AND_FETCH(data))
+
+#	define ATOMIC_FETCH_32BIT(data) ((unsigned) ATOMIC_ADD((data), 0xffffffff))
+
+#	define ATOMIC_STORE_1_TO_32BIT(data) \
+({ \
+	my_typeof(data) tmp; \
+	sem_wait(&atomicSem); \
+	tmp = data; \
+	data = 1; \
+	sem_post(&atomicSem); \
+	tmp; \
+})
+
+#	define ATOMIC_STORE_0_TO_INT(data) \
+({ \
+	my_typeof(data) tmp; \
+	sem_wait(&atomicSem); \
+	tmp = data; \
+	data = 0; \
+	sem_post(&atomicSem); \
+	tmp; \
+})
+
+#	define ATOMIC_STORE_1_TO_INT(data) \
+({ \
+	my_typeof(data) tmp; \
+	sem_wait(&atomicSem); \
+	tmp = data; \
+	data = 1; \
+	sem_post(&atomicSem); \
+	tmp; \
+})
+
+#	define ATOMIC_CAS(data, oldVal, newVal) \
+({ \
+	int ret; \
+	sem_wait(&atomicSem); \
+	if(data != oldVal) ret = 0; \
+	else \
+	{ \
+		data = newVal; \
+		ret = 1; \
+	} \
+	sem_post(&atomicSem); \
+	ret; \
+})
+
+#	define ATOMIC_CAS_VAL(data, oldVal, newVal) \
+({ \
+	sem_wait(&atomicSem); \
+	if(data == oldVal) \
+	{ \
+		data = newVal; \
+	} \
+	sem_post(&atomicSem); \
+	data; \
+})
+
+#else /* not HAVE_SEMAPHORE_H */
 	/* note that we gained parctical proof that theoretical problems DO occur
 	 * if we do not properly address them. See this blog post for details:
 	 * http://blog.gerhards.net/2009/01/rsyslog-data-race-analysis.html
@@ -68,5 +184,6 @@
 #	define ATOMIC_FETCH_32BIT(data) (data)
 #	define ATOMIC_STORE_1_TO_32BIT(data) (data) = 1
 #endif
+#endif
 
 #endif /* #ifndef INCLUDED_ATOMIC_H */
diff --git a/runtime/rsyslog.c b/runtime/rsyslog.c
index a76ae25..921ad0b 100644
--- a/runtime/rsyslog.c
+++ b/runtime/rsyslog.c
@@ -81,6 +81,7 @@
 #include "rule.h"
 #include "ruleset.h"
 #include "parser.h"
+#include "atomic.h"
 
 /* forward definitions */
 static rsRetVal dfltErrLogger(int, uchar *errMsg);
@@ -140,6 +141,12 @@ rsrtInit(char **ppErrObj, obj_if_t *pObjIF)
 		CHKiRet(objClassInit(NULL)); /* *THIS* *MUST* always be the first class initilizer being called! */
 		CHKiRet(objGetObjInterface(pObjIF)); /* this provides the root pointer for all other queries */
 
+#ifndef HAVE_ATOMIC_BUILTINS
+#ifdef HAVE_SEMAPHORE_H
+		CHKiRet(atomicSemInit());
+#endif /* HAVE_SEMAPHORE_H */
+#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
+
 		/* initialize core classes. We must be very careful with the order of events. Some
 		 * classes use others and if we do not initialize them in the right order, we may end
 		 * up with an invalid call. The most important thing that can happen is that an error
@@ -216,6 +223,13 @@ rsrtExit(void)
 		glblClassExit();
 		rulesetClassExit();
 		ruleClassExit();
+
+#ifndef HAVE_ATOMIC_BUILTINS
+#ifdef HAVE_SEMAPHORE_H
+		atomicSemExit();
+#endif /* HAVE_SEMAPHORE_H */
+#endif /* !defined(HAVE_ATOMIC_BUILTINS) */
+
 		objClassExit(); /* *THIS* *MUST/SHOULD?* always be the first class initilizer being called (except debug)! */
 	}
 
-- 
1.6.4.2



From rgerhards at hq.adiscon.com  Tue Mar 23 11:22:29 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 23 Mar 2010 11:22:29 +0100
Subject: [rsyslog] patch for architectures without atomic builtins
In-Reply-To: <4BA398F2.4080704@dresearch.de>
References: <DB561678F08F554EB0D9F21B70E268A505AE8B@exchange.intern.dresearch.de>
	<alpine.DEB.2.00.1003181555170.10411@asgard.lang.hm>
	<4BA398F2.4080704@dresearch.de>
Message-ID: <1269339749.3320.0.camel@localhost>

thanks again for the contribution, I have just applied it to the master
branch. I will see if I can backport it to v4, what I think would be
useful.

Rainer

On Fri, 2010-03-19 at 16:32 +0100, Steffen Sledz wrote:
> david at lang.hm write:
> > can you re-send with the patch inline?
> 
> I've seen that you use git for rsyslog. I can send our
> patch with git-send-email if you have an email address
> for it. Or should i use the mailing list address for
> this?
> 
> Steffen
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com




From mikolaj at kucharski.name  Thu Mar 25 22:08:43 2010
From: mikolaj at kucharski.name (Mikolaj Kucharski)
Date: Thu, 25 Mar 2010 21:08:43 +0000
Subject: [rsyslog] last message repeated n times from remote hosts
In-Reply-To: <20100210120230.GI3838@x40.openbsd.home.lan>
References: <20100210120230.GI3838@x40.openbsd.home.lan>
Message-ID: <20100325210843.GK3838@x40.openbsd.home.lan>

Any comments?

On Wed, Feb 10, 2010 at 12:02:30PM +0000, Mikolaj Kucharski wrote:
> Hi,
> 
> I have few Linux based machines with sysklogd installed, and I have
> central syslog server based on CentOS 5 with rsyslog-2.0.6-1.el5.
> 
> I have issue with missing hostname when sysklog sends "last message
> repeated N times" over the wire to rsyslog server.
> 
> Let's have a look. I used logger(1) to repeatedly sent one message few
> times followed by one different message. Here is what I see in the log
> file on my central rsyslog server:
> 
> Feb 10 11:39:46 10.101.43.124 root: remote test start
> Feb 10 11:39:54 last message repeated 14 times
> Feb 10 11:39:54 10.101.43.124 root: remote test end
> 
> 
> and here is tcpdump(8) log from the source (10.101.43.124) machine:
> 
> 
> 11:39:46.642297 IP 10.101.43.124.syslog > 10.101.40.116.syslog: SYSLOG user.notice, length: 28
>         0x0000:  4500 0038 0000 4000 4011 d1fb 0a65 2b7c  E..8.. at .@....e+|
>         0x0010:  0a65 2874 0202 0202 0024 68ef 3c31 333e  .e(t.....$h.<13>
>         0x0020:  726f 6f74 3a20 7265 6d6f 7465 2074 6573  root:.remote.tes
>         0x0030:  7420 7374 6172 740a                      t.start.
> 11:39:54.904820 IP 10.101.43.124.syslog > 10.101.40.116.syslog: SYSLOG user.notice, length: 35
>         0x0000:  4500 003f 0000 4000 4011 d1f4 0a65 2b7c  E..?.. at .@....e+|
>         0x0010:  0a65 2874 0202 0202 002b 68f6 3c31 333e  .e(t.....+h.<13>
>         0x0020:  6c61 7374 206d 6573 7361 6765 2072 6570  last.message.rep
>         0x0030:  6561 7465 6420 3134 2074 696d 6573 0a    eated.14.times.
> 11:39:54.904826 IP 10.101.43.124.syslog > 10.101.40.116.syslog: SYSLOG user.notice, length: 26
>         0x0000:  4500 0036 0000 4000 4011 d1fd 0a65 2b7c  E..6.. at .@....e+|
>         0x0010:  0a65 2874 0202 0202 0022 68ed 3c31 333e  .e(t....."h.<13>
>         0x0020:  726f 6f74 3a20 7265 6d6f 7465 2074 6573  root:.remote.tes
>         0x0030:  7420 656e 640a                           t.end.
> 
> 
> I searched the list, and saw a comment which say the fault is on the
> sysklogd end as it never sends hostname in the repeated-n-times packet,
> but from above I cannot see that it ever sends the packet with hostname,
> so I think the issues is on rsyslog side, and not on the sysklogd.
> 
> Could someone shed some light on my issue, as I would like to see all
> the time the source IP or hostname of incomming messages to rsyslog
> daemon.
> 
> Is this missing source hostname/IP a bug of rsyslog?
> 
> Is there any way to workaround that?
> 
> 
> Thanks.
> 
> 
> PS1. I cannot change client machines, I cannot reinstall them with
> different syslog implementation, the only machine where I have
> permission to do modifications is central rsyslog server.
> 
> PS2. I know about DNS and RevDNS and yes, above server doesn't have
> revDNS setup.

-- 
best regards
q#


From pablo at docecosas.com  Fri Mar 26 13:02:41 2010
From: pablo at docecosas.com (Pablo Martinez Schroder)
Date: Fri, 26 Mar 2010 13:02:41 +0100
Subject: [rsyslog] Rsyslog is not using FQDN in all messages
Message-ID: <fe120686aaa393fb1425df1a5fc29530.squirrel@webmail.docecosas.com>

Hi,

I'm using rsyslog-5.4.0 and I having a issue that I cannot understand. I
want to implement a central logging server using stunnel, and I need to
use FQDN always, I don't want to have HOSTNAME trunked to the hostname so
I enable $PreserveFQDN but only some of the messages seems to use the
whole hostname. I've seen this issue in multiple versions of rsyslog, so
I'm quite sure is not caused by rsyslog-5.4.0

I've tested on multiples versions of CentOS and currently I'm testing
rsyslog-5.4.0 on a CentOS 5.4. To to the tests I'm logging all the
messages in a local file, so the interesting part of my /etc/rsyslog.conf
is like this:

$ModLoad immark   # provides --MARK-- message capability
$ModLoad imuxsock # provides support for local system logging (e.g. via
logger command)
$ModLoad imklog   # kernel logging (formerly provided by rklogd)

$PreserveFQDN on

*.* /var/log/everything.log

I start rsyslogd with the "-c 4" option, and if I enable $PreserveFQDN in
the conf messages from kernel and rsyslogd are saved with the proper FQDN
but the rest of the messages are stored as if they were originated from
the hostname, without domain.

# tail /var/log/everything.log
010-03-26T07:51:29.513679-04:00 syslog-test.scrambled.com kernel: imklog
5.4.0, log source = /proc/kmsg started.
2010-03-26T07:51:29.684129-04:00 syslog-test.scrambled.com rsyslogd:
[origin software="rsyslogd" swVersion="5.4.0" x-pid="1977"
x-info="http://www.rsyslog.com"] start
2010-03-26T07:51:29.707163-04:00 syslog-test stunnel:
LOG5[1508:1099114816]: ssyslog connected from 127.0.0.1:35446
2010-03-26T07:51:30.610043-04:00 syslog-test.scrambled.com kernel: Kernel
logging (proc) stopped.
2010-03-26T07:51:30.610519-04:00 syslog-test.scrambled.com rsyslogd:
[origin software="rsyslogd" swVersion="5.4.0" x-pid="1977"
x-info="http://www.rsyslog.com"] exiting on signal 2.

In this case, the only one using "syslog-test" (without the
".scrambled.com" part is stunnel, but all the logs generated from sshd,
pam, cron, etc are generated from "syslog-test" instead of
"syslog-test.scrambled.com".

I only have localhost in the hosts file and if I run hostname I get
"syslog-test.scrambled.com". I even rebooted the server after changing
$PreserveFQDN, just in case it was some weird thing of applications
'caching' the simple hostname (it doesn't make sense, I know). Is there
anything that I'm doing wrong? Is it possible for rsyslog to use the FQDN
for all the logs?

I believe that I need that the FQDN is used locally for it to work with a
remote server (so the HOSTNAME variabvle in the template has the FQDN)

Many thanks in advance.
-- 
Pablo Martinez Schroder




From tbergfeld at hq.adiscon.com  Fri Mar 26 16:04:56 2010
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Fri, 26 Mar 2010 16:04:56 +0100
Subject: [rsyslog] rsyslog 4.6.2 (v4-stable) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103B99@GRFEXC.intern.adiscon.com>

Hi all,

We have just released rsyslog 4.6.2, a member of the v4-stable branch. This
is a bug-fixing release containing some important fixes and also two new
features. It is a recommended update for all users of the v4-stable. See
Changelog for more details.

ChangeLog:

http://www.rsyslog.com/Article449.phtml

Download:

http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-198.phtml


As always, feedback is appreciated.

Best regards,
Tom Bergfeld
--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com


From david at lang.hm  Fri Mar 26 19:19:02 2010
From: david at lang.hm (david at lang.hm)
Date: Fri, 26 Mar 2010 11:19:02 -0700 (PDT)
Subject: [rsyslog] Rsyslog is not using FQDN in all messages
In-Reply-To: <fe120686aaa393fb1425df1a5fc29530.squirrel@webmail.docecosas.com>
References: <fe120686aaa393fb1425df1a5fc29530.squirrel@webmail.docecosas.com>
Message-ID: <alpine.DEB.2.00.1003261116320.12764@asgard.lang.hm>

My first question would be if the logs have the FQDN in them in the first 
place.

you may want to create a log with the format %raw% and see what is 
arriving to your box.

David Lang

On Fri, 26 Mar 2010, Pablo Martinez Schroder wrote:

> Hi,
>
> I'm using rsyslog-5.4.0 and I having a issue that I cannot understand. I
> want to implement a central logging server using stunnel, and I need to
> use FQDN always, I don't want to have HOSTNAME trunked to the hostname so
> I enable $PreserveFQDN but only some of the messages seems to use the
> whole hostname. I've seen this issue in multiple versions of rsyslog, so
> I'm quite sure is not caused by rsyslog-5.4.0
>
> I've tested on multiples versions of CentOS and currently I'm testing
> rsyslog-5.4.0 on a CentOS 5.4. To to the tests I'm logging all the
> messages in a local file, so the interesting part of my /etc/rsyslog.conf
> is like this:
>
> $ModLoad immark   # provides --MARK-- message capability
> $ModLoad imuxsock # provides support for local system logging (e.g. via
> logger command)
> $ModLoad imklog   # kernel logging (formerly provided by rklogd)
>
> $PreserveFQDN on
>
> *.* /var/log/everything.log
>
> I start rsyslogd with the "-c 4" option, and if I enable $PreserveFQDN in
> the conf messages from kernel and rsyslogd are saved with the proper FQDN
> but the rest of the messages are stored as if they were originated from
> the hostname, without domain.
>
> # tail /var/log/everything.log
> 010-03-26T07:51:29.513679-04:00 syslog-test.scrambled.com kernel: imklog
> 5.4.0, log source = /proc/kmsg started.
> 2010-03-26T07:51:29.684129-04:00 syslog-test.scrambled.com rsyslogd:
> [origin software="rsyslogd" swVersion="5.4.0" x-pid="1977"
> x-info="http://www.rsyslog.com"] start
> 2010-03-26T07:51:29.707163-04:00 syslog-test stunnel:
> LOG5[1508:1099114816]: ssyslog connected from 127.0.0.1:35446
> 2010-03-26T07:51:30.610043-04:00 syslog-test.scrambled.com kernel: Kernel
> logging (proc) stopped.
> 2010-03-26T07:51:30.610519-04:00 syslog-test.scrambled.com rsyslogd:
> [origin software="rsyslogd" swVersion="5.4.0" x-pid="1977"
> x-info="http://www.rsyslog.com"] exiting on signal 2.
>
> In this case, the only one using "syslog-test" (without the
> ".scrambled.com" part is stunnel, but all the logs generated from sshd,
> pam, cron, etc are generated from "syslog-test" instead of
> "syslog-test.scrambled.com".
>
> I only have localhost in the hosts file and if I run hostname I get
> "syslog-test.scrambled.com". I even rebooted the server after changing
> $PreserveFQDN, just in case it was some weird thing of applications
> 'caching' the simple hostname (it doesn't make sense, I know). Is there
> anything that I'm doing wrong? Is it possible for rsyslog to use the FQDN
> for all the logs?
>
> I believe that I need that the FQDN is used locally for it to work with a
> remote server (so the HOSTNAME variabvle in the template has the FQDN)
>
> Many thanks in advance.
>


From pablo at docecosas.com  Fri Mar 26 19:28:16 2010
From: pablo at docecosas.com (Pablo Martinez Schroder)
Date: Fri, 26 Mar 2010 19:28:16 +0100
Subject: [rsyslog] Rsyslog is not using FQDN in all messages
In-Reply-To: <alpine.DEB.2.00.1003261116320.12764@asgard.lang.hm>
References: <fe120686aaa393fb1425df1a5fc29530.squirrel@webmail.docecosas.com>
	<alpine.DEB.2.00.1003261116320.12764@asgard.lang.hm>
Message-ID: <bd858f6056203841b7c277706d5f5106.squirrel@webmail.docecosas.com>


> My first question would be if the logs have the FQDN in them in the first
> place.
>
> you may want to create a log with the format %raw% and see what is
> arriving to your box.

I'm currently testing this issue on local, with the configuration below,
and the logs saved are a mix: some of them have a hostname like
syslog-test.scrambled.com (only the logs generated from kernel and
rsyslog) and all the other lines publish syslog-test as hostname.

$PreserveFQDN on

*.* /var/log/everything.log

>> I'm using rsyslog-5.4.0 and I having a issue that I cannot understand. I
>> want to implement a central logging server using stunnel, and I need to
>> use FQDN always, I don't want to have HOSTNAME trunked to the hostname
>> so
>> I enable $PreserveFQDN but only some of the messages seems to use the
>> whole hostname. I've seen this issue in multiple versions of rsyslog, so
>> I'm quite sure is not caused by rsyslog-5.4.0
>>
>> I start rsyslogd with the "-c 4" option, and if I enable $PreserveFQDN
>> in
>> the conf messages from kernel and rsyslogd are saved with the proper
>> FQDN
>> but the rest of the messages are stored as if they were originated from
>> the hostname, without domain.

-- 
Pablo Martinez Schroder



From ralph at crongeyer.com  Sun Mar 28 16:29:20 2010
From: ralph at crongeyer.com (Ralph Crongeyer)
Date: Sun, 28 Mar 2010 10:29:20 -0400
Subject: [rsyslog] High cpu usage - rsyslog 5.4.0.
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>
References: <5fa6e0144d8003c7c72edff17f9f1675@webmail.crongeyer.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>
Message-ID: <4BAF67C0.4000506@crongeyer.com>

List,
I tried 5.4.0 last weekend on two machines and ran it for almost a week. 
During that time I noticed that every day when I got home from work both 
machines server/log collector and my client machines fans were racing 
(both laptops, Compaq Armada M700's) ? I logged in ran top and found 
rsyslog consuming 99% of the cpu on both machines? After I restarted 
rsyslog on both they seemed to run normal. The next day the same thing. 
By this time I was watching more closely. I noticed that it took about 
an 45 to 60 minutes and rsyslog would have consumed 95 to 99% of the cpu 
on both machines?

I tried to reboot both machines but it would still happen. I have 
switched back to 5.3.7 which doesn't have this problem.

I'm running Debian Lenny on both machines.
I compiled rsyslog with this:
./configure --prefix=/usr --sysconfdir=/etc --enable-relp 
--enable-imtemplate --enable-omtemplate --enable-rfc3195 --enable-imdiag 
--enable-mail --enable-zlib --enable-mysql --enable-gnutls 
--enable-imdiag --enable-imfile --enable-omprog --enable-omudpspoof 
--enable-omstdout --enable-diagtools
make
checkinstall -D make install

I used the same commands to make 5.3.7 and 5.4.0.

Is anyone else seeing high cpu loads with 5.4.0?

Thanks,
Ralph

-- 
Reminds me of my expedition into the wilds of Afghanistan. We lost our 
corkscrew and were compelled to live on food and water for several days. - 
WC Fields



From mbiebl at gmail.com  Sun Mar 28 16:49:49 2010
From: mbiebl at gmail.com (Michael Biebl)
Date: Sun, 28 Mar 2010 16:49:49 +0200
Subject: [rsyslog] High cpu usage - rsyslog 5.4.0.
In-Reply-To: <4BAF67C0.4000506@crongeyer.com>
References: <5fa6e0144d8003c7c72edff17f9f1675@webmail.crongeyer.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>
	<4BAF67C0.4000506@crongeyer.com>
Message-ID: <aa76cdac1003280749p4673e404t7e5aefd63e102348@mail.gmail.com>

2010/3/28 Ralph Crongeyer <ralph at crongeyer.com>:
> I'm running Debian Lenny on both machines.

> Is anyone else seeing high cpu loads with 5.4.0?
>

I can confirm his (on Debian unstable) and already notified Rainer
about this particular issue.
Some preliminary testing here seems to indicate it's an issue with pipes.

When I removed
daemon.*;mail.*;\
 news.err;\
 *.=debug;*.=info;\
 *.=notice;*.=warn      |/dev/xconsole

from my rsyslog.conf, I could no longer reproduce the problem.

Would be interesting to know, if you can confirm this.

Cheers,
Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From ralph at crongeyer.com  Sun Mar 28 16:52:54 2010
From: ralph at crongeyer.com (Ralph Crongeyer)
Date: Sun, 28 Mar 2010 10:52:54 -0400
Subject: [rsyslog] High cpu usage - rsyslog 5.4.0.
In-Reply-To: <aa76cdac1003280749p4673e404t7e5aefd63e102348@mail.gmail.com>
References: <5fa6e0144d8003c7c72edff17f9f1675@webmail.crongeyer.com>	<9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>	<4BAF67C0.4000506@crongeyer.com>
	<aa76cdac1003280749p4673e404t7e5aefd63e102348@mail.gmail.com>
Message-ID: <4BAF6D46.9090009@crongeyer.com>

I'll do it right now. Just a second.

Thanks,
Ralph

Michael Biebl wrote:
> 2010/3/28 Ralph Crongeyer <ralph at crongeyer.com>:
>   
>> I'm running Debian Lenny on both machines.
>>     
>
>   
>> Is anyone else seeing high cpu loads with 5.4.0?
>>
>>     
>
> I can confirm his (on Debian unstable) and already notified Rainer
> about this particular issue.
> Some preliminary testing here seems to indicate it's an issue with pipes.
>
> When I removed
> daemon.*;mail.*;\
>  news.err;\
>  *.=debug;*.=info;\
>  *.=notice;*.=warn      |/dev/xconsole
>
> from my rsyslog.conf, I could no longer reproduce the problem.
>
> Would be interesting to know, if you can confirm this.
>
> Cheers,
> Michael
>
>   


-- 
Reminds me of my expedition into the wilds of Afghanistan. We lost our 
corkscrew and were compelled to live on food and water for several days. - 
WC Fields



From ralph at crongeyer.com  Sun Mar 28 17:09:02 2010
From: ralph at crongeyer.com (Ralph Crongeyer)
Date: Sun, 28 Mar 2010 11:09:02 -0400
Subject: [rsyslog] High cpu usage - rsyslog 5.4.0.
In-Reply-To: <aa76cdac1003280749p4673e404t7e5aefd63e102348@mail.gmail.com>
References: <5fa6e0144d8003c7c72edff17f9f1675@webmail.crongeyer.com>	<9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>	<4BAF67C0.4000506@crongeyer.com>
	<aa76cdac1003280749p4673e404t7e5aefd63e102348@mail.gmail.com>
Message-ID: <4BAF710E.8080506@crongeyer.com>

Ok,
I stoped rsyslog apt-get removed the 5.3.7 package, installed the 5.4.0 
package:

root at logs:/opt# aptitude show rsyslog
Package: rsyslog
State: installed
Automatically installed: no
Version: 5.4.0-1
Priority: extra
Section: extra
Maintainer: ralph at crongeyer.com
Uncompressed Size: 3047k
Depends: libc6 (>= 2.7-1), zlib1g (>= 1:1.1.4), lsb-base (>= 3.2-14)
Description: enhanced multi-threaded syslogd

commented out the lines from rsyslog.conf:
#daemon.*;mail.*;\
#       news.err;\
#       *.=debug;*.=info;\
#       *.=notice;*.=warn       |/dev/xconsole


and restarted rsyslog:
root at logs:/opt# /etc/init.d/rsyslog restart
Stopping enhanced syslogd: rsyslogd.
Starting enhanced syslogd: rsyslogd.

I'll get back to you in about an hour.

Thanks,
Ralph

Michael Biebl wrote:
> 2010/3/28 Ralph Crongeyer <ralph at crongeyer.com>:
>   
>> I'm running Debian Lenny on both machines.
>>     
>
>   
>> Is anyone else seeing high cpu loads with 5.4.0?
>>
>>     
>
> I can confirm his (on Debian unstable) and already notified Rainer
> about this particular issue.
> Some preliminary testing here seems to indicate it's an issue with pipes.
>
> When I removed
> daemon.*;mail.*;\
>  news.err;\
>  *.=debug;*.=info;\
>  *.=notice;*.=warn      |/dev/xconsole
>
> from my rsyslog.conf, I could no longer reproduce the problem.
>
> Would be interesting to know, if you can confirm this.
>
> Cheers,
> Michael
>
>   


-- 
Reminds me of my expedition into the wilds of Afghanistan. We lost our 
corkscrew and were compelled to live on food and water for several days. - 
WC Fields



From ralph at crongeyer.com  Sun Mar 28 17:54:21 2010
From: ralph at crongeyer.com (Ralph Crongeyer)
Date: Sun, 28 Mar 2010 11:54:21 -0400
Subject: [rsyslog] rsyslog + mysql + base(snort)
Message-ID: <4BAF7BAD.8020502@crongeyer.com>

All,
I'm would like to try to have a setup like this. Have Snort log to 
rsyslog and then have rsyslog log to a mysql Base schema database.
I know that people use Barnyard and or Barnyard2 for this setup to 
offload the writing to mysql to barnyard so that barnyard could receive 
snort logs and spool them if necessary before writing to mysql should 
mysql not be able to keep up.
It seems to me that rsyslog's spooling capability could eliminate the 
need for barnyard.
How would one go about applying a (for lack of better words) particular 
database schema so that rsyslog could write to the base database?

Does anyone have any thoughts on this?

Thanks,
Ralph

-- 
Reminds me of my expedition into the wilds of Afghanistan. We lost our 
corkscrew and were compelled to live on food and water for several days. - 
WC Fields



From ralph at crongeyer.com  Sun Mar 28 18:44:35 2010
From: ralph at crongeyer.com (Ralph Crongeyer)
Date: Sun, 28 Mar 2010 12:44:35 -0400
Subject: [rsyslog] High cpu usage - rsyslog 5.4.0.
In-Reply-To: <4BAF710E.8080506@crongeyer.com>
References: <5fa6e0144d8003c7c72edff17f9f1675@webmail.crongeyer.com>	<9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>	<4BAF67C0.4000506@crongeyer.com>	<aa76cdac1003280749p4673e404t7e5aefd63e102348@mail.gmail.com>
	<4BAF710E.8080506@crongeyer.com>
Message-ID: <4BAF8773.1020000@crongeyer.com>

It's confirmed!

I've been running for over an hour and a half now and:
load average: 0.06, 0.06, 0.02

Looks good so far.

Let me know if you guys (Michael, Rainer) want me to run in debug mode 
without the lines commented out to see if I can catch the problem for 
more troubleshooting.

Thanks,
Ralph

Ralph Crongeyer wrote:
> Ok,
> I stoped rsyslog apt-get removed the 5.3.7 package, installed the 5.4.0 
> package:
>
> root at logs:/opt# aptitude show rsyslog
> Package: rsyslog
> State: installed
> Automatically installed: no
> Version: 5.4.0-1
> Priority: extra
> Section: extra
> Maintainer: ralph at crongeyer.com
> Uncompressed Size: 3047k
> Depends: libc6 (>= 2.7-1), zlib1g (>= 1:1.1.4), lsb-base (>= 3.2-14)
> Description: enhanced multi-threaded syslogd
>
> commented out the lines from rsyslog.conf:
> #daemon.*;mail.*;\
> #       news.err;\
> #       *.=debug;*.=info;\
> #       *.=notice;*.=warn       |/dev/xconsole
>
>
> and restarted rsyslog:
> root at logs:/opt# /etc/init.d/rsyslog restart
> Stopping enhanced syslogd: rsyslogd.
> Starting enhanced syslogd: rsyslogd.
>
> I'll get back to you in about an hour.
>
> Thanks,
> Ralph
>
> Michael Biebl wrote:
>   
>> 2010/3/28 Ralph Crongeyer <ralph at crongeyer.com>:
>>   
>>     
>>> I'm running Debian Lenny on both machines.
>>>     
>>>       
>>   
>>     
>>> Is anyone else seeing high cpu loads with 5.4.0?
>>>
>>>     
>>>       
>> I can confirm his (on Debian unstable) and already notified Rainer
>> about this particular issue.
>> Some preliminary testing here seems to indicate it's an issue with pipes.
>>
>> When I removed
>> daemon.*;mail.*;\
>>  news.err;\
>>  *.=debug;*.=info;\
>>  *.=notice;*.=warn      |/dev/xconsole
>>
>> from my rsyslog.conf, I could no longer reproduce the problem.
>>
>> Would be interesting to know, if you can confirm this.
>>
>> Cheers,
>> Michael
>>
>>   
>>     
>
>
>   


-- 
Reminds me of my expedition into the wilds of Afghanistan. We lost our 
corkscrew and were compelled to live on food and water for several days. - 
WC Fields



From david at lang.hm  Sun Mar 28 21:10:39 2010
From: david at lang.hm (david at lang.hm)
Date: Sun, 28 Mar 2010 12:10:39 -0700 (PDT)
Subject: [rsyslog] High cpu usage - rsyslog 5.4.0.
In-Reply-To: <4BAF67C0.4000506@crongeyer.com>
References: <5fa6e0144d8003c7c72edff17f9f1675@webmail.crongeyer.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>
	<4BAF67C0.4000506@crongeyer.com>
Message-ID: <alpine.DEB.2.00.1003281209061.10653@asgard.lang.hm>

On Sun, 28 Mar 2010, Ralph Crongeyer wrote:

> List,
> I tried 5.4.0 last weekend on two machines and ran it for almost a week.
> During that time I noticed that every day when I got home from work both
> machines server/log collector and my client machines fans were racing
> (both laptops, Compaq Armada M700's) ? I logged in ran top and found
> rsyslog consuming 99% of the cpu on both machines? After I restarted
> rsyslog on both they seemed to run normal. The next day the same thing.
> By this time I was watching more closely. I noticed that it took about
> an 45 to 60 minutes and rsyslog would have consumed 95 to 99% of the cpu
> on both machines?
>
> I tried to reboot both machines but it would still happen. I have
> switched back to 5.3.7 which doesn't have this problem.
>
> I'm running Debian Lenny on both machines.
> I compiled rsyslog with this:
> ./configure --prefix=/usr --sysconfdir=/etc --enable-relp
> --enable-imtemplate --enable-omtemplate --enable-rfc3195 --enable-imdiag
> --enable-mail --enable-zlib --enable-mysql --enable-gnutls
> --enable-imdiag --enable-imfile --enable-omprog --enable-omudpspoof
> --enable-omstdout --enable-diagtools
> make
> checkinstall -D make install
>
> I used the same commands to make 5.3.7 and 5.4.0.
>
> Is anyone else seeing high cpu loads with 5.4.0?

my ubuntu laptop ends up with hundreds of thousands of messages/sec and 
eats up 100% cpu with an error message about access not being allowed.

other than that I have had no problems with excessive CPU, even under 
heavy loads.

David Lang

> Thanks,
> Ralph
>
>


From david at lang.hm  Sun Mar 28 21:13:19 2010
From: david at lang.hm (david at lang.hm)
Date: Sun, 28 Mar 2010 12:13:19 -0700 (PDT)
Subject: [rsyslog] rsyslog + mysql + base(snort)
In-Reply-To: <4BAF7BAD.8020502@crongeyer.com>
References: <4BAF7BAD.8020502@crongeyer.com>
Message-ID: <alpine.DEB.2.00.1003281211250.10653@asgard.lang.hm>

I was just talking about snort logging friday with this in mind..

My understanding of the snort logs include a significant binary component. 
This would not be trivial to deal with in rsyslog without encoding it 
first.

David Lang

  On Sun, 28 Mar 2010, Ralph Crongeyer wrote:

> All,
> I'm would like to try to have a setup like this. Have Snort log to
> rsyslog and then have rsyslog log to a mysql Base schema database.
> I know that people use Barnyard and or Barnyard2 for this setup to
> offload the writing to mysql to barnyard so that barnyard could receive
> snort logs and spool them if necessary before writing to mysql should
> mysql not be able to keep up.
> It seems to me that rsyslog's spooling capability could eliminate the
> need for barnyard.
> How would one go about applying a (for lack of better words) particular
> database schema so that rsyslog could write to the base database?
>
> Does anyone have any thoughts on this?
>
> Thanks,
> Ralph
>
>


From rgerhards at hq.adiscon.com  Mon Mar 29 07:11:10 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 29 Mar 2010 07:11:10 +0200
Subject: [rsyslog] High cpu usage - rsyslog 5.4.0.
References: <5fa6e0144d8003c7c72edff17f9f1675@webmail.crongeyer.com>	<9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>	<4BAF67C0.4000506@crongeyer.com>	<aa76cdac1003280749p4673e404t7e5aefd63e102348@mail.gmail.com><4BAF710E.8080506@crongeyer.com>
	<4BAF8773.1020000@crongeyer.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103BAA@GRFEXC.intern.adiscon.com>

Hi,

thanks for the information. I have crafted a big patch package for v4 the
last two weeks. I will try to merge it in to v5 this week (I guess this will
be a bit harder than usual). It may fix the issue.

If you have a moment or two, it would be interesting to know if 5.5.2 has the
same problem.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Ralph Crongeyer
> Sent: Sunday, March 28, 2010 6:45 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] High cpu usage - rsyslog 5.4.0.
> 
> It's confirmed!
> 
> I've been running for over an hour and a half now and:
> load average: 0.06, 0.06, 0.02
> 
> Looks good so far.
> 
> Let me know if you guys (Michael, Rainer) want me to run in debug mode
> without the lines commented out to see if I can catch the problem for
> more troubleshooting.
> 
> Thanks,
> Ralph
> 
> Ralph Crongeyer wrote:
> > Ok,
> > I stoped rsyslog apt-get removed the 5.3.7 package, installed the
> 5.4.0
> > package:
> >
> > root at logs:/opt# aptitude show rsyslog
> > Package: rsyslog
> > State: installed
> > Automatically installed: no
> > Version: 5.4.0-1
> > Priority: extra
> > Section: extra
> > Maintainer: ralph at crongeyer.com
> > Uncompressed Size: 3047k
> > Depends: libc6 (>= 2.7-1), zlib1g (>= 1:1.1.4), lsb-base (>= 3.2-14)
> > Description: enhanced multi-threaded syslogd
> >
> > commented out the lines from rsyslog.conf:
> > #daemon.*;mail.*;\
> > #       news.err;\
> > #       *.=debug;*.=info;\
> > #       *.=notice;*.=warn       |/dev/xconsole
> >
> >
> > and restarted rsyslog:
> > root at logs:/opt# /etc/init.d/rsyslog restart
> > Stopping enhanced syslogd: rsyslogd.
> > Starting enhanced syslogd: rsyslogd.
> >
> > I'll get back to you in about an hour.
> >
> > Thanks,
> > Ralph
> >
> > Michael Biebl wrote:
> >
> >> 2010/3/28 Ralph Crongeyer <ralph at crongeyer.com>:
> >>
> >>
> >>> I'm running Debian Lenny on both machines.
> >>>
> >>>
> >>
> >>
> >>> Is anyone else seeing high cpu loads with 5.4.0?
> >>>
> >>>
> >>>
> >> I can confirm his (on Debian unstable) and already notified Rainer
> >> about this particular issue.
> >> Some preliminary testing here seems to indicate it's an issue with
> pipes.
> >>
> >> When I removed
> >> daemon.*;mail.*;\
> >>  news.err;\
> >>  *.=debug;*.=info;\
> >>  *.=notice;*.=warn      |/dev/xconsole
> >>
> >> from my rsyslog.conf, I could no longer reproduce the problem.
> >>
> >> Would be interesting to know, if you can confirm this.
> >>
> >> Cheers,
> >> Michael
> >>
> >>
> >>
> >
> >
> >
> 
> 
> --
> Reminds me of my expedition into the wilds of Afghanistan. We lost our
> corkscrew and were compelled to live on food and water for several
> days. -
> WC Fields
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Mon Mar 29 17:29:37 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 29 Mar 2010 17:29:37 +0200
Subject: [rsyslog] High cpu usage - rsyslog 5.4.0.
References: <5fa6e0144d8003c7c72edff17f9f1675@webmail.crongeyer.com>	<9B6E2A8877C38245BFB15CC491A11DA7103702@GRFEXC.intern.adiscon.com>	<4BAF67C0.4000506@crongeyer.com>	<aa76cdac1003280749p4673e404t7e5aefd63e102348@mail.gmail.com><4BAF710E.8080506@crongeyer.com><4BAF8773.1020000@crongeyer.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103BAA@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103BBC@GRFEXC.intern.adiscon.com>

I managed to merge the v4 changes into v5-stable. Can you pull from it's git
head (git.adiscon.com/git/rsyslog.gig) and check if it works? Or do you need
a release tarball.

Note that I need to finally fix an issue in v4 before (the released version
has an interim/work-around fix) before I can take up more work on v5, but
chances are not too bad that the issue is fixed.

The master branch will receive the new patches ASAP, but that merge is a
little bit more complicated.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Monday, March 29, 2010 7:11 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] High cpu usage - rsyslog 5.4.0.
> 
> Hi,
> 
> thanks for the information. I have crafted a big patch package for v4
> the
> last two weeks. I will try to merge it in to v5 this week (I guess this
> will
> be a bit harder than usual). It may fix the issue.
> 
> If you have a moment or two, it would be interesting to know if 5.5.2
> has the
> same problem.
> 
> Rainer
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Ralph Crongeyer
> > Sent: Sunday, March 28, 2010 6:45 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] High cpu usage - rsyslog 5.4.0.
> >
> > It's confirmed!
> >
> > I've been running for over an hour and a half now and:
> > load average: 0.06, 0.06, 0.02
> >
> > Looks good so far.
> >
> > Let me know if you guys (Michael, Rainer) want me to run in debug
> mode
> > without the lines commented out to see if I can catch the problem for
> > more troubleshooting.
> >
> > Thanks,
> > Ralph
> >
> > Ralph Crongeyer wrote:
> > > Ok,
> > > I stoped rsyslog apt-get removed the 5.3.7 package, installed the
> > 5.4.0
> > > package:
> > >
> > > root at logs:/opt# aptitude show rsyslog
> > > Package: rsyslog
> > > State: installed
> > > Automatically installed: no
> > > Version: 5.4.0-1
> > > Priority: extra
> > > Section: extra
> > > Maintainer: ralph at crongeyer.com
> > > Uncompressed Size: 3047k
> > > Depends: libc6 (>= 2.7-1), zlib1g (>= 1:1.1.4), lsb-base (>= 3.2-
> 14)
> > > Description: enhanced multi-threaded syslogd
> > >
> > > commented out the lines from rsyslog.conf:
> > > #daemon.*;mail.*;\
> > > #       news.err;\
> > > #       *.=debug;*.=info;\
> > > #       *.=notice;*.=warn       |/dev/xconsole
> > >
> > >
> > > and restarted rsyslog:
> > > root at logs:/opt# /etc/init.d/rsyslog restart
> > > Stopping enhanced syslogd: rsyslogd.
> > > Starting enhanced syslogd: rsyslogd.
> > >
> > > I'll get back to you in about an hour.
> > >
> > > Thanks,
> > > Ralph
> > >
> > > Michael Biebl wrote:
> > >
> > >> 2010/3/28 Ralph Crongeyer <ralph at crongeyer.com>:
> > >>
> > >>
> > >>> I'm running Debian Lenny on both machines.
> > >>>
> > >>>
> > >>
> > >>
> > >>> Is anyone else seeing high cpu loads with 5.4.0?
> > >>>
> > >>>
> > >>>
> > >> I can confirm his (on Debian unstable) and already notified Rainer
> > >> about this particular issue.
> > >> Some preliminary testing here seems to indicate it's an issue with
> > pipes.
> > >>
> > >> When I removed
> > >> daemon.*;mail.*;\
> > >>  news.err;\
> > >>  *.=debug;*.=info;\
> > >>  *.=notice;*.=warn      |/dev/xconsole
> > >>
> > >> from my rsyslog.conf, I could no longer reproduce the problem.
> > >>
> > >> Would be interesting to know, if you can confirm this.
> > >>
> > >> Cheers,
> > >> Michael
> > >>
> > >>
> > >>
> > >
> > >
> > >
> >
> >
> > --
> > Reminds me of my expedition into the wilds of Afghanistan. We lost
> our
> > corkscrew and were compelled to live on food and water for several
> > days. -
> > WC Fields
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From boonsiong0527 at gmail.com  Tue Mar 30 08:40:57 2010
From: boonsiong0527 at gmail.com (BoOnSiOnG)
Date: Tue, 30 Mar 2010 14:40:57 +0800
Subject: [rsyslog] imfile module
In-Reply-To: <mailman.230.1269930120.2883.rsyslog@lists.adiscon.com>
References: <mailman.230.1269930120.2883.rsyslog@lists.adiscon.com>
Message-ID: <bb92aee11003292340s7fc3ad29q8f143c251a492041@mail.gmail.com>

Hello all,

currently I'm using imfile module to monitor apache Tomcat
catalina.out, I noticed rsyslog will append date time and machine name
to every line, is it possible to configure rsyslog do not alter the
original log?

Mar 29 04:13:08 rsyslogd : INFO: Using <Listener
className="org.apache.jk.config.ApacheConfig"
modJk="PATH_TO_MOD_JK.SO_OR_DLL" />
Mar 29 04:13:08 rsyslogd : Mar 29, 2010 4:13:08 AM
org.apache.coyote.http11.Http11Protocol start
Mar 29 04:13:08 rsyslogd : INFO: Starting Coyote HTTP/1.1 on http-8080
Mar 29 04:13:08 rsyslogd : Mar 29, 2010 4:13:08 AM
org.apache.jk.common.ChannelSocket init
Mar 29 04:13:08 rsyslogd : INFO: JK: ajp13 listening on /0.0.0.0:8009
Mar 29 04:13:08 rsyslogd : Mar 29, 2010 4:13:08 AM
org.apache.jk.server.JkMain start
Mar 29 04:13:08 rsyslogd : INFO: Jk running ID=0 time=0/14 config=null
Mar 29 04:13:08 rsyslogd : Mar 29, 2010 4:13:08 AM
org.apache.catalina.startup.Catalina start

my configuration,
$InputFileName /srv/tomcat/logs/catalina.out
$InputFileTag :
$InputFileStateFile stat-file-tomcat-01
$InputFileSeverity info
$InputFileFacility local3
$InputRunFileMonitor

Thanks. :)


From david at lang.hm  Tue Mar 30 09:03:37 2010
From: david at lang.hm (david at lang.hm)
Date: Tue, 30 Mar 2010 00:03:37 -0700 (PDT)
Subject: [rsyslog] imfile module
In-Reply-To: <bb92aee11003292340s7fc3ad29q8f143c251a492041@mail.gmail.com>
References: <mailman.230.1269930120.2883.rsyslog@lists.adiscon.com>
	<bb92aee11003292340s7fc3ad29q8f143c251a492041@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1003300000170.14525@asgard.lang.hm>

On Tue, 30 Mar 2010, BoOnSiOnG wrote:

> Hello all,
>
> currently I'm using imfile module to monitor apache Tomcat
> catalina.out, I noticed rsyslog will append date time and machine name
> to every line, is it possible to configure rsyslog do not alter the
> original log?

the way to do this would be to change the output format when writing the 
log.

David Lang

> Mar 29 04:13:08 rsyslogd : INFO: Using <Listener
> className="org.apache.jk.config.ApacheConfig"
> modJk="PATH_TO_MOD_JK.SO_OR_DLL" />
> Mar 29 04:13:08 rsyslogd : Mar 29, 2010 4:13:08 AM
> org.apache.coyote.http11.Http11Protocol start
> Mar 29 04:13:08 rsyslogd : INFO: Starting Coyote HTTP/1.1 on http-8080
> Mar 29 04:13:08 rsyslogd : Mar 29, 2010 4:13:08 AM
> org.apache.jk.common.ChannelSocket init
> Mar 29 04:13:08 rsyslogd : INFO: JK: ajp13 listening on /0.0.0.0:8009
> Mar 29 04:13:08 rsyslogd : Mar 29, 2010 4:13:08 AM
> org.apache.jk.server.JkMain start
> Mar 29 04:13:08 rsyslogd : INFO: Jk running ID=0 time=0/14 config=null
> Mar 29 04:13:08 rsyslogd : Mar 29, 2010 4:13:08 AM
> org.apache.catalina.startup.Catalina start
>
> my configuration,
> $InputFileName /srv/tomcat/logs/catalina.out
> $InputFileTag :
> $InputFileStateFile stat-file-tomcat-01
> $InputFileSeverity info
> $InputFileFacility local3
> $InputRunFileMonitor
>
> Thanks. :)
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>