From rgerhards at hq.adiscon.com  Sun Aug  1 11:30:17 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Sun, 1 Aug 2010 11:30:17 +0200
Subject: [rsyslog] log forwarding through unix sockets
References: <4C456476.4010703@redhat.com>
	<AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710403E@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Ryan Lynch
> Sent: Friday, July 30, 2010 3:29 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] log forwarding through unix sockets
> 
> On Tue, Jul 20, 2010 at 04:55, Ales Kozumplik <akozumpl at redhat.com>
> wrote:
> > 1) KVM opens a SOCK_STREAM on the host end but rsyslogd is only able
> to
> > read data from SOCK_DGRAM. This has two consequences: first, to be
> able
> > to attach rsyslog on the host end one first needs to copy the data
> > between the two socket types, e.g. using socat. Second, messages
> longer
> > than 1024 characters are sometimes split into two. The second message
> is
> > thus missing the syslog header and the receiving rsyslogd doesn't
> know
> > where to file it. Is there a recommended workaround for those things
> > (maybe a parameter I overlooked in the docs tellling rsyslogd to use
> > SOCK_STREAM)?
> 
> I ran into a similar problem. Doesn't it seem wierd that RSyslog:
>  - can write TO a pipe, but it can't natively read FROM a pipe.
>  - can read FROM a UNIX socket, but can't natively write TO a UNIX
> socket.
> The only protocols that Rsyslog can both read to AND write from are
> network sockets (UDP, TCP, RELP) and real files.
> 

Quite honestly, I think this issue never came up until recently -- at least I
didn't notice. It should be fairly simple to add a plugin to write to unix
sockets, though...
 
> I don't think the problem is your forwarding format--I don't think
> it's possible for RSyslog to handle a HOSTNAME field, properly, in
> messages received via socket.
> 
> Based on my own tests, I believe that 'imuxsock' and 'imudp' use
> different logic to parse incoming messages. 'imuxsock' always assumes
> that the hostname is the local host, so it doesn't have the
> conditional logic to differentiate between "forwarded" messages (which
> have an extra HOSTNAME field between the timestamp and tag) versus
> regular local messages (no HOSTNAME).

The problem is that you do not have any reliable indication what is a
hostname and what not. So far, I worked on the assumption that anything
received via a local socket complies to the "usual format" for such messages.
This means no hostname is present.

However, there is a simple setting (parse hostname yes/no) which is part of
the message object. Imuxsock sets it to "no", remote receivers set it to
"yes". It should be fairly trivial to add a config setting which enables
imuxsock to set it to "yes" as well.

> This is a pretty reasonable
> assumption, really--the local UNIX socket doesn't traditionally have
> any way to receive messages forwarded from other hosts.
> 
> Rainer could probably confirm this, or we could compare those two
> modules' sources.
> 
> RSyslog is a fantastic piece of software, and its feature set has come
> a long way. But all of Rainer's excellent work has expanded our
> imaginations about what Syslog can do, and maybe our imaginations are
> advancing faster than he can code new features.

The problem simply is that we are tackeling more complex things, and this
also means things need more planning than in the past. For example, the
latest round of performance enhancements took me 2 to 3 month of reading
scientific papers and experimenting. This all resulted in a relatively small
delta to the code, but this delta does not judge the work that went into it.
Also, being primarily a lone guy, I need to focus on some things and leave
others untouched -- otherwise I'll never make any progress (ever-swapping
between different features). The current scope is configurability and
performance, so other things have to wait (where I try to fit in small and
useful items as it fits). 

What is being discussed here is small and useful ;) However, my summer break
is approaching, and I'd like to finish a couple of loose ends (first config
iteration, getting a new v4-beta ready). Also I (finally ;)) need to write an
article for the upcoming Linux conference in Nuremberg, with the deadline
rapidly approaching. So these are obviously my priority.

Also I have to admit that I weigh features by their importance to the
community at large. Those often requested receive higher priority, so they
will preempt less requested features from being implemented when they have
similar workload. Also, I prefer features requested/need by non-profits over
those that have a clearly for-profit user base. The reasoning here is that
the non-profit orgs will probably otherwise never get their features, whereas
the enterprises can task someone with implementing such a features. And if an
enterprise doesn't think it is worth tasking someone with implementing a
feature, that is a clear indication that the enterprise doesn't actually
think the feature is worth the effort. So I think there is a lot of logic in
assigning priorities in this way ;)

Rainer


From akozumpl at redhat.com  Mon Aug  2 13:11:20 2010
From: akozumpl at redhat.com (Ales Kozumplik)
Date: Mon, 02 Aug 2010 13:11:20 +0200
Subject: [rsyslog] log forwarding through unix sockets
In-Reply-To: <AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com>
References: <4C456476.4010703@redhat.com>
	<AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com>
Message-ID: <4C56A7D8.5020901@redhat.com>

On 07/30/2010 03:28 PM, Ryan Lynch wrote:
> I like your method, too. And thank you for mentioning 'socat', that's
> what gave me the idea to go in this direction, in the first place.

Thanks. To tell you the truth at the end we found a way to forward from 
qemu to a TCP socket, and I am happy I don't have to deal with unix 
sockets any more.

> Based on my own tests, I believe that 'imuxsock' and 'imudp' use
> different logic to parse incoming messages. 'imuxsock' always assumes

That's exactly my feeling about this. I just think one either should be 
able to tell every input module what format should be expected (instead 
of letting rsyslogd try some guessing method), or that every input 
module should by default understand a standard "officially recommended" 
forwarding format.

Ales


From joel.merrick at gmail.com  Mon Aug  2 13:13:35 2010
From: joel.merrick at gmail.com (Joel Merrick)
Date: Mon, 2 Aug 2010 12:13:35 +0100
Subject: [rsyslog] Problem with mysql template
In-Reply-To: <AANLkTimu-Cm-+_BWE+xRCNO6xDVxt2cf38jAcOiFxZBb@mail.gmail.com>
References: <AANLkTimu-Cm-+_BWE+xRCNO6xDVxt2cf38jAcOiFxZBb@mail.gmail.com>
Message-ID: <AANLkTinbjtOgBXxFz35RKX+oohfhg+-fPU8U_CtkLyL5@mail.gmail.com>

Do I take it from the lack of replies that this should work? :)



On Thu, Jul 29, 2010 at 11:53 AM, Joel Merrick <joel.merrick at gmail.com> wrote:
> Hi all,
>
> I'm trying to create a very quick mail log searching solution. The
> idea is to pre-hash the database by using the last 2 alpha-numeric
> characters on the message id.
>
> This isn't inserting to the database and I can't think why.. I've
> already built the tables with the structure logs-aa.. logs-ZZ etc..
>
>
> $template OurDBLog,"INSERT INTO
> logs-'%msg:R,ERE,1,NULL:[A-Za-z0-9]{6}-[A-Za-z0-9]{6}-([A-Za-z0-9]{2})--end%'
> (messageid, host, \
> ? ? send_host, created_at, payload, subject) values \
> ? ? ('%msg:R,ERE,0,ZERO:[A-Za-z0-9]{6}-[A-Za-z0-9]{6}-[A-Za-z0-9]{2}--end%','%HOSTNAME%',
> '%msg:R,ERE,0,ZERO:H=.*\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}]--end%',
> \
> ? ? '%timegenerated:::date-mysql%', '%msg%',
> '%msg:R,ERE,1,ZERO:T="(.+)"--end%')",SQL
>
>
> Any idea?
>
>
> On a side note, how can I get extra verbosity out of rsyslog so I'm
> not blindly trying to insert and then check via mysql
>
> Cheers,
> Joel
>
>
> --
> $ echo "kpfmAdpoofdufevq/dp/vl" | perl -pe 's/(.)/chr(ord($1)-1)/ge'
>



-- 
$ echo "kpfmAdpoofdufevq/dp/vl" | perl -pe 's/(.)/chr(ord($1)-1)/ge'


From rgerhards at hq.adiscon.com  Mon Aug  2 13:14:41 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 2 Aug 2010 13:14:41 +0200
Subject: [rsyslog] log forwarding through unix sockets
References: <4C456476.4010703@redhat.com><AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com>
	<4C56A7D8.5020901@redhat.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104042@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Ales Kozumplik
> Sent: Monday, August 02, 2010 1:11 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] log forwarding through unix sockets
> > Based on my own tests, I believe that 'imuxsock' and 'imudp' use
> > different logic to parse incoming messages. 'imuxsock' always assumes
> 
> That's exactly my feeling about this. 

I elaborated on that yesterday -- not sure if you saw that posting.

> I just think one either should be
> able to tell every input module what format should be expected (instead
> of letting rsyslogd try some guessing method), or that every input
> module should by default understand a standard "officially recommended"
> forwarding format.

The problem is that syslog() does not emit the format recommended by RFC3164,
so no message provided over that channel includes a hostname. Things would be
heavily broken if I would simply ignore that fact.

Rainer


From rgerhards at hq.adiscon.com  Mon Aug  2 13:16:40 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 2 Aug 2010 13:16:40 +0200
Subject: [rsyslog] Problem with mysql template
References: <AANLkTimu-Cm-+_BWE+xRCNO6xDVxt2cf38jAcOiFxZBb@mail.gmail.com>
	<AANLkTinbjtOgBXxFz35RKX+oohfhg+-fPU8U_CtkLyL5@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104043@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Joel Merrick
> Sent: Monday, August 02, 2010 1:14 PM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] Problem with mysql template
> 
> Do I take it from the lack of replies that this should work? :)

simply overlooked from my side ;)

> 
> On Thu, Jul 29, 2010 at 11:53 AM, Joel Merrick <joel.merrick at gmail.com>
> wrote:
> > Hi all,
> >
> > I'm trying to create a very quick mail log searching solution. The
> > idea is to pre-hash the database by using the last 2 alpha-numeric
> > characters on the message id.
> >
> > This isn't inserting to the database and I can't think why.. I've
> > already built the tables with the structure logs-aa.. logs-ZZ etc..
> >
> >
> > $template OurDBLog,"INSERT INTO
> > logs-'%msg:R,ERE,1,NULL:[A-Za-z0-9]{6}-[A-Za-z0-9]{6}-([A-Za-z0-
> 9]{2})--end%'
> > (messageid, host, \
> > ? ? send_host, created_at, payload, subject) values \
> > ? ? ('%msg:R,ERE,0,ZERO:[A-Za-z0-9]{6}-[A-Za-z0-9]{6}-[A-Za-z0-9]{2}-
> -end%','%HOSTNAME%',
> > '%msg:R,ERE,0,ZERO:H=.*\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-
> 9]{1,3}]--end%',
> > \
> > ? ? '%timegenerated:::date-mysql%', '%msg%',
> > '%msg:R,ERE,1,ZERO:T="(.+)"--end%')",SQL
> >
> >
> > Any idea?
> >
> >
> > On a side note, how can I get extra verbosity out of rsyslog so I'm
> > not blindly trying to insert and then check via mysql

Two ways to proceed: 

a) write to a file with the same template
then you can check if the sql is like you intended it to be

b) run rsyslog in debug mode (doc available in doc set and on site)

It may also be worth trying out the regex checker on the rsyslog site.

Rainer


From akozumpl at redhat.com  Mon Aug  2 13:21:14 2010
From: akozumpl at redhat.com (Ales Kozumplik)
Date: Mon, 02 Aug 2010 13:21:14 +0200
Subject: [rsyslog] log forwarding through unix sockets
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710403E@GRFEXC.intern.adiscon.com>
References: <4C456476.4010703@redhat.com>	<AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA710403E@GRFEXC.intern.adiscon.com>
Message-ID: <4C56AA2A.1010709@redhat.com>

On 08/01/2010 11:30 AM, Rainer Gerhards wrote:
> Also I have to admit that I weigh features by their importance to the
> community at large. Those often requested receive higher priority, so they
> will preempt less requested features from being implemented when they have
> similar workload. Also, I prefer features requested/need by non-profits over
> those that have a clearly for-profit user base. The reasoning here is that
> the non-profit orgs will probably otherwise never get their features, whereas
> the enterprises can task someone with implementing such a features. And if an
> enterprise doesn't think it is worth tasking someone with implementing a
> feature, that is a clear indication that the enterprise doesn't actually
> think the feature is worth the effort. So I think there is a lot of logic in
> assigning priorities in this way ;)
>

Hi Rainer,

I'd just add that the feature is targeted for Fedora 14 and the 
community will benefit because improved installer logging allows the 
Fedora QA to work more efficiently and so make a greater Fedora.

Saying that I agree that some obscure unix socket logging features 
shouldn't be of high priority. Just an idea for a backlog.

Ales


From joel.merrick at gmail.com  Mon Aug  2 13:26:05 2010
From: joel.merrick at gmail.com (Joel Merrick)
Date: Mon, 2 Aug 2010 12:26:05 +0100
Subject: [rsyslog] Problem with mysql template
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7104043@GRFEXC.intern.adiscon.com>
References: <AANLkTimu-Cm-+_BWE+xRCNO6xDVxt2cf38jAcOiFxZBb@mail.gmail.com>
	<AANLkTinbjtOgBXxFz35RKX+oohfhg+-fPU8U_CtkLyL5@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7104043@GRFEXC.intern.adiscon.com>
Message-ID: <AANLkTikMf-oe381Ab8eUg4HtWJzbdQOcye0a+sjqns+S@mail.gmail.com>

On Mon, Aug 2, 2010 at 12:16 PM, Rainer Gerhards
<rgerhards at hq.adiscon.com> wrote:
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Joel Merrick
>> Sent: Monday, August 02, 2010 1:14 PM
>> To: rsyslog at lists.adiscon.com
>> Subject: Re: [rsyslog] Problem with mysql template
>>
>> Do I take it from the lack of replies that this should work? :)
>
> simply overlooked from my side ;)
>

No worries, I know you're very busy!


> Two ways to proceed:
>
> a) write to a file with the same template
> then you can check if the sql is like you intended it to be
>

The rest of the SQL is correct, it's when I add the part that adds the
last 2 alphanumeric characters to the table name when it breaks..


> b) run rsyslog in debug mode (doc available in doc set and on site)
>

Will do, cheers for the tips.

> It may also be worth trying out the regex checker on the rsyslog site.


Yes, used this extensively and I'm getting the matches correct.

>
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>



-- 
$ echo "kpfmAdpoofdufevq/dp/vl" | perl -pe 's/(.)/chr(ord($1)-1)/ge'


From rgerhards at hq.adiscon.com  Mon Aug  2 13:26:42 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 2 Aug 2010 13:26:42 +0200
Subject: [rsyslog] log forwarding through unix sockets
References: <4C456476.4010703@redhat.com>	<AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA710403E@GRFEXC.intern.adiscon.com>
	<4C56AA2A.1010709@redhat.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104044@GRFEXC.intern.adiscon.com>

Hi Ales,

> I'd just add that the feature is targeted for Fedora 14 and the
> community will benefit because improved installer logging allows the
> Fedora QA to work more efficiently and so make a greater Fedora.

that's good info! What's the latest date you need to have the feature?

> Saying that I agree that some obscure unix socket logging features
> shouldn't be of high priority. Just an idea for a backlog.

I don't think it is a lot of work, maybe just 2 days. But I am not sure if I
can begin to work on it before end of August. If work on my conference papers
goes well, I may be able to do it before I leave to vacation.

Rainer


From akozumpl at redhat.com  Mon Aug  2 17:03:27 2010
From: akozumpl at redhat.com (Ales Kozumplik)
Date: Mon, 02 Aug 2010 17:03:27 +0200
Subject: [rsyslog] log forwarding through unix sockets
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7104044@GRFEXC.intern.adiscon.com>
References: <4C456476.4010703@redhat.com>	<AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA710403E@GRFEXC.intern.adiscon.com>	<4C56AA2A.1010709@redhat.com>
	<9B6E2A8877C38245BFB15CC491A11DA7104044@GRFEXC.intern.adiscon.com>
Message-ID: <4C56DE3F.6010408@redhat.com>

On 08/02/2010 01:26 PM, Rainer Gerhards wrote:
> I don't think it is a lot of work, maybe just 2 days. But I am not sure if I
> can begin to work on it before end of August. If work on my conference papers
> goes well, I may be able to do it before I leave to vacation.

Thanks, like I said in a reply earlier today, we made qemu forward the 
logs to a TCP port on the host. That way we avoid all of the problems we 
had, so from Anaconda's point of view rsyslog works just fine at the moment.

Ales


From sean at conman.org  Mon Aug  2 22:46:21 2010
From: sean at conman.org (Sean Conner)
Date: Mon, 2 Aug 2010 16:46:21 -0400
Subject: [rsyslog] log forwarding through unix sockets
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7104042@GRFEXC.intern.adiscon.com>
References: <4C56A7D8.5020901@redhat.com>
	<9B6E2A8877C38245BFB15CC491A11DA7104042@GRFEXC.intern.adiscon.com>
Message-ID: <20100802204621.GA7853@brevard.conman.org>

It was thus said that the Great Rainer Gerhards once stated:
> 
> The problem is that syslog() does not emit the format recommended by RFC3164,
> so no message provided over that channel includes a hostname. Things would be
> heavily broken if I would simply ignore that fact.

  Can't it be added as part of the parsing?  Even if it's just "localhost"? 
Or a parameter to the imuxsock module?  My own syslogd fills in missing
fields with sane (at least to me) defaults.

  -spc 



From david at lang.hm  Tue Aug  3 00:03:11 2010
From: david at lang.hm (david at lang.hm)
Date: Mon, 2 Aug 2010 15:03:11 -0700 (PDT)
Subject: [rsyslog] omspoof module
Message-ID: <alpine.DEB.2.00.1008021459190.29820@asgard.lang.hm>

In trying to use the module omspoof with 5.5.6 it looks like there is a 
problem.

the debug file shows a couple errors (entry points not present in module) 
followed by 'tried selector action for omudpspoof: -2001'

then a couple of lines later when defining 
$ActionUDPSpoofSourceNameTemplate it gets the error 'unkown config, did 
you forget to load a module?'

David Lang
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: debug.txt
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20100802/5e4384f9/attachment.txt>

From alexandre.chapellon at mana.pf  Tue Aug  3 05:26:49 2010
From: alexandre.chapellon at mana.pf (Alexandre Chapellon)
Date: Mon, 02 Aug 2010 17:26:49 -1000
Subject: [rsyslog] replaying QIF files
Message-ID: <1280806009.3863.40.camel@mcbain.mana.lan>

Hello I have rsyslog installed on a server to inject data in postgresql
database.
Few days ago SQL injection stopped (because of a syntax error: bad
template)
As I configured disk Queue for the ompgsql action I have all the non
injected messages in thousans of QIF Files.
Here is the cfg:
##########################
$template LogToPqsql, "insert into blablabla' );", sql
$WorkDirectory /var/spool/rsyslog/work
$ActionQueueType LinkedList
$ActionQueueFileName dbq
$ActionResumeRetryCount -1

local1.* :ompgsql:blablablabla
##########################

Unfortunately after restarting rsyslog, it does not reads the files to
inject it back to pg.
Looking in the work dir I can't see file with the .qi extension.
If I manually rename the first QIF file to dbq.qi, rsyslog crash with a
segfault.
Then I tried installing 4.6.3 instead of my old 4.4.2. Rsyslog do not
crash anymore but complains about something wrong with my QIF file (i
guess):

##########################
...
5150.884531712:40800950: deserializer has possibly been able to re-sync
and recover, state 0
5150.884531712:40800950: objDeserializePropBag error -2037 during header
- trying to recover
5150.884531712:40800950: strm 0x689a10: file 10 read 421 bytes
5150.884531712:40800950: strm 0x689a10: file 10 read 0 bytes
5150.884531712:40800950: deserializer has possibly been able to re-sync
and recover, state -2026
5150.884531712:40800950: strm 0x689a10: file
10(/var/spool/rsyslog/work/dbq.qi) closing
5150.884531712:40800950: action 3 queue[DA]: error -2026 reading .qi
file - can not read persisted info (if any)
5150.884531712:40800950: action 3 queue[DA]: persisting queue to disk, 0
entries...
5150.884531712:40800950: action 3 queue: error -2026 creating disk queue
- giving up.
...
##########################

Does anyone have any idea?



From rgerhards at hq.adiscon.com  Tue Aug  3 07:56:57 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 3 Aug 2010 07:56:57 +0200
Subject: [rsyslog] omspoof module
References: <alpine.DEB.2.00.1008021459190.29820@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710404A@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, August 03, 2010 12:03 AM
> To: rsyslog-users
> Subject: [rsyslog] omspoof module
> 
> In trying to use the module omspoof with 5.5.6 it looks like there is a
> problem.
> 
> the debug file shows a couple errors (entry points not present in
> module)

That is perfectly valid. This is just information for me, so that I know
which of the optional entry points are present or not.

> followed by 'tried selector action for omudpspoof: -2001'

This is also valid. rsyslog (bare of a better method) pushes the action part
to the plugin. The plugin either returns OK (if it recognizes the format) or
RS_RET_CONFLINE_UNPROCESSED = -2001 if it does not recognize it. So for each
action, you'll see a couple of them. Thus my guess is that there simply is a
syntax error in the :omudpspoof: part (typo?).

> 
> then a couple of lines later when defining
> $ActionUDPSpoofSourceNameTemplate it gets the error 'unkown config, did
> you forget to load a module?'

Mhhh... that doesn't sound normal. Do you have a longer excerpt of the debug
log for me?

Rainer


From rgerhards at hq.adiscon.com  Tue Aug  3 08:45:41 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 3 Aug 2010 08:45:41 +0200
Subject: [rsyslog] omspoof module
References: <alpine.DEB.2.00.1008021459190.29820@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710404C@GRFEXC.intern.adiscon.com>


> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, August 03, 2010 12:03 AM
> To: rsyslog-users
> Subject: [rsyslog] omspoof module
> 
> In trying to use the module omspoof with 5.5.6 it looks like there is a
> problem.
> 
> the debug file shows a couple errors (entry points not present in
> module)
> followed by 'tried selector action for omudpspoof: -2001'
> 
> then a couple of lines later when defining
> $ActionUDPSpoofSourceNameTemplate it gets the error 'unkown config, did
> you forget to load a module?'


David, I overlooked this as well: it is $ActionOMUDP..., not $ActionUDP.... I
just saw it when I set up my lab... With the current config format (and also
with the new scoping), I can not produce a better error message, because it
actually could be either case...

Rainer


From sledz at dresearch.de  Tue Aug  3 09:12:17 2010
From: sledz at dresearch.de (Steffen Sledz)
Date: Tue, 03 Aug 2010 09:12:17 +0200
Subject: [rsyslog] [PATCH] break potential infinite loop in actionDoRetry
In-Reply-To: <1280494375-11930-1-git-send-email-sledz@dresearch.de>
References: <4C527173.9090805@dresearch.de>
	<1280494375-11930-1-git-send-email-sledz@dresearch.de>
Message-ID: <4C57C151.2020402@dresearch.de>

Am 30.07.2010 14:52, schrieb Steffen Sledz:
> If a module always returns RS_RET_OK (like ompipe does) the actionDoRetry
> loop may not have leaved faked ACT_STATE_SUSP state in case iResumeOKinRow
> had ever reached a count of 1000.

In our test the suggested patch seems to fix this problem. :) But now we hit the next one. :(

If we use a LinkedList DA-Queue for this channel suspended messages may be transmitted out-of-order. This may not be a problem for database consumers. But in our context this is really bad.

Is this a bug or an intended behaviour?

Is it possible to force strict order of messages?

Steffen



From david at lang.hm  Tue Aug  3 09:21:38 2010
From: david at lang.hm (david at lang.hm)
Date: Tue, 3 Aug 2010 00:21:38 -0700 (PDT)
Subject: [rsyslog] [PATCH] break potential infinite loop in actionDoRetry
In-Reply-To: <4C57C151.2020402@dresearch.de>
References: <4C527173.9090805@dresearch.de>
	<1280494375-11930-1-git-send-email-sledz@dresearch.de>
	<4C57C151.2020402@dresearch.de>
Message-ID: <alpine.DEB.2.00.1008030016210.25170@asgard.lang.hm>

On Tue, 3 Aug 2010, Steffen Sledz wrote:

> Am 30.07.2010 14:52, schrieb Steffen Sledz:
>> If a module always returns RS_RET_OK (like ompipe does) the actionDoRetry
>> loop may not have leaved faked ACT_STATE_SUSP state in case iResumeOKinRow
>> had ever reached a count of 1000.
>
> In our test the suggested patch seems to fix this problem. :) But now we hit the next one. :(
>
> If we use a LinkedList DA-Queue for this channel suspended messages may be transmitted out-of-order. This may not be a problem for database consumers. But in our context this is really bad.
>
> Is this a bug or an intended behaviour?

intended behavior,

the issue came up beforeand there are many different ways that messages 
can end up being processed out of order (including the network re-ordering 
UDP packets before they arrive)

part of the huge performance advantage that rsyslog has is that it is able 
to use multiple threads to process messages (both inbound and outbound), 
trying to maintain the order would cripple rsyslog (and still not be good 
enough in many cases)

when I took a class on the Simple Event Correlator (a very good tool for 
log analysis), the instructor had a specific example in the beginning to 
point out that you can't count on logs being processed by you in the order 
they are generated, and this was a couple of years ago before rsyslog 
started becoming so popular. This example was with plain old 
single-threaded syslogd.

David Lang

> Is it possible to force strict order of messages?
>
> Steffen
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From sledz at dresearch.de  Tue Aug  3 10:29:43 2010
From: sledz at dresearch.de (Steffen Sledz)
Date: Tue, 03 Aug 2010 10:29:43 +0200
Subject: [rsyslog] [PATCH] break potential infinite loop in actionDoRetry
In-Reply-To: <alpine.DEB.2.00.1008030016210.25170@asgard.lang.hm>
References: <4C527173.9090805@dresearch.de>	<1280494375-11930-1-git-send-email-sledz@dresearch.de>	<4C57C151.2020402@dresearch.de>
	<alpine.DEB.2.00.1008030016210.25170@asgard.lang.hm>
Message-ID: <4C57D377.1020107@dresearch.de>

Am 03.08.2010 09:21, schrieb david at lang.hm:
>> If we use a LinkedList DA-Queue for this channel suspended
>> messages may be transmitted out-of-order. This may not be
>> a problem for database consumers. But in our context this
>> is really bad.
>>
>> Is this a bug or an intended behaviour?
> 
> intended behavior,
> 
> the issue came up beforeand there are many different ways that messages 
> can end up being processed out of order (including the network re-ordering 
> UDP packets before they arrive)
> 
> part of the huge performance advantage that rsyslog has is that it is able 
> to use multiple threads to process messages (both inbound and outbound), 
> trying to maintain the order would cripple rsyslog (and still not be good 
> enough in many cases)
> 
> when I took a class on the Simple Event Correlator (a very good tool for 
> log analysis), the instructor had a specific example in the beginning to 
> point out that you can't count on logs being processed by you in the order 
> they are generated, and this was a couple of years ago before rsyslog 
> started becoming so popular. This example was with plain old 
> single-threaded syslogd.

In my opinion this may occur if messages came from different/concurrent sources. But it should be possible to keep messages from the same source in strict order (with performance loss).

So again my question. Is it possible to force strict order of messages?

Steffen



From rgerhards at hq.adiscon.com  Tue Aug  3 10:42:58 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 3 Aug 2010 10:42:58 +0200
Subject: [rsyslog] [PATCH] break potential infinite loop in actionDoRetry
References: <4C527173.9090805@dresearch.de>	<1280494375-11930-1-git-send-email-sledz@dresearch.de>	<4C57C151.2020402@dresearch.de><alpine.DEB.2.00.1008030016210.25170@asgard.lang.hm>
	<4C57D377.1020107@dresearch.de>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104054@GRFEXC.intern.adiscon.com>


> In my opinion this may occur if messages came from different/concurrent
> sources. 

Depends on your environment. If you have a single UDP listener, reordering
may occur on the wire before you even receive it. Sequence as seen on the
input/API level is not sequence as the message may be originated. Not even so
for local unix sockets, this is depending on scheduling order, which you
cannot control in userland. So I can back David's point: it is dangerous to
assume that a log stream has strict sequence. This may be the case for an
single hop (obviously for messages within a single TCP stream), but not
further than that. In general, you need to define an order relation. 

As a side note, a not-so-small part of the conference paper I am writing
right now is about these wrong perception of log sequence that users commonly
have. I think I can post it myself after the conference (this is the end of
september timeframe).

> But it should be possible to keep messages from the same
> source in strict order (with performance loss).


> 
> So again my question. Is it possible to force strict order of messages?


I think you can come close to what you want:

You need to dig into the queue docs. Set everything to single worker thread
(maybe even use direct mode). If you use a recent build, set all batch sizes
to one (this is conservative, in theory it should work with larger batches as
well, iff everything is single-threaded).

Note that when you use a disk assisted queue, the queue engine always puts
some messages out of order when switching to/from Disk mode. Newer builds
also consume messages from memory and disk concurrently and thus potentially
out of queue order. So you need to avoid disk assisted mode. If you need disk
functionality, use pure disk mode, only.

Make sure you do not define more than one input per rsyslog instance. Else OS
scheduling order for rsyslog threads (input vs. output) potentially gets you
into trouble. Make sure you do not define any async output actions. Do not
write to the same file (or action in general) from more than one action or
more than one instance of rsyslog.

That should give you a somewhat strict sequence of messages. Depending on the
use case/volume, you'll see no to unacceptable performance loss.

Rainer


From sledz at dresearch.de  Tue Aug  3 11:06:23 2010
From: sledz at dresearch.de (Steffen Sledz)
Date: Tue, 03 Aug 2010 11:06:23 +0200
Subject: [rsyslog] [PATCH] break potential infinite loop in actionDoRetry
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7104054@GRFEXC.intern.adiscon.com>
References: <4C527173.9090805@dresearch.de>	<1280494375-11930-1-git-send-email-sledz@dresearch.de>	<4C57C151.2020402@dresearch.de><alpine.DEB.2.00.1008030016210.25170@asgard.lang.hm>	<4C57D377.1020107@dresearch.de>
	<9B6E2A8877C38245BFB15CC491A11DA7104054@GRFEXC.intern.adiscon.com>
Message-ID: <4C57DC0F.8070404@dresearch.de>

Am 03.08.2010 10:42, schrieb Rainer Gerhards:
> Depends on your environment. If you have a single UDP
> listener, reordering may occur on the wire before you
> even receive it....

As you can guess in our context UDP is not involved, just the fifo.

> Note that when you use a disk assisted queue, the queue
> engine always puts some messages out of order when switching
> to/from Disk mode. Newer builds also consume messages from
> memory and disk concurrently and thus potentially out of
> queue order.

I think that's what we see.

So i believe we have to decide between out-of-order messages or loss of some messages when we do not use a disk assisted queue. :(

Steffen



From rgerhards at hq.adiscon.com  Tue Aug  3 11:08:18 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 3 Aug 2010 11:08:18 +0200
Subject: [rsyslog] [PATCH] break potential infinite loop in actionDoRetry
References: <4C527173.9090805@dresearch.de>	<1280494375-11930-1-git-send-email-sledz@dresearch.de>	<4C57C151.2020402@dresearch.de><alpine.DEB.2.00.1008030016210.25170@asgard.lang.hm>	<4C57D377.1020107@dresearch.de><9B6E2A8877C38245BFB15CC491A11DA7104054@GRFEXC.intern.adiscon.com>
	<4C57DC0F.8070404@dresearch.de>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104055@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Steffen Sledz
> Sent: Tuesday, August 03, 2010 11:06 AM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] [PATCH] break potential infinite loop in
> actionDoRetry
> 
> Am 03.08.2010 10:42, schrieb Rainer Gerhards:
> > Depends on your environment. If you have a single UDP
> > listener, reordering may occur on the wire before you
> > even receive it....
> 
> As you can guess in our context UDP is not involved, just the fifo.

The you have the issue of context switching between producers (assuming there
is more than one).

> 
> > Note that when you use a disk assisted queue, the queue
> > engine always puts some messages out of order when switching
> > to/from Disk mode. Newer builds also consume messages from
> > memory and disk concurrently and thus potentially out of
> > queue order.
> 
> I think that's what we see.
> 
> So i believe we have to decide between out-of-order messages or loss of
> some messages when we do not use a disk assisted queue. :(

Or use a disk-only queue. It's performance is obviously very bad, but may be
sufficient depending on volume.

Rainer


From rgerhards at hq.adiscon.com  Thu Aug  5 07:53:36 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 5 Aug 2010 07:53:36 +0200
Subject: [rsyslog] [PATCH] break potential infinite loop in actionDoRetry
References: <4C527173.9090805@dresearch.de>
	<1280494375-11930-1-git-send-email-sledz@dresearch.de>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710405D@GRFEXC.intern.adiscon.com>

I just wanted to let you know that I have integrated the patch. Many thanks
for your effort!

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Steffen Sledz
> Sent: Friday, July 30, 2010 2:53 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] [PATCH] break potential infinite loop in
> actionDoRetry
> 
> If a module always returns RS_RET_OK (like ompipe does) the
> actionDoRetry
> loop may not have leaved faked ACT_STATE_SUSP state in case
> iResumeOKinRow
> had ever reached a count of 1000.
> 
> Signed-off-by: Steffen Sledz <sledz at dresearch.de>
> ---
>  action.c |    1 +
>  1 files changed, 1 insertions(+), 0 deletions(-)
> 
> diff --git a/action.c b/action.c
> index 32a07dc..90ec1bf 100644
> --- a/action.c
> +++ b/action.c
> @@ -508,6 +508,7 @@ static rsRetVal actionDoRetry(action_t *pThis,
> time_t ttNow)
>  		iRet = pThis->pMod->tryResume(pThis->pModData);
>  		if((pThis->iResumeOKinRow > 999) && (pThis->iResumeOKinRow
> % 1000 == 0)) {
>  			bTreatOKasSusp = 1;
> +			pThis->iResumeOKinRow = 0;
>  		} else {
>  			bTreatOKasSusp = 0;
>  		}
> --
> 1.6.4.2
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From tbergfeld at hq.adiscon.com  Thu Aug  5 14:18:52 2010
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Thu, 5 Aug 2010 14:18:52 +0200
Subject: [rsyslog] rsyslog 3.22.2 (v3-stable) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104067@GRFEXC.intern.adiscon.com>

Hi all,

We have just released rsyslog 3.22.2, a member of the v3-stable branch.

This is a stability update, integrating bug fixes that have become available
during the past month. No new functionality is included.

See ChangeLog for more details.

ChangeLog:

http://www.rsyslog.com/changelog-for-3-2-2-v3-stable/

Download:

http://www.rsyslog.com/rsyslog-3-22-2-v3-stable/

As always, feedback is appreciated.

Best regards,
Tom Bergfeld
--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .


From tbergfeld at hq.adiscon.com  Thu Aug  5 14:19:49 2010
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Thu, 5 Aug 2010 14:19:49 +0200
Subject: [rsyslog] rsyslog 4.6.4 (v4-stable) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104068@GRFEXC.intern.adiscon.com>

Hi all,

We have just released rsyslog 4.6.4, a member of the v4-stable branch.

This is a stability update without new features.

See ChangeLog for more details.

ChangeLog:

http://www.rsyslog.com/changelog-for-4-6-4-v4-stable/

Download:

http://www.rsyslog.com/rsyslog-4-6-4-v4-stable/

As always, feedback is appreciated.

Best regards,
Tom Bergfeld
--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .


From rgerhards at hq.adiscon.com  Thu Aug  5 15:45:04 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 5 Aug 2010 15:45:04 +0200
Subject: [rsyslog] spoofing module configuration
References: <alpine.DEB.2.00.1007091435070.14221@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710406C@GRFEXC.intern.adiscon.com>

David,

I have now changed the default:

http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=e4c926bfbca3aab3fe34fc9c
fedb7343423de016

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Friday, July 09, 2010 11:43 PM
> To: rsyslog-users
> Subject: [rsyslog] spoofing module configuration
> 
> in reading the spoofing module configuration it strikes me that the
> defaults can be significantly improved.
> 
> the common case for needing to so spoofing is that you are spoofing the
> original source IP address
> 
> so the current configuration equivalent commands
> 
> $template spoofaddr, "%fromhost-ip%"
> $ActionUDPSpoofSourceNameTemplate spoofaddr
> 
> could be made the default (or call it RSYSLOG_spoofaddr to keep from
> polluting the namespace) and the result would be far simpler for people
> to
> configure, becomging simply
> 
> $modload omudpspoof
> $ActionUDPSpoofTargetHost server.example.com
> *.* :omudpspoof:
> 
> it could be simplified even further if there was some way to specify
> the
> destination on the action line (like the @ and @@  functions do today,
> could we use @S@ to indicate spoofing?)
> 
> 
> changing the defaults should have no problems with backwards
> compatibility, adding/changing how the desitnation is specified could
> break backward compatibility, but this is a very new piece of
> functionality and the simplification may be worth it (what versions
> have
> this available?)
> 
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Thu Aug  5 17:32:17 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 5 Aug 2010 17:32:17 +0200
Subject: [rsyslog] log forwarding through unix sockets
References: <4C456476.4010703@redhat.com><AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com>
	<4C56A7D8.5020901@redhat.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710406E@GRFEXC.intern.adiscon.com>

Just some more information for anyone interested. I today looked at the
module. First of all, there is a directive to set the hostname
($InputUnixListenSocketHostName) but I also found out that there exist
undocumented functionality to activate hostname parsing. In theory, this is
done by putting a colon in front of the socket name. In practice, there seems
to be a bug that prevents this from working at all (the colon is not
removed). So it probably was good this was not documented ;)

I'll see that I fix that first, so that we have some basic functionality in
place.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Ales Kozumplik
> Sent: Monday, August 02, 2010 1:11 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] log forwarding through unix sockets
> 
> On 07/30/2010 03:28 PM, Ryan Lynch wrote:
> > I like your method, too. And thank you for mentioning 'socat', that's
> > what gave me the idea to go in this direction, in the first place.
> 
> Thanks. To tell you the truth at the end we found a way to forward from
> qemu to a TCP socket, and I am happy I don't have to deal with unix
> sockets any more.
> 
> > Based on my own tests, I believe that 'imuxsock' and 'imudp' use
> > different logic to parse incoming messages. 'imuxsock' always assumes
> 
> That's exactly my feeling about this. I just think one either should be
> able to tell every input module what format should be expected (instead
> of letting rsyslogd try some guessing method), or that every input
> module should by default understand a standard "officially recommended"
> forwarding format.
> 
> Ales
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From david at lang.hm  Thu Aug  5 17:50:12 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 5 Aug 2010 08:50:12 -0700 (PDT)
Subject: [rsyslog] log forwarding through unix sockets
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710406E@GRFEXC.intern.adiscon.com>
References: <4C456476.4010703@redhat.com><AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com>
	<4C56A7D8.5020901@redhat.com>
	<9B6E2A8877C38245BFB15CC491A11DA710406E@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.2.00.1008050849340.25170@asgard.lang.hm>

On Thu, 5 Aug 2010, Rainer Gerhards wrote:

> Just some more information for anyone interested. I today looked at the
> module. First of all, there is a directive to set the hostname
> ($InputUnixListenSocketHostName) but I also found out that there exist
> undocumented functionality to activate hostname parsing. In theory, this is
> done by putting a colon in front of the socket name. In practice, there seems
> to be a bug that prevents this from working at all (the colon is not
> removed). So it probably was good this was not documented ;)

so if you use a socket name that starts with a colon it should work right? 
;-)

David Lang

> I'll see that I fix that first, so that we have some basic functionality in
> place.
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Ales Kozumplik
>> Sent: Monday, August 02, 2010 1:11 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] log forwarding through unix sockets
>>
>> On 07/30/2010 03:28 PM, Ryan Lynch wrote:
>>> I like your method, too. And thank you for mentioning 'socat', that's
>>> what gave me the idea to go in this direction, in the first place.
>>
>> Thanks. To tell you the truth at the end we found a way to forward from
>> qemu to a TCP socket, and I am happy I don't have to deal with unix
>> sockets any more.
>>
>>> Based on my own tests, I believe that 'imuxsock' and 'imudp' use
>>> different logic to parse incoming messages. 'imuxsock' always assumes
>>
>> That's exactly my feeling about this. I just think one either should be
>> able to tell every input module what format should be expected (instead
>> of letting rsyslogd try some guessing method), or that every input
>> module should by default understand a standard "officially recommended"
>> forwarding format.
>>
>> Ales
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Thu Aug  5 17:51:41 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 5 Aug 2010 17:51:41 +0200
Subject: [rsyslog] log forwarding through unix sockets
References: <4C456476.4010703@redhat.com><AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com><4C56A7D8.5020901@redhat.com><9B6E2A8877C38245BFB15CC491A11DA710406E@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.00.1008050849340.25170@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104070@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Thursday, August 05, 2010 5:50 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] log forwarding through unix sockets
> 
> On Thu, 5 Aug 2010, Rainer Gerhards wrote:
> 
> > Just some more information for anyone interested. I today looked at
> the
> > module. First of all, there is a directive to set the hostname
> > ($InputUnixListenSocketHostName) but I also found out that there
> exist
> > undocumented functionality to activate hostname parsing. In theory,
> this is
> > done by putting a colon in front of the socket name. In practice,
> there seems
> > to be a bug that prevents this from working at all (the colon is not
> > removed). So it probably was good this was not documented ;)
> 
> so if you use a socket name that starts with a colon it should work
> right?
> ;-)

indeed... ;) [but I've not yet tested it, right now thinking about creating
an automated test...]

Rainer


From rgerhards at hq.adiscon.com  Fri Aug  6 17:53:04 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 6 Aug 2010 17:53:04 +0200
Subject: [rsyslog] log forwarding through unix sockets
References: <4C456476.4010703@redhat.com><AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com><4C56A7D8.5020901@redhat.com><9B6E2A8877C38245BFB15CC491A11DA710406E@GRFEXC.intern.adiscon.com><alpine.DEB.2.00.1008050849340.25170@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7104070@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710407E@GRFEXC.intern.adiscon.com>

As a first step (and because I needed something to do decent testing ;)) I
have implemented a new module omuxsock which now can natively write to unix
domain sockets. That should close part of the gap that was discussed here.

I hope I will find time to look at imuxsock and the hostname parsing code
next week (but that will be *very* busy).

Doc:
http://www.rsyslog.com/doc/omuxsock.html

v4 Commit (v5 also already merged):
http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=80ff634c841d692c1d9f335b
88e225d6ce7317f7

I plan to remove this plugin next week with the upcoming v5 beta.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Thursday, August 05, 2010 5:52 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] log forwarding through unix sockets
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> > Sent: Thursday, August 05, 2010 5:50 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] log forwarding through unix sockets
> >
> > On Thu, 5 Aug 2010, Rainer Gerhards wrote:
> >
> > > Just some more information for anyone interested. I today looked at
> > the
> > > module. First of all, there is a directive to set the hostname
> > > ($InputUnixListenSocketHostName) but I also found out that there
> > exist
> > > undocumented functionality to activate hostname parsing. In theory,
> > this is
> > > done by putting a colon in front of the socket name. In practice,
> > there seems
> > > to be a bug that prevents this from working at all (the colon is
> not
> > > removed). So it probably was good this was not documented ;)
> >
> > so if you use a socket name that starts with a colon it should work
> > right?
> > ;-)
> 
> indeed... ;) [but I've not yet tested it, right now thinking about
> creating
> an automated test...]
> 
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Fri Aug  6 17:55:10 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 6 Aug 2010 17:55:10 +0200
Subject: [rsyslog] log forwarding through unix sockets
References: <4C456476.4010703@redhat.com><AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com><4C56A7D8.5020901@redhat.com><9B6E2A8877C38245BFB15CC491A11DA710406E@GRFEXC.intern.adiscon.com><alpine.DEB.2.00.1008050849340.25170@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA7104070@GRFEXC.intern.adiscon.com>
	<9B6E2A8877C38245BFB15CC491A11DA710407E@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710407F@GRFEXC.intern.adiscon.com>

> I plan to remove this plugin next week with the upcoming v5 beta.

lol: "release", of course, NOT "remove" ;)

Rainer


From shdashbeta at gmail.com  Sat Aug  7 01:39:02 2010
From: shdashbeta at gmail.com (S H)
Date: Fri, 6 Aug 2010 19:39:02 -0400
Subject: [rsyslog] Tuning for performance
Message-ID: <AANLkTimb_hVcG8DvFkC0cmOSP3w23X-g=s-nON56a+ge@mail.gmail.com>

Hello,

I'm newish to the world of rsyslog. I've used it for regular
syslogging stuff with dynamically generated filenames and the like,
but I've never gone in depth with its configuration. Now, however, I'm
working on a project that will involve very high message rates and am
trying to figure out how to tune the system for the kind of throughput
(>200k/sec) documented at
http://mperedim.wordpress.com/2010/01/21/rsyslog-evaluation/

I have a pair of hardware test servers outfitted with dual quad-core
Xeon processors and 8GB of RAM. The network connection is only
100Mbit, but that doesn't seem to be my bottleneck yet. I'm using
loggen for my tests. The servers are running rsyslog 5.5.6.

serverB is the one listening for connections. It's using the
configuration pasted below.

serverA is the one running the test:
# loggen --verbose -r 20000 -I 10 -s 512 -S 207.150.202.100 10514
average rate = 5663.95 msg/sec, count=56644, time=10.007, (last) msg
size=512, bandwidth=2831.98 kB/sec

Increasing the rate (-r) doesn't change the average rate. Switching to
UDP or performing the test on localhost yield very similar results.
What's really strange is that I left the servers alone for about an
hour to work on another project and when I came back the rates were
roughly double - 10-13k/sec. As I tested, however, they gradually fell
back to the 5-6k levels you see here. Restarting the rsyslog process
makes no difference. So I've tuned something incorrectly but I have no
idea what.

iperf shows 100Mbit between the servers. I can double or halve the
message size without affecting the rate, so actual message rate is the
problem -- not bandwidth.

Any help would be wonderful.

-SH

# rsyslog.conf:

$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022

# different rulesets even though I haven't seen any performance effects
$RuleSet remote10514
$RulesetCreateMainQueue on # create ruleset-specific queue
$MainMsgQueueSize 100000
$MainMsgQueueDequeueBatchSize 1024
$RepeatedMsgReduction off

*.* /dev/null
& ~

$ModLoad imtcp
$InputTCPServerBindRuleset remote10514
$InputTCPServerRun 10514


$RuleSet RSYSLOG_DefaultRuleset
$RepeatedMsgReduction off

*.* /var/log/test.log


From rgerhards at hq.adiscon.com  Sat Aug  7 10:14:18 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Sat, 7 Aug 2010 10:14:18 +0200
Subject: [rsyslog] Tuning for performance
References: <AANLkTimb_hVcG8DvFkC0cmOSP3w23X-g=s-nON56a+ge@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104080@GRFEXC.intern.adiscon.com>

This sounds very strange, even the early v4 version could work at higher
rates. Do you use 512 byte messages, only? Could you start with a very basic
rsyslog.conf to get a baseline? Just loading imtcp, starting the listener,
and a single 

*.* /path/to/some/file

Config.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of S H
> Sent: Saturday, August 07, 2010 1:39 AM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] Tuning for performance
> 
> Hello,
> 
> I'm newish to the world of rsyslog. I've used it for regular
> syslogging stuff with dynamically generated filenames and the like,
> but I've never gone in depth with its configuration. Now, however, I'm
> working on a project that will involve very high message rates and am
> trying to figure out how to tune the system for the kind of throughput
> (>200k/sec) documented at
> http://mperedim.wordpress.com/2010/01/21/rsyslog-evaluation/
> 
> I have a pair of hardware test servers outfitted with dual quad-core
> Xeon processors and 8GB of RAM. The network connection is only
> 100Mbit, but that doesn't seem to be my bottleneck yet. I'm using
> loggen for my tests. The servers are running rsyslog 5.5.6.
> 
> serverB is the one listening for connections. It's using the
> configuration pasted below.
> 
> serverA is the one running the test:
> # loggen --verbose -r 20000 -I 10 -s 512 -S 207.150.202.100 10514
> average rate = 5663.95 msg/sec, count=56644, time=10.007, (last) msg
> size=512, bandwidth=2831.98 kB/sec
> 
> Increasing the rate (-r) doesn't change the average rate. Switching to
> UDP or performing the test on localhost yield very similar results.
> What's really strange is that I left the servers alone for about an
> hour to work on another project and when I came back the rates were
> roughly double - 10-13k/sec. As I tested, however, they gradually fell
> back to the 5-6k levels you see here. Restarting the rsyslog process
> makes no difference. So I've tuned something incorrectly but I have no
> idea what.
> 
> iperf shows 100Mbit between the servers. I can double or halve the
> message size without affecting the rate, so actual message rate is the
> problem -- not bandwidth.
> 
> Any help would be wonderful.
> 
> -SH
> 
> # rsyslog.conf:
> 
> $FileOwner syslog
> $FileGroup adm
> $FileCreateMode 0640
> $DirCreateMode 0755
> $Umask 0022
> 
> # different rulesets even though I haven't seen any performance effects
> $RuleSet remote10514
> $RulesetCreateMainQueue on # create ruleset-specific queue
> $MainMsgQueueSize 100000
> $MainMsgQueueDequeueBatchSize 1024
> $RepeatedMsgReduction off
> 
> *.* /dev/null
> & ~
> 
> $ModLoad imtcp
> $InputTCPServerBindRuleset remote10514
> $InputTCPServerRun 10514
> 
> 
> $RuleSet RSYSLOG_DefaultRuleset
> $RepeatedMsgReduction off
> 
> *.* /var/log/test.log
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From shdashbeta at gmail.com  Sat Aug  7 17:10:46 2010
From: shdashbeta at gmail.com (S H)
Date: Sat, 7 Aug 2010 11:10:46 -0400
Subject: [rsyslog] Tuning for performance
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7104080@GRFEXC.intern.adiscon.com>
References: <AANLkTimb_hVcG8DvFkC0cmOSP3w23X-g=s-nON56a+ge@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7104080@GRFEXC.intern.adiscon.com>
Message-ID: <AANLkTikhRY6iEowuRJ0hmMUX28-20y++Vd7tYSXHQU2j@mail.gmail.com>

I think the problem is with loggen, or at least the version I'm using.
I wrote a simple little utility in Python to exercise the native
syslog() calls and was seeing about 240k/second with a simple
configuration.

Once I've got more detailed benchmarks, I'll post them to this thread.
Thanks for the help.

-SH

On Sat, Aug 7, 2010 at 4:14 AM, Rainer Gerhards
<rgerhards at hq.adiscon.com> wrote:
> This sounds very strange, even the early v4 version could work at higher
> rates. Do you use 512 byte messages, only? Could you start with a very basic
> rsyslog.conf to get a baseline? Just loading imtcp, starting the listener,
> and a single
>
> *.* /path/to/some/file
>
> Config.
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of S H
>> Sent: Saturday, August 07, 2010 1:39 AM
>> To: rsyslog at lists.adiscon.com
>> Subject: [rsyslog] Tuning for performance
>>
>> Hello,
>>
>> I'm newish to the world of rsyslog. I've used it for regular
>> syslogging stuff with dynamically generated filenames and the like,
>> but I've never gone in depth with its configuration. Now, however, I'm
>> working on a project that will involve very high message rates and am
>> trying to figure out how to tune the system for the kind of throughput
>> (>200k/sec) documented at
>> http://mperedim.wordpress.com/2010/01/21/rsyslog-evaluation/
>>
>> I have a pair of hardware test servers outfitted with dual quad-core
>> Xeon processors and 8GB of RAM. The network connection is only
>> 100Mbit, but that doesn't seem to be my bottleneck yet. I'm using
>> loggen for my tests. The servers are running rsyslog 5.5.6.
>>
>> serverB is the one listening for connections. It's using the
>> configuration pasted below.
>>
>> serverA is the one running the test:
>> # loggen --verbose -r 20000 -I 10 -s 512 -S 207.150.202.100 10514
>> average rate = 5663.95 msg/sec, count=56644, time=10.007, (last) msg
>> size=512, bandwidth=2831.98 kB/sec
>>
>> Increasing the rate (-r) doesn't change the average rate. Switching to
>> UDP or performing the test on localhost yield very similar results.
>> What's really strange is that I left the servers alone for about an
>> hour to work on another project and when I came back the rates were
>> roughly double - 10-13k/sec. As I tested, however, they gradually fell
>> back to the 5-6k levels you see here. Restarting the rsyslog process
>> makes no difference. So I've tuned something incorrectly but I have no
>> idea what.
>>
>> iperf shows 100Mbit between the servers. I can double or halve the
>> message size without affecting the rate, so actual message rate is the
>> problem -- not bandwidth.
>>
>> Any help would be wonderful.
>>
>> -SH
>>
>> # rsyslog.conf:
>>
>> $FileOwner syslog
>> $FileGroup adm
>> $FileCreateMode 0640
>> $DirCreateMode 0755
>> $Umask 0022
>>
>> # different rulesets even though I haven't seen any performance effects
>> $RuleSet remote10514
>> $RulesetCreateMainQueue on # create ruleset-specific queue
>> $MainMsgQueueSize 100000
>> $MainMsgQueueDequeueBatchSize 1024
>> $RepeatedMsgReduction off
>>
>> *.* /dev/null
>> & ~
>>
>> $ModLoad imtcp
>> $InputTCPServerBindRuleset remote10514
>> $InputTCPServerRun 10514
>>
>>
>> $RuleSet RSYSLOG_DefaultRuleset
>> $RepeatedMsgReduction off
>>
>> *.* /var/log/test.log
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From david at lang.hm  Sun Aug  8 02:46:11 2010
From: david at lang.hm (david at lang.hm)
Date: Sat, 7 Aug 2010 17:46:11 -0700 (PDT)
Subject: [rsyslog] Tuning for performance
In-Reply-To: <AANLkTikhRY6iEowuRJ0hmMUX28-20y++Vd7tYSXHQU2j@mail.gmail.com>
References: <AANLkTimb_hVcG8DvFkC0cmOSP3w23X-g=s-nON56a+ge@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7104080@GRFEXC.intern.adiscon.com>
	<AANLkTikhRY6iEowuRJ0hmMUX28-20y++Vd7tYSXHQU2j@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1008071743560.22036@asgard.lang.hm>

I've also had problems generating enough traffic to stress rsyslog, what I 
ended up doing is using tcpreplay. I send out sample data, capture it with 
tcpdump, and then use tcpreplay to replay the packets over the network at 
different speeds.

this only works well for UDP packets (as tcpreplay does not maintain all 
the state needed to do a real tcp connection.)

David Lang

  On Sat, 7 Aug 2010, S H wrote:

> Date: Sat, 7 Aug 2010 11:10:46 -0400
> From: S H <shdashbeta at gmail.com>
> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> To: rsyslog-users <rsyslog at lists.adiscon.com>
> Subject: Re: [rsyslog] Tuning for performance
> 
> I think the problem is with loggen, or at least the version I'm using.
> I wrote a simple little utility in Python to exercise the native
> syslog() calls and was seeing about 240k/second with a simple
> configuration.
>
> Once I've got more detailed benchmarks, I'll post them to this thread.
> Thanks for the help.
>
> -SH
>
> On Sat, Aug 7, 2010 at 4:14 AM, Rainer Gerhards
> <rgerhards at hq.adiscon.com> wrote:
>> This sounds very strange, even the early v4 version could work at higher
>> rates. Do you use 512 byte messages, only? Could you start with a very basic
>> rsyslog.conf to get a baseline? Just loading imtcp, starting the listener,
>> and a single
>>
>> *.* /path/to/some/file
>>
>> Config.
>>
>> Rainer
>>
>>> -----Original Message-----
>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>> bounces at lists.adiscon.com] On Behalf Of S H
>>> Sent: Saturday, August 07, 2010 1:39 AM
>>> To: rsyslog at lists.adiscon.com
>>> Subject: [rsyslog] Tuning for performance
>>>
>>> Hello,
>>>
>>> I'm newish to the world of rsyslog. I've used it for regular
>>> syslogging stuff with dynamically generated filenames and the like,
>>> but I've never gone in depth with its configuration. Now, however, I'm
>>> working on a project that will involve very high message rates and am
>>> trying to figure out how to tune the system for the kind of throughput
>>> (>200k/sec) documented at
>>> http://mperedim.wordpress.com/2010/01/21/rsyslog-evaluation/
>>>
>>> I have a pair of hardware test servers outfitted with dual quad-core
>>> Xeon processors and 8GB of RAM. The network connection is only
>>> 100Mbit, but that doesn't seem to be my bottleneck yet. I'm using
>>> loggen for my tests. The servers are running rsyslog 5.5.6.
>>>
>>> serverB is the one listening for connections. It's using the
>>> configuration pasted below.
>>>
>>> serverA is the one running the test:
>>> # loggen --verbose -r 20000 -I 10 -s 512 -S 207.150.202.100 10514
>>> average rate = 5663.95 msg/sec, count=56644, time=10.007, (last) msg
>>> size=512, bandwidth=2831.98 kB/sec
>>>
>>> Increasing the rate (-r) doesn't change the average rate. Switching to
>>> UDP or performing the test on localhost yield very similar results.
>>> What's really strange is that I left the servers alone for about an
>>> hour to work on another project and when I came back the rates were
>>> roughly double - 10-13k/sec. As I tested, however, they gradually fell
>>> back to the 5-6k levels you see here. Restarting the rsyslog process
>>> makes no difference. So I've tuned something incorrectly but I have no
>>> idea what.
>>>
>>> iperf shows 100Mbit between the servers. I can double or halve the
>>> message size without affecting the rate, so actual message rate is the
>>> problem -- not bandwidth.
>>>
>>> Any help would be wonderful.
>>>
>>> -SH
>>>
>>> # rsyslog.conf:
>>>
>>> $FileOwner syslog
>>> $FileGroup adm
>>> $FileCreateMode 0640
>>> $DirCreateMode 0755
>>> $Umask 0022
>>>
>>> # different rulesets even though I haven't seen any performance effects
>>> $RuleSet remote10514
>>> $RulesetCreateMainQueue on # create ruleset-specific queue
>>> $MainMsgQueueSize 100000
>>> $MainMsgQueueDequeueBatchSize 1024
>>> $RepeatedMsgReduction off
>>>
>>> *.* /dev/null
>>> & ~
>>>
>>> $ModLoad imtcp
>>> $InputTCPServerBindRuleset remote10514
>>> $InputTCPServerRun 10514
>>>
>>>
>>> $RuleSet RSYSLOG_DefaultRuleset
>>> $RepeatedMsgReduction off
>>>
>>> *.* /var/log/test.log
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From tbergfeld at hq.adiscon.com  Mon Aug  9 14:47:50 2010
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Mon, 9 Aug 2010 14:47:50 +0200
Subject: [rsyslog] rsyslog 5.5.7 (v5-beta) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104091@GRFEXC.intern.adiscon.com>

Hi all,

We have just released rsyslog 5.5.7, the new v5-beta.
This is a new beta version, scheduled to replace the current stable hopefully
soon. It has some bugfixes over 5.5.6, and also introduces a new module,
omuxsock, that provides the ability to natively write to Unix domain sockets.
Feedback on this version is highly appreciated. We would like to promote it
to stable by mid to end September if we do not hear anything that prevents us
from doing so.
See ChangeLog for more details.

ChangeLog:

http://www.rsyslog.com/changelog-for-5-5-7-v5-beta/

Download:

http://www.rsyslog.com/rsyslog-5-5-7-v5-beta/

As always, feedback is appreciated.

Best regards,
Tom Bergfeld
--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .


From joe.smithian at gmail.com  Tue Aug 10 19:19:00 2010
From: joe.smithian at gmail.com (Joe Smithian)
Date: Tue, 10 Aug 2010 13:19:00 -0400
Subject: [rsyslog] rsyslog truncates host name in the log messages if host
	name contains dots
In-Reply-To: <mailman.434.1281448570.2883.rsyslog@lists.adiscon.com>
References: <mailman.434.1281448570.2883.rsyslog@lists.adiscon.com>
Message-ID: <AANLkTi=oduvpO4P1jY9kb9FGRw5SQYXfZy1HP4-4eGd_@mail.gmail.com>

Hello,

If Linux hostname contains dots "." e.g. an IP address : 192.168.1.1 then
rsyslog prints "192" as hostname in the log messages:

$template MySyslFormat2,"%$YEAR% %timegenerated% *%HOSTNAME%*
%syslogfacility-text% %syslogseverity-text%  %syslogtag%%msg%\n"


2010 Aug 10 09:36:02 *192* kern info  kernel: Kernel logging (proc) stopped.


Is there any way to change the rsyslog behavior to show the full hostname if
it contains dots "."?


Thanks

Joe


From rgerhards at hq.adiscon.com  Wed Aug 11 12:01:02 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 11 Aug 2010 12:01:02 +0200
Subject: [rsyslog] help requested -- connect() performance
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD1F1@GRFEXC.intern.adiscon.com>

Hi all,

I yesterday stumbled over behavior that I cannot explain. Mabye someone here
on the list has a hint.

There is a testing tool "tcpflood", which I use to carry out the tcp based
tests inside the automated testbench. That tool has evolved over time and has
many options. One option is to open a large number of connections and then
send data. I have seen that when I request 1,000 connections, it takes quite
some while to open them (with the CPU being almost idle during that time). So
far, I did not try to dig into this as the only bad effect it causes is that
the testbench runs a little bit longer.

Yesterday, for some other reason, I ran the very same tool, with the very
same options under strace. And I was totally stunned to see that the
connect() calls were now finished in almost no time. When I removed strace,
it took long again (30..60 seconds, not exactly measured). If I ran it under
strace, maybe a second or two. This is totally reproducible.

Does anybody have a clue?

Thanks,
Rainer

PS: if someone is interested in reproducing this (or the details): the test
in question is ./tests/manytcp.sh


From tbergfeld at hq.adiscon.com  Thu Aug 12 14:47:47 2010
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Thu, 12 Aug 2010 14:47:47 +0200
Subject: [rsyslog] rsyslog 6.1.0 (devel) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD20B@GRFEXC.intern.adiscon.com>

Hi all,

We have just released rsyslog 6.1.0, the new v6-devel.
This is the first v6 release of rsyslog, beginning a new development focus.
In v6, configuration shall become much more straigthforward. This release
already offers action scoping, which keeps related things closer together.
Other than that, it is roughly functionally equivalent to the latest v5
releases. However, there is imptcp available, a Linux-optimized plain tcp
input which in theory should perform faster than imtcp (at the price of some
reduced functionality).

Note that v6 requieres modifications to pre v6-plugins in order to work.

ChangeLog:

http://www.rsyslog.com/changelog-for-6-1-0-devel/

Download:

http://www.rsyslog.com/rsyslog-6-1-0-v6-devel/

As always, feedback is appreciated.

Best regards,
Tom Bergfeld
--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .



From sivan at omniqueue.com  Sun Aug 22 15:17:50 2010
From: sivan at omniqueue.com (Sivan Greenberg)
Date: Sun, 22 Aug 2010 16:17:50 +0300
Subject: [rsyslog] Setting up rsyslog 4.6.4 with RELP *and* TLS
Message-ID: <AANLkTinzqWFW+uyP0mS-XkrOiKgoaGNUDF75h-GsQ4Jh@mail.gmail.com>

Hi List,

 I've successfully built 4.6.4 with support for TLS and RELP. When I
worked with the imtcp module everything was fine and setting TLS to be
enabled through a snippet like this:


$DefaultNetstreamDriver gtls
$DefaultNetstreamDriverCAFile /root/tls/ca.pem
$DefaultNetstreamDriverCertFile /root/tls/client-cert.pem
$DefaultNetstreamDriverKeyFile /root/tls/client-key.pem
$ModLoad imuxsock
$ModLoad omrelp
$ModLoad imtcp

$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer server.example.com
$ActionSendStreamDriverMode 1
*.* @@:omrelp:server.example.com:10514


----

However, when I restart on the client and use logger to send a line
nothing happens, there's also no error reported or anything similar on
the logging output created by rsyslog.

This is the server snippet:

$ModLoad imuxsock
$Modload imtcp
$Modload imrelp
$DefaultNetstreamDriver gtls

$DefaultNetstreamDriverCAFile /root/tls/ca.pem
$DefaultNetstreamDriverCertFile /root/tls/server-cert.pem
$DefaultNetstreamDriverKeyFile /root/tls/server-key.pem

$InputTCPServerStreamDriverAuthMode x509/name
$InputTCPServerStreamDriverPermittedPeer *.example.com
$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
$InputRELPServerRun 10514 # start up listener at port 10514
$InputTCPServerRun 10514


I realize that I might have to add the:
RSYSLOG_ForwardFormat to the forward line, but given that I do add it
and it works without TLS, what is the way to enable it to use TLS
together with RELP ?

Any feedback will be highly appreciated!

Thanks,

Sivan


From joe at joetify.com  Thu Aug 26 16:42:45 2010
From: joe at joetify.com (Joe Williams)
Date: Thu, 26 Aug 2010 07:42:45 -0700
Subject: [rsyslog] symlinks
Message-ID: <DD10BDA2-03BA-4D5B-B299-38D41D6FD314@joetify.com>


I have some hostnamed directories and would like to symlink to them to something shorter and more meaningful. Is it possible to have rsyslog create symlinks to make certain logs easier to get too? Alternatively, can I configure rsyslog to match on certain hostnames and instead of using the hostname use a directory name I choose.

Thanks.

-Joe



Name: Joseph A. Williams
Email: joe at joetify.com
Blog: http://www.joeandmotorboat.com/
Twitter: http://twitter.com/williamsjoe



From gravyface at gmail.com  Thu Aug 26 18:12:09 2010
From: gravyface at gmail.com (GravyFace)
Date: Thu, 26 Aug 2010 12:12:09 -0400
Subject: [rsyslog] Should ActionExecOnlyOnceEveryInterval be working in 4.2?
Message-ID: <AANLkTikW4smA5e3LZQP0W7qCim8cveCBNZ9qCJ0CVJDY@mail.gmail.com>

>From the changelogs, it sounds like it was fixed in 4.4.3, and then
included in 4.6.x I believe.

Ubuntu 10.04 ships with 4.2, but in my config, it's not working --
emails get sent out for each match on :msg, regardless of
ActionExecOnlyOnceEveryInterval.

Config as follows:

$ActionMailSMTPServer 127.0.0.1
$ActionMailFrom root at example.com
$ActionMailTo somebody at example.com
$template mailSubject,"Alert: Inbound Packet Denied"
$template mailBody,"DETAILS:\r\n%timereported% %msg%"
$ActionMailSubject mailSubject
# make sure we receive a mail only once per hour
$ActionExecOnlyOnceEveryInterval 3600
:msg, contains, "Deny tcp src inside" :ommail:;mailBody


From sivan at omniqueue.com  Sun Aug 29 10:43:51 2010
From: sivan at omniqueue.com (Sivan Greenberg)
Date: Sun, 29 Aug 2010 11:43:51 +0300
Subject: [rsyslog] Reducing the time a msg spends in the main unreliable
	main queue to a minimum.
Message-ID: <AANLkTi=FcoYYG7LHGF=Mp7epjQ3=YNmdJkMqeoSNR+SC@mail.gmail.com>

Hi List,

 I have some critical msgs being transferred using rsyslog. Hence, I
would like to minimize the time a msg spends in the unreliable by
definition main queue. My understanding is that only action queues
which are where msgs arrive when dispatched from main queue  can be
set to be disk only. Therefor my assumption that only they can be made
truly reliable.

Any suggestions / feedback will be highly appreciated.

-Sivan


From rgerhards at hq.adiscon.com  Mon Aug 30 08:19:21 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 30 Aug 2010 08:19:21 +0200
Subject: [rsyslog] Reducing the time a msg spends in the main
	unreliablemain queue to a minimum.
References: <AANLkTi=FcoYYG7LHGF=Mp7epjQ3=YNmdJkMqeoSNR+SC@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD275@GRFEXC.intern.adiscon.com>

Hi Sivan,

the main message queue supports the same set of properties and capabilities
like the action queue. The directives just start with $MainMsg... instead of
$Action...

So you can set the main queue to pure disk mode as well.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Sivan Greenberg
> Sent: Sunday, August 29, 2010 10:44 AM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] Reducing the time a msg spends in the main
> unreliablemain queue to a minimum.
> 
> Hi List,
> 
>  I have some critical msgs being transferred using rsyslog. Hence, I
> would like to minimize the time a msg spends in the unreliable by
> definition main queue. My understanding is that only action queues
> which are where msgs arrive when dispatched from main queue  can be
> set to be disk only. Therefor my assumption that only they can be made
> truly reliable.
> 
> Any suggestions / feedback will be highly appreciated.
> 
> -Sivan
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From sivan at omniqueue.com  Mon Aug 30 08:33:13 2010
From: sivan at omniqueue.com (Sivan Greenberg)
Date: Mon, 30 Aug 2010 09:33:13 +0300
Subject: [rsyslog] Reducing the time a msg spends in the main
 unreliablemain queue to a minimum.
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD275@GRFEXC.intern.adiscon.com>
References: <AANLkTi=FcoYYG7LHGF=Mp7epjQ3=YNmdJkMqeoSNR+SC@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA71DD275@GRFEXC.intern.adiscon.com>
Message-ID: <AANLkTinY7CMNap1cQrPS5JevbkN4VEa3qdcjD23VtBkO@mail.gmail.com>

Hi Rainer,

I see, however, that would cause a major performance downgrade
thinking of it, and filtering seems to be only done on the main queue,
so to be able to use the same rsyslog instance for both critical and
non critical stuff I think I might be better leaving it buffered to
some extent (is there a directive to define how long/ how much memory)
and filter critical msgs to the pure disk msg queue and the rest.

Do you think this can be done? Could you kindly give pointer how to
configure for this?

Many thanks!

-Sivan

On Mon, Aug 30, 2010 at 9:19 AM, Rainer Gerhards
<rgerhards at hq.adiscon.com> wrote:
> Hi Sivan,
>
> the main message queue supports the same set of properties and capabilities
> like the action queue. The directives just start with $MainMsg... instead of
> $Action...
>
> So you can set the main queue to pure disk mode as well.
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Sivan Greenberg
>> Sent: Sunday, August 29, 2010 10:44 AM
>> To: rsyslog at lists.adiscon.com
>> Subject: [rsyslog] Reducing the time a msg spends in the main
>> unreliablemain queue to a minimum.
>>
>> Hi List,
>>
>> ?I have some critical msgs being transferred using rsyslog. Hence, I
>> would like to minimize the time a msg spends in the unreliable by
>> definition main queue. My understanding is that only action queues
>> which are where msgs arrive when dispatched from main queue ?can be
>> set to be disk only. Therefor my assumption that only they can be made
>> truly reliable.
>>
>> Any suggestions / feedback will be highly appreciated.
>>
>> -Sivan
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Mon Aug 30 12:38:39 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 30 Aug 2010 12:38:39 +0200
Subject: [rsyslog] Reducing the time a msg spends in the main
	unreliablemain queue to a minimum.
References: <AANLkTi=FcoYYG7LHGF=Mp7epjQ3=YNmdJkMqeoSNR+SC@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA71DD275@GRFEXC.intern.adiscon.com>
	<AANLkTinY7CMNap1cQrPS5JevbkN4VEa3qdcjD23VtBkO@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD278@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Sivan Greenberg
> Sent: Monday, August 30, 2010 8:33 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Reducing the time a msg spends in the main
> unreliablemain queue to a minimum.
> 
> Hi Rainer,
> 
> I see, however, that would cause a major performance downgrade

Definitely, very, very severe (you can't get all at once in life ;))

> thinking of it, and filtering seems to be only done on the main queue,
> so to be able to use the same rsyslog instance for both critical and
> non critical stuff I think I might be better leaving it buffered to
> some extent (is there a directive to define how long/ how much memory)

There are many. I suggest you read the queue doc and experiment with some oft
he settings. This is pretty complicated stuff (because you intend to do
complicated things), so it is vital to get good working knowledge before
deploying it to production. Queue doc is:

http://www.rsyslog.com/doc/queues.html

There are also a number of tutorials and descriptions on the workflow inside
the doc set. I'd personally expect that you need at least 2 to 3 days of
careful studying before you get a sufficient idea of how to configure a very
demanding environment.

> and filter critical msgs to the pure disk msg queue and the rest.
> 
> Do you think this can be done?

Yes

> Could you kindly give pointer how to
> configure for this?

As I said, very high reliability is rather complicated, it's a real
consulting job. So, sorry, I don't have a canned response. First step would
be to look at the actual needs, I think you already did this for most of the
part.

For example, if you can submit messages from the originator to the receiver
based on importance, you could use multiple rulesets and use a dedicated
ruleset using a disk-based queue for those important messages. But as I said
... doing it right involves a lot of planning, far more than setting a few
parameters. Otherwise you often end up with something that looks like it were
reliable, but is not really. You need to look at the whole picture. Once this
is done, you have a spec on which messages are what important and how many
loss is acceptable. From there on, you can work to turn that into a rsyslog
conf.

I recently wrote a blog post that may be worth mentioning for your case:

http://blog.gerhards.net/2010/08/rsyslog-queues-reliability-and.html

HTH
Rainer

> 
> Many thanks!
> 
> -Sivan
> 
> On Mon, Aug 30, 2010 at 9:19 AM, Rainer Gerhards
> <rgerhards at hq.adiscon.com> wrote:
> > Hi Sivan,
> >
> > the main message queue supports the same set of properties and
> capabilities
> > like the action queue. The directives just start with $MainMsg...
> instead of
> > $Action...
> >
> > So you can set the main queue to pure disk mode as well.
> >
> > Rainer
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Sivan Greenberg
> >> Sent: Sunday, August 29, 2010 10:44 AM
> >> To: rsyslog at lists.adiscon.com
> >> Subject: [rsyslog] Reducing the time a msg spends in the main
> >> unreliablemain queue to a minimum.
> >>
> >> Hi List,
> >>
> >> ?I have some critical msgs being transferred using rsyslog. Hence, I
> >> would like to minimize the time a msg spends in the unreliable by
> >> definition main queue. My understanding is that only action queues
> >> which are where msgs arrive when dispatched from main queue ?can be
> >> set to be disk only. Therefor my assumption that only they can be
> made
> >> truly reliable.
> >>
> >> Any suggestions / feedback will be highly appreciated.
> >>
> >> -Sivan
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Sun Aug  1 11:30:17 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Sun, 1 Aug 2010 11:30:17 +0200
Subject: [rsyslog] log forwarding through unix sockets
References: <4C456476.4010703@redhat.com>
	<AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710403E@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Ryan Lynch
> Sent: Friday, July 30, 2010 3:29 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] log forwarding through unix sockets
> 
> On Tue, Jul 20, 2010 at 04:55, Ales Kozumplik <akozumpl at redhat.com>
> wrote:
> > 1) KVM opens a SOCK_STREAM on the host end but rsyslogd is only able
> to
> > read data from SOCK_DGRAM. This has two consequences: first, to be
> able
> > to attach rsyslog on the host end one first needs to copy the data
> > between the two socket types, e.g. using socat. Second, messages
> longer
> > than 1024 characters are sometimes split into two. The second message
> is
> > thus missing the syslog header and the receiving rsyslogd doesn't
> know
> > where to file it. Is there a recommended workaround for those things
> > (maybe a parameter I overlooked in the docs tellling rsyslogd to use
> > SOCK_STREAM)?
> 
> I ran into a similar problem. Doesn't it seem wierd that RSyslog:
>  - can write TO a pipe, but it can't natively read FROM a pipe.
>  - can read FROM a UNIX socket, but can't natively write TO a UNIX
> socket.
> The only protocols that Rsyslog can both read to AND write from are
> network sockets (UDP, TCP, RELP) and real files.
> 

Quite honestly, I think this issue never came up until recently -- at least I
didn't notice. It should be fairly simple to add a plugin to write to unix
sockets, though...
 
> I don't think the problem is your forwarding format--I don't think
> it's possible for RSyslog to handle a HOSTNAME field, properly, in
> messages received via socket.
> 
> Based on my own tests, I believe that 'imuxsock' and 'imudp' use
> different logic to parse incoming messages. 'imuxsock' always assumes
> that the hostname is the local host, so it doesn't have the
> conditional logic to differentiate between "forwarded" messages (which
> have an extra HOSTNAME field between the timestamp and tag) versus
> regular local messages (no HOSTNAME).

The problem is that you do not have any reliable indication what is a
hostname and what not. So far, I worked on the assumption that anything
received via a local socket complies to the "usual format" for such messages.
This means no hostname is present.

However, there is a simple setting (parse hostname yes/no) which is part of
the message object. Imuxsock sets it to "no", remote receivers set it to
"yes". It should be fairly trivial to add a config setting which enables
imuxsock to set it to "yes" as well.

> This is a pretty reasonable
> assumption, really--the local UNIX socket doesn't traditionally have
> any way to receive messages forwarded from other hosts.
> 
> Rainer could probably confirm this, or we could compare those two
> modules' sources.
> 
> RSyslog is a fantastic piece of software, and its feature set has come
> a long way. But all of Rainer's excellent work has expanded our
> imaginations about what Syslog can do, and maybe our imaginations are
> advancing faster than he can code new features.

The problem simply is that we are tackeling more complex things, and this
also means things need more planning than in the past. For example, the
latest round of performance enhancements took me 2 to 3 month of reading
scientific papers and experimenting. This all resulted in a relatively small
delta to the code, but this delta does not judge the work that went into it.
Also, being primarily a lone guy, I need to focus on some things and leave
others untouched -- otherwise I'll never make any progress (ever-swapping
between different features). The current scope is configurability and
performance, so other things have to wait (where I try to fit in small and
useful items as it fits). 

What is being discussed here is small and useful ;) However, my summer break
is approaching, and I'd like to finish a couple of loose ends (first config
iteration, getting a new v4-beta ready). Also I (finally ;)) need to write an
article for the upcoming Linux conference in Nuremberg, with the deadline
rapidly approaching. So these are obviously my priority.

Also I have to admit that I weigh features by their importance to the
community at large. Those often requested receive higher priority, so they
will preempt less requested features from being implemented when they have
similar workload. Also, I prefer features requested/need by non-profits over
those that have a clearly for-profit user base. The reasoning here is that
the non-profit orgs will probably otherwise never get their features, whereas
the enterprises can task someone with implementing such a features. And if an
enterprise doesn't think it is worth tasking someone with implementing a
feature, that is a clear indication that the enterprise doesn't actually
think the feature is worth the effort. So I think there is a lot of logic in
assigning priorities in this way ;)

Rainer


From akozumpl at redhat.com  Mon Aug  2 13:11:20 2010
From: akozumpl at redhat.com (Ales Kozumplik)
Date: Mon, 02 Aug 2010 13:11:20 +0200
Subject: [rsyslog] log forwarding through unix sockets
In-Reply-To: <AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com>
References: <4C456476.4010703@redhat.com>
	<AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com>
Message-ID: <4C56A7D8.5020901@redhat.com>

On 07/30/2010 03:28 PM, Ryan Lynch wrote:
> I like your method, too. And thank you for mentioning 'socat', that's
> what gave me the idea to go in this direction, in the first place.

Thanks. To tell you the truth at the end we found a way to forward from 
qemu to a TCP socket, and I am happy I don't have to deal with unix 
sockets any more.

> Based on my own tests, I believe that 'imuxsock' and 'imudp' use
> different logic to parse incoming messages. 'imuxsock' always assumes

That's exactly my feeling about this. I just think one either should be 
able to tell every input module what format should be expected (instead 
of letting rsyslogd try some guessing method), or that every input 
module should by default understand a standard "officially recommended" 
forwarding format.

Ales


From joel.merrick at gmail.com  Mon Aug  2 13:13:35 2010
From: joel.merrick at gmail.com (Joel Merrick)
Date: Mon, 2 Aug 2010 12:13:35 +0100
Subject: [rsyslog] Problem with mysql template
In-Reply-To: <AANLkTimu-Cm-+_BWE+xRCNO6xDVxt2cf38jAcOiFxZBb@mail.gmail.com>
References: <AANLkTimu-Cm-+_BWE+xRCNO6xDVxt2cf38jAcOiFxZBb@mail.gmail.com>
Message-ID: <AANLkTinbjtOgBXxFz35RKX+oohfhg+-fPU8U_CtkLyL5@mail.gmail.com>

Do I take it from the lack of replies that this should work? :)



On Thu, Jul 29, 2010 at 11:53 AM, Joel Merrick <joel.merrick at gmail.com> wrote:
> Hi all,
>
> I'm trying to create a very quick mail log searching solution. The
> idea is to pre-hash the database by using the last 2 alpha-numeric
> characters on the message id.
>
> This isn't inserting to the database and I can't think why.. I've
> already built the tables with the structure logs-aa.. logs-ZZ etc..
>
>
> $template OurDBLog,"INSERT INTO
> logs-'%msg:R,ERE,1,NULL:[A-Za-z0-9]{6}-[A-Za-z0-9]{6}-([A-Za-z0-9]{2})--end%'
> (messageid, host, \
> ? ? send_host, created_at, payload, subject) values \
> ? ? ('%msg:R,ERE,0,ZERO:[A-Za-z0-9]{6}-[A-Za-z0-9]{6}-[A-Za-z0-9]{2}--end%','%HOSTNAME%',
> '%msg:R,ERE,0,ZERO:H=.*\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}]--end%',
> \
> ? ? '%timegenerated:::date-mysql%', '%msg%',
> '%msg:R,ERE,1,ZERO:T="(.+)"--end%')",SQL
>
>
> Any idea?
>
>
> On a side note, how can I get extra verbosity out of rsyslog so I'm
> not blindly trying to insert and then check via mysql
>
> Cheers,
> Joel
>
>
> --
> $ echo "kpfmAdpoofdufevq/dp/vl" | perl -pe 's/(.)/chr(ord($1)-1)/ge'
>



-- 
$ echo "kpfmAdpoofdufevq/dp/vl" | perl -pe 's/(.)/chr(ord($1)-1)/ge'


From rgerhards at hq.adiscon.com  Mon Aug  2 13:14:41 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 2 Aug 2010 13:14:41 +0200
Subject: [rsyslog] log forwarding through unix sockets
References: <4C456476.4010703@redhat.com><AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com>
	<4C56A7D8.5020901@redhat.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104042@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Ales Kozumplik
> Sent: Monday, August 02, 2010 1:11 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] log forwarding through unix sockets
> > Based on my own tests, I believe that 'imuxsock' and 'imudp' use
> > different logic to parse incoming messages. 'imuxsock' always assumes
> 
> That's exactly my feeling about this. 

I elaborated on that yesterday -- not sure if you saw that posting.

> I just think one either should be
> able to tell every input module what format should be expected (instead
> of letting rsyslogd try some guessing method), or that every input
> module should by default understand a standard "officially recommended"
> forwarding format.

The problem is that syslog() does not emit the format recommended by RFC3164,
so no message provided over that channel includes a hostname. Things would be
heavily broken if I would simply ignore that fact.

Rainer


From rgerhards at hq.adiscon.com  Mon Aug  2 13:16:40 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 2 Aug 2010 13:16:40 +0200
Subject: [rsyslog] Problem with mysql template
References: <AANLkTimu-Cm-+_BWE+xRCNO6xDVxt2cf38jAcOiFxZBb@mail.gmail.com>
	<AANLkTinbjtOgBXxFz35RKX+oohfhg+-fPU8U_CtkLyL5@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104043@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Joel Merrick
> Sent: Monday, August 02, 2010 1:14 PM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] Problem with mysql template
> 
> Do I take it from the lack of replies that this should work? :)

simply overlooked from my side ;)

> 
> On Thu, Jul 29, 2010 at 11:53 AM, Joel Merrick <joel.merrick at gmail.com>
> wrote:
> > Hi all,
> >
> > I'm trying to create a very quick mail log searching solution. The
> > idea is to pre-hash the database by using the last 2 alpha-numeric
> > characters on the message id.
> >
> > This isn't inserting to the database and I can't think why.. I've
> > already built the tables with the structure logs-aa.. logs-ZZ etc..
> >
> >
> > $template OurDBLog,"INSERT INTO
> > logs-'%msg:R,ERE,1,NULL:[A-Za-z0-9]{6}-[A-Za-z0-9]{6}-([A-Za-z0-
> 9]{2})--end%'
> > (messageid, host, \
> > ? ? send_host, created_at, payload, subject) values \
> > ? ? ('%msg:R,ERE,0,ZERO:[A-Za-z0-9]{6}-[A-Za-z0-9]{6}-[A-Za-z0-9]{2}-
> -end%','%HOSTNAME%',
> > '%msg:R,ERE,0,ZERO:H=.*\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-
> 9]{1,3}]--end%',
> > \
> > ? ? '%timegenerated:::date-mysql%', '%msg%',
> > '%msg:R,ERE,1,ZERO:T="(.+)"--end%')",SQL
> >
> >
> > Any idea?
> >
> >
> > On a side note, how can I get extra verbosity out of rsyslog so I'm
> > not blindly trying to insert and then check via mysql

Two ways to proceed: 

a) write to a file with the same template
then you can check if the sql is like you intended it to be

b) run rsyslog in debug mode (doc available in doc set and on site)

It may also be worth trying out the regex checker on the rsyslog site.

Rainer


From akozumpl at redhat.com  Mon Aug  2 13:21:14 2010
From: akozumpl at redhat.com (Ales Kozumplik)
Date: Mon, 02 Aug 2010 13:21:14 +0200
Subject: [rsyslog] log forwarding through unix sockets
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710403E@GRFEXC.intern.adiscon.com>
References: <4C456476.4010703@redhat.com>	<AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA710403E@GRFEXC.intern.adiscon.com>
Message-ID: <4C56AA2A.1010709@redhat.com>

On 08/01/2010 11:30 AM, Rainer Gerhards wrote:
> Also I have to admit that I weigh features by their importance to the
> community at large. Those often requested receive higher priority, so they
> will preempt less requested features from being implemented when they have
> similar workload. Also, I prefer features requested/need by non-profits over
> those that have a clearly for-profit user base. The reasoning here is that
> the non-profit orgs will probably otherwise never get their features, whereas
> the enterprises can task someone with implementing such a features. And if an
> enterprise doesn't think it is worth tasking someone with implementing a
> feature, that is a clear indication that the enterprise doesn't actually
> think the feature is worth the effort. So I think there is a lot of logic in
> assigning priorities in this way ;)
>

Hi Rainer,

I'd just add that the feature is targeted for Fedora 14 and the 
community will benefit because improved installer logging allows the 
Fedora QA to work more efficiently and so make a greater Fedora.

Saying that I agree that some obscure unix socket logging features 
shouldn't be of high priority. Just an idea for a backlog.

Ales


From joel.merrick at gmail.com  Mon Aug  2 13:26:05 2010
From: joel.merrick at gmail.com (Joel Merrick)
Date: Mon, 2 Aug 2010 12:26:05 +0100
Subject: [rsyslog] Problem with mysql template
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7104043@GRFEXC.intern.adiscon.com>
References: <AANLkTimu-Cm-+_BWE+xRCNO6xDVxt2cf38jAcOiFxZBb@mail.gmail.com>
	<AANLkTinbjtOgBXxFz35RKX+oohfhg+-fPU8U_CtkLyL5@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7104043@GRFEXC.intern.adiscon.com>
Message-ID: <AANLkTikMf-oe381Ab8eUg4HtWJzbdQOcye0a+sjqns+S@mail.gmail.com>

On Mon, Aug 2, 2010 at 12:16 PM, Rainer Gerhards
<rgerhards at hq.adiscon.com> wrote:
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Joel Merrick
>> Sent: Monday, August 02, 2010 1:14 PM
>> To: rsyslog at lists.adiscon.com
>> Subject: Re: [rsyslog] Problem with mysql template
>>
>> Do I take it from the lack of replies that this should work? :)
>
> simply overlooked from my side ;)
>

No worries, I know you're very busy!


> Two ways to proceed:
>
> a) write to a file with the same template
> then you can check if the sql is like you intended it to be
>

The rest of the SQL is correct, it's when I add the part that adds the
last 2 alphanumeric characters to the table name when it breaks..


> b) run rsyslog in debug mode (doc available in doc set and on site)
>

Will do, cheers for the tips.

> It may also be worth trying out the regex checker on the rsyslog site.


Yes, used this extensively and I'm getting the matches correct.

>
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>



-- 
$ echo "kpfmAdpoofdufevq/dp/vl" | perl -pe 's/(.)/chr(ord($1)-1)/ge'


From rgerhards at hq.adiscon.com  Mon Aug  2 13:26:42 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 2 Aug 2010 13:26:42 +0200
Subject: [rsyslog] log forwarding through unix sockets
References: <4C456476.4010703@redhat.com>	<AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA710403E@GRFEXC.intern.adiscon.com>
	<4C56AA2A.1010709@redhat.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104044@GRFEXC.intern.adiscon.com>

Hi Ales,

> I'd just add that the feature is targeted for Fedora 14 and the
> community will benefit because improved installer logging allows the
> Fedora QA to work more efficiently and so make a greater Fedora.

that's good info! What's the latest date you need to have the feature?

> Saying that I agree that some obscure unix socket logging features
> shouldn't be of high priority. Just an idea for a backlog.

I don't think it is a lot of work, maybe just 2 days. But I am not sure if I
can begin to work on it before end of August. If work on my conference papers
goes well, I may be able to do it before I leave to vacation.

Rainer


From akozumpl at redhat.com  Mon Aug  2 17:03:27 2010
From: akozumpl at redhat.com (Ales Kozumplik)
Date: Mon, 02 Aug 2010 17:03:27 +0200
Subject: [rsyslog] log forwarding through unix sockets
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7104044@GRFEXC.intern.adiscon.com>
References: <4C456476.4010703@redhat.com>	<AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA710403E@GRFEXC.intern.adiscon.com>	<4C56AA2A.1010709@redhat.com>
	<9B6E2A8877C38245BFB15CC491A11DA7104044@GRFEXC.intern.adiscon.com>
Message-ID: <4C56DE3F.6010408@redhat.com>

On 08/02/2010 01:26 PM, Rainer Gerhards wrote:
> I don't think it is a lot of work, maybe just 2 days. But I am not sure if I
> can begin to work on it before end of August. If work on my conference papers
> goes well, I may be able to do it before I leave to vacation.

Thanks, like I said in a reply earlier today, we made qemu forward the 
logs to a TCP port on the host. That way we avoid all of the problems we 
had, so from Anaconda's point of view rsyslog works just fine at the moment.

Ales


From sean at conman.org  Mon Aug  2 22:46:21 2010
From: sean at conman.org (Sean Conner)
Date: Mon, 2 Aug 2010 16:46:21 -0400
Subject: [rsyslog] log forwarding through unix sockets
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7104042@GRFEXC.intern.adiscon.com>
References: <4C56A7D8.5020901@redhat.com>
	<9B6E2A8877C38245BFB15CC491A11DA7104042@GRFEXC.intern.adiscon.com>
Message-ID: <20100802204621.GA7853@brevard.conman.org>

It was thus said that the Great Rainer Gerhards once stated:
> 
> The problem is that syslog() does not emit the format recommended by RFC3164,
> so no message provided over that channel includes a hostname. Things would be
> heavily broken if I would simply ignore that fact.

  Can't it be added as part of the parsing?  Even if it's just "localhost"? 
Or a parameter to the imuxsock module?  My own syslogd fills in missing
fields with sane (at least to me) defaults.

  -spc 



From david at lang.hm  Tue Aug  3 00:03:11 2010
From: david at lang.hm (david at lang.hm)
Date: Mon, 2 Aug 2010 15:03:11 -0700 (PDT)
Subject: [rsyslog] omspoof module
Message-ID: <alpine.DEB.2.00.1008021459190.29820@asgard.lang.hm>

In trying to use the module omspoof with 5.5.6 it looks like there is a 
problem.

the debug file shows a couple errors (entry points not present in module) 
followed by 'tried selector action for omudpspoof: -2001'

then a couple of lines later when defining 
$ActionUDPSpoofSourceNameTemplate it gets the error 'unkown config, did 
you forget to load a module?'

David Lang
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: debug.txt
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20100802/5e4384f9/attachment-0001.txt>

From alexandre.chapellon at mana.pf  Tue Aug  3 05:26:49 2010
From: alexandre.chapellon at mana.pf (Alexandre Chapellon)
Date: Mon, 02 Aug 2010 17:26:49 -1000
Subject: [rsyslog] replaying QIF files
Message-ID: <1280806009.3863.40.camel@mcbain.mana.lan>

Hello I have rsyslog installed on a server to inject data in postgresql
database.
Few days ago SQL injection stopped (because of a syntax error: bad
template)
As I configured disk Queue for the ompgsql action I have all the non
injected messages in thousans of QIF Files.
Here is the cfg:
##########################
$template LogToPqsql, "insert into blablabla' );", sql
$WorkDirectory /var/spool/rsyslog/work
$ActionQueueType LinkedList
$ActionQueueFileName dbq
$ActionResumeRetryCount -1

local1.* :ompgsql:blablablabla
##########################

Unfortunately after restarting rsyslog, it does not reads the files to
inject it back to pg.
Looking in the work dir I can't see file with the .qi extension.
If I manually rename the first QIF file to dbq.qi, rsyslog crash with a
segfault.
Then I tried installing 4.6.3 instead of my old 4.4.2. Rsyslog do not
crash anymore but complains about something wrong with my QIF file (i
guess):

##########################
...
5150.884531712:40800950: deserializer has possibly been able to re-sync
and recover, state 0
5150.884531712:40800950: objDeserializePropBag error -2037 during header
- trying to recover
5150.884531712:40800950: strm 0x689a10: file 10 read 421 bytes
5150.884531712:40800950: strm 0x689a10: file 10 read 0 bytes
5150.884531712:40800950: deserializer has possibly been able to re-sync
and recover, state -2026
5150.884531712:40800950: strm 0x689a10: file
10(/var/spool/rsyslog/work/dbq.qi) closing
5150.884531712:40800950: action 3 queue[DA]: error -2026 reading .qi
file - can not read persisted info (if any)
5150.884531712:40800950: action 3 queue[DA]: persisting queue to disk, 0
entries...
5150.884531712:40800950: action 3 queue: error -2026 creating disk queue
- giving up.
...
##########################

Does anyone have any idea?



From rgerhards at hq.adiscon.com  Tue Aug  3 07:56:57 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 3 Aug 2010 07:56:57 +0200
Subject: [rsyslog] omspoof module
References: <alpine.DEB.2.00.1008021459190.29820@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710404A@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, August 03, 2010 12:03 AM
> To: rsyslog-users
> Subject: [rsyslog] omspoof module
> 
> In trying to use the module omspoof with 5.5.6 it looks like there is a
> problem.
> 
> the debug file shows a couple errors (entry points not present in
> module)

That is perfectly valid. This is just information for me, so that I know
which of the optional entry points are present or not.

> followed by 'tried selector action for omudpspoof: -2001'

This is also valid. rsyslog (bare of a better method) pushes the action part
to the plugin. The plugin either returns OK (if it recognizes the format) or
RS_RET_CONFLINE_UNPROCESSED = -2001 if it does not recognize it. So for each
action, you'll see a couple of them. Thus my guess is that there simply is a
syntax error in the :omudpspoof: part (typo?).

> 
> then a couple of lines later when defining
> $ActionUDPSpoofSourceNameTemplate it gets the error 'unkown config, did
> you forget to load a module?'

Mhhh... that doesn't sound normal. Do you have a longer excerpt of the debug
log for me?

Rainer


From rgerhards at hq.adiscon.com  Tue Aug  3 08:45:41 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 3 Aug 2010 08:45:41 +0200
Subject: [rsyslog] omspoof module
References: <alpine.DEB.2.00.1008021459190.29820@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710404C@GRFEXC.intern.adiscon.com>


> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, August 03, 2010 12:03 AM
> To: rsyslog-users
> Subject: [rsyslog] omspoof module
> 
> In trying to use the module omspoof with 5.5.6 it looks like there is a
> problem.
> 
> the debug file shows a couple errors (entry points not present in
> module)
> followed by 'tried selector action for omudpspoof: -2001'
> 
> then a couple of lines later when defining
> $ActionUDPSpoofSourceNameTemplate it gets the error 'unkown config, did
> you forget to load a module?'


David, I overlooked this as well: it is $ActionOMUDP..., not $ActionUDP.... I
just saw it when I set up my lab... With the current config format (and also
with the new scoping), I can not produce a better error message, because it
actually could be either case...

Rainer


From sledz at dresearch.de  Tue Aug  3 09:12:17 2010
From: sledz at dresearch.de (Steffen Sledz)
Date: Tue, 03 Aug 2010 09:12:17 +0200
Subject: [rsyslog] [PATCH] break potential infinite loop in actionDoRetry
In-Reply-To: <1280494375-11930-1-git-send-email-sledz@dresearch.de>
References: <4C527173.9090805@dresearch.de>
	<1280494375-11930-1-git-send-email-sledz@dresearch.de>
Message-ID: <4C57C151.2020402@dresearch.de>

Am 30.07.2010 14:52, schrieb Steffen Sledz:
> If a module always returns RS_RET_OK (like ompipe does) the actionDoRetry
> loop may not have leaved faked ACT_STATE_SUSP state in case iResumeOKinRow
> had ever reached a count of 1000.

In our test the suggested patch seems to fix this problem. :) But now we hit the next one. :(

If we use a LinkedList DA-Queue for this channel suspended messages may be transmitted out-of-order. This may not be a problem for database consumers. But in our context this is really bad.

Is this a bug or an intended behaviour?

Is it possible to force strict order of messages?

Steffen



From david at lang.hm  Tue Aug  3 09:21:38 2010
From: david at lang.hm (david at lang.hm)
Date: Tue, 3 Aug 2010 00:21:38 -0700 (PDT)
Subject: [rsyslog] [PATCH] break potential infinite loop in actionDoRetry
In-Reply-To: <4C57C151.2020402@dresearch.de>
References: <4C527173.9090805@dresearch.de>
	<1280494375-11930-1-git-send-email-sledz@dresearch.de>
	<4C57C151.2020402@dresearch.de>
Message-ID: <alpine.DEB.2.00.1008030016210.25170@asgard.lang.hm>

On Tue, 3 Aug 2010, Steffen Sledz wrote:

> Am 30.07.2010 14:52, schrieb Steffen Sledz:
>> If a module always returns RS_RET_OK (like ompipe does) the actionDoRetry
>> loop may not have leaved faked ACT_STATE_SUSP state in case iResumeOKinRow
>> had ever reached a count of 1000.
>
> In our test the suggested patch seems to fix this problem. :) But now we hit the next one. :(
>
> If we use a LinkedList DA-Queue for this channel suspended messages may be transmitted out-of-order. This may not be a problem for database consumers. But in our context this is really bad.
>
> Is this a bug or an intended behaviour?

intended behavior,

the issue came up beforeand there are many different ways that messages 
can end up being processed out of order (including the network re-ordering 
UDP packets before they arrive)

part of the huge performance advantage that rsyslog has is that it is able 
to use multiple threads to process messages (both inbound and outbound), 
trying to maintain the order would cripple rsyslog (and still not be good 
enough in many cases)

when I took a class on the Simple Event Correlator (a very good tool for 
log analysis), the instructor had a specific example in the beginning to 
point out that you can't count on logs being processed by you in the order 
they are generated, and this was a couple of years ago before rsyslog 
started becoming so popular. This example was with plain old 
single-threaded syslogd.

David Lang

> Is it possible to force strict order of messages?
>
> Steffen
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From sledz at dresearch.de  Tue Aug  3 10:29:43 2010
From: sledz at dresearch.de (Steffen Sledz)
Date: Tue, 03 Aug 2010 10:29:43 +0200
Subject: [rsyslog] [PATCH] break potential infinite loop in actionDoRetry
In-Reply-To: <alpine.DEB.2.00.1008030016210.25170@asgard.lang.hm>
References: <4C527173.9090805@dresearch.de>	<1280494375-11930-1-git-send-email-sledz@dresearch.de>	<4C57C151.2020402@dresearch.de>
	<alpine.DEB.2.00.1008030016210.25170@asgard.lang.hm>
Message-ID: <4C57D377.1020107@dresearch.de>

Am 03.08.2010 09:21, schrieb david at lang.hm:
>> If we use a LinkedList DA-Queue for this channel suspended
>> messages may be transmitted out-of-order. This may not be
>> a problem for database consumers. But in our context this
>> is really bad.
>>
>> Is this a bug or an intended behaviour?
> 
> intended behavior,
> 
> the issue came up beforeand there are many different ways that messages 
> can end up being processed out of order (including the network re-ordering 
> UDP packets before they arrive)
> 
> part of the huge performance advantage that rsyslog has is that it is able 
> to use multiple threads to process messages (both inbound and outbound), 
> trying to maintain the order would cripple rsyslog (and still not be good 
> enough in many cases)
> 
> when I took a class on the Simple Event Correlator (a very good tool for 
> log analysis), the instructor had a specific example in the beginning to 
> point out that you can't count on logs being processed by you in the order 
> they are generated, and this was a couple of years ago before rsyslog 
> started becoming so popular. This example was with plain old 
> single-threaded syslogd.

In my opinion this may occur if messages came from different/concurrent sources. But it should be possible to keep messages from the same source in strict order (with performance loss).

So again my question. Is it possible to force strict order of messages?

Steffen



From rgerhards at hq.adiscon.com  Tue Aug  3 10:42:58 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 3 Aug 2010 10:42:58 +0200
Subject: [rsyslog] [PATCH] break potential infinite loop in actionDoRetry
References: <4C527173.9090805@dresearch.de>	<1280494375-11930-1-git-send-email-sledz@dresearch.de>	<4C57C151.2020402@dresearch.de><alpine.DEB.2.00.1008030016210.25170@asgard.lang.hm>
	<4C57D377.1020107@dresearch.de>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104054@GRFEXC.intern.adiscon.com>


> In my opinion this may occur if messages came from different/concurrent
> sources. 

Depends on your environment. If you have a single UDP listener, reordering
may occur on the wire before you even receive it. Sequence as seen on the
input/API level is not sequence as the message may be originated. Not even so
for local unix sockets, this is depending on scheduling order, which you
cannot control in userland. So I can back David's point: it is dangerous to
assume that a log stream has strict sequence. This may be the case for an
single hop (obviously for messages within a single TCP stream), but not
further than that. In general, you need to define an order relation. 

As a side note, a not-so-small part of the conference paper I am writing
right now is about these wrong perception of log sequence that users commonly
have. I think I can post it myself after the conference (this is the end of
september timeframe).

> But it should be possible to keep messages from the same
> source in strict order (with performance loss).


> 
> So again my question. Is it possible to force strict order of messages?


I think you can come close to what you want:

You need to dig into the queue docs. Set everything to single worker thread
(maybe even use direct mode). If you use a recent build, set all batch sizes
to one (this is conservative, in theory it should work with larger batches as
well, iff everything is single-threaded).

Note that when you use a disk assisted queue, the queue engine always puts
some messages out of order when switching to/from Disk mode. Newer builds
also consume messages from memory and disk concurrently and thus potentially
out of queue order. So you need to avoid disk assisted mode. If you need disk
functionality, use pure disk mode, only.

Make sure you do not define more than one input per rsyslog instance. Else OS
scheduling order for rsyslog threads (input vs. output) potentially gets you
into trouble. Make sure you do not define any async output actions. Do not
write to the same file (or action in general) from more than one action or
more than one instance of rsyslog.

That should give you a somewhat strict sequence of messages. Depending on the
use case/volume, you'll see no to unacceptable performance loss.

Rainer


From sledz at dresearch.de  Tue Aug  3 11:06:23 2010
From: sledz at dresearch.de (Steffen Sledz)
Date: Tue, 03 Aug 2010 11:06:23 +0200
Subject: [rsyslog] [PATCH] break potential infinite loop in actionDoRetry
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7104054@GRFEXC.intern.adiscon.com>
References: <4C527173.9090805@dresearch.de>	<1280494375-11930-1-git-send-email-sledz@dresearch.de>	<4C57C151.2020402@dresearch.de><alpine.DEB.2.00.1008030016210.25170@asgard.lang.hm>	<4C57D377.1020107@dresearch.de>
	<9B6E2A8877C38245BFB15CC491A11DA7104054@GRFEXC.intern.adiscon.com>
Message-ID: <4C57DC0F.8070404@dresearch.de>

Am 03.08.2010 10:42, schrieb Rainer Gerhards:
> Depends on your environment. If you have a single UDP
> listener, reordering may occur on the wire before you
> even receive it....

As you can guess in our context UDP is not involved, just the fifo.

> Note that when you use a disk assisted queue, the queue
> engine always puts some messages out of order when switching
> to/from Disk mode. Newer builds also consume messages from
> memory and disk concurrently and thus potentially out of
> queue order.

I think that's what we see.

So i believe we have to decide between out-of-order messages or loss of some messages when we do not use a disk assisted queue. :(

Steffen



From rgerhards at hq.adiscon.com  Tue Aug  3 11:08:18 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 3 Aug 2010 11:08:18 +0200
Subject: [rsyslog] [PATCH] break potential infinite loop in actionDoRetry
References: <4C527173.9090805@dresearch.de>	<1280494375-11930-1-git-send-email-sledz@dresearch.de>	<4C57C151.2020402@dresearch.de><alpine.DEB.2.00.1008030016210.25170@asgard.lang.hm>	<4C57D377.1020107@dresearch.de><9B6E2A8877C38245BFB15CC491A11DA7104054@GRFEXC.intern.adiscon.com>
	<4C57DC0F.8070404@dresearch.de>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104055@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Steffen Sledz
> Sent: Tuesday, August 03, 2010 11:06 AM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] [PATCH] break potential infinite loop in
> actionDoRetry
> 
> Am 03.08.2010 10:42, schrieb Rainer Gerhards:
> > Depends on your environment. If you have a single UDP
> > listener, reordering may occur on the wire before you
> > even receive it....
> 
> As you can guess in our context UDP is not involved, just the fifo.

The you have the issue of context switching between producers (assuming there
is more than one).

> 
> > Note that when you use a disk assisted queue, the queue
> > engine always puts some messages out of order when switching
> > to/from Disk mode. Newer builds also consume messages from
> > memory and disk concurrently and thus potentially out of
> > queue order.
> 
> I think that's what we see.
> 
> So i believe we have to decide between out-of-order messages or loss of
> some messages when we do not use a disk assisted queue. :(

Or use a disk-only queue. It's performance is obviously very bad, but may be
sufficient depending on volume.

Rainer


From rgerhards at hq.adiscon.com  Thu Aug  5 07:53:36 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 5 Aug 2010 07:53:36 +0200
Subject: [rsyslog] [PATCH] break potential infinite loop in actionDoRetry
References: <4C527173.9090805@dresearch.de>
	<1280494375-11930-1-git-send-email-sledz@dresearch.de>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710405D@GRFEXC.intern.adiscon.com>

I just wanted to let you know that I have integrated the patch. Many thanks
for your effort!

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Steffen Sledz
> Sent: Friday, July 30, 2010 2:53 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] [PATCH] break potential infinite loop in
> actionDoRetry
> 
> If a module always returns RS_RET_OK (like ompipe does) the
> actionDoRetry
> loop may not have leaved faked ACT_STATE_SUSP state in case
> iResumeOKinRow
> had ever reached a count of 1000.
> 
> Signed-off-by: Steffen Sledz <sledz at dresearch.de>
> ---
>  action.c |    1 +
>  1 files changed, 1 insertions(+), 0 deletions(-)
> 
> diff --git a/action.c b/action.c
> index 32a07dc..90ec1bf 100644
> --- a/action.c
> +++ b/action.c
> @@ -508,6 +508,7 @@ static rsRetVal actionDoRetry(action_t *pThis,
> time_t ttNow)
>  		iRet = pThis->pMod->tryResume(pThis->pModData);
>  		if((pThis->iResumeOKinRow > 999) && (pThis->iResumeOKinRow
> % 1000 == 0)) {
>  			bTreatOKasSusp = 1;
> +			pThis->iResumeOKinRow = 0;
>  		} else {
>  			bTreatOKasSusp = 0;
>  		}
> --
> 1.6.4.2
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From tbergfeld at hq.adiscon.com  Thu Aug  5 14:18:52 2010
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Thu, 5 Aug 2010 14:18:52 +0200
Subject: [rsyslog] rsyslog 3.22.2 (v3-stable) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104067@GRFEXC.intern.adiscon.com>

Hi all,

We have just released rsyslog 3.22.2, a member of the v3-stable branch.

This is a stability update, integrating bug fixes that have become available
during the past month. No new functionality is included.

See ChangeLog for more details.

ChangeLog:

http://www.rsyslog.com/changelog-for-3-2-2-v3-stable/

Download:

http://www.rsyslog.com/rsyslog-3-22-2-v3-stable/

As always, feedback is appreciated.

Best regards,
Tom Bergfeld
--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .


From tbergfeld at hq.adiscon.com  Thu Aug  5 14:19:49 2010
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Thu, 5 Aug 2010 14:19:49 +0200
Subject: [rsyslog] rsyslog 4.6.4 (v4-stable) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104068@GRFEXC.intern.adiscon.com>

Hi all,

We have just released rsyslog 4.6.4, a member of the v4-stable branch.

This is a stability update without new features.

See ChangeLog for more details.

ChangeLog:

http://www.rsyslog.com/changelog-for-4-6-4-v4-stable/

Download:

http://www.rsyslog.com/rsyslog-4-6-4-v4-stable/

As always, feedback is appreciated.

Best regards,
Tom Bergfeld
--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .


From rgerhards at hq.adiscon.com  Thu Aug  5 15:45:04 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 5 Aug 2010 15:45:04 +0200
Subject: [rsyslog] spoofing module configuration
References: <alpine.DEB.2.00.1007091435070.14221@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710406C@GRFEXC.intern.adiscon.com>

David,

I have now changed the default:

http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=e4c926bfbca3aab3fe34fc9c
fedb7343423de016

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Friday, July 09, 2010 11:43 PM
> To: rsyslog-users
> Subject: [rsyslog] spoofing module configuration
> 
> in reading the spoofing module configuration it strikes me that the
> defaults can be significantly improved.
> 
> the common case for needing to so spoofing is that you are spoofing the
> original source IP address
> 
> so the current configuration equivalent commands
> 
> $template spoofaddr, "%fromhost-ip%"
> $ActionUDPSpoofSourceNameTemplate spoofaddr
> 
> could be made the default (or call it RSYSLOG_spoofaddr to keep from
> polluting the namespace) and the result would be far simpler for people
> to
> configure, becomging simply
> 
> $modload omudpspoof
> $ActionUDPSpoofTargetHost server.example.com
> *.* :omudpspoof:
> 
> it could be simplified even further if there was some way to specify
> the
> destination on the action line (like the @ and @@  functions do today,
> could we use @S@ to indicate spoofing?)
> 
> 
> changing the defaults should have no problems with backwards
> compatibility, adding/changing how the desitnation is specified could
> break backward compatibility, but this is a very new piece of
> functionality and the simplification may be worth it (what versions
> have
> this available?)
> 
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Thu Aug  5 17:32:17 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 5 Aug 2010 17:32:17 +0200
Subject: [rsyslog] log forwarding through unix sockets
References: <4C456476.4010703@redhat.com><AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com>
	<4C56A7D8.5020901@redhat.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710406E@GRFEXC.intern.adiscon.com>

Just some more information for anyone interested. I today looked at the
module. First of all, there is a directive to set the hostname
($InputUnixListenSocketHostName) but I also found out that there exist
undocumented functionality to activate hostname parsing. In theory, this is
done by putting a colon in front of the socket name. In practice, there seems
to be a bug that prevents this from working at all (the colon is not
removed). So it probably was good this was not documented ;)

I'll see that I fix that first, so that we have some basic functionality in
place.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Ales Kozumplik
> Sent: Monday, August 02, 2010 1:11 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] log forwarding through unix sockets
> 
> On 07/30/2010 03:28 PM, Ryan Lynch wrote:
> > I like your method, too. And thank you for mentioning 'socat', that's
> > what gave me the idea to go in this direction, in the first place.
> 
> Thanks. To tell you the truth at the end we found a way to forward from
> qemu to a TCP socket, and I am happy I don't have to deal with unix
> sockets any more.
> 
> > Based on my own tests, I believe that 'imuxsock' and 'imudp' use
> > different logic to parse incoming messages. 'imuxsock' always assumes
> 
> That's exactly my feeling about this. I just think one either should be
> able to tell every input module what format should be expected (instead
> of letting rsyslogd try some guessing method), or that every input
> module should by default understand a standard "officially recommended"
> forwarding format.
> 
> Ales
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From david at lang.hm  Thu Aug  5 17:50:12 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 5 Aug 2010 08:50:12 -0700 (PDT)
Subject: [rsyslog] log forwarding through unix sockets
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710406E@GRFEXC.intern.adiscon.com>
References: <4C456476.4010703@redhat.com><AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com>
	<4C56A7D8.5020901@redhat.com>
	<9B6E2A8877C38245BFB15CC491A11DA710406E@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.2.00.1008050849340.25170@asgard.lang.hm>

On Thu, 5 Aug 2010, Rainer Gerhards wrote:

> Just some more information for anyone interested. I today looked at the
> module. First of all, there is a directive to set the hostname
> ($InputUnixListenSocketHostName) but I also found out that there exist
> undocumented functionality to activate hostname parsing. In theory, this is
> done by putting a colon in front of the socket name. In practice, there seems
> to be a bug that prevents this from working at all (the colon is not
> removed). So it probably was good this was not documented ;)

so if you use a socket name that starts with a colon it should work right? 
;-)

David Lang

> I'll see that I fix that first, so that we have some basic functionality in
> place.
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Ales Kozumplik
>> Sent: Monday, August 02, 2010 1:11 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] log forwarding through unix sockets
>>
>> On 07/30/2010 03:28 PM, Ryan Lynch wrote:
>>> I like your method, too. And thank you for mentioning 'socat', that's
>>> what gave me the idea to go in this direction, in the first place.
>>
>> Thanks. To tell you the truth at the end we found a way to forward from
>> qemu to a TCP socket, and I am happy I don't have to deal with unix
>> sockets any more.
>>
>>> Based on my own tests, I believe that 'imuxsock' and 'imudp' use
>>> different logic to parse incoming messages. 'imuxsock' always assumes
>>
>> That's exactly my feeling about this. I just think one either should be
>> able to tell every input module what format should be expected (instead
>> of letting rsyslogd try some guessing method), or that every input
>> module should by default understand a standard "officially recommended"
>> forwarding format.
>>
>> Ales
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Thu Aug  5 17:51:41 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 5 Aug 2010 17:51:41 +0200
Subject: [rsyslog] log forwarding through unix sockets
References: <4C456476.4010703@redhat.com><AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com><4C56A7D8.5020901@redhat.com><9B6E2A8877C38245BFB15CC491A11DA710406E@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.00.1008050849340.25170@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104070@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Thursday, August 05, 2010 5:50 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] log forwarding through unix sockets
> 
> On Thu, 5 Aug 2010, Rainer Gerhards wrote:
> 
> > Just some more information for anyone interested. I today looked at
> the
> > module. First of all, there is a directive to set the hostname
> > ($InputUnixListenSocketHostName) but I also found out that there
> exist
> > undocumented functionality to activate hostname parsing. In theory,
> this is
> > done by putting a colon in front of the socket name. In practice,
> there seems
> > to be a bug that prevents this from working at all (the colon is not
> > removed). So it probably was good this was not documented ;)
> 
> so if you use a socket name that starts with a colon it should work
> right?
> ;-)

indeed... ;) [but I've not yet tested it, right now thinking about creating
an automated test...]

Rainer


From rgerhards at hq.adiscon.com  Fri Aug  6 17:53:04 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 6 Aug 2010 17:53:04 +0200
Subject: [rsyslog] log forwarding through unix sockets
References: <4C456476.4010703@redhat.com><AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com><4C56A7D8.5020901@redhat.com><9B6E2A8877C38245BFB15CC491A11DA710406E@GRFEXC.intern.adiscon.com><alpine.DEB.2.00.1008050849340.25170@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7104070@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710407E@GRFEXC.intern.adiscon.com>

As a first step (and because I needed something to do decent testing ;)) I
have implemented a new module omuxsock which now can natively write to unix
domain sockets. That should close part of the gap that was discussed here.

I hope I will find time to look at imuxsock and the hostname parsing code
next week (but that will be *very* busy).

Doc:
http://www.rsyslog.com/doc/omuxsock.html

v4 Commit (v5 also already merged):
http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=80ff634c841d692c1d9f335b
88e225d6ce7317f7

I plan to remove this plugin next week with the upcoming v5 beta.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Thursday, August 05, 2010 5:52 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] log forwarding through unix sockets
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> > Sent: Thursday, August 05, 2010 5:50 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] log forwarding through unix sockets
> >
> > On Thu, 5 Aug 2010, Rainer Gerhards wrote:
> >
> > > Just some more information for anyone interested. I today looked at
> > the
> > > module. First of all, there is a directive to set the hostname
> > > ($InputUnixListenSocketHostName) but I also found out that there
> > exist
> > > undocumented functionality to activate hostname parsing. In theory,
> > this is
> > > done by putting a colon in front of the socket name. In practice,
> > there seems
> > > to be a bug that prevents this from working at all (the colon is
> not
> > > removed). So it probably was good this was not documented ;)
> >
> > so if you use a socket name that starts with a colon it should work
> > right?
> > ;-)
> 
> indeed... ;) [but I've not yet tested it, right now thinking about
> creating
> an automated test...]
> 
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Fri Aug  6 17:55:10 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 6 Aug 2010 17:55:10 +0200
Subject: [rsyslog] log forwarding through unix sockets
References: <4C456476.4010703@redhat.com><AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com><4C56A7D8.5020901@redhat.com><9B6E2A8877C38245BFB15CC491A11DA710406E@GRFEXC.intern.adiscon.com><alpine.DEB.2.00.1008050849340.25170@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA7104070@GRFEXC.intern.adiscon.com>
	<9B6E2A8877C38245BFB15CC491A11DA710407E@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710407F@GRFEXC.intern.adiscon.com>

> I plan to remove this plugin next week with the upcoming v5 beta.

lol: "release", of course, NOT "remove" ;)

Rainer


From shdashbeta at gmail.com  Sat Aug  7 01:39:02 2010
From: shdashbeta at gmail.com (S H)
Date: Fri, 6 Aug 2010 19:39:02 -0400
Subject: [rsyslog] Tuning for performance
Message-ID: <AANLkTimb_hVcG8DvFkC0cmOSP3w23X-g=s-nON56a+ge@mail.gmail.com>

Hello,

I'm newish to the world of rsyslog. I've used it for regular
syslogging stuff with dynamically generated filenames and the like,
but I've never gone in depth with its configuration. Now, however, I'm
working on a project that will involve very high message rates and am
trying to figure out how to tune the system for the kind of throughput
(>200k/sec) documented at
http://mperedim.wordpress.com/2010/01/21/rsyslog-evaluation/

I have a pair of hardware test servers outfitted with dual quad-core
Xeon processors and 8GB of RAM. The network connection is only
100Mbit, but that doesn't seem to be my bottleneck yet. I'm using
loggen for my tests. The servers are running rsyslog 5.5.6.

serverB is the one listening for connections. It's using the
configuration pasted below.

serverA is the one running the test:
# loggen --verbose -r 20000 -I 10 -s 512 -S 207.150.202.100 10514
average rate = 5663.95 msg/sec, count=56644, time=10.007, (last) msg
size=512, bandwidth=2831.98 kB/sec

Increasing the rate (-r) doesn't change the average rate. Switching to
UDP or performing the test on localhost yield very similar results.
What's really strange is that I left the servers alone for about an
hour to work on another project and when I came back the rates were
roughly double - 10-13k/sec. As I tested, however, they gradually fell
back to the 5-6k levels you see here. Restarting the rsyslog process
makes no difference. So I've tuned something incorrectly but I have no
idea what.

iperf shows 100Mbit between the servers. I can double or halve the
message size without affecting the rate, so actual message rate is the
problem -- not bandwidth.

Any help would be wonderful.

-SH

# rsyslog.conf:

$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022

# different rulesets even though I haven't seen any performance effects
$RuleSet remote10514
$RulesetCreateMainQueue on # create ruleset-specific queue
$MainMsgQueueSize 100000
$MainMsgQueueDequeueBatchSize 1024
$RepeatedMsgReduction off

*.* /dev/null
& ~

$ModLoad imtcp
$InputTCPServerBindRuleset remote10514
$InputTCPServerRun 10514


$RuleSet RSYSLOG_DefaultRuleset
$RepeatedMsgReduction off

*.* /var/log/test.log


From rgerhards at hq.adiscon.com  Sat Aug  7 10:14:18 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Sat, 7 Aug 2010 10:14:18 +0200
Subject: [rsyslog] Tuning for performance
References: <AANLkTimb_hVcG8DvFkC0cmOSP3w23X-g=s-nON56a+ge@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104080@GRFEXC.intern.adiscon.com>

This sounds very strange, even the early v4 version could work at higher
rates. Do you use 512 byte messages, only? Could you start with a very basic
rsyslog.conf to get a baseline? Just loading imtcp, starting the listener,
and a single 

*.* /path/to/some/file

Config.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of S H
> Sent: Saturday, August 07, 2010 1:39 AM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] Tuning for performance
> 
> Hello,
> 
> I'm newish to the world of rsyslog. I've used it for regular
> syslogging stuff with dynamically generated filenames and the like,
> but I've never gone in depth with its configuration. Now, however, I'm
> working on a project that will involve very high message rates and am
> trying to figure out how to tune the system for the kind of throughput
> (>200k/sec) documented at
> http://mperedim.wordpress.com/2010/01/21/rsyslog-evaluation/
> 
> I have a pair of hardware test servers outfitted with dual quad-core
> Xeon processors and 8GB of RAM. The network connection is only
> 100Mbit, but that doesn't seem to be my bottleneck yet. I'm using
> loggen for my tests. The servers are running rsyslog 5.5.6.
> 
> serverB is the one listening for connections. It's using the
> configuration pasted below.
> 
> serverA is the one running the test:
> # loggen --verbose -r 20000 -I 10 -s 512 -S 207.150.202.100 10514
> average rate = 5663.95 msg/sec, count=56644, time=10.007, (last) msg
> size=512, bandwidth=2831.98 kB/sec
> 
> Increasing the rate (-r) doesn't change the average rate. Switching to
> UDP or performing the test on localhost yield very similar results.
> What's really strange is that I left the servers alone for about an
> hour to work on another project and when I came back the rates were
> roughly double - 10-13k/sec. As I tested, however, they gradually fell
> back to the 5-6k levels you see here. Restarting the rsyslog process
> makes no difference. So I've tuned something incorrectly but I have no
> idea what.
> 
> iperf shows 100Mbit between the servers. I can double or halve the
> message size without affecting the rate, so actual message rate is the
> problem -- not bandwidth.
> 
> Any help would be wonderful.
> 
> -SH
> 
> # rsyslog.conf:
> 
> $FileOwner syslog
> $FileGroup adm
> $FileCreateMode 0640
> $DirCreateMode 0755
> $Umask 0022
> 
> # different rulesets even though I haven't seen any performance effects
> $RuleSet remote10514
> $RulesetCreateMainQueue on # create ruleset-specific queue
> $MainMsgQueueSize 100000
> $MainMsgQueueDequeueBatchSize 1024
> $RepeatedMsgReduction off
> 
> *.* /dev/null
> & ~
> 
> $ModLoad imtcp
> $InputTCPServerBindRuleset remote10514
> $InputTCPServerRun 10514
> 
> 
> $RuleSet RSYSLOG_DefaultRuleset
> $RepeatedMsgReduction off
> 
> *.* /var/log/test.log
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From shdashbeta at gmail.com  Sat Aug  7 17:10:46 2010
From: shdashbeta at gmail.com (S H)
Date: Sat, 7 Aug 2010 11:10:46 -0400
Subject: [rsyslog] Tuning for performance
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7104080@GRFEXC.intern.adiscon.com>
References: <AANLkTimb_hVcG8DvFkC0cmOSP3w23X-g=s-nON56a+ge@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7104080@GRFEXC.intern.adiscon.com>
Message-ID: <AANLkTikhRY6iEowuRJ0hmMUX28-20y++Vd7tYSXHQU2j@mail.gmail.com>

I think the problem is with loggen, or at least the version I'm using.
I wrote a simple little utility in Python to exercise the native
syslog() calls and was seeing about 240k/second with a simple
configuration.

Once I've got more detailed benchmarks, I'll post them to this thread.
Thanks for the help.

-SH

On Sat, Aug 7, 2010 at 4:14 AM, Rainer Gerhards
<rgerhards at hq.adiscon.com> wrote:
> This sounds very strange, even the early v4 version could work at higher
> rates. Do you use 512 byte messages, only? Could you start with a very basic
> rsyslog.conf to get a baseline? Just loading imtcp, starting the listener,
> and a single
>
> *.* /path/to/some/file
>
> Config.
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of S H
>> Sent: Saturday, August 07, 2010 1:39 AM
>> To: rsyslog at lists.adiscon.com
>> Subject: [rsyslog] Tuning for performance
>>
>> Hello,
>>
>> I'm newish to the world of rsyslog. I've used it for regular
>> syslogging stuff with dynamically generated filenames and the like,
>> but I've never gone in depth with its configuration. Now, however, I'm
>> working on a project that will involve very high message rates and am
>> trying to figure out how to tune the system for the kind of throughput
>> (>200k/sec) documented at
>> http://mperedim.wordpress.com/2010/01/21/rsyslog-evaluation/
>>
>> I have a pair of hardware test servers outfitted with dual quad-core
>> Xeon processors and 8GB of RAM. The network connection is only
>> 100Mbit, but that doesn't seem to be my bottleneck yet. I'm using
>> loggen for my tests. The servers are running rsyslog 5.5.6.
>>
>> serverB is the one listening for connections. It's using the
>> configuration pasted below.
>>
>> serverA is the one running the test:
>> # loggen --verbose -r 20000 -I 10 -s 512 -S 207.150.202.100 10514
>> average rate = 5663.95 msg/sec, count=56644, time=10.007, (last) msg
>> size=512, bandwidth=2831.98 kB/sec
>>
>> Increasing the rate (-r) doesn't change the average rate. Switching to
>> UDP or performing the test on localhost yield very similar results.
>> What's really strange is that I left the servers alone for about an
>> hour to work on another project and when I came back the rates were
>> roughly double - 10-13k/sec. As I tested, however, they gradually fell
>> back to the 5-6k levels you see here. Restarting the rsyslog process
>> makes no difference. So I've tuned something incorrectly but I have no
>> idea what.
>>
>> iperf shows 100Mbit between the servers. I can double or halve the
>> message size without affecting the rate, so actual message rate is the
>> problem -- not bandwidth.
>>
>> Any help would be wonderful.
>>
>> -SH
>>
>> # rsyslog.conf:
>>
>> $FileOwner syslog
>> $FileGroup adm
>> $FileCreateMode 0640
>> $DirCreateMode 0755
>> $Umask 0022
>>
>> # different rulesets even though I haven't seen any performance effects
>> $RuleSet remote10514
>> $RulesetCreateMainQueue on # create ruleset-specific queue
>> $MainMsgQueueSize 100000
>> $MainMsgQueueDequeueBatchSize 1024
>> $RepeatedMsgReduction off
>>
>> *.* /dev/null
>> & ~
>>
>> $ModLoad imtcp
>> $InputTCPServerBindRuleset remote10514
>> $InputTCPServerRun 10514
>>
>>
>> $RuleSet RSYSLOG_DefaultRuleset
>> $RepeatedMsgReduction off
>>
>> *.* /var/log/test.log
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From david at lang.hm  Sun Aug  8 02:46:11 2010
From: david at lang.hm (david at lang.hm)
Date: Sat, 7 Aug 2010 17:46:11 -0700 (PDT)
Subject: [rsyslog] Tuning for performance
In-Reply-To: <AANLkTikhRY6iEowuRJ0hmMUX28-20y++Vd7tYSXHQU2j@mail.gmail.com>
References: <AANLkTimb_hVcG8DvFkC0cmOSP3w23X-g=s-nON56a+ge@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7104080@GRFEXC.intern.adiscon.com>
	<AANLkTikhRY6iEowuRJ0hmMUX28-20y++Vd7tYSXHQU2j@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1008071743560.22036@asgard.lang.hm>

I've also had problems generating enough traffic to stress rsyslog, what I 
ended up doing is using tcpreplay. I send out sample data, capture it with 
tcpdump, and then use tcpreplay to replay the packets over the network at 
different speeds.

this only works well for UDP packets (as tcpreplay does not maintain all 
the state needed to do a real tcp connection.)

David Lang

  On Sat, 7 Aug 2010, S H wrote:

> Date: Sat, 7 Aug 2010 11:10:46 -0400
> From: S H <shdashbeta at gmail.com>
> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> To: rsyslog-users <rsyslog at lists.adiscon.com>
> Subject: Re: [rsyslog] Tuning for performance
> 
> I think the problem is with loggen, or at least the version I'm using.
> I wrote a simple little utility in Python to exercise the native
> syslog() calls and was seeing about 240k/second with a simple
> configuration.
>
> Once I've got more detailed benchmarks, I'll post them to this thread.
> Thanks for the help.
>
> -SH
>
> On Sat, Aug 7, 2010 at 4:14 AM, Rainer Gerhards
> <rgerhards at hq.adiscon.com> wrote:
>> This sounds very strange, even the early v4 version could work at higher
>> rates. Do you use 512 byte messages, only? Could you start with a very basic
>> rsyslog.conf to get a baseline? Just loading imtcp, starting the listener,
>> and a single
>>
>> *.* /path/to/some/file
>>
>> Config.
>>
>> Rainer
>>
>>> -----Original Message-----
>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>> bounces at lists.adiscon.com] On Behalf Of S H
>>> Sent: Saturday, August 07, 2010 1:39 AM
>>> To: rsyslog at lists.adiscon.com
>>> Subject: [rsyslog] Tuning for performance
>>>
>>> Hello,
>>>
>>> I'm newish to the world of rsyslog. I've used it for regular
>>> syslogging stuff with dynamically generated filenames and the like,
>>> but I've never gone in depth with its configuration. Now, however, I'm
>>> working on a project that will involve very high message rates and am
>>> trying to figure out how to tune the system for the kind of throughput
>>> (>200k/sec) documented at
>>> http://mperedim.wordpress.com/2010/01/21/rsyslog-evaluation/
>>>
>>> I have a pair of hardware test servers outfitted with dual quad-core
>>> Xeon processors and 8GB of RAM. The network connection is only
>>> 100Mbit, but that doesn't seem to be my bottleneck yet. I'm using
>>> loggen for my tests. The servers are running rsyslog 5.5.6.
>>>
>>> serverB is the one listening for connections. It's using the
>>> configuration pasted below.
>>>
>>> serverA is the one running the test:
>>> # loggen --verbose -r 20000 -I 10 -s 512 -S 207.150.202.100 10514
>>> average rate = 5663.95 msg/sec, count=56644, time=10.007, (last) msg
>>> size=512, bandwidth=2831.98 kB/sec
>>>
>>> Increasing the rate (-r) doesn't change the average rate. Switching to
>>> UDP or performing the test on localhost yield very similar results.
>>> What's really strange is that I left the servers alone for about an
>>> hour to work on another project and when I came back the rates were
>>> roughly double - 10-13k/sec. As I tested, however, they gradually fell
>>> back to the 5-6k levels you see here. Restarting the rsyslog process
>>> makes no difference. So I've tuned something incorrectly but I have no
>>> idea what.
>>>
>>> iperf shows 100Mbit between the servers. I can double or halve the
>>> message size without affecting the rate, so actual message rate is the
>>> problem -- not bandwidth.
>>>
>>> Any help would be wonderful.
>>>
>>> -SH
>>>
>>> # rsyslog.conf:
>>>
>>> $FileOwner syslog
>>> $FileGroup adm
>>> $FileCreateMode 0640
>>> $DirCreateMode 0755
>>> $Umask 0022
>>>
>>> # different rulesets even though I haven't seen any performance effects
>>> $RuleSet remote10514
>>> $RulesetCreateMainQueue on # create ruleset-specific queue
>>> $MainMsgQueueSize 100000
>>> $MainMsgQueueDequeueBatchSize 1024
>>> $RepeatedMsgReduction off
>>>
>>> *.* /dev/null
>>> & ~
>>>
>>> $ModLoad imtcp
>>> $InputTCPServerBindRuleset remote10514
>>> $InputTCPServerRun 10514
>>>
>>>
>>> $RuleSet RSYSLOG_DefaultRuleset
>>> $RepeatedMsgReduction off
>>>
>>> *.* /var/log/test.log
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From tbergfeld at hq.adiscon.com  Mon Aug  9 14:47:50 2010
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Mon, 9 Aug 2010 14:47:50 +0200
Subject: [rsyslog] rsyslog 5.5.7 (v5-beta) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104091@GRFEXC.intern.adiscon.com>

Hi all,

We have just released rsyslog 5.5.7, the new v5-beta.
This is a new beta version, scheduled to replace the current stable hopefully
soon. It has some bugfixes over 5.5.6, and also introduces a new module,
omuxsock, that provides the ability to natively write to Unix domain sockets.
Feedback on this version is highly appreciated. We would like to promote it
to stable by mid to end September if we do not hear anything that prevents us
from doing so.
See ChangeLog for more details.

ChangeLog:

http://www.rsyslog.com/changelog-for-5-5-7-v5-beta/

Download:

http://www.rsyslog.com/rsyslog-5-5-7-v5-beta/

As always, feedback is appreciated.

Best regards,
Tom Bergfeld
--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .


From joe.smithian at gmail.com  Tue Aug 10 19:19:00 2010
From: joe.smithian at gmail.com (Joe Smithian)
Date: Tue, 10 Aug 2010 13:19:00 -0400
Subject: [rsyslog] rsyslog truncates host name in the log messages if host
	name contains dots
In-Reply-To: <mailman.434.1281448570.2883.rsyslog@lists.adiscon.com>
References: <mailman.434.1281448570.2883.rsyslog@lists.adiscon.com>
Message-ID: <AANLkTi=oduvpO4P1jY9kb9FGRw5SQYXfZy1HP4-4eGd_@mail.gmail.com>

Hello,

If Linux hostname contains dots "." e.g. an IP address : 192.168.1.1 then
rsyslog prints "192" as hostname in the log messages:

$template MySyslFormat2,"%$YEAR% %timegenerated% *%HOSTNAME%*
%syslogfacility-text% %syslogseverity-text%  %syslogtag%%msg%\n"


2010 Aug 10 09:36:02 *192* kern info  kernel: Kernel logging (proc) stopped.


Is there any way to change the rsyslog behavior to show the full hostname if
it contains dots "."?


Thanks

Joe


From rgerhards at hq.adiscon.com  Wed Aug 11 12:01:02 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 11 Aug 2010 12:01:02 +0200
Subject: [rsyslog] help requested -- connect() performance
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD1F1@GRFEXC.intern.adiscon.com>

Hi all,

I yesterday stumbled over behavior that I cannot explain. Mabye someone here
on the list has a hint.

There is a testing tool "tcpflood", which I use to carry out the tcp based
tests inside the automated testbench. That tool has evolved over time and has
many options. One option is to open a large number of connections and then
send data. I have seen that when I request 1,000 connections, it takes quite
some while to open them (with the CPU being almost idle during that time). So
far, I did not try to dig into this as the only bad effect it causes is that
the testbench runs a little bit longer.

Yesterday, for some other reason, I ran the very same tool, with the very
same options under strace. And I was totally stunned to see that the
connect() calls were now finished in almost no time. When I removed strace,
it took long again (30..60 seconds, not exactly measured). If I ran it under
strace, maybe a second or two. This is totally reproducible.

Does anybody have a clue?

Thanks,
Rainer

PS: if someone is interested in reproducing this (or the details): the test
in question is ./tests/manytcp.sh


From tbergfeld at hq.adiscon.com  Thu Aug 12 14:47:47 2010
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Thu, 12 Aug 2010 14:47:47 +0200
Subject: [rsyslog] rsyslog 6.1.0 (devel) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD20B@GRFEXC.intern.adiscon.com>

Hi all,

We have just released rsyslog 6.1.0, the new v6-devel.
This is the first v6 release of rsyslog, beginning a new development focus.
In v6, configuration shall become much more straigthforward. This release
already offers action scoping, which keeps related things closer together.
Other than that, it is roughly functionally equivalent to the latest v5
releases. However, there is imptcp available, a Linux-optimized plain tcp
input which in theory should perform faster than imtcp (at the price of some
reduced functionality).

Note that v6 requieres modifications to pre v6-plugins in order to work.

ChangeLog:

http://www.rsyslog.com/changelog-for-6-1-0-devel/

Download:

http://www.rsyslog.com/rsyslog-6-1-0-v6-devel/

As always, feedback is appreciated.

Best regards,
Tom Bergfeld
--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .



From sivan at omniqueue.com  Sun Aug 22 15:17:50 2010
From: sivan at omniqueue.com (Sivan Greenberg)
Date: Sun, 22 Aug 2010 16:17:50 +0300
Subject: [rsyslog] Setting up rsyslog 4.6.4 with RELP *and* TLS
Message-ID: <AANLkTinzqWFW+uyP0mS-XkrOiKgoaGNUDF75h-GsQ4Jh@mail.gmail.com>

Hi List,

 I've successfully built 4.6.4 with support for TLS and RELP. When I
worked with the imtcp module everything was fine and setting TLS to be
enabled through a snippet like this:


$DefaultNetstreamDriver gtls
$DefaultNetstreamDriverCAFile /root/tls/ca.pem
$DefaultNetstreamDriverCertFile /root/tls/client-cert.pem
$DefaultNetstreamDriverKeyFile /root/tls/client-key.pem
$ModLoad imuxsock
$ModLoad omrelp
$ModLoad imtcp

$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer server.example.com
$ActionSendStreamDriverMode 1
*.* @@:omrelp:server.example.com:10514


----

However, when I restart on the client and use logger to send a line
nothing happens, there's also no error reported or anything similar on
the logging output created by rsyslog.

This is the server snippet:

$ModLoad imuxsock
$Modload imtcp
$Modload imrelp
$DefaultNetstreamDriver gtls

$DefaultNetstreamDriverCAFile /root/tls/ca.pem
$DefaultNetstreamDriverCertFile /root/tls/server-cert.pem
$DefaultNetstreamDriverKeyFile /root/tls/server-key.pem

$InputTCPServerStreamDriverAuthMode x509/name
$InputTCPServerStreamDriverPermittedPeer *.example.com
$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
$InputRELPServerRun 10514 # start up listener at port 10514
$InputTCPServerRun 10514


I realize that I might have to add the:
RSYSLOG_ForwardFormat to the forward line, but given that I do add it
and it works without TLS, what is the way to enable it to use TLS
together with RELP ?

Any feedback will be highly appreciated!

Thanks,

Sivan


From joe at joetify.com  Thu Aug 26 16:42:45 2010
From: joe at joetify.com (Joe Williams)
Date: Thu, 26 Aug 2010 07:42:45 -0700
Subject: [rsyslog] symlinks
Message-ID: <DD10BDA2-03BA-4D5B-B299-38D41D6FD314@joetify.com>


I have some hostnamed directories and would like to symlink to them to something shorter and more meaningful. Is it possible to have rsyslog create symlinks to make certain logs easier to get too? Alternatively, can I configure rsyslog to match on certain hostnames and instead of using the hostname use a directory name I choose.

Thanks.

-Joe



Name: Joseph A. Williams
Email: joe at joetify.com
Blog: http://www.joeandmotorboat.com/
Twitter: http://twitter.com/williamsjoe



From gravyface at gmail.com  Thu Aug 26 18:12:09 2010
From: gravyface at gmail.com (GravyFace)
Date: Thu, 26 Aug 2010 12:12:09 -0400
Subject: [rsyslog] Should ActionExecOnlyOnceEveryInterval be working in 4.2?
Message-ID: <AANLkTikW4smA5e3LZQP0W7qCim8cveCBNZ9qCJ0CVJDY@mail.gmail.com>

>From the changelogs, it sounds like it was fixed in 4.4.3, and then
included in 4.6.x I believe.

Ubuntu 10.04 ships with 4.2, but in my config, it's not working --
emails get sent out for each match on :msg, regardless of
ActionExecOnlyOnceEveryInterval.

Config as follows:

$ActionMailSMTPServer 127.0.0.1
$ActionMailFrom root at example.com
$ActionMailTo somebody at example.com
$template mailSubject,"Alert: Inbound Packet Denied"
$template mailBody,"DETAILS:\r\n%timereported% %msg%"
$ActionMailSubject mailSubject
# make sure we receive a mail only once per hour
$ActionExecOnlyOnceEveryInterval 3600
:msg, contains, "Deny tcp src inside" :ommail:;mailBody


From sivan at omniqueue.com  Sun Aug 29 10:43:51 2010
From: sivan at omniqueue.com (Sivan Greenberg)
Date: Sun, 29 Aug 2010 11:43:51 +0300
Subject: [rsyslog] Reducing the time a msg spends in the main unreliable
	main queue to a minimum.
Message-ID: <AANLkTi=FcoYYG7LHGF=Mp7epjQ3=YNmdJkMqeoSNR+SC@mail.gmail.com>

Hi List,

 I have some critical msgs being transferred using rsyslog. Hence, I
would like to minimize the time a msg spends in the unreliable by
definition main queue. My understanding is that only action queues
which are where msgs arrive when dispatched from main queue  can be
set to be disk only. Therefor my assumption that only they can be made
truly reliable.

Any suggestions / feedback will be highly appreciated.

-Sivan


From rgerhards at hq.adiscon.com  Mon Aug 30 08:19:21 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 30 Aug 2010 08:19:21 +0200
Subject: [rsyslog] Reducing the time a msg spends in the main
	unreliablemain queue to a minimum.
References: <AANLkTi=FcoYYG7LHGF=Mp7epjQ3=YNmdJkMqeoSNR+SC@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD275@GRFEXC.intern.adiscon.com>

Hi Sivan,

the main message queue supports the same set of properties and capabilities
like the action queue. The directives just start with $MainMsg... instead of
$Action...

So you can set the main queue to pure disk mode as well.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Sivan Greenberg
> Sent: Sunday, August 29, 2010 10:44 AM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] Reducing the time a msg spends in the main
> unreliablemain queue to a minimum.
> 
> Hi List,
> 
>  I have some critical msgs being transferred using rsyslog. Hence, I
> would like to minimize the time a msg spends in the unreliable by
> definition main queue. My understanding is that only action queues
> which are where msgs arrive when dispatched from main queue  can be
> set to be disk only. Therefor my assumption that only they can be made
> truly reliable.
> 
> Any suggestions / feedback will be highly appreciated.
> 
> -Sivan
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From sivan at omniqueue.com  Mon Aug 30 08:33:13 2010
From: sivan at omniqueue.com (Sivan Greenberg)
Date: Mon, 30 Aug 2010 09:33:13 +0300
Subject: [rsyslog] Reducing the time a msg spends in the main
 unreliablemain queue to a minimum.
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD275@GRFEXC.intern.adiscon.com>
References: <AANLkTi=FcoYYG7LHGF=Mp7epjQ3=YNmdJkMqeoSNR+SC@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA71DD275@GRFEXC.intern.adiscon.com>
Message-ID: <AANLkTinY7CMNap1cQrPS5JevbkN4VEa3qdcjD23VtBkO@mail.gmail.com>

Hi Rainer,

I see, however, that would cause a major performance downgrade
thinking of it, and filtering seems to be only done on the main queue,
so to be able to use the same rsyslog instance for both critical and
non critical stuff I think I might be better leaving it buffered to
some extent (is there a directive to define how long/ how much memory)
and filter critical msgs to the pure disk msg queue and the rest.

Do you think this can be done? Could you kindly give pointer how to
configure for this?

Many thanks!

-Sivan

On Mon, Aug 30, 2010 at 9:19 AM, Rainer Gerhards
<rgerhards at hq.adiscon.com> wrote:
> Hi Sivan,
>
> the main message queue supports the same set of properties and capabilities
> like the action queue. The directives just start with $MainMsg... instead of
> $Action...
>
> So you can set the main queue to pure disk mode as well.
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Sivan Greenberg
>> Sent: Sunday, August 29, 2010 10:44 AM
>> To: rsyslog at lists.adiscon.com
>> Subject: [rsyslog] Reducing the time a msg spends in the main
>> unreliablemain queue to a minimum.
>>
>> Hi List,
>>
>> ?I have some critical msgs being transferred using rsyslog. Hence, I
>> would like to minimize the time a msg spends in the unreliable by
>> definition main queue. My understanding is that only action queues
>> which are where msgs arrive when dispatched from main queue ?can be
>> set to be disk only. Therefor my assumption that only they can be made
>> truly reliable.
>>
>> Any suggestions / feedback will be highly appreciated.
>>
>> -Sivan
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Mon Aug 30 12:38:39 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 30 Aug 2010 12:38:39 +0200
Subject: [rsyslog] Reducing the time a msg spends in the main
	unreliablemain queue to a minimum.
References: <AANLkTi=FcoYYG7LHGF=Mp7epjQ3=YNmdJkMqeoSNR+SC@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA71DD275@GRFEXC.intern.adiscon.com>
	<AANLkTinY7CMNap1cQrPS5JevbkN4VEa3qdcjD23VtBkO@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD278@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Sivan Greenberg
> Sent: Monday, August 30, 2010 8:33 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Reducing the time a msg spends in the main
> unreliablemain queue to a minimum.
> 
> Hi Rainer,
> 
> I see, however, that would cause a major performance downgrade

Definitely, very, very severe (you can't get all at once in life ;))

> thinking of it, and filtering seems to be only done on the main queue,
> so to be able to use the same rsyslog instance for both critical and
> non critical stuff I think I might be better leaving it buffered to
> some extent (is there a directive to define how long/ how much memory)

There are many. I suggest you read the queue doc and experiment with some oft
he settings. This is pretty complicated stuff (because you intend to do
complicated things), so it is vital to get good working knowledge before
deploying it to production. Queue doc is:

http://www.rsyslog.com/doc/queues.html

There are also a number of tutorials and descriptions on the workflow inside
the doc set. I'd personally expect that you need at least 2 to 3 days of
careful studying before you get a sufficient idea of how to configure a very
demanding environment.

> and filter critical msgs to the pure disk msg queue and the rest.
> 
> Do you think this can be done?

Yes

> Could you kindly give pointer how to
> configure for this?

As I said, very high reliability is rather complicated, it's a real
consulting job. So, sorry, I don't have a canned response. First step would
be to look at the actual needs, I think you already did this for most of the
part.

For example, if you can submit messages from the originator to the receiver
based on importance, you could use multiple rulesets and use a dedicated
ruleset using a disk-based queue for those important messages. But as I said
... doing it right involves a lot of planning, far more than setting a few
parameters. Otherwise you often end up with something that looks like it were
reliable, but is not really. You need to look at the whole picture. Once this
is done, you have a spec on which messages are what important and how many
loss is acceptable. From there on, you can work to turn that into a rsyslog
conf.

I recently wrote a blog post that may be worth mentioning for your case:

http://blog.gerhards.net/2010/08/rsyslog-queues-reliability-and.html

HTH
Rainer

> 
> Many thanks!
> 
> -Sivan
> 
> On Mon, Aug 30, 2010 at 9:19 AM, Rainer Gerhards
> <rgerhards at hq.adiscon.com> wrote:
> > Hi Sivan,
> >
> > the main message queue supports the same set of properties and
> capabilities
> > like the action queue. The directives just start with $MainMsg...
> instead of
> > $Action...
> >
> > So you can set the main queue to pure disk mode as well.
> >
> > Rainer
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Sivan Greenberg
> >> Sent: Sunday, August 29, 2010 10:44 AM
> >> To: rsyslog at lists.adiscon.com
> >> Subject: [rsyslog] Reducing the time a msg spends in the main
> >> unreliablemain queue to a minimum.
> >>
> >> Hi List,
> >>
> >> ?I have some critical msgs being transferred using rsyslog. Hence, I
> >> would like to minimize the time a msg spends in the unreliable by
> >> definition main queue. My understanding is that only action queues
> >> which are where msgs arrive when dispatched from main queue ?can be
> >> set to be disk only. Therefor my assumption that only they can be
> made
> >> truly reliable.
> >>
> >> Any suggestions / feedback will be highly appreciated.
> >>
> >> -Sivan
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Sun Aug  1 11:30:17 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Sun, 1 Aug 2010 11:30:17 +0200
Subject: [rsyslog] log forwarding through unix sockets
References: <4C456476.4010703@redhat.com>
	<AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710403E@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Ryan Lynch
> Sent: Friday, July 30, 2010 3:29 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] log forwarding through unix sockets
> 
> On Tue, Jul 20, 2010 at 04:55, Ales Kozumplik <akozumpl at redhat.com>
> wrote:
> > 1) KVM opens a SOCK_STREAM on the host end but rsyslogd is only able
> to
> > read data from SOCK_DGRAM. This has two consequences: first, to be
> able
> > to attach rsyslog on the host end one first needs to copy the data
> > between the two socket types, e.g. using socat. Second, messages
> longer
> > than 1024 characters are sometimes split into two. The second message
> is
> > thus missing the syslog header and the receiving rsyslogd doesn't
> know
> > where to file it. Is there a recommended workaround for those things
> > (maybe a parameter I overlooked in the docs tellling rsyslogd to use
> > SOCK_STREAM)?
> 
> I ran into a similar problem. Doesn't it seem wierd that RSyslog:
>  - can write TO a pipe, but it can't natively read FROM a pipe.
>  - can read FROM a UNIX socket, but can't natively write TO a UNIX
> socket.
> The only protocols that Rsyslog can both read to AND write from are
> network sockets (UDP, TCP, RELP) and real files.
> 

Quite honestly, I think this issue never came up until recently -- at least I
didn't notice. It should be fairly simple to add a plugin to write to unix
sockets, though...
 
> I don't think the problem is your forwarding format--I don't think
> it's possible for RSyslog to handle a HOSTNAME field, properly, in
> messages received via socket.
> 
> Based on my own tests, I believe that 'imuxsock' and 'imudp' use
> different logic to parse incoming messages. 'imuxsock' always assumes
> that the hostname is the local host, so it doesn't have the
> conditional logic to differentiate between "forwarded" messages (which
> have an extra HOSTNAME field between the timestamp and tag) versus
> regular local messages (no HOSTNAME).

The problem is that you do not have any reliable indication what is a
hostname and what not. So far, I worked on the assumption that anything
received via a local socket complies to the "usual format" for such messages.
This means no hostname is present.

However, there is a simple setting (parse hostname yes/no) which is part of
the message object. Imuxsock sets it to "no", remote receivers set it to
"yes". It should be fairly trivial to add a config setting which enables
imuxsock to set it to "yes" as well.

> This is a pretty reasonable
> assumption, really--the local UNIX socket doesn't traditionally have
> any way to receive messages forwarded from other hosts.
> 
> Rainer could probably confirm this, or we could compare those two
> modules' sources.
> 
> RSyslog is a fantastic piece of software, and its feature set has come
> a long way. But all of Rainer's excellent work has expanded our
> imaginations about what Syslog can do, and maybe our imaginations are
> advancing faster than he can code new features.

The problem simply is that we are tackeling more complex things, and this
also means things need more planning than in the past. For example, the
latest round of performance enhancements took me 2 to 3 month of reading
scientific papers and experimenting. This all resulted in a relatively small
delta to the code, but this delta does not judge the work that went into it.
Also, being primarily a lone guy, I need to focus on some things and leave
others untouched -- otherwise I'll never make any progress (ever-swapping
between different features). The current scope is configurability and
performance, so other things have to wait (where I try to fit in small and
useful items as it fits). 

What is being discussed here is small and useful ;) However, my summer break
is approaching, and I'd like to finish a couple of loose ends (first config
iteration, getting a new v4-beta ready). Also I (finally ;)) need to write an
article for the upcoming Linux conference in Nuremberg, with the deadline
rapidly approaching. So these are obviously my priority.

Also I have to admit that I weigh features by their importance to the
community at large. Those often requested receive higher priority, so they
will preempt less requested features from being implemented when they have
similar workload. Also, I prefer features requested/need by non-profits over
those that have a clearly for-profit user base. The reasoning here is that
the non-profit orgs will probably otherwise never get their features, whereas
the enterprises can task someone with implementing such a features. And if an
enterprise doesn't think it is worth tasking someone with implementing a
feature, that is a clear indication that the enterprise doesn't actually
think the feature is worth the effort. So I think there is a lot of logic in
assigning priorities in this way ;)

Rainer


From akozumpl at redhat.com  Mon Aug  2 13:11:20 2010
From: akozumpl at redhat.com (Ales Kozumplik)
Date: Mon, 02 Aug 2010 13:11:20 +0200
Subject: [rsyslog] log forwarding through unix sockets
In-Reply-To: <AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com>
References: <4C456476.4010703@redhat.com>
	<AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com>
Message-ID: <4C56A7D8.5020901@redhat.com>

On 07/30/2010 03:28 PM, Ryan Lynch wrote:
> I like your method, too. And thank you for mentioning 'socat', that's
> what gave me the idea to go in this direction, in the first place.

Thanks. To tell you the truth at the end we found a way to forward from 
qemu to a TCP socket, and I am happy I don't have to deal with unix 
sockets any more.

> Based on my own tests, I believe that 'imuxsock' and 'imudp' use
> different logic to parse incoming messages. 'imuxsock' always assumes

That's exactly my feeling about this. I just think one either should be 
able to tell every input module what format should be expected (instead 
of letting rsyslogd try some guessing method), or that every input 
module should by default understand a standard "officially recommended" 
forwarding format.

Ales


From joel.merrick at gmail.com  Mon Aug  2 13:13:35 2010
From: joel.merrick at gmail.com (Joel Merrick)
Date: Mon, 2 Aug 2010 12:13:35 +0100
Subject: [rsyslog] Problem with mysql template
In-Reply-To: <AANLkTimu-Cm-+_BWE+xRCNO6xDVxt2cf38jAcOiFxZBb@mail.gmail.com>
References: <AANLkTimu-Cm-+_BWE+xRCNO6xDVxt2cf38jAcOiFxZBb@mail.gmail.com>
Message-ID: <AANLkTinbjtOgBXxFz35RKX+oohfhg+-fPU8U_CtkLyL5@mail.gmail.com>

Do I take it from the lack of replies that this should work? :)



On Thu, Jul 29, 2010 at 11:53 AM, Joel Merrick <joel.merrick at gmail.com> wrote:
> Hi all,
>
> I'm trying to create a very quick mail log searching solution. The
> idea is to pre-hash the database by using the last 2 alpha-numeric
> characters on the message id.
>
> This isn't inserting to the database and I can't think why.. I've
> already built the tables with the structure logs-aa.. logs-ZZ etc..
>
>
> $template OurDBLog,"INSERT INTO
> logs-'%msg:R,ERE,1,NULL:[A-Za-z0-9]{6}-[A-Za-z0-9]{6}-([A-Za-z0-9]{2})--end%'
> (messageid, host, \
> ? ? send_host, created_at, payload, subject) values \
> ? ? ('%msg:R,ERE,0,ZERO:[A-Za-z0-9]{6}-[A-Za-z0-9]{6}-[A-Za-z0-9]{2}--end%','%HOSTNAME%',
> '%msg:R,ERE,0,ZERO:H=.*\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}]--end%',
> \
> ? ? '%timegenerated:::date-mysql%', '%msg%',
> '%msg:R,ERE,1,ZERO:T="(.+)"--end%')",SQL
>
>
> Any idea?
>
>
> On a side note, how can I get extra verbosity out of rsyslog so I'm
> not blindly trying to insert and then check via mysql
>
> Cheers,
> Joel
>
>
> --
> $ echo "kpfmAdpoofdufevq/dp/vl" | perl -pe 's/(.)/chr(ord($1)-1)/ge'
>



-- 
$ echo "kpfmAdpoofdufevq/dp/vl" | perl -pe 's/(.)/chr(ord($1)-1)/ge'


From rgerhards at hq.adiscon.com  Mon Aug  2 13:14:41 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 2 Aug 2010 13:14:41 +0200
Subject: [rsyslog] log forwarding through unix sockets
References: <4C456476.4010703@redhat.com><AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com>
	<4C56A7D8.5020901@redhat.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104042@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Ales Kozumplik
> Sent: Monday, August 02, 2010 1:11 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] log forwarding through unix sockets
> > Based on my own tests, I believe that 'imuxsock' and 'imudp' use
> > different logic to parse incoming messages. 'imuxsock' always assumes
> 
> That's exactly my feeling about this. 

I elaborated on that yesterday -- not sure if you saw that posting.

> I just think one either should be
> able to tell every input module what format should be expected (instead
> of letting rsyslogd try some guessing method), or that every input
> module should by default understand a standard "officially recommended"
> forwarding format.

The problem is that syslog() does not emit the format recommended by RFC3164,
so no message provided over that channel includes a hostname. Things would be
heavily broken if I would simply ignore that fact.

Rainer


From rgerhards at hq.adiscon.com  Mon Aug  2 13:16:40 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 2 Aug 2010 13:16:40 +0200
Subject: [rsyslog] Problem with mysql template
References: <AANLkTimu-Cm-+_BWE+xRCNO6xDVxt2cf38jAcOiFxZBb@mail.gmail.com>
	<AANLkTinbjtOgBXxFz35RKX+oohfhg+-fPU8U_CtkLyL5@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104043@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Joel Merrick
> Sent: Monday, August 02, 2010 1:14 PM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] Problem with mysql template
> 
> Do I take it from the lack of replies that this should work? :)

simply overlooked from my side ;)

> 
> On Thu, Jul 29, 2010 at 11:53 AM, Joel Merrick <joel.merrick at gmail.com>
> wrote:
> > Hi all,
> >
> > I'm trying to create a very quick mail log searching solution. The
> > idea is to pre-hash the database by using the last 2 alpha-numeric
> > characters on the message id.
> >
> > This isn't inserting to the database and I can't think why.. I've
> > already built the tables with the structure logs-aa.. logs-ZZ etc..
> >
> >
> > $template OurDBLog,"INSERT INTO
> > logs-'%msg:R,ERE,1,NULL:[A-Za-z0-9]{6}-[A-Za-z0-9]{6}-([A-Za-z0-
> 9]{2})--end%'
> > (messageid, host, \
> > ? ? send_host, created_at, payload, subject) values \
> > ? ? ('%msg:R,ERE,0,ZERO:[A-Za-z0-9]{6}-[A-Za-z0-9]{6}-[A-Za-z0-9]{2}-
> -end%','%HOSTNAME%',
> > '%msg:R,ERE,0,ZERO:H=.*\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-
> 9]{1,3}]--end%',
> > \
> > ? ? '%timegenerated:::date-mysql%', '%msg%',
> > '%msg:R,ERE,1,ZERO:T="(.+)"--end%')",SQL
> >
> >
> > Any idea?
> >
> >
> > On a side note, how can I get extra verbosity out of rsyslog so I'm
> > not blindly trying to insert and then check via mysql

Two ways to proceed: 

a) write to a file with the same template
then you can check if the sql is like you intended it to be

b) run rsyslog in debug mode (doc available in doc set and on site)

It may also be worth trying out the regex checker on the rsyslog site.

Rainer


From akozumpl at redhat.com  Mon Aug  2 13:21:14 2010
From: akozumpl at redhat.com (Ales Kozumplik)
Date: Mon, 02 Aug 2010 13:21:14 +0200
Subject: [rsyslog] log forwarding through unix sockets
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710403E@GRFEXC.intern.adiscon.com>
References: <4C456476.4010703@redhat.com>	<AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA710403E@GRFEXC.intern.adiscon.com>
Message-ID: <4C56AA2A.1010709@redhat.com>

On 08/01/2010 11:30 AM, Rainer Gerhards wrote:
> Also I have to admit that I weigh features by their importance to the
> community at large. Those often requested receive higher priority, so they
> will preempt less requested features from being implemented when they have
> similar workload. Also, I prefer features requested/need by non-profits over
> those that have a clearly for-profit user base. The reasoning here is that
> the non-profit orgs will probably otherwise never get their features, whereas
> the enterprises can task someone with implementing such a features. And if an
> enterprise doesn't think it is worth tasking someone with implementing a
> feature, that is a clear indication that the enterprise doesn't actually
> think the feature is worth the effort. So I think there is a lot of logic in
> assigning priorities in this way ;)
>

Hi Rainer,

I'd just add that the feature is targeted for Fedora 14 and the 
community will benefit because improved installer logging allows the 
Fedora QA to work more efficiently and so make a greater Fedora.

Saying that I agree that some obscure unix socket logging features 
shouldn't be of high priority. Just an idea for a backlog.

Ales


From joel.merrick at gmail.com  Mon Aug  2 13:26:05 2010
From: joel.merrick at gmail.com (Joel Merrick)
Date: Mon, 2 Aug 2010 12:26:05 +0100
Subject: [rsyslog] Problem with mysql template
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7104043@GRFEXC.intern.adiscon.com>
References: <AANLkTimu-Cm-+_BWE+xRCNO6xDVxt2cf38jAcOiFxZBb@mail.gmail.com>
	<AANLkTinbjtOgBXxFz35RKX+oohfhg+-fPU8U_CtkLyL5@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7104043@GRFEXC.intern.adiscon.com>
Message-ID: <AANLkTikMf-oe381Ab8eUg4HtWJzbdQOcye0a+sjqns+S@mail.gmail.com>

On Mon, Aug 2, 2010 at 12:16 PM, Rainer Gerhards
<rgerhards at hq.adiscon.com> wrote:
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Joel Merrick
>> Sent: Monday, August 02, 2010 1:14 PM
>> To: rsyslog at lists.adiscon.com
>> Subject: Re: [rsyslog] Problem with mysql template
>>
>> Do I take it from the lack of replies that this should work? :)
>
> simply overlooked from my side ;)
>

No worries, I know you're very busy!


> Two ways to proceed:
>
> a) write to a file with the same template
> then you can check if the sql is like you intended it to be
>

The rest of the SQL is correct, it's when I add the part that adds the
last 2 alphanumeric characters to the table name when it breaks..


> b) run rsyslog in debug mode (doc available in doc set and on site)
>

Will do, cheers for the tips.

> It may also be worth trying out the regex checker on the rsyslog site.


Yes, used this extensively and I'm getting the matches correct.

>
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>



-- 
$ echo "kpfmAdpoofdufevq/dp/vl" | perl -pe 's/(.)/chr(ord($1)-1)/ge'


From rgerhards at hq.adiscon.com  Mon Aug  2 13:26:42 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 2 Aug 2010 13:26:42 +0200
Subject: [rsyslog] log forwarding through unix sockets
References: <4C456476.4010703@redhat.com>	<AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA710403E@GRFEXC.intern.adiscon.com>
	<4C56AA2A.1010709@redhat.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104044@GRFEXC.intern.adiscon.com>

Hi Ales,

> I'd just add that the feature is targeted for Fedora 14 and the
> community will benefit because improved installer logging allows the
> Fedora QA to work more efficiently and so make a greater Fedora.

that's good info! What's the latest date you need to have the feature?

> Saying that I agree that some obscure unix socket logging features
> shouldn't be of high priority. Just an idea for a backlog.

I don't think it is a lot of work, maybe just 2 days. But I am not sure if I
can begin to work on it before end of August. If work on my conference papers
goes well, I may be able to do it before I leave to vacation.

Rainer


From akozumpl at redhat.com  Mon Aug  2 17:03:27 2010
From: akozumpl at redhat.com (Ales Kozumplik)
Date: Mon, 02 Aug 2010 17:03:27 +0200
Subject: [rsyslog] log forwarding through unix sockets
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7104044@GRFEXC.intern.adiscon.com>
References: <4C456476.4010703@redhat.com>	<AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA710403E@GRFEXC.intern.adiscon.com>	<4C56AA2A.1010709@redhat.com>
	<9B6E2A8877C38245BFB15CC491A11DA7104044@GRFEXC.intern.adiscon.com>
Message-ID: <4C56DE3F.6010408@redhat.com>

On 08/02/2010 01:26 PM, Rainer Gerhards wrote:
> I don't think it is a lot of work, maybe just 2 days. But I am not sure if I
> can begin to work on it before end of August. If work on my conference papers
> goes well, I may be able to do it before I leave to vacation.

Thanks, like I said in a reply earlier today, we made qemu forward the 
logs to a TCP port on the host. That way we avoid all of the problems we 
had, so from Anaconda's point of view rsyslog works just fine at the moment.

Ales


From sean at conman.org  Mon Aug  2 22:46:21 2010
From: sean at conman.org (Sean Conner)
Date: Mon, 2 Aug 2010 16:46:21 -0400
Subject: [rsyslog] log forwarding through unix sockets
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7104042@GRFEXC.intern.adiscon.com>
References: <4C56A7D8.5020901@redhat.com>
	<9B6E2A8877C38245BFB15CC491A11DA7104042@GRFEXC.intern.adiscon.com>
Message-ID: <20100802204621.GA7853@brevard.conman.org>

It was thus said that the Great Rainer Gerhards once stated:
> 
> The problem is that syslog() does not emit the format recommended by RFC3164,
> so no message provided over that channel includes a hostname. Things would be
> heavily broken if I would simply ignore that fact.

  Can't it be added as part of the parsing?  Even if it's just "localhost"? 
Or a parameter to the imuxsock module?  My own syslogd fills in missing
fields with sane (at least to me) defaults.

  -spc 



From david at lang.hm  Tue Aug  3 00:03:11 2010
From: david at lang.hm (david at lang.hm)
Date: Mon, 2 Aug 2010 15:03:11 -0700 (PDT)
Subject: [rsyslog] omspoof module
Message-ID: <alpine.DEB.2.00.1008021459190.29820@asgard.lang.hm>

In trying to use the module omspoof with 5.5.6 it looks like there is a 
problem.

the debug file shows a couple errors (entry points not present in module) 
followed by 'tried selector action for omudpspoof: -2001'

then a couple of lines later when defining 
$ActionUDPSpoofSourceNameTemplate it gets the error 'unkown config, did 
you forget to load a module?'

David Lang
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: debug.txt
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20100802/5e4384f9/attachment-0002.txt>

From alexandre.chapellon at mana.pf  Tue Aug  3 05:26:49 2010
From: alexandre.chapellon at mana.pf (Alexandre Chapellon)
Date: Mon, 02 Aug 2010 17:26:49 -1000
Subject: [rsyslog] replaying QIF files
Message-ID: <1280806009.3863.40.camel@mcbain.mana.lan>

Hello I have rsyslog installed on a server to inject data in postgresql
database.
Few days ago SQL injection stopped (because of a syntax error: bad
template)
As I configured disk Queue for the ompgsql action I have all the non
injected messages in thousans of QIF Files.
Here is the cfg:
##########################
$template LogToPqsql, "insert into blablabla' );", sql
$WorkDirectory /var/spool/rsyslog/work
$ActionQueueType LinkedList
$ActionQueueFileName dbq
$ActionResumeRetryCount -1

local1.* :ompgsql:blablablabla
##########################

Unfortunately after restarting rsyslog, it does not reads the files to
inject it back to pg.
Looking in the work dir I can't see file with the .qi extension.
If I manually rename the first QIF file to dbq.qi, rsyslog crash with a
segfault.
Then I tried installing 4.6.3 instead of my old 4.4.2. Rsyslog do not
crash anymore but complains about something wrong with my QIF file (i
guess):

##########################
...
5150.884531712:40800950: deserializer has possibly been able to re-sync
and recover, state 0
5150.884531712:40800950: objDeserializePropBag error -2037 during header
- trying to recover
5150.884531712:40800950: strm 0x689a10: file 10 read 421 bytes
5150.884531712:40800950: strm 0x689a10: file 10 read 0 bytes
5150.884531712:40800950: deserializer has possibly been able to re-sync
and recover, state -2026
5150.884531712:40800950: strm 0x689a10: file
10(/var/spool/rsyslog/work/dbq.qi) closing
5150.884531712:40800950: action 3 queue[DA]: error -2026 reading .qi
file - can not read persisted info (if any)
5150.884531712:40800950: action 3 queue[DA]: persisting queue to disk, 0
entries...
5150.884531712:40800950: action 3 queue: error -2026 creating disk queue
- giving up.
...
##########################

Does anyone have any idea?



From rgerhards at hq.adiscon.com  Tue Aug  3 07:56:57 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 3 Aug 2010 07:56:57 +0200
Subject: [rsyslog] omspoof module
References: <alpine.DEB.2.00.1008021459190.29820@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710404A@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, August 03, 2010 12:03 AM
> To: rsyslog-users
> Subject: [rsyslog] omspoof module
> 
> In trying to use the module omspoof with 5.5.6 it looks like there is a
> problem.
> 
> the debug file shows a couple errors (entry points not present in
> module)

That is perfectly valid. This is just information for me, so that I know
which of the optional entry points are present or not.

> followed by 'tried selector action for omudpspoof: -2001'

This is also valid. rsyslog (bare of a better method) pushes the action part
to the plugin. The plugin either returns OK (if it recognizes the format) or
RS_RET_CONFLINE_UNPROCESSED = -2001 if it does not recognize it. So for each
action, you'll see a couple of them. Thus my guess is that there simply is a
syntax error in the :omudpspoof: part (typo?).

> 
> then a couple of lines later when defining
> $ActionUDPSpoofSourceNameTemplate it gets the error 'unkown config, did
> you forget to load a module?'

Mhhh... that doesn't sound normal. Do you have a longer excerpt of the debug
log for me?

Rainer


From rgerhards at hq.adiscon.com  Tue Aug  3 08:45:41 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 3 Aug 2010 08:45:41 +0200
Subject: [rsyslog] omspoof module
References: <alpine.DEB.2.00.1008021459190.29820@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710404C@GRFEXC.intern.adiscon.com>


> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, August 03, 2010 12:03 AM
> To: rsyslog-users
> Subject: [rsyslog] omspoof module
> 
> In trying to use the module omspoof with 5.5.6 it looks like there is a
> problem.
> 
> the debug file shows a couple errors (entry points not present in
> module)
> followed by 'tried selector action for omudpspoof: -2001'
> 
> then a couple of lines later when defining
> $ActionUDPSpoofSourceNameTemplate it gets the error 'unkown config, did
> you forget to load a module?'


David, I overlooked this as well: it is $ActionOMUDP..., not $ActionUDP.... I
just saw it when I set up my lab... With the current config format (and also
with the new scoping), I can not produce a better error message, because it
actually could be either case...

Rainer


From sledz at dresearch.de  Tue Aug  3 09:12:17 2010
From: sledz at dresearch.de (Steffen Sledz)
Date: Tue, 03 Aug 2010 09:12:17 +0200
Subject: [rsyslog] [PATCH] break potential infinite loop in actionDoRetry
In-Reply-To: <1280494375-11930-1-git-send-email-sledz@dresearch.de>
References: <4C527173.9090805@dresearch.de>
	<1280494375-11930-1-git-send-email-sledz@dresearch.de>
Message-ID: <4C57C151.2020402@dresearch.de>

Am 30.07.2010 14:52, schrieb Steffen Sledz:
> If a module always returns RS_RET_OK (like ompipe does) the actionDoRetry
> loop may not have leaved faked ACT_STATE_SUSP state in case iResumeOKinRow
> had ever reached a count of 1000.

In our test the suggested patch seems to fix this problem. :) But now we hit the next one. :(

If we use a LinkedList DA-Queue for this channel suspended messages may be transmitted out-of-order. This may not be a problem for database consumers. But in our context this is really bad.

Is this a bug or an intended behaviour?

Is it possible to force strict order of messages?

Steffen



From david at lang.hm  Tue Aug  3 09:21:38 2010
From: david at lang.hm (david at lang.hm)
Date: Tue, 3 Aug 2010 00:21:38 -0700 (PDT)
Subject: [rsyslog] [PATCH] break potential infinite loop in actionDoRetry
In-Reply-To: <4C57C151.2020402@dresearch.de>
References: <4C527173.9090805@dresearch.de>
	<1280494375-11930-1-git-send-email-sledz@dresearch.de>
	<4C57C151.2020402@dresearch.de>
Message-ID: <alpine.DEB.2.00.1008030016210.25170@asgard.lang.hm>

On Tue, 3 Aug 2010, Steffen Sledz wrote:

> Am 30.07.2010 14:52, schrieb Steffen Sledz:
>> If a module always returns RS_RET_OK (like ompipe does) the actionDoRetry
>> loop may not have leaved faked ACT_STATE_SUSP state in case iResumeOKinRow
>> had ever reached a count of 1000.
>
> In our test the suggested patch seems to fix this problem. :) But now we hit the next one. :(
>
> If we use a LinkedList DA-Queue for this channel suspended messages may be transmitted out-of-order. This may not be a problem for database consumers. But in our context this is really bad.
>
> Is this a bug or an intended behaviour?

intended behavior,

the issue came up beforeand there are many different ways that messages 
can end up being processed out of order (including the network re-ordering 
UDP packets before they arrive)

part of the huge performance advantage that rsyslog has is that it is able 
to use multiple threads to process messages (both inbound and outbound), 
trying to maintain the order would cripple rsyslog (and still not be good 
enough in many cases)

when I took a class on the Simple Event Correlator (a very good tool for 
log analysis), the instructor had a specific example in the beginning to 
point out that you can't count on logs being processed by you in the order 
they are generated, and this was a couple of years ago before rsyslog 
started becoming so popular. This example was with plain old 
single-threaded syslogd.

David Lang

> Is it possible to force strict order of messages?
>
> Steffen
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From sledz at dresearch.de  Tue Aug  3 10:29:43 2010
From: sledz at dresearch.de (Steffen Sledz)
Date: Tue, 03 Aug 2010 10:29:43 +0200
Subject: [rsyslog] [PATCH] break potential infinite loop in actionDoRetry
In-Reply-To: <alpine.DEB.2.00.1008030016210.25170@asgard.lang.hm>
References: <4C527173.9090805@dresearch.de>	<1280494375-11930-1-git-send-email-sledz@dresearch.de>	<4C57C151.2020402@dresearch.de>
	<alpine.DEB.2.00.1008030016210.25170@asgard.lang.hm>
Message-ID: <4C57D377.1020107@dresearch.de>

Am 03.08.2010 09:21, schrieb david at lang.hm:
>> If we use a LinkedList DA-Queue for this channel suspended
>> messages may be transmitted out-of-order. This may not be
>> a problem for database consumers. But in our context this
>> is really bad.
>>
>> Is this a bug or an intended behaviour?
> 
> intended behavior,
> 
> the issue came up beforeand there are many different ways that messages 
> can end up being processed out of order (including the network re-ordering 
> UDP packets before they arrive)
> 
> part of the huge performance advantage that rsyslog has is that it is able 
> to use multiple threads to process messages (both inbound and outbound), 
> trying to maintain the order would cripple rsyslog (and still not be good 
> enough in many cases)
> 
> when I took a class on the Simple Event Correlator (a very good tool for 
> log analysis), the instructor had a specific example in the beginning to 
> point out that you can't count on logs being processed by you in the order 
> they are generated, and this was a couple of years ago before rsyslog 
> started becoming so popular. This example was with plain old 
> single-threaded syslogd.

In my opinion this may occur if messages came from different/concurrent sources. But it should be possible to keep messages from the same source in strict order (with performance loss).

So again my question. Is it possible to force strict order of messages?

Steffen



From rgerhards at hq.adiscon.com  Tue Aug  3 10:42:58 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 3 Aug 2010 10:42:58 +0200
Subject: [rsyslog] [PATCH] break potential infinite loop in actionDoRetry
References: <4C527173.9090805@dresearch.de>	<1280494375-11930-1-git-send-email-sledz@dresearch.de>	<4C57C151.2020402@dresearch.de><alpine.DEB.2.00.1008030016210.25170@asgard.lang.hm>
	<4C57D377.1020107@dresearch.de>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104054@GRFEXC.intern.adiscon.com>


> In my opinion this may occur if messages came from different/concurrent
> sources. 

Depends on your environment. If you have a single UDP listener, reordering
may occur on the wire before you even receive it. Sequence as seen on the
input/API level is not sequence as the message may be originated. Not even so
for local unix sockets, this is depending on scheduling order, which you
cannot control in userland. So I can back David's point: it is dangerous to
assume that a log stream has strict sequence. This may be the case for an
single hop (obviously for messages within a single TCP stream), but not
further than that. In general, you need to define an order relation. 

As a side note, a not-so-small part of the conference paper I am writing
right now is about these wrong perception of log sequence that users commonly
have. I think I can post it myself after the conference (this is the end of
september timeframe).

> But it should be possible to keep messages from the same
> source in strict order (with performance loss).


> 
> So again my question. Is it possible to force strict order of messages?


I think you can come close to what you want:

You need to dig into the queue docs. Set everything to single worker thread
(maybe even use direct mode). If you use a recent build, set all batch sizes
to one (this is conservative, in theory it should work with larger batches as
well, iff everything is single-threaded).

Note that when you use a disk assisted queue, the queue engine always puts
some messages out of order when switching to/from Disk mode. Newer builds
also consume messages from memory and disk concurrently and thus potentially
out of queue order. So you need to avoid disk assisted mode. If you need disk
functionality, use pure disk mode, only.

Make sure you do not define more than one input per rsyslog instance. Else OS
scheduling order for rsyslog threads (input vs. output) potentially gets you
into trouble. Make sure you do not define any async output actions. Do not
write to the same file (or action in general) from more than one action or
more than one instance of rsyslog.

That should give you a somewhat strict sequence of messages. Depending on the
use case/volume, you'll see no to unacceptable performance loss.

Rainer


From sledz at dresearch.de  Tue Aug  3 11:06:23 2010
From: sledz at dresearch.de (Steffen Sledz)
Date: Tue, 03 Aug 2010 11:06:23 +0200
Subject: [rsyslog] [PATCH] break potential infinite loop in actionDoRetry
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7104054@GRFEXC.intern.adiscon.com>
References: <4C527173.9090805@dresearch.de>	<1280494375-11930-1-git-send-email-sledz@dresearch.de>	<4C57C151.2020402@dresearch.de><alpine.DEB.2.00.1008030016210.25170@asgard.lang.hm>	<4C57D377.1020107@dresearch.de>
	<9B6E2A8877C38245BFB15CC491A11DA7104054@GRFEXC.intern.adiscon.com>
Message-ID: <4C57DC0F.8070404@dresearch.de>

Am 03.08.2010 10:42, schrieb Rainer Gerhards:
> Depends on your environment. If you have a single UDP
> listener, reordering may occur on the wire before you
> even receive it....

As you can guess in our context UDP is not involved, just the fifo.

> Note that when you use a disk assisted queue, the queue
> engine always puts some messages out of order when switching
> to/from Disk mode. Newer builds also consume messages from
> memory and disk concurrently and thus potentially out of
> queue order.

I think that's what we see.

So i believe we have to decide between out-of-order messages or loss of some messages when we do not use a disk assisted queue. :(

Steffen



From rgerhards at hq.adiscon.com  Tue Aug  3 11:08:18 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 3 Aug 2010 11:08:18 +0200
Subject: [rsyslog] [PATCH] break potential infinite loop in actionDoRetry
References: <4C527173.9090805@dresearch.de>	<1280494375-11930-1-git-send-email-sledz@dresearch.de>	<4C57C151.2020402@dresearch.de><alpine.DEB.2.00.1008030016210.25170@asgard.lang.hm>	<4C57D377.1020107@dresearch.de><9B6E2A8877C38245BFB15CC491A11DA7104054@GRFEXC.intern.adiscon.com>
	<4C57DC0F.8070404@dresearch.de>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104055@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Steffen Sledz
> Sent: Tuesday, August 03, 2010 11:06 AM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] [PATCH] break potential infinite loop in
> actionDoRetry
> 
> Am 03.08.2010 10:42, schrieb Rainer Gerhards:
> > Depends on your environment. If you have a single UDP
> > listener, reordering may occur on the wire before you
> > even receive it....
> 
> As you can guess in our context UDP is not involved, just the fifo.

The you have the issue of context switching between producers (assuming there
is more than one).

> 
> > Note that when you use a disk assisted queue, the queue
> > engine always puts some messages out of order when switching
> > to/from Disk mode. Newer builds also consume messages from
> > memory and disk concurrently and thus potentially out of
> > queue order.
> 
> I think that's what we see.
> 
> So i believe we have to decide between out-of-order messages or loss of
> some messages when we do not use a disk assisted queue. :(

Or use a disk-only queue. It's performance is obviously very bad, but may be
sufficient depending on volume.

Rainer


From rgerhards at hq.adiscon.com  Thu Aug  5 07:53:36 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 5 Aug 2010 07:53:36 +0200
Subject: [rsyslog] [PATCH] break potential infinite loop in actionDoRetry
References: <4C527173.9090805@dresearch.de>
	<1280494375-11930-1-git-send-email-sledz@dresearch.de>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710405D@GRFEXC.intern.adiscon.com>

I just wanted to let you know that I have integrated the patch. Many thanks
for your effort!

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Steffen Sledz
> Sent: Friday, July 30, 2010 2:53 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] [PATCH] break potential infinite loop in
> actionDoRetry
> 
> If a module always returns RS_RET_OK (like ompipe does) the
> actionDoRetry
> loop may not have leaved faked ACT_STATE_SUSP state in case
> iResumeOKinRow
> had ever reached a count of 1000.
> 
> Signed-off-by: Steffen Sledz <sledz at dresearch.de>
> ---
>  action.c |    1 +
>  1 files changed, 1 insertions(+), 0 deletions(-)
> 
> diff --git a/action.c b/action.c
> index 32a07dc..90ec1bf 100644
> --- a/action.c
> +++ b/action.c
> @@ -508,6 +508,7 @@ static rsRetVal actionDoRetry(action_t *pThis,
> time_t ttNow)
>  		iRet = pThis->pMod->tryResume(pThis->pModData);
>  		if((pThis->iResumeOKinRow > 999) && (pThis->iResumeOKinRow
> % 1000 == 0)) {
>  			bTreatOKasSusp = 1;
> +			pThis->iResumeOKinRow = 0;
>  		} else {
>  			bTreatOKasSusp = 0;
>  		}
> --
> 1.6.4.2
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From tbergfeld at hq.adiscon.com  Thu Aug  5 14:18:52 2010
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Thu, 5 Aug 2010 14:18:52 +0200
Subject: [rsyslog] rsyslog 3.22.2 (v3-stable) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104067@GRFEXC.intern.adiscon.com>

Hi all,

We have just released rsyslog 3.22.2, a member of the v3-stable branch.

This is a stability update, integrating bug fixes that have become available
during the past month. No new functionality is included.

See ChangeLog for more details.

ChangeLog:

http://www.rsyslog.com/changelog-for-3-2-2-v3-stable/

Download:

http://www.rsyslog.com/rsyslog-3-22-2-v3-stable/

As always, feedback is appreciated.

Best regards,
Tom Bergfeld
--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .


From tbergfeld at hq.adiscon.com  Thu Aug  5 14:19:49 2010
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Thu, 5 Aug 2010 14:19:49 +0200
Subject: [rsyslog] rsyslog 4.6.4 (v4-stable) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104068@GRFEXC.intern.adiscon.com>

Hi all,

We have just released rsyslog 4.6.4, a member of the v4-stable branch.

This is a stability update without new features.

See ChangeLog for more details.

ChangeLog:

http://www.rsyslog.com/changelog-for-4-6-4-v4-stable/

Download:

http://www.rsyslog.com/rsyslog-4-6-4-v4-stable/

As always, feedback is appreciated.

Best regards,
Tom Bergfeld
--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .


From rgerhards at hq.adiscon.com  Thu Aug  5 15:45:04 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 5 Aug 2010 15:45:04 +0200
Subject: [rsyslog] spoofing module configuration
References: <alpine.DEB.2.00.1007091435070.14221@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710406C@GRFEXC.intern.adiscon.com>

David,

I have now changed the default:

http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=e4c926bfbca3aab3fe34fc9c
fedb7343423de016

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Friday, July 09, 2010 11:43 PM
> To: rsyslog-users
> Subject: [rsyslog] spoofing module configuration
> 
> in reading the spoofing module configuration it strikes me that the
> defaults can be significantly improved.
> 
> the common case for needing to so spoofing is that you are spoofing the
> original source IP address
> 
> so the current configuration equivalent commands
> 
> $template spoofaddr, "%fromhost-ip%"
> $ActionUDPSpoofSourceNameTemplate spoofaddr
> 
> could be made the default (or call it RSYSLOG_spoofaddr to keep from
> polluting the namespace) and the result would be far simpler for people
> to
> configure, becomging simply
> 
> $modload omudpspoof
> $ActionUDPSpoofTargetHost server.example.com
> *.* :omudpspoof:
> 
> it could be simplified even further if there was some way to specify
> the
> destination on the action line (like the @ and @@  functions do today,
> could we use @S@ to indicate spoofing?)
> 
> 
> changing the defaults should have no problems with backwards
> compatibility, adding/changing how the desitnation is specified could
> break backward compatibility, but this is a very new piece of
> functionality and the simplification may be worth it (what versions
> have
> this available?)
> 
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Thu Aug  5 17:32:17 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 5 Aug 2010 17:32:17 +0200
Subject: [rsyslog] log forwarding through unix sockets
References: <4C456476.4010703@redhat.com><AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com>
	<4C56A7D8.5020901@redhat.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710406E@GRFEXC.intern.adiscon.com>

Just some more information for anyone interested. I today looked at the
module. First of all, there is a directive to set the hostname
($InputUnixListenSocketHostName) but I also found out that there exist
undocumented functionality to activate hostname parsing. In theory, this is
done by putting a colon in front of the socket name. In practice, there seems
to be a bug that prevents this from working at all (the colon is not
removed). So it probably was good this was not documented ;)

I'll see that I fix that first, so that we have some basic functionality in
place.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Ales Kozumplik
> Sent: Monday, August 02, 2010 1:11 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] log forwarding through unix sockets
> 
> On 07/30/2010 03:28 PM, Ryan Lynch wrote:
> > I like your method, too. And thank you for mentioning 'socat', that's
> > what gave me the idea to go in this direction, in the first place.
> 
> Thanks. To tell you the truth at the end we found a way to forward from
> qemu to a TCP socket, and I am happy I don't have to deal with unix
> sockets any more.
> 
> > Based on my own tests, I believe that 'imuxsock' and 'imudp' use
> > different logic to parse incoming messages. 'imuxsock' always assumes
> 
> That's exactly my feeling about this. I just think one either should be
> able to tell every input module what format should be expected (instead
> of letting rsyslogd try some guessing method), or that every input
> module should by default understand a standard "officially recommended"
> forwarding format.
> 
> Ales
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From david at lang.hm  Thu Aug  5 17:50:12 2010
From: david at lang.hm (david at lang.hm)
Date: Thu, 5 Aug 2010 08:50:12 -0700 (PDT)
Subject: [rsyslog] log forwarding through unix sockets
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710406E@GRFEXC.intern.adiscon.com>
References: <4C456476.4010703@redhat.com><AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com>
	<4C56A7D8.5020901@redhat.com>
	<9B6E2A8877C38245BFB15CC491A11DA710406E@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.2.00.1008050849340.25170@asgard.lang.hm>

On Thu, 5 Aug 2010, Rainer Gerhards wrote:

> Just some more information for anyone interested. I today looked at the
> module. First of all, there is a directive to set the hostname
> ($InputUnixListenSocketHostName) but I also found out that there exist
> undocumented functionality to activate hostname parsing. In theory, this is
> done by putting a colon in front of the socket name. In practice, there seems
> to be a bug that prevents this from working at all (the colon is not
> removed). So it probably was good this was not documented ;)

so if you use a socket name that starts with a colon it should work right? 
;-)

David Lang

> I'll see that I fix that first, so that we have some basic functionality in
> place.
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Ales Kozumplik
>> Sent: Monday, August 02, 2010 1:11 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] log forwarding through unix sockets
>>
>> On 07/30/2010 03:28 PM, Ryan Lynch wrote:
>>> I like your method, too. And thank you for mentioning 'socat', that's
>>> what gave me the idea to go in this direction, in the first place.
>>
>> Thanks. To tell you the truth at the end we found a way to forward from
>> qemu to a TCP socket, and I am happy I don't have to deal with unix
>> sockets any more.
>>
>>> Based on my own tests, I believe that 'imuxsock' and 'imudp' use
>>> different logic to parse incoming messages. 'imuxsock' always assumes
>>
>> That's exactly my feeling about this. I just think one either should be
>> able to tell every input module what format should be expected (instead
>> of letting rsyslogd try some guessing method), or that every input
>> module should by default understand a standard "officially recommended"
>> forwarding format.
>>
>> Ales
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Thu Aug  5 17:51:41 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 5 Aug 2010 17:51:41 +0200
Subject: [rsyslog] log forwarding through unix sockets
References: <4C456476.4010703@redhat.com><AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com><4C56A7D8.5020901@redhat.com><9B6E2A8877C38245BFB15CC491A11DA710406E@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.00.1008050849340.25170@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104070@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Thursday, August 05, 2010 5:50 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] log forwarding through unix sockets
> 
> On Thu, 5 Aug 2010, Rainer Gerhards wrote:
> 
> > Just some more information for anyone interested. I today looked at
> the
> > module. First of all, there is a directive to set the hostname
> > ($InputUnixListenSocketHostName) but I also found out that there
> exist
> > undocumented functionality to activate hostname parsing. In theory,
> this is
> > done by putting a colon in front of the socket name. In practice,
> there seems
> > to be a bug that prevents this from working at all (the colon is not
> > removed). So it probably was good this was not documented ;)
> 
> so if you use a socket name that starts with a colon it should work
> right?
> ;-)

indeed... ;) [but I've not yet tested it, right now thinking about creating
an automated test...]

Rainer


From rgerhards at hq.adiscon.com  Fri Aug  6 17:53:04 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 6 Aug 2010 17:53:04 +0200
Subject: [rsyslog] log forwarding through unix sockets
References: <4C456476.4010703@redhat.com><AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com><4C56A7D8.5020901@redhat.com><9B6E2A8877C38245BFB15CC491A11DA710406E@GRFEXC.intern.adiscon.com><alpine.DEB.2.00.1008050849340.25170@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7104070@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710407E@GRFEXC.intern.adiscon.com>

As a first step (and because I needed something to do decent testing ;)) I
have implemented a new module omuxsock which now can natively write to unix
domain sockets. That should close part of the gap that was discussed here.

I hope I will find time to look at imuxsock and the hostname parsing code
next week (but that will be *very* busy).

Doc:
http://www.rsyslog.com/doc/omuxsock.html

v4 Commit (v5 also already merged):
http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=80ff634c841d692c1d9f335b
88e225d6ce7317f7

I plan to remove this plugin next week with the upcoming v5 beta.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Thursday, August 05, 2010 5:52 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] log forwarding through unix sockets
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> > Sent: Thursday, August 05, 2010 5:50 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] log forwarding through unix sockets
> >
> > On Thu, 5 Aug 2010, Rainer Gerhards wrote:
> >
> > > Just some more information for anyone interested. I today looked at
> > the
> > > module. First of all, there is a directive to set the hostname
> > > ($InputUnixListenSocketHostName) but I also found out that there
> > exist
> > > undocumented functionality to activate hostname parsing. In theory,
> > this is
> > > done by putting a colon in front of the socket name. In practice,
> > there seems
> > > to be a bug that prevents this from working at all (the colon is
> not
> > > removed). So it probably was good this was not documented ;)
> >
> > so if you use a socket name that starts with a colon it should work
> > right?
> > ;-)
> 
> indeed... ;) [but I've not yet tested it, right now thinking about
> creating
> an automated test...]
> 
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Fri Aug  6 17:55:10 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 6 Aug 2010 17:55:10 +0200
Subject: [rsyslog] log forwarding through unix sockets
References: <4C456476.4010703@redhat.com><AANLkTik_JCeOjeNNcEwPYoyfo9RxRj3hqr94RwZFER=3@mail.gmail.com><4C56A7D8.5020901@redhat.com><9B6E2A8877C38245BFB15CC491A11DA710406E@GRFEXC.intern.adiscon.com><alpine.DEB.2.00.1008050849340.25170@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA7104070@GRFEXC.intern.adiscon.com>
	<9B6E2A8877C38245BFB15CC491A11DA710407E@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710407F@GRFEXC.intern.adiscon.com>

> I plan to remove this plugin next week with the upcoming v5 beta.

lol: "release", of course, NOT "remove" ;)

Rainer


From shdashbeta at gmail.com  Sat Aug  7 01:39:02 2010
From: shdashbeta at gmail.com (S H)
Date: Fri, 6 Aug 2010 19:39:02 -0400
Subject: [rsyslog] Tuning for performance
Message-ID: <AANLkTimb_hVcG8DvFkC0cmOSP3w23X-g=s-nON56a+ge@mail.gmail.com>

Hello,

I'm newish to the world of rsyslog. I've used it for regular
syslogging stuff with dynamically generated filenames and the like,
but I've never gone in depth with its configuration. Now, however, I'm
working on a project that will involve very high message rates and am
trying to figure out how to tune the system for the kind of throughput
(>200k/sec) documented at
http://mperedim.wordpress.com/2010/01/21/rsyslog-evaluation/

I have a pair of hardware test servers outfitted with dual quad-core
Xeon processors and 8GB of RAM. The network connection is only
100Mbit, but that doesn't seem to be my bottleneck yet. I'm using
loggen for my tests. The servers are running rsyslog 5.5.6.

serverB is the one listening for connections. It's using the
configuration pasted below.

serverA is the one running the test:
# loggen --verbose -r 20000 -I 10 -s 512 -S 207.150.202.100 10514
average rate = 5663.95 msg/sec, count=56644, time=10.007, (last) msg
size=512, bandwidth=2831.98 kB/sec

Increasing the rate (-r) doesn't change the average rate. Switching to
UDP or performing the test on localhost yield very similar results.
What's really strange is that I left the servers alone for about an
hour to work on another project and when I came back the rates were
roughly double - 10-13k/sec. As I tested, however, they gradually fell
back to the 5-6k levels you see here. Restarting the rsyslog process
makes no difference. So I've tuned something incorrectly but I have no
idea what.

iperf shows 100Mbit between the servers. I can double or halve the
message size without affecting the rate, so actual message rate is the
problem -- not bandwidth.

Any help would be wonderful.

-SH

# rsyslog.conf:

$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022

# different rulesets even though I haven't seen any performance effects
$RuleSet remote10514
$RulesetCreateMainQueue on # create ruleset-specific queue
$MainMsgQueueSize 100000
$MainMsgQueueDequeueBatchSize 1024
$RepeatedMsgReduction off

*.* /dev/null
& ~

$ModLoad imtcp
$InputTCPServerBindRuleset remote10514
$InputTCPServerRun 10514


$RuleSet RSYSLOG_DefaultRuleset
$RepeatedMsgReduction off

*.* /var/log/test.log


From rgerhards at hq.adiscon.com  Sat Aug  7 10:14:18 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Sat, 7 Aug 2010 10:14:18 +0200
Subject: [rsyslog] Tuning for performance
References: <AANLkTimb_hVcG8DvFkC0cmOSP3w23X-g=s-nON56a+ge@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104080@GRFEXC.intern.adiscon.com>

This sounds very strange, even the early v4 version could work at higher
rates. Do you use 512 byte messages, only? Could you start with a very basic
rsyslog.conf to get a baseline? Just loading imtcp, starting the listener,
and a single 

*.* /path/to/some/file

Config.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of S H
> Sent: Saturday, August 07, 2010 1:39 AM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] Tuning for performance
> 
> Hello,
> 
> I'm newish to the world of rsyslog. I've used it for regular
> syslogging stuff with dynamically generated filenames and the like,
> but I've never gone in depth with its configuration. Now, however, I'm
> working on a project that will involve very high message rates and am
> trying to figure out how to tune the system for the kind of throughput
> (>200k/sec) documented at
> http://mperedim.wordpress.com/2010/01/21/rsyslog-evaluation/
> 
> I have a pair of hardware test servers outfitted with dual quad-core
> Xeon processors and 8GB of RAM. The network connection is only
> 100Mbit, but that doesn't seem to be my bottleneck yet. I'm using
> loggen for my tests. The servers are running rsyslog 5.5.6.
> 
> serverB is the one listening for connections. It's using the
> configuration pasted below.
> 
> serverA is the one running the test:
> # loggen --verbose -r 20000 -I 10 -s 512 -S 207.150.202.100 10514
> average rate = 5663.95 msg/sec, count=56644, time=10.007, (last) msg
> size=512, bandwidth=2831.98 kB/sec
> 
> Increasing the rate (-r) doesn't change the average rate. Switching to
> UDP or performing the test on localhost yield very similar results.
> What's really strange is that I left the servers alone for about an
> hour to work on another project and when I came back the rates were
> roughly double - 10-13k/sec. As I tested, however, they gradually fell
> back to the 5-6k levels you see here. Restarting the rsyslog process
> makes no difference. So I've tuned something incorrectly but I have no
> idea what.
> 
> iperf shows 100Mbit between the servers. I can double or halve the
> message size without affecting the rate, so actual message rate is the
> problem -- not bandwidth.
> 
> Any help would be wonderful.
> 
> -SH
> 
> # rsyslog.conf:
> 
> $FileOwner syslog
> $FileGroup adm
> $FileCreateMode 0640
> $DirCreateMode 0755
> $Umask 0022
> 
> # different rulesets even though I haven't seen any performance effects
> $RuleSet remote10514
> $RulesetCreateMainQueue on # create ruleset-specific queue
> $MainMsgQueueSize 100000
> $MainMsgQueueDequeueBatchSize 1024
> $RepeatedMsgReduction off
> 
> *.* /dev/null
> & ~
> 
> $ModLoad imtcp
> $InputTCPServerBindRuleset remote10514
> $InputTCPServerRun 10514
> 
> 
> $RuleSet RSYSLOG_DefaultRuleset
> $RepeatedMsgReduction off
> 
> *.* /var/log/test.log
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From shdashbeta at gmail.com  Sat Aug  7 17:10:46 2010
From: shdashbeta at gmail.com (S H)
Date: Sat, 7 Aug 2010 11:10:46 -0400
Subject: [rsyslog] Tuning for performance
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7104080@GRFEXC.intern.adiscon.com>
References: <AANLkTimb_hVcG8DvFkC0cmOSP3w23X-g=s-nON56a+ge@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7104080@GRFEXC.intern.adiscon.com>
Message-ID: <AANLkTikhRY6iEowuRJ0hmMUX28-20y++Vd7tYSXHQU2j@mail.gmail.com>

I think the problem is with loggen, or at least the version I'm using.
I wrote a simple little utility in Python to exercise the native
syslog() calls and was seeing about 240k/second with a simple
configuration.

Once I've got more detailed benchmarks, I'll post them to this thread.
Thanks for the help.

-SH

On Sat, Aug 7, 2010 at 4:14 AM, Rainer Gerhards
<rgerhards at hq.adiscon.com> wrote:
> This sounds very strange, even the early v4 version could work at higher
> rates. Do you use 512 byte messages, only? Could you start with a very basic
> rsyslog.conf to get a baseline? Just loading imtcp, starting the listener,
> and a single
>
> *.* /path/to/some/file
>
> Config.
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of S H
>> Sent: Saturday, August 07, 2010 1:39 AM
>> To: rsyslog at lists.adiscon.com
>> Subject: [rsyslog] Tuning for performance
>>
>> Hello,
>>
>> I'm newish to the world of rsyslog. I've used it for regular
>> syslogging stuff with dynamically generated filenames and the like,
>> but I've never gone in depth with its configuration. Now, however, I'm
>> working on a project that will involve very high message rates and am
>> trying to figure out how to tune the system for the kind of throughput
>> (>200k/sec) documented at
>> http://mperedim.wordpress.com/2010/01/21/rsyslog-evaluation/
>>
>> I have a pair of hardware test servers outfitted with dual quad-core
>> Xeon processors and 8GB of RAM. The network connection is only
>> 100Mbit, but that doesn't seem to be my bottleneck yet. I'm using
>> loggen for my tests. The servers are running rsyslog 5.5.6.
>>
>> serverB is the one listening for connections. It's using the
>> configuration pasted below.
>>
>> serverA is the one running the test:
>> # loggen --verbose -r 20000 -I 10 -s 512 -S 207.150.202.100 10514
>> average rate = 5663.95 msg/sec, count=56644, time=10.007, (last) msg
>> size=512, bandwidth=2831.98 kB/sec
>>
>> Increasing the rate (-r) doesn't change the average rate. Switching to
>> UDP or performing the test on localhost yield very similar results.
>> What's really strange is that I left the servers alone for about an
>> hour to work on another project and when I came back the rates were
>> roughly double - 10-13k/sec. As I tested, however, they gradually fell
>> back to the 5-6k levels you see here. Restarting the rsyslog process
>> makes no difference. So I've tuned something incorrectly but I have no
>> idea what.
>>
>> iperf shows 100Mbit between the servers. I can double or halve the
>> message size without affecting the rate, so actual message rate is the
>> problem -- not bandwidth.
>>
>> Any help would be wonderful.
>>
>> -SH
>>
>> # rsyslog.conf:
>>
>> $FileOwner syslog
>> $FileGroup adm
>> $FileCreateMode 0640
>> $DirCreateMode 0755
>> $Umask 0022
>>
>> # different rulesets even though I haven't seen any performance effects
>> $RuleSet remote10514
>> $RulesetCreateMainQueue on # create ruleset-specific queue
>> $MainMsgQueueSize 100000
>> $MainMsgQueueDequeueBatchSize 1024
>> $RepeatedMsgReduction off
>>
>> *.* /dev/null
>> & ~
>>
>> $ModLoad imtcp
>> $InputTCPServerBindRuleset remote10514
>> $InputTCPServerRun 10514
>>
>>
>> $RuleSet RSYSLOG_DefaultRuleset
>> $RepeatedMsgReduction off
>>
>> *.* /var/log/test.log
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From david at lang.hm  Sun Aug  8 02:46:11 2010
From: david at lang.hm (david at lang.hm)
Date: Sat, 7 Aug 2010 17:46:11 -0700 (PDT)
Subject: [rsyslog] Tuning for performance
In-Reply-To: <AANLkTikhRY6iEowuRJ0hmMUX28-20y++Vd7tYSXHQU2j@mail.gmail.com>
References: <AANLkTimb_hVcG8DvFkC0cmOSP3w23X-g=s-nON56a+ge@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7104080@GRFEXC.intern.adiscon.com>
	<AANLkTikhRY6iEowuRJ0hmMUX28-20y++Vd7tYSXHQU2j@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1008071743560.22036@asgard.lang.hm>

I've also had problems generating enough traffic to stress rsyslog, what I 
ended up doing is using tcpreplay. I send out sample data, capture it with 
tcpdump, and then use tcpreplay to replay the packets over the network at 
different speeds.

this only works well for UDP packets (as tcpreplay does not maintain all 
the state needed to do a real tcp connection.)

David Lang

  On Sat, 7 Aug 2010, S H wrote:

> Date: Sat, 7 Aug 2010 11:10:46 -0400
> From: S H <shdashbeta at gmail.com>
> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> To: rsyslog-users <rsyslog at lists.adiscon.com>
> Subject: Re: [rsyslog] Tuning for performance
> 
> I think the problem is with loggen, or at least the version I'm using.
> I wrote a simple little utility in Python to exercise the native
> syslog() calls and was seeing about 240k/second with a simple
> configuration.
>
> Once I've got more detailed benchmarks, I'll post them to this thread.
> Thanks for the help.
>
> -SH
>
> On Sat, Aug 7, 2010 at 4:14 AM, Rainer Gerhards
> <rgerhards at hq.adiscon.com> wrote:
>> This sounds very strange, even the early v4 version could work at higher
>> rates. Do you use 512 byte messages, only? Could you start with a very basic
>> rsyslog.conf to get a baseline? Just loading imtcp, starting the listener,
>> and a single
>>
>> *.* /path/to/some/file
>>
>> Config.
>>
>> Rainer
>>
>>> -----Original Message-----
>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>> bounces at lists.adiscon.com] On Behalf Of S H
>>> Sent: Saturday, August 07, 2010 1:39 AM
>>> To: rsyslog at lists.adiscon.com
>>> Subject: [rsyslog] Tuning for performance
>>>
>>> Hello,
>>>
>>> I'm newish to the world of rsyslog. I've used it for regular
>>> syslogging stuff with dynamically generated filenames and the like,
>>> but I've never gone in depth with its configuration. Now, however, I'm
>>> working on a project that will involve very high message rates and am
>>> trying to figure out how to tune the system for the kind of throughput
>>> (>200k/sec) documented at
>>> http://mperedim.wordpress.com/2010/01/21/rsyslog-evaluation/
>>>
>>> I have a pair of hardware test servers outfitted with dual quad-core
>>> Xeon processors and 8GB of RAM. The network connection is only
>>> 100Mbit, but that doesn't seem to be my bottleneck yet. I'm using
>>> loggen for my tests. The servers are running rsyslog 5.5.6.
>>>
>>> serverB is the one listening for connections. It's using the
>>> configuration pasted below.
>>>
>>> serverA is the one running the test:
>>> # loggen --verbose -r 20000 -I 10 -s 512 -S 207.150.202.100 10514
>>> average rate = 5663.95 msg/sec, count=56644, time=10.007, (last) msg
>>> size=512, bandwidth=2831.98 kB/sec
>>>
>>> Increasing the rate (-r) doesn't change the average rate. Switching to
>>> UDP or performing the test on localhost yield very similar results.
>>> What's really strange is that I left the servers alone for about an
>>> hour to work on another project and when I came back the rates were
>>> roughly double - 10-13k/sec. As I tested, however, they gradually fell
>>> back to the 5-6k levels you see here. Restarting the rsyslog process
>>> makes no difference. So I've tuned something incorrectly but I have no
>>> idea what.
>>>
>>> iperf shows 100Mbit between the servers. I can double or halve the
>>> message size without affecting the rate, so actual message rate is the
>>> problem -- not bandwidth.
>>>
>>> Any help would be wonderful.
>>>
>>> -SH
>>>
>>> # rsyslog.conf:
>>>
>>> $FileOwner syslog
>>> $FileGroup adm
>>> $FileCreateMode 0640
>>> $DirCreateMode 0755
>>> $Umask 0022
>>>
>>> # different rulesets even though I haven't seen any performance effects
>>> $RuleSet remote10514
>>> $RulesetCreateMainQueue on # create ruleset-specific queue
>>> $MainMsgQueueSize 100000
>>> $MainMsgQueueDequeueBatchSize 1024
>>> $RepeatedMsgReduction off
>>>
>>> *.* /dev/null
>>> & ~
>>>
>>> $ModLoad imtcp
>>> $InputTCPServerBindRuleset remote10514
>>> $InputTCPServerRun 10514
>>>
>>>
>>> $RuleSet RSYSLOG_DefaultRuleset
>>> $RepeatedMsgReduction off
>>>
>>> *.* /var/log/test.log
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From tbergfeld at hq.adiscon.com  Mon Aug  9 14:47:50 2010
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Mon, 9 Aug 2010 14:47:50 +0200
Subject: [rsyslog] rsyslog 5.5.7 (v5-beta) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7104091@GRFEXC.intern.adiscon.com>

Hi all,

We have just released rsyslog 5.5.7, the new v5-beta.
This is a new beta version, scheduled to replace the current stable hopefully
soon. It has some bugfixes over 5.5.6, and also introduces a new module,
omuxsock, that provides the ability to natively write to Unix domain sockets.
Feedback on this version is highly appreciated. We would like to promote it
to stable by mid to end September if we do not hear anything that prevents us
from doing so.
See ChangeLog for more details.

ChangeLog:

http://www.rsyslog.com/changelog-for-5-5-7-v5-beta/

Download:

http://www.rsyslog.com/rsyslog-5-5-7-v5-beta/

As always, feedback is appreciated.

Best regards,
Tom Bergfeld
--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .


From joe.smithian at gmail.com  Tue Aug 10 19:19:00 2010
From: joe.smithian at gmail.com (Joe Smithian)
Date: Tue, 10 Aug 2010 13:19:00 -0400
Subject: [rsyslog] rsyslog truncates host name in the log messages if host
	name contains dots
In-Reply-To: <mailman.434.1281448570.2883.rsyslog@lists.adiscon.com>
References: <mailman.434.1281448570.2883.rsyslog@lists.adiscon.com>
Message-ID: <AANLkTi=oduvpO4P1jY9kb9FGRw5SQYXfZy1HP4-4eGd_@mail.gmail.com>

Hello,

If Linux hostname contains dots "." e.g. an IP address : 192.168.1.1 then
rsyslog prints "192" as hostname in the log messages:

$template MySyslFormat2,"%$YEAR% %timegenerated% *%HOSTNAME%*
%syslogfacility-text% %syslogseverity-text%  %syslogtag%%msg%\n"


2010 Aug 10 09:36:02 *192* kern info  kernel: Kernel logging (proc) stopped.


Is there any way to change the rsyslog behavior to show the full hostname if
it contains dots "."?


Thanks

Joe


From rgerhards at hq.adiscon.com  Wed Aug 11 12:01:02 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 11 Aug 2010 12:01:02 +0200
Subject: [rsyslog] help requested -- connect() performance
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD1F1@GRFEXC.intern.adiscon.com>

Hi all,

I yesterday stumbled over behavior that I cannot explain. Mabye someone here
on the list has a hint.

There is a testing tool "tcpflood", which I use to carry out the tcp based
tests inside the automated testbench. That tool has evolved over time and has
many options. One option is to open a large number of connections and then
send data. I have seen that when I request 1,000 connections, it takes quite
some while to open them (with the CPU being almost idle during that time). So
far, I did not try to dig into this as the only bad effect it causes is that
the testbench runs a little bit longer.

Yesterday, for some other reason, I ran the very same tool, with the very
same options under strace. And I was totally stunned to see that the
connect() calls were now finished in almost no time. When I removed strace,
it took long again (30..60 seconds, not exactly measured). If I ran it under
strace, maybe a second or two. This is totally reproducible.

Does anybody have a clue?

Thanks,
Rainer

PS: if someone is interested in reproducing this (or the details): the test
in question is ./tests/manytcp.sh


From tbergfeld at hq.adiscon.com  Thu Aug 12 14:47:47 2010
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Thu, 12 Aug 2010 14:47:47 +0200
Subject: [rsyslog] rsyslog 6.1.0 (devel) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD20B@GRFEXC.intern.adiscon.com>

Hi all,

We have just released rsyslog 6.1.0, the new v6-devel.
This is the first v6 release of rsyslog, beginning a new development focus.
In v6, configuration shall become much more straigthforward. This release
already offers action scoping, which keeps related things closer together.
Other than that, it is roughly functionally equivalent to the latest v5
releases. However, there is imptcp available, a Linux-optimized plain tcp
input which in theory should perform faster than imtcp (at the price of some
reduced functionality).

Note that v6 requieres modifications to pre v6-plugins in order to work.

ChangeLog:

http://www.rsyslog.com/changelog-for-6-1-0-devel/

Download:

http://www.rsyslog.com/rsyslog-6-1-0-v6-devel/

As always, feedback is appreciated.

Best regards,
Tom Bergfeld
--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .



From sivan at omniqueue.com  Sun Aug 22 15:17:50 2010
From: sivan at omniqueue.com (Sivan Greenberg)
Date: Sun, 22 Aug 2010 16:17:50 +0300
Subject: [rsyslog] Setting up rsyslog 4.6.4 with RELP *and* TLS
Message-ID: <AANLkTinzqWFW+uyP0mS-XkrOiKgoaGNUDF75h-GsQ4Jh@mail.gmail.com>

Hi List,

 I've successfully built 4.6.4 with support for TLS and RELP. When I
worked with the imtcp module everything was fine and setting TLS to be
enabled through a snippet like this:


$DefaultNetstreamDriver gtls
$DefaultNetstreamDriverCAFile /root/tls/ca.pem
$DefaultNetstreamDriverCertFile /root/tls/client-cert.pem
$DefaultNetstreamDriverKeyFile /root/tls/client-key.pem
$ModLoad imuxsock
$ModLoad omrelp
$ModLoad imtcp

$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer server.example.com
$ActionSendStreamDriverMode 1
*.* @@:omrelp:server.example.com:10514


----

However, when I restart on the client and use logger to send a line
nothing happens, there's also no error reported or anything similar on
the logging output created by rsyslog.

This is the server snippet:

$ModLoad imuxsock
$Modload imtcp
$Modload imrelp
$DefaultNetstreamDriver gtls

$DefaultNetstreamDriverCAFile /root/tls/ca.pem
$DefaultNetstreamDriverCertFile /root/tls/server-cert.pem
$DefaultNetstreamDriverKeyFile /root/tls/server-key.pem

$InputTCPServerStreamDriverAuthMode x509/name
$InputTCPServerStreamDriverPermittedPeer *.example.com
$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
$InputRELPServerRun 10514 # start up listener at port 10514
$InputTCPServerRun 10514


I realize that I might have to add the:
RSYSLOG_ForwardFormat to the forward line, but given that I do add it
and it works without TLS, what is the way to enable it to use TLS
together with RELP ?

Any feedback will be highly appreciated!

Thanks,

Sivan


From joe at joetify.com  Thu Aug 26 16:42:45 2010
From: joe at joetify.com (Joe Williams)
Date: Thu, 26 Aug 2010 07:42:45 -0700
Subject: [rsyslog] symlinks
Message-ID: <DD10BDA2-03BA-4D5B-B299-38D41D6FD314@joetify.com>


I have some hostnamed directories and would like to symlink to them to something shorter and more meaningful. Is it possible to have rsyslog create symlinks to make certain logs easier to get too? Alternatively, can I configure rsyslog to match on certain hostnames and instead of using the hostname use a directory name I choose.

Thanks.

-Joe



Name: Joseph A. Williams
Email: joe at joetify.com
Blog: http://www.joeandmotorboat.com/
Twitter: http://twitter.com/williamsjoe



From gravyface at gmail.com  Thu Aug 26 18:12:09 2010
From: gravyface at gmail.com (GravyFace)
Date: Thu, 26 Aug 2010 12:12:09 -0400
Subject: [rsyslog] Should ActionExecOnlyOnceEveryInterval be working in 4.2?
Message-ID: <AANLkTikW4smA5e3LZQP0W7qCim8cveCBNZ9qCJ0CVJDY@mail.gmail.com>

>From the changelogs, it sounds like it was fixed in 4.4.3, and then
included in 4.6.x I believe.

Ubuntu 10.04 ships with 4.2, but in my config, it's not working --
emails get sent out for each match on :msg, regardless of
ActionExecOnlyOnceEveryInterval.

Config as follows:

$ActionMailSMTPServer 127.0.0.1
$ActionMailFrom root at example.com
$ActionMailTo somebody at example.com
$template mailSubject,"Alert: Inbound Packet Denied"
$template mailBody,"DETAILS:\r\n%timereported% %msg%"
$ActionMailSubject mailSubject
# make sure we receive a mail only once per hour
$ActionExecOnlyOnceEveryInterval 3600
:msg, contains, "Deny tcp src inside" :ommail:;mailBody


From sivan at omniqueue.com  Sun Aug 29 10:43:51 2010
From: sivan at omniqueue.com (Sivan Greenberg)
Date: Sun, 29 Aug 2010 11:43:51 +0300
Subject: [rsyslog] Reducing the time a msg spends in the main unreliable
	main queue to a minimum.
Message-ID: <AANLkTi=FcoYYG7LHGF=Mp7epjQ3=YNmdJkMqeoSNR+SC@mail.gmail.com>

Hi List,

 I have some critical msgs being transferred using rsyslog. Hence, I
would like to minimize the time a msg spends in the unreliable by
definition main queue. My understanding is that only action queues
which are where msgs arrive when dispatched from main queue  can be
set to be disk only. Therefor my assumption that only they can be made
truly reliable.

Any suggestions / feedback will be highly appreciated.

-Sivan


From rgerhards at hq.adiscon.com  Mon Aug 30 08:19:21 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 30 Aug 2010 08:19:21 +0200
Subject: [rsyslog] Reducing the time a msg spends in the main
	unreliablemain queue to a minimum.
References: <AANLkTi=FcoYYG7LHGF=Mp7epjQ3=YNmdJkMqeoSNR+SC@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD275@GRFEXC.intern.adiscon.com>

Hi Sivan,

the main message queue supports the same set of properties and capabilities
like the action queue. The directives just start with $MainMsg... instead of
$Action...

So you can set the main queue to pure disk mode as well.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Sivan Greenberg
> Sent: Sunday, August 29, 2010 10:44 AM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] Reducing the time a msg spends in the main
> unreliablemain queue to a minimum.
> 
> Hi List,
> 
>  I have some critical msgs being transferred using rsyslog. Hence, I
> would like to minimize the time a msg spends in the unreliable by
> definition main queue. My understanding is that only action queues
> which are where msgs arrive when dispatched from main queue  can be
> set to be disk only. Therefor my assumption that only they can be made
> truly reliable.
> 
> Any suggestions / feedback will be highly appreciated.
> 
> -Sivan
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From sivan at omniqueue.com  Mon Aug 30 08:33:13 2010
From: sivan at omniqueue.com (Sivan Greenberg)
Date: Mon, 30 Aug 2010 09:33:13 +0300
Subject: [rsyslog] Reducing the time a msg spends in the main
 unreliablemain queue to a minimum.
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD275@GRFEXC.intern.adiscon.com>
References: <AANLkTi=FcoYYG7LHGF=Mp7epjQ3=YNmdJkMqeoSNR+SC@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA71DD275@GRFEXC.intern.adiscon.com>
Message-ID: <AANLkTinY7CMNap1cQrPS5JevbkN4VEa3qdcjD23VtBkO@mail.gmail.com>

Hi Rainer,

I see, however, that would cause a major performance downgrade
thinking of it, and filtering seems to be only done on the main queue,
so to be able to use the same rsyslog instance for both critical and
non critical stuff I think I might be better leaving it buffered to
some extent (is there a directive to define how long/ how much memory)
and filter critical msgs to the pure disk msg queue and the rest.

Do you think this can be done? Could you kindly give pointer how to
configure for this?

Many thanks!

-Sivan

On Mon, Aug 30, 2010 at 9:19 AM, Rainer Gerhards
<rgerhards at hq.adiscon.com> wrote:
> Hi Sivan,
>
> the main message queue supports the same set of properties and capabilities
> like the action queue. The directives just start with $MainMsg... instead of
> $Action...
>
> So you can set the main queue to pure disk mode as well.
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Sivan Greenberg
>> Sent: Sunday, August 29, 2010 10:44 AM
>> To: rsyslog at lists.adiscon.com
>> Subject: [rsyslog] Reducing the time a msg spends in the main
>> unreliablemain queue to a minimum.
>>
>> Hi List,
>>
>> ?I have some critical msgs being transferred using rsyslog. Hence, I
>> would like to minimize the time a msg spends in the unreliable by
>> definition main queue. My understanding is that only action queues
>> which are where msgs arrive when dispatched from main queue ?can be
>> set to be disk only. Therefor my assumption that only they can be made
>> truly reliable.
>>
>> Any suggestions / feedback will be highly appreciated.
>>
>> -Sivan
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Mon Aug 30 12:38:39 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 30 Aug 2010 12:38:39 +0200
Subject: [rsyslog] Reducing the time a msg spends in the main
	unreliablemain queue to a minimum.
References: <AANLkTi=FcoYYG7LHGF=Mp7epjQ3=YNmdJkMqeoSNR+SC@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA71DD275@GRFEXC.intern.adiscon.com>
	<AANLkTinY7CMNap1cQrPS5JevbkN4VEa3qdcjD23VtBkO@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD278@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Sivan Greenberg
> Sent: Monday, August 30, 2010 8:33 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Reducing the time a msg spends in the main
> unreliablemain queue to a minimum.
> 
> Hi Rainer,
> 
> I see, however, that would cause a major performance downgrade

Definitely, very, very severe (you can't get all at once in life ;))

> thinking of it, and filtering seems to be only done on the main queue,
> so to be able to use the same rsyslog instance for both critical and
> non critical stuff I think I might be better leaving it buffered to
> some extent (is there a directive to define how long/ how much memory)

There are many. I suggest you read the queue doc and experiment with some oft
he settings. This is pretty complicated stuff (because you intend to do
complicated things), so it is vital to get good working knowledge before
deploying it to production. Queue doc is:

http://www.rsyslog.com/doc/queues.html

There are also a number of tutorials and descriptions on the workflow inside
the doc set. I'd personally expect that you need at least 2 to 3 days of
careful studying before you get a sufficient idea of how to configure a very
demanding environment.

> and filter critical msgs to the pure disk msg queue and the rest.
> 
> Do you think this can be done?

Yes

> Could you kindly give pointer how to
> configure for this?

As I said, very high reliability is rather complicated, it's a real
consulting job. So, sorry, I don't have a canned response. First step would
be to look at the actual needs, I think you already did this for most of the
part.

For example, if you can submit messages from the originator to the receiver
based on importance, you could use multiple rulesets and use a dedicated
ruleset using a disk-based queue for those important messages. But as I said
... doing it right involves a lot of planning, far more than setting a few
parameters. Otherwise you often end up with something that looks like it were
reliable, but is not really. You need to look at the whole picture. Once this
is done, you have a spec on which messages are what important and how many
loss is acceptable. From there on, you can work to turn that into a rsyslog
conf.

I recently wrote a blog post that may be worth mentioning for your case:

http://blog.gerhards.net/2010/08/rsyslog-queues-reliability-and.html

HTH
Rainer

> 
> Many thanks!
> 
> -Sivan
> 
> On Mon, Aug 30, 2010 at 9:19 AM, Rainer Gerhards
> <rgerhards at hq.adiscon.com> wrote:
> > Hi Sivan,
> >
> > the main message queue supports the same set of properties and
> capabilities
> > like the action queue. The directives just start with $MainMsg...
> instead of
> > $Action...
> >
> > So you can set the main queue to pure disk mode as well.
> >
> > Rainer
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Sivan Greenberg
> >> Sent: Sunday, August 29, 2010 10:44 AM
> >> To: rsyslog at lists.adiscon.com
> >> Subject: [rsyslog] Reducing the time a msg spends in the main
> >> unreliablemain queue to a minimum.
> >>
> >> Hi List,
> >>
> >> ?I have some critical msgs being transferred using rsyslog. Hence, I
> >> would like to minimize the time a msg spends in the unreliable by
> >> definition main queue. My understanding is that only action queues
> >> which are where msgs arrive when dispatched from main queue ?can be
> >> set to be disk only. Therefor my assumption that only they can be
> made
> >> truly reliable.
> >>
> >> Any suggestions / feedback will be highly appreciated.
> >>
> >> -Sivan
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From raubvogel at gmail.com  Mon Aug 30 20:05:26 2010
From: raubvogel at gmail.com (Mauricio Tavares)
Date: Mon, 30 Aug 2010 14:05:26 -0400
Subject: [rsyslog] /etc/rsyslog.d questions
Message-ID: <AANLkTiniNfq+_iYF+qeCGNObh=QTmygSQks5Jd9QHhfe@mail.gmail.com>

I do not know if this is very ubuntu-specific, but in Ubuntu 10.04 LTS
(shipped with version 4.2.0) in addition to the rsylog.conf file there
is also a /etc/rsyslog.d directory. How does the /etc/rsyslog.d dir
works? Specifically, do you start with the 50-default.conf and then
add a, say, 55-mine.conf which adds the changes from the default you
want to use (Ex: different log file for auth)?


From david at lang.hm  Mon Aug 30 21:29:06 2010
From: david at lang.hm (david at lang.hm)
Date: Mon, 30 Aug 2010 12:29:06 -0700 (PDT)
Subject: [rsyslog] /etc/rsyslog.d questions
In-Reply-To: <AANLkTiniNfq+_iYF+qeCGNObh=QTmygSQks5Jd9QHhfe@mail.gmail.com>
References: <AANLkTiniNfq+_iYF+qeCGNObh=QTmygSQks5Jd9QHhfe@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1008301227080.12449@asgard.lang.hm>

On Mon, 30 Aug 2010, Mauricio Tavares wrote:

> I do not know if this is very ubuntu-specific, but in Ubuntu 10.04 LTS
> (shipped with version 4.2.0) in addition to the rsylog.conf file there
> is also a /etc/rsyslog.d directory. How does the /etc/rsyslog.d dir
> works? Specifically, do you start with the 50-default.conf and then
> add a, say, 55-mine.conf which adds the changes from the default you
> want to use (Ex: different log file for auth)?

the ubuntu default rsyslog.conf includes a line to include all files from 
the rsyslog.d directory. they are included in alphabetical order.

since it reads all of the files in, putting conflicting instructions in 
the file can result in confused behavior.

personally, I think that using includes like this is a bad idea (and I 
have some fairly complex config files), but if you have really large 
config files I could see this helping, just make sure that you don't do 
anything that conflicts between the files.

David Lang


From david at lang.hm  Mon Aug 30 21:37:35 2010
From: david at lang.hm (david at lang.hm)
Date: Mon, 30 Aug 2010 12:37:35 -0700 (PDT)
Subject: [rsyslog] Reducing the time a msg spends in the main
 unreliablemain queue to a minimum.
In-Reply-To: <AANLkTinY7CMNap1cQrPS5JevbkN4VEa3qdcjD23VtBkO@mail.gmail.com>
References: <AANLkTi=FcoYYG7LHGF=Mp7epjQ3=YNmdJkMqeoSNR+SC@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA71DD275@GRFEXC.intern.adiscon.com>
	<AANLkTinY7CMNap1cQrPS5JevbkN4VEa3qdcjD23VtBkO@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1008301235090.12449@asgard.lang.hm>

On Mon, 30 Aug 2010, Sivan Greenberg wrote:

> Hi Rainer,
>
> I see, however, that would cause a major performance downgrade
> thinking of it, and filtering seems to be only done on the main queue,
> so to be able to use the same rsyslog instance for both critical and
> non critical stuff I think I might be better leaving it buffered to
> some extent (is there a directive to define how long/ how much memory)
> and filter critical msgs to the pure disk msg queue and the rest.
>
> Do you think this can be done? Could you kindly give pointer how to
> configure for this?

What I would suggest is running two parallel logging infrastructures, one 
for your normal 'unreliable' logs, and a second for your 'audit-grade' 
logs. That way you can configure the 'augid-grade' instances to use the 
disk queue, relp, etc to make them fully reliable (and slow), while your 
other instance can take the shortcuts that result in the high perfomance 
that rsyslog is able to get.

David Lang

> Many thanks!
>
> -Sivan
>
> On Mon, Aug 30, 2010 at 9:19 AM, Rainer Gerhards
> <rgerhards at hq.adiscon.com> wrote:
>> Hi Sivan,
>>
>> the main message queue supports the same set of properties and capabilities
>> like the action queue. The directives just start with $MainMsg... instead of
>> $Action...
>>
>> So you can set the main queue to pure disk mode as well.
>>
>> Rainer
>>
>>> -----Original Message-----
>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>> bounces at lists.adiscon.com] On Behalf Of Sivan Greenberg
>>> Sent: Sunday, August 29, 2010 10:44 AM
>>> To: rsyslog at lists.adiscon.com
>>> Subject: [rsyslog] Reducing the time a msg spends in the main
>>> unreliablemain queue to a minimum.
>>>
>>> Hi List,
>>>
>>> ?I have some critical msgs being transferred using rsyslog. Hence, I
>>> would like to minimize the time a msg spends in the unreliable by
>>> definition main queue. My understanding is that only action queues
>>> which are where msgs arrive when dispatched from main queue ?can be
>>> set to be disk only. Therefor my assumption that only they can be made
>>> truly reliable.
>>>
>>> Any suggestions / feedback will be highly appreciated.
>>>
>>> -Sivan
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From gravyface at gmail.com  Mon Aug 30 21:41:08 2010
From: gravyface at gmail.com (GravyFace)
Date: Mon, 30 Aug 2010 15:41:08 -0400
Subject: [rsyslog] /etc/rsyslog.d questions
In-Reply-To: <alpine.DEB.2.00.1008301227080.12449@asgard.lang.hm>
References: <AANLkTiniNfq+_iYF+qeCGNObh=QTmygSQks5Jd9QHhfe@mail.gmail.com>
	<alpine.DEB.2.00.1008301227080.12449@asgard.lang.hm>
Message-ID: <AANLkTi=0c5j1gXqa2mHHFJXA0OAvoh3--QdGg0gowYfj@mail.gmail.com>

On Mon, Aug 30, 2010 at 3:29 PM,  <david at lang.hm> wrote:
> On Mon, 30 Aug 2010, Mauricio Tavares wrote:
>
>> I do not know if this is very ubuntu-specific, but in Ubuntu 10.04 LTS
>> (shipped with version 4.2.0) in addition to the rsylog.conf file there
>> is also a /etc/rsyslog.d directory. How does the /etc/rsyslog.d dir
>> works? Specifically, do you start with the 50-default.conf and then
>> add a, say, 55-mine.conf which adds the changes from the default you
>> want to use (Ex: different log file for auth)?

Just an FYI, I discovered that the 4.2 version that ships with Ubuntu
10.04 doesn't include a later fix for the
ActionExecOnlyOnceEveryInterval property.  If you're planning on using
the ommail (SMTP alerts) feature, you'll need to upgrade to 4.6.4.


From raubvogel at gmail.com  Mon Aug 30 21:57:35 2010
From: raubvogel at gmail.com (Mauricio Tavares)
Date: Mon, 30 Aug 2010 15:57:35 -0400
Subject: [rsyslog] /etc/rsyslog.d questions
In-Reply-To: <alpine.DEB.2.00.1008301227080.12449@asgard.lang.hm>
References: <AANLkTiniNfq+_iYF+qeCGNObh=QTmygSQks5Jd9QHhfe@mail.gmail.com>
	<alpine.DEB.2.00.1008301227080.12449@asgard.lang.hm>
Message-ID: <AANLkTimudoAvzULFJ3RwaRmcdAoUn6mFY7N7L7ScnkX2@mail.gmail.com>

On Mon, Aug 30, 2010 at 3:29 PM,  <david at lang.hm> wrote:
> On Mon, 30 Aug 2010, Mauricio Tavares wrote:
>
>> I do not know if this is very ubuntu-specific, but in Ubuntu 10.04 LTS
>> (shipped with version 4.2.0) in addition to the rsylog.conf file there
>> is also a /etc/rsyslog.d directory. How does the /etc/rsyslog.d dir
>> works? Specifically, do you start with the 50-default.conf and then
>> add a, say, 55-mine.conf which adds the changes from the default you
>> want to use (Ex: different log file for auth)?
>
> the ubuntu default rsyslog.conf includes a line to include all files from
> the rsyslog.d directory. they are included in alphabetical order.
>
> since it reads all of the files in, putting conflicting instructions in
> the file can result in confused behavior.
>
      I see. For some reason I thought that whichever changes came in
the last conf file to be read win.

> personally, I think that using includes like this is a bad idea (and I
> have some fairly complex config files), but if you have really large
> config files I could see this helping, just make sure that you don't do
> anything that conflicts between the files.
>
       When I saw the ubuntu approach of having different .conf files
for whatever reason, I thought they were trying to do something like
apache (and nginx) virtual sites directory. Honestly I am just
starting with rsyslog (was using syslog-ng and now want to see if this
is better for my needs). if it makes sense to use just *one* file, I
am game.

I think for now I will be humble and just edit the 50-default.conf
file to keep it ubuntu-proper.

> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From david at lang.hm  Mon Aug 30 22:01:07 2010
From: david at lang.hm (david at lang.hm)
Date: Mon, 30 Aug 2010 13:01:07 -0700 (PDT)
Subject: [rsyslog] /etc/rsyslog.d questions
In-Reply-To: <AANLkTimudoAvzULFJ3RwaRmcdAoUn6mFY7N7L7ScnkX2@mail.gmail.com>
References: <AANLkTiniNfq+_iYF+qeCGNObh=QTmygSQks5Jd9QHhfe@mail.gmail.com>
	<alpine.DEB.2.00.1008301227080.12449@asgard.lang.hm>
	<AANLkTimudoAvzULFJ3RwaRmcdAoUn6mFY7N7L7ScnkX2@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1008301259400.12449@asgard.lang.hm>

On Mon, 30 Aug 2010, Mauricio Tavares wrote:

> On Mon, Aug 30, 2010 at 3:29 PM,  <david at lang.hm> wrote:
>> On Mon, 30 Aug 2010, Mauricio Tavares wrote:
>>
>>> I do not know if this is very ubuntu-specific, but in Ubuntu 10.04 LTS
>>> (shipped with version 4.2.0) in addition to the rsylog.conf file there
>>> is also a /etc/rsyslog.d directory. How does the /etc/rsyslog.d dir
>>> works? Specifically, do you start with the 50-default.conf and then
>>> add a, say, 55-mine.conf which adds the changes from the default you
>>> want to use (Ex: different log file for auth)?
>>
>> the ubuntu default rsyslog.conf includes a line to include all files from
>> the rsyslog.d directory. they are included in alphabetical order.
>>
>> since it reads all of the files in, putting conflicting instructions in
>> the file can result in confused behavior.
>>
>      I see. For some reason I thought that whichever changes came in
> the last conf file to be read win.

sometimes yes, sometimes the things that appear in earlier config files 
have an effect, no matter what appears in later config files.

>> personally, I think that using includes like this is a bad idea (and I
>> have some fairly complex config files), but if you have really large
>> config files I could see this helping, just make sure that you don't do
>> anything that conflicts between the files.
>>
>       When I saw the ubuntu approach of having different .conf files
> for whatever reason, I thought they were trying to do something like
> apache (and nginx) virtual sites directory. Honestly I am just
> starting with rsyslog (was using syslog-ng and now want to see if this
> is better for my needs). if it makes sense to use just *one* file, I
> am game.
>
> I think for now I will be humble and just edit the 50-default.conf
> file to keep it ubuntu-proper.

either that or just edit /etc/rsyslog.conf (and remove the include 
statement)

David Lang


From raubvogel at gmail.com  Mon Aug 30 21:59:19 2010
From: raubvogel at gmail.com (Mauricio Tavares)
Date: Mon, 30 Aug 2010 15:59:19 -0400
Subject: [rsyslog] /etc/rsyslog.d questions
In-Reply-To: <AANLkTi=0c5j1gXqa2mHHFJXA0OAvoh3--QdGg0gowYfj@mail.gmail.com>
References: <AANLkTiniNfq+_iYF+qeCGNObh=QTmygSQks5Jd9QHhfe@mail.gmail.com>
	<alpine.DEB.2.00.1008301227080.12449@asgard.lang.hm>
	<AANLkTi=0c5j1gXqa2mHHFJXA0OAvoh3--QdGg0gowYfj@mail.gmail.com>
Message-ID: <AANLkTinZD=pZpz+vfTAoxND_p9qFK81OGCp9LDMv7EXi@mail.gmail.com>

On Mon, Aug 30, 2010 at 3:41 PM, GravyFace <gravyface at gmail.com> wrote:
> On Mon, Aug 30, 2010 at 3:29 PM, ?<david at lang.hm> wrote:
>> On Mon, 30 Aug 2010, Mauricio Tavares wrote:
>>
>>> I do not know if this is very ubuntu-specific, but in Ubuntu 10.04 LTS
>>> (shipped with version 4.2.0) in addition to the rsylog.conf file there
>>> is also a /etc/rsyslog.d directory. How does the /etc/rsyslog.d dir
>>> works? Specifically, do you start with the 50-default.conf and then
>>> add a, say, 55-mine.conf which adds the changes from the default you
>>> want to use (Ex: different log file for auth)?
>
> Just an FYI, I discovered that the 4.2 version that ships with Ubuntu
> 10.04 doesn't include a later fix for the
> ActionExecOnlyOnceEveryInterval property. ?If you're planning on using
> the ommail (SMTP alerts) feature, you'll need to upgrade to 4.6.4.
>
      Thanks for the warning! I do not think we will be using SMTP for
now, so I will have some time to get used to rsyslog. And then figure
out the best way for it to let us know of something wicked this way
came.

 _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From raubvogel at gmail.com  Tue Aug 31 00:15:32 2010
From: raubvogel at gmail.com (Mauricio Tavares)
Date: Mon, 30 Aug 2010 18:15:32 -0400
Subject: [rsyslog] $template to create filename, ownership,
	and other file-related questions
Message-ID: <AANLkTikxZcNDdVNjoG0ATp7JdG3Z2vAbnF7e61cZ4sJA@mail.gmail.com>

In my ongoing quest to understand rsyslog, I have a few log
file-related questions:

1) What does the "-" in

news.err                        /var/log/news/news.err
news.notice                     -/var/log/news/news.notice

stand for? I probably passed through its explanation a few times but
did not see it.

2) Let's say I want to generate dynamic file names. Based on the man
page, I create the following template:

$template AuthFile,"/var/log/auth/auth-%$YEAR%-%$MONTH%-%$DAY%.log"

And apply it to, say, auth:

# auth,authpriv.*                       /var/log/auth.log
# auth,authpriv.*                 -?AuthFile
auth,authpriv.*                 ?AuthFile

What I noticed is that instead of the log file be owned by syslog:adm,
it is owned by syslog:syslog. Did I miss anything?

3) If I create a log file with dynamic filename as in the previous
question and want to have it linked to, say, /var/log/auth.log, can I
do that from within rsyslog or should I do it using a external program
(cron comes to mind)?


From david at lang.hm  Tue Aug 31 00:43:23 2010
From: david at lang.hm (david at lang.hm)
Date: Mon, 30 Aug 2010 15:43:23 -0700 (PDT)
Subject: [rsyslog] $template to create filename, ownership,
 and other file-related questions
In-Reply-To: <AANLkTikxZcNDdVNjoG0ATp7JdG3Z2vAbnF7e61cZ4sJA@mail.gmail.com>
References: <AANLkTikxZcNDdVNjoG0ATp7JdG3Z2vAbnF7e61cZ4sJA@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1008301540510.12449@asgard.lang.hm>

On Mon, 30 Aug 2010, Mauricio Tavares wrote:

> In my ongoing quest to understand rsyslog, I have a few log
> file-related questions:
>
> 1) What does the "-" in
>
> news.err                        /var/log/news/news.err
> news.notice                     -/var/log/news/news.notice
>
> stand for? I probably passed through its explanation a few times but
> did not see it.

actually, in rsyslog the - has no effect (and can even cause problems)

in other syslog implementations the - tells syslog that it doesn't have to 
do a fsync after writing each message to disk, it can just keep writing 
and let the OS buffer them and write them to disk.

This is the default in rsyslog.

> 2) Let's say I want to generate dynamic file names. Based on the man
> page, I create the following template:
>
> $template AuthFile,"/var/log/auth/auth-%$YEAR%-%$MONTH%-%$DAY%.log"
>
> And apply it to, say, auth:
>
> # auth,authpriv.*                       /var/log/auth.log
> # auth,authpriv.*                 -?AuthFile
> auth,authpriv.*                 ?AuthFile
>
> What I noticed is that instead of the log file be owned by syslog:adm,
> it is owned by syslog:syslog. Did I miss anything?

I believe that there are options to define what the file ownership is.

> 3) If I create a log file with dynamic filename as in the previous
> question and want to have it linked to, say, /var/log/auth.log, can I
> do that from within rsyslog or should I do it using a external program
> (cron comes to mind)?

I don't know any way do create links from inside rsyslog.

David Lang


From raubvogel at gmail.com  Tue Aug 31 00:52:20 2010
From: raubvogel at gmail.com (Mauricio Tavares)
Date: Mon, 30 Aug 2010 18:52:20 -0400
Subject: [rsyslog] $template to create filename, ownership,
 and other file-related questions
In-Reply-To: <alpine.DEB.2.00.1008301540510.12449@asgard.lang.hm>
References: <AANLkTikxZcNDdVNjoG0ATp7JdG3Z2vAbnF7e61cZ4sJA@mail.gmail.com>
	<alpine.DEB.2.00.1008301540510.12449@asgard.lang.hm>
Message-ID: <AANLkTina-AsorrDRZ+zu5Z9o3=ntRv1AsqX=LkKybNug@mail.gmail.com>

On Mon, Aug 30, 2010 at 6:43 PM,  <david at lang.hm> wrote:
> On Mon, 30 Aug 2010, Mauricio Tavares wrote:
>
>> In my ongoing quest to understand rsyslog, I have a few log
>> file-related questions:
>>
>> 1) What does the "-" in
>>
>> news.err ? ? ? ? ? ? ? ? ? ? ? ?/var/log/news/news.err
>> news.notice ? ? ? ? ? ? ? ? ? ? -/var/log/news/news.notice
>>
>> stand for? I probably passed through its explanation a few times but
>> did not see it.
>
> actually, in rsyslog the - has no effect (and can even cause problems)
>
      How could it cause problems?

> in other syslog implementations the - tells syslog that it doesn't have to
> do a fsync after writing each message to disk, it can just keep writing
> and let the OS buffer them and write them to disk.
>
> This is the default in rsyslog.
>
      Now it makes sense.  Thanks!

>> 2) Let's say I want to generate dynamic file names. Based on the man
>> page, I create the following template:
>>
>> $template AuthFile,"/var/log/auth/auth-%$YEAR%-%$MONTH%-%$DAY%.log"
>>
>> And apply it to, say, auth:
>>
>> # auth,authpriv.* ? ? ? ? ? ? ? ? ? ? ? /var/log/auth.log
>> # auth,authpriv.* ? ? ? ? ? ? ? ? -?AuthFile
>> auth,authpriv.* ? ? ? ? ? ? ? ? ?AuthFile
>>
>> What I noticed is that instead of the log file be owned by syslog:adm,
>> it is owned by syslog:syslog. Did I miss anything?
>
> I believe that there are options to define what the file ownership is.
>
      in /etc/rsyslog.conf I have

#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog

If I tell it to write to /var/log/auth.log, rsyslog seems to honor
$FileOwner and $FileGroup. But when I tell it to create a dynamic
filename log file, it seems to be using the user:group it dropped to
once it started, namely syslog:syslog:

raub at ubuntu1004-x64:/etc/rsyslog.d$ ls -l /var/log/auth.log
/var/log/auth/auth-2010-08-30.log
-rw-r----- 1 syslog syslog     0 2010-08-30 17:33
/var/log/auth/auth-2010-08-30.log
-rw-r----- 1 syslog adm    36701 2010-08-30 17:38 /var/log/auth.log
raub at ubuntu1004-x64:/etc/rsyslog.d$

And that is what confuses me.

>> 3) If I create a log file with dynamic filename as in the previous
>> question and want to have it linked to, say, /var/log/auth.log, can I
>> do that from within rsyslog or should I do it using a external program
>> (cron comes to mind)?
>
> I don't know any way do create links from inside rsyslog.
>
      Not worries; there are other ways to do that. =)

> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From david at lang.hm  Tue Aug 31 01:02:45 2010
From: david at lang.hm (david at lang.hm)
Date: Mon, 30 Aug 2010 16:02:45 -0700 (PDT)
Subject: [rsyslog] $template to create filename, ownership,
 and other file-related questions
In-Reply-To: <AANLkTina-AsorrDRZ+zu5Z9o3=ntRv1AsqX=LkKybNug@mail.gmail.com>
References: <AANLkTikxZcNDdVNjoG0ATp7JdG3Z2vAbnF7e61cZ4sJA@mail.gmail.com>
	<alpine.DEB.2.00.1008301540510.12449@asgard.lang.hm>
	<AANLkTina-AsorrDRZ+zu5Z9o3=ntRv1AsqX=LkKybNug@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1008301557360.12449@asgard.lang.hm>

On Mon, 30 Aug 2010, Mauricio Tavares wrote:

> On Mon, Aug 30, 2010 at 6:43 PM,  <david at lang.hm> wrote:
>> On Mon, 30 Aug 2010, Mauricio Tavares wrote:
>>
>>> In my ongoing quest to understand rsyslog, I have a few log
>>> file-related questions:
>>>
>>> 1) What does the "-" in
>>>
>>> news.err ? ? ? ? ? ? ? ? ? ? ? ?/var/log/news/news.err
>>> news.notice ? ? ? ? ? ? ? ? ? ? -/var/log/news/news.notice
>>>
>>> stand for? I probably passed through its explanation a few times but
>>> did not see it.
>>
>> actually, in rsyslog the - has no effect (and can even cause problems)
>>
>      How could it cause problems?

in sysklogd syslog.conf - at ip is valid, in rsyslog this isn't valid and 
causes  a problem. I don't remember if the problem is that rsyslog doesn't 
send the logs, or that it has some other problem.

>> in other syslog implementations the - tells syslog that it doesn't have to
>> do a fsync after writing each message to disk, it can just keep writing
>> and let the OS buffer them and write them to disk.
>>
>> This is the default in rsyslog.
>>
>      Now it makes sense.  Thanks!
>
>>> 2) Let's say I want to generate dynamic file names. Based on the man
>>> page, I create the following template:
>>>
>>> $template AuthFile,"/var/log/auth/auth-%$YEAR%-%$MONTH%-%$DAY%.log"
>>>
>>> And apply it to, say, auth:
>>>
>>> # auth,authpriv.* ? ? ? ? ? ? ? ? ? ? ? /var/log/auth.log
>>> # auth,authpriv.* ? ? ? ? ? ? ? ? -?AuthFile
>>> auth,authpriv.* ? ? ? ? ? ? ? ? ?AuthFile
>>>
>>> What I noticed is that instead of the log file be owned by syslog:adm,
>>> it is owned by syslog:syslog. Did I miss anything?
>>
>> I believe that there are options to define what the file ownership is.
>>
>      in /etc/rsyslog.conf I have
>
> #
> # Set the default permissions for all log files.
> #
> $FileOwner syslog
> $FileGroup adm
> $FileCreateMode 0640
> $DirCreateMode 0755
> $Umask 0022
> $PrivDropToUser syslog
> $PrivDropToGroup syslog
>
> If I tell it to write to /var/log/auth.log, rsyslog seems to honor
> $FileOwner and $FileGroup. But when I tell it to create a dynamic
> filename log file, it seems to be using the user:group it dropped to
> once it started, namely syslog:syslog:

that makes sense, once it drops privilages it may not have the ability to 
use different credentials.

the privilage dropping feature of rsyslog was a quick-and-dirty 
modification, there are many things that do not work with it because after 
the privilages are dropped the process doesn't have the rights to do what 
it's trying to do.

David Lang

> raub at ubuntu1004-x64:/etc/rsyslog.d$ ls -l /var/log/auth.log
> /var/log/auth/auth-2010-08-30.log
> -rw-r----- 1 syslog syslog     0 2010-08-30 17:33
> /var/log/auth/auth-2010-08-30.log
> -rw-r----- 1 syslog adm    36701 2010-08-30 17:38 /var/log/auth.log
> raub at ubuntu1004-x64:/etc/rsyslog.d$
>
> And that is what confuses me.
>
>>> 3) If I create a log file with dynamic filename as in the previous
>>> question and want to have it linked to, say, /var/log/auth.log, can I
>>> do that from within rsyslog or should I do it using a external program
>>> (cron comes to mind)?
>>
>> I don't know any way do create links from inside rsyslog.
>>
>      Not worries; there are other ways to do that. =)
>
>> David Lang
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From raubvogel at gmail.com  Tue Aug 31 02:02:57 2010
From: raubvogel at gmail.com (Mauricio Tavares)
Date: Mon, 30 Aug 2010 20:02:57 -0400
Subject: [rsyslog] $template to create filename, ownership,
 and other file-related questions
In-Reply-To: <alpine.DEB.2.00.1008301557360.12449@asgard.lang.hm>
References: <AANLkTikxZcNDdVNjoG0ATp7JdG3Z2vAbnF7e61cZ4sJA@mail.gmail.com>
	<alpine.DEB.2.00.1008301540510.12449@asgard.lang.hm>
	<AANLkTina-AsorrDRZ+zu5Z9o3=ntRv1AsqX=LkKybNug@mail.gmail.com>
	<alpine.DEB.2.00.1008301557360.12449@asgard.lang.hm>
Message-ID: <AANLkTi=q2cjizeMtDwyQO8af3jdy6w+v=B79qXiGmMM4@mail.gmail.com>

On Mon, Aug 30, 2010 at 7:02 PM,  <david at lang.hm> wrote:
> On Mon, 30 Aug 2010, Mauricio Tavares wrote:
>
>> On Mon, Aug 30, 2010 at 6:43 PM, ?<david at lang.hm> wrote:
>>>
>>> On Mon, 30 Aug 2010, Mauricio Tavares wrote:
>>>
>>>> In my ongoing quest to understand rsyslog, I have a few log
>>>> file-related questions:
>>>>
>>>> 1) What does the "-" in
>>>>
>>>> news.err ? ? ? ? ? ? ? ? ? ? ? ?/var/log/news/news.err
>>>> news.notice ? ? ? ? ? ? ? ? ? ? -/var/log/news/news.notice
>>>>
>>>> stand for? I probably passed through its explanation a few times but
>>>> did not see it.
>>>
>>> actually, in rsyslog the - has no effect (and can even cause problems)
>>>
>> ? ? How could it cause problems?
>
> in sysklogd syslog.conf - at ip is valid, in rsyslog this isn't valid and
> causes ?a problem. I don't remember if the problem is that rsyslog doesn't
> send the logs, or that it has some other problem.
>
      Ok. But would that be a problem if instead of being forwarded to
another machine it is just being saved to a local file? I ask since
the ubuntu config file for rsyslog only has entries to save to log
file with and without the "-".

>>> in other syslog implementations the - tells syslog that it doesn't have
>>> to
>>> do a fsync after writing each message to disk, it can just keep writing
>>> and let the OS buffer them and write them to disk.
>>>
>>> This is the default in rsyslog.
>>>
>> ? ? Now it makes sense. ?Thanks!
>>
>>>> 2) Let's say I want to generate dynamic file names. Based on the man
>>>> page, I create the following template:
>>>>
>>>> $template AuthFile,"/var/log/auth/auth-%$YEAR%-%$MONTH%-%$DAY%.log"
>>>>
>>>> And apply it to, say, auth:
>>>>
>>>> # auth,authpriv.* ? ? ? ? ? ? ? ? ? ? ? /var/log/auth.log
>>>> # auth,authpriv.* ? ? ? ? ? ? ? ? -?AuthFile
>>>> auth,authpriv.* ? ? ? ? ? ? ? ? ?AuthFile
>>>>
>>>> What I noticed is that instead of the log file be owned by syslog:adm,
>>>> it is owned by syslog:syslog. Did I miss anything?
>>>
>>> I believe that there are options to define what the file ownership is.
>>>
>> ? ? in /etc/rsyslog.conf I have
>>
>> #
>> # Set the default permissions for all log files.
>> #
>> $FileOwner syslog
>> $FileGroup adm
>> $FileCreateMode 0640
>> $DirCreateMode 0755
>> $Umask 0022
>> $PrivDropToUser syslog
>> $PrivDropToGroup syslog
>>
>> If I tell it to write to /var/log/auth.log, rsyslog seems to honor
>> $FileOwner and $FileGroup. But when I tell it to create a dynamic
>> filename log file, it seems to be using the user:group it dropped to
>> once it started, namely syslog:syslog:
>
> that makes sense, once it drops privilages it may not have the ability to
> use different credentials.
>
> the privilage dropping feature of rsyslog was a quick-and-dirty
> modification, there are many things that do not work with it because after
> the privilages are dropped the process doesn't have the rights to do what
> it's trying to do.
>
      That said, in examples like
http://wiki.rsyslog.com/index.php/DailyLogRotation, they are using
dynamic filenames. I would think those examples reflect used
configurations. Could it be they are older/newer examples, so that
either the issue I am having did not exist or was solved? Or perhaps
it is just me who did not set it up right.

> David Lang
>
>> raub at ubuntu1004-x64:/etc/rsyslog.d$ ls -l /var/log/auth.log
>> /var/log/auth/auth-2010-08-30.log
>> -rw-r----- 1 syslog syslog ? ? 0 2010-08-30 17:33
>> /var/log/auth/auth-2010-08-30.log
>> -rw-r----- 1 syslog adm ? ?36701 2010-08-30 17:38 /var/log/auth.log
>> raub at ubuntu1004-x64:/etc/rsyslog.d$
>>
>> And that is what confuses me.
>>
>>>> 3) If I create a log file with dynamic filename as in the previous
>>>> question and want to have it linked to, say, /var/log/auth.log, can I
>>>> do that from within rsyslog or should I do it using a external program
>>>> (cron comes to mind)?
>>>
>>> I don't know any way do create links from inside rsyslog.
>>>
>> ? ? Not worries; there are other ways to do that. =)
>>
>>> David Lang
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>
>


From david at lang.hm  Tue Aug 31 02:18:08 2010
From: david at lang.hm (david at lang.hm)
Date: Mon, 30 Aug 2010 17:18:08 -0700 (PDT)
Subject: [rsyslog] $template to create filename, ownership,
 and other file-related questions
In-Reply-To: <AANLkTi=q2cjizeMtDwyQO8af3jdy6w+v=B79qXiGmMM4@mail.gmail.com>
References: <AANLkTikxZcNDdVNjoG0ATp7JdG3Z2vAbnF7e61cZ4sJA@mail.gmail.com>
	<alpine.DEB.2.00.1008301540510.12449@asgard.lang.hm>
	<AANLkTina-AsorrDRZ+zu5Z9o3=ntRv1AsqX=LkKybNug@mail.gmail.com>
	<alpine.DEB.2.00.1008301557360.12449@asgard.lang.hm>
	<AANLkTi=q2cjizeMtDwyQO8af3jdy6w+v=B79qXiGmMM4@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1008301713290.12449@asgard.lang.hm>

On Mon, 30 Aug 2010, Mauricio Tavares wrote:

> On Mon, Aug 30, 2010 at 7:02 PM,  <david at lang.hm> wrote:
>> On Mon, 30 Aug 2010, Mauricio Tavares wrote:
>>
>>> On Mon, Aug 30, 2010 at 6:43 PM, ?<david at lang.hm> wrote:
>>>>
>>>> On Mon, 30 Aug 2010, Mauricio Tavares wrote:
>>>>
>>>>> In my ongoing quest to understand rsyslog, I have a few log
>>>>> file-related questions:
>>>>>
>>>>> 1) What does the "-" in
>>>>>
>>>>> news.err ? ? ? ? ? ? ? ? ? ? ? ?/var/log/news/news.err
>>>>> news.notice ? ? ? ? ? ? ? ? ? ? -/var/log/news/news.notice
>>>>>
>>>>> stand for? I probably passed through its explanation a few times but
>>>>> did not see it.
>>>>
>>>> actually, in rsyslog the - has no effect (and can even cause problems)
>>>>
>>> ? ? How could it cause problems?
>>
>> in sysklogd syslog.conf - at ip is valid, in rsyslog this isn't valid and
>> causes ?a problem. I don't remember if the problem is that rsyslog doesn't
>> send the logs, or that it has some other problem.
>>
>      Ok. But would that be a problem if instead of being forwarded to
> another machine it is just being saved to a local file? I ask since
> the ubuntu config file for rsyslog only has entries to save to log
> file with and without the "-".

In that case I believe that the '-' does nothing and causes no harm.

>>>> in other syslog implementations the - tells syslog that it doesn't have
>>>> to
>>>> do a fsync after writing each message to disk, it can just keep writing
>>>> and let the OS buffer them and write them to disk.
>>>>
>>>> This is the default in rsyslog.
>>>>
>>> ? ? Now it makes sense. ?Thanks!
>>>
>>>>> 2) Let's say I want to generate dynamic file names. Based on the man
>>>>> page, I create the following template:
>>>>>
>>>>> $template AuthFile,"/var/log/auth/auth-%$YEAR%-%$MONTH%-%$DAY%.log"
>>>>>
>>>>> And apply it to, say, auth:
>>>>>
>>>>> # auth,authpriv.* ? ? ? ? ? ? ? ? ? ? ? /var/log/auth.log
>>>>> # auth,authpriv.* ? ? ? ? ? ? ? ? -?AuthFile
>>>>> auth,authpriv.* ? ? ? ? ? ? ? ? ?AuthFile
>>>>>
>>>>> What I noticed is that instead of the log file be owned by syslog:adm,
>>>>> it is owned by syslog:syslog. Did I miss anything?
>>>>
>>>> I believe that there are options to define what the file ownership is.
>>>>
>>> ? ? in /etc/rsyslog.conf I have
>>>
>>> #
>>> # Set the default permissions for all log files.
>>> #
>>> $FileOwner syslog
>>> $FileGroup adm
>>> $FileCreateMode 0640
>>> $DirCreateMode 0755
>>> $Umask 0022
>>> $PrivDropToUser syslog
>>> $PrivDropToGroup syslog
>>>
>>> If I tell it to write to /var/log/auth.log, rsyslog seems to honor
>>> $FileOwner and $FileGroup. But when I tell it to create a dynamic
>>> filename log file, it seems to be using the user:group it dropped to
>>> once it started, namely syslog:syslog:
>>
>> that makes sense, once it drops privilages it may not have the ability to
>> use different credentials.
>>
>> the privilage dropping feature of rsyslog was a quick-and-dirty
>> modification, there are many things that do not work with it because after
>> the privilages are dropped the process doesn't have the rights to do what
>> it's trying to do.
>>
>      That said, in examples like
> http://wiki.rsyslog.com/index.php/DailyLogRotation, they are using
> dynamic filenames. I would think those examples reflect used
> configurations. Could it be they are older/newer examples, so that
> either the issue I am having did not exist or was solved? Or perhaps
> it is just me who did not set it up right.

but that example says nothing about dropping privilage.

There are a lot of things in rsyslog that work perfectly without dropping 
privilage, but don't work once privilages have been dropped. Unfortunantly 
such things have not been documented as such (mostly they've been 
identified as not working, questions are asked here or on the message 
board and the response is very similar to what I posted, just with the 
authority of the developers saying "yeah, that won't work with reduced 
privilages")

David Lang

From rgerhards at hq.adiscon.com  Tue Aug 31 07:58:57 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 31 Aug 2010 07:58:57 +0200
Subject: [rsyslog] $template to create filename, ownership,
	and other file-related questions
References: <AANLkTikxZcNDdVNjoG0ATp7JdG3Z2vAbnF7e61cZ4sJA@mail.gmail.com><alpine.DEB.2.00.1008301540510.12449@asgard.lang.hm><AANLkTina-AsorrDRZ+zu5Z9o3=ntRv1AsqX=LkKybNug@mail.gmail.com><alpine.DEB.2.00.1008301557360.12449@asgard.lang.hm><AANLkTi=q2cjizeMtDwyQO8af3jdy6w+v=B79qXiGmMM4@mail.gmail.com>
	<alpine.DEB.2.00.1008301713290.12449@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD289@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, August 31, 2010 2:18 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] $template to create filename, ownership, and
> other file-related questions
> 
> On Mon, 30 Aug 2010, Mauricio Tavares wrote:
> 
> > On Mon, Aug 30, 2010 at 7:02 PM,  <david at lang.hm> wrote:
> >> On Mon, 30 Aug 2010, Mauricio Tavares wrote:
> >>
> >>> On Mon, Aug 30, 2010 at 6:43 PM, ?<david at lang.hm> wrote:
> >>>>
> >>>> On Mon, 30 Aug 2010, Mauricio Tavares wrote:
> >>>>
> >>>>> In my ongoing quest to understand rsyslog, I have a few log
> >>>>> file-related questions:
> >>>>>
> >>>>> 1) What does the "-" in
> >>>>>
> >>>>> news.err ? ? ? ? ? ? ? ? ? ? ? ?/var/log/news/news.err
> >>>>> news.notice ? ? ? ? ? ? ? ? ? ? -/var/log/news/news.notice
> >>>>>
> >>>>> stand for? I probably passed through its explanation a few times
> but
> >>>>> did not see it.
> >>>>
> >>>> actually, in rsyslog the - has no effect (and can even cause
> problems)
> >>>>
> >>> ? ? How could it cause problems?
> >>
> >> in sysklogd syslog.conf - at ip is valid, in rsyslog this isn't valid
> and
> >> causes ?a problem. I don't remember if the problem is that rsyslog
> doesn't
> >> send the logs, or that it has some other problem.
> >>
> >      Ok. But would that be a problem if instead of being forwarded to
> > another machine it is just being saved to a local file? I ask since
> > the ubuntu config file for rsyslog only has entries to save to log
> > file with and without the "-".
> 
> In that case I believe that the '-' does nothing and causes no harm.

What it does depends on startup switches (see v3 compatibility doc:
http://www.rsyslog.com/doc/v3compatibility.html ) as well as config. By
default, it does nothing (see v3 comp. doc to learn why this is good). With
"$ActionFileEnableSync on" you can enable old-style behavior.
> 
> >>>> in other syslog implementations the - tells syslog that it doesn't
> have
> >>>> to
> >>>> do a fsync after writing each message to disk, it can just keep
> writing
> >>>> and let the OS buffer them and write them to disk.
> >>>>
> >>>> This is the default in rsyslog.
> >>>>
> >>> ? ? Now it makes sense. ?Thanks!
> >>>
> >>>>> 2) Let's say I want to generate dynamic file names. Based on the
> man
> >>>>> page, I create the following template:
> >>>>>
> >>>>> $template AuthFile,"/var/log/auth/auth-%$YEAR%-%$MONTH%-
> %$DAY%.log"
> >>>>>
> >>>>> And apply it to, say, auth:
> >>>>>
> >>>>> # auth,authpriv.* ? ? ? ? ? ? ? ? ? ? ? /var/log/auth.log
> >>>>> # auth,authpriv.* ? ? ? ? ? ? ? ? -?AuthFile
> >>>>> auth,authpriv.* ? ? ? ? ? ? ? ? ?AuthFile
> >>>>>
> >>>>> What I noticed is that instead of the log file be owned by
> syslog:adm,
> >>>>> it is owned by syslog:syslog. Did I miss anything?
> >>>>
> >>>> I believe that there are options to define what the file ownership
> is.
> >>>>
> >>> ? ? in /etc/rsyslog.conf I have
> >>>
> >>> #
> >>> # Set the default permissions for all log files.
> >>> #
> >>> $FileOwner syslog
> >>> $FileGroup adm
> >>> $FileCreateMode 0640
> >>> $DirCreateMode 0755
> >>> $Umask 0022
> >>> $PrivDropToUser syslog
> >>> $PrivDropToGroup syslog
> >>>
> >>> If I tell it to write to /var/log/auth.log, rsyslog seems to honor
> >>> $FileOwner and $FileGroup. But when I tell it to create a dynamic
> >>> filename log file, it seems to be using the user:group it dropped
> to
> >>> once it started, namely syslog:syslog:
> >>
> >> that makes sense, once it drops privilages it may not have the
> ability to
> >> use different credentials.
> >>
> >> the privilage dropping feature of rsyslog was a quick-and-dirty
> >> modification, there are many things that do not work with it because
> after
> >> the privilages are dropped the process doesn't have the rights to do
> what
> >> it's trying to do.
> >>
> >      That said, in examples like
> > http://wiki.rsyslog.com/index.php/DailyLogRotation, they are using
> > dynamic filenames. I would think those examples reflect used
> > configurations. Could it be they are older/newer examples, so that
> > either the issue I am having did not exist or was solved? Or perhaps
> > it is just me who did not set it up right.
> 
> but that example says nothing about dropping privilage.
> 
> There are a lot of things in rsyslog that work perfectly without
> dropping
> privilage, but don't work once privilages have been dropped.
> Unfortunantly
> such things have not been documented as such (mostly they've been
> identified as not working, questions are asked here or on the message
> board and the response is very similar to what I posted, just with the
> authority of the developers saying "yeah, that won't work with reduced
> privilages")

David is right on privilege drop. Most of these things are simply impossible
after you have dropped privileges. I am not sure if this is the case in your
scenario. Please remove the privilege drop for a test and let us know if the
files then have proper user and group.

Rainer
> 
> David Lang


From rgerhards at hq.adiscon.com  Tue Aug 31 08:00:30 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 31 Aug 2010 08:00:30 +0200
Subject: [rsyslog] $template to create filename, ownership,
	and other file-related questions
References: <AANLkTikxZcNDdVNjoG0ATp7JdG3Z2vAbnF7e61cZ4sJA@mail.gmail.com><alpine.DEB.2.00.1008301540510.12449@asgard.lang.hm><AANLkTina-AsorrDRZ+zu5Z9o3=ntRv1AsqX=LkKybNug@mail.gmail.com>
	<alpine.DEB.2.00.1008301557360.12449@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD28A@GRFEXC.intern.adiscon.com>

> in sysklogd syslog.conf - at ip is valid

Really? What does it do? As far as I remember, in sysklogd "-" was tied to
files as well as in rsyslog (keep in mind that I started with the sysklogd
config parser).

Rainer



From david at lang.hm  Tue Aug 31 08:07:01 2010
From: david at lang.hm (david at lang.hm)
Date: Mon, 30 Aug 2010 23:07:01 -0700 (PDT)
Subject: [rsyslog] $template to create filename, ownership,
 and other file-related questions
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD28A@GRFEXC.intern.adiscon.com>
References: <AANLkTikxZcNDdVNjoG0ATp7JdG3Z2vAbnF7e61cZ4sJA@mail.gmail.com><alpine.DEB.2.00.1008301540510.12449@asgard.lang.hm><AANLkTina-AsorrDRZ+zu5Z9o3=ntRv1AsqX=LkKybNug@mail.gmail.com>
	<alpine.DEB.2.00.1008301557360.12449@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA71DD28A@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.2.00.1008302305240.12449@asgard.lang.hm>

On Tue, 31 Aug 2010, Rainer Gerhards wrote:

>> in sysklogd syslog.conf - at ip is valid
>
> Really? What does it do? As far as I remember, in sysklogd "-" was tied to
> files as well as in rsyslog (keep in mind that I started with the sysklogd
> config parser).

it may not do anything, but it accepts it without any problem. I recently 
upgraded a bunch of systems and discovered a week or so later that I 
wasn't getting logs from them and the reasons ended up being that I had 
copied the - from the old syslog.conf file into the rsyslog.conf file. 
removing it and restarting rsyslog made the logs flow again.

David Lang


From rgerhards at hq.adiscon.com  Tue Aug 31 09:03:03 2010
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 31 Aug 2010 09:03:03 +0200
Subject: [rsyslog] $template to create filename, ownership,
	and other file-related questions
References: <AANLkTikxZcNDdVNjoG0ATp7JdG3Z2vAbnF7e61cZ4sJA@mail.gmail.com><alpine.DEB.2.00.1008301540510.12449@asgard.lang.hm><AANLkTina-AsorrDRZ+zu5Z9o3=ntRv1AsqX=LkKybNug@mail.gmail.com><alpine.DEB.2.00.1008301557360.12449@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA71DD28A@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.00.1008302305240.12449@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD28B@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, August 31, 2010 8:07 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] $template to create filename, ownership, and
> other file-related questions
> 
> On Tue, 31 Aug 2010, Rainer Gerhards wrote:
> 
> >> in sysklogd syslog.conf - at ip is valid
> >
> > Really? What does it do? As far as I remember, in sysklogd "-" was
> tied to
> > files as well as in rsyslog (keep in mind that I started with the
> sysklogd
> > config parser).
> 
> it may not do anything, but it accepts it without any problem. I
> recently
> upgraded a bunch of systems and discovered a week or so later that I
> wasn't getting logs from them and the reasons ended up being that I had
> copied the - from the old syslog.conf file into the rsyslog.conf file.
> removing it and restarting rsyslog made the logs flow again.


That's interesting. IMHO it is a config error which is simply not caught by
the sysklogd parser. It would be somewhat hard to emulate that in rsyslog, at
least with the new config system. But if it is considered worth it, I could
look into this issue.

Rainer


From raubvogel at gmail.com  Tue Aug 31 15:27:31 2010
From: raubvogel at gmail.com (Mauricio Tavares)
Date: Tue, 31 Aug 2010 09:27:31 -0400
Subject: [rsyslog] $template to create filename, ownership,
 and other file-related questions
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD28B@GRFEXC.intern.adiscon.com>
References: <AANLkTikxZcNDdVNjoG0ATp7JdG3Z2vAbnF7e61cZ4sJA@mail.gmail.com>
	<alpine.DEB.2.00.1008301540510.12449@asgard.lang.hm>
	<AANLkTina-AsorrDRZ+zu5Z9o3=ntRv1AsqX=LkKybNug@mail.gmail.com>
	<alpine.DEB.2.00.1008301557360.12449@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA71DD28A@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.00.1008302305240.12449@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA71DD28B@GRFEXC.intern.adiscon.com>
Message-ID: <AANLkTin+3chFJdv_Sbr8MHT=WuNVczi1uN7n=APDQ1iO@mail.gmail.com>

On Tue, Aug 31, 2010 at 3:03 AM, Rainer Gerhards
<rgerhards at hq.adiscon.com> wrote:
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Tuesday, August 31, 2010 8:07 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] $template to create filename, ownership, and
>> other file-related questions
>>
>> On Tue, 31 Aug 2010, Rainer Gerhards wrote:
>>
>> >> in sysklogd syslog.conf - at ip is valid
>> >
>> > Really? What does it do? As far as I remember, in sysklogd "-" was
>> tied to
>> > files as well as in rsyslog (keep in mind that I started with the
>> sysklogd
>> > config parser).
>>
>> it may not do anything, but it accepts it without any problem. I
>> recently
>> upgraded a bunch of systems and discovered a week or so later that I
>> wasn't getting logs from them and the reasons ended up being that I had
>> copied the - from the old syslog.conf file into the rsyslog.conf file.
>> removing it and restarting rsyslog made the logs flow again.
>
> That's interesting. IMHO it is a config error which is simply not caught by
> the sysklogd parser. It would be somewhat hard to emulate that in rsyslog, at
> least with the new config system. But if it is considered worth it, I could
> look into this issue.
>
      How about just mentioning it in the rsyslog.conf man page or
something like that? i guess the other question is how compatible to
sysklogd do you want to be and when to say enough is enough?

> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From raubvogel at gmail.com  Tue Aug 31 15:46:32 2010
From: raubvogel at gmail.com (Mauricio Tavares)
Date: Tue, 31 Aug 2010 09:46:32 -0400
Subject: [rsyslog] $template to create filename, ownership,
 and other file-related questions
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD289@GRFEXC.intern.adiscon.com>
References: <AANLkTikxZcNDdVNjoG0ATp7JdG3Z2vAbnF7e61cZ4sJA@mail.gmail.com>
	<alpine.DEB.2.00.1008301540510.12449@asgard.lang.hm>
	<AANLkTina-AsorrDRZ+zu5Z9o3=ntRv1AsqX=LkKybNug@mail.gmail.com>
	<alpine.DEB.2.00.1008301557360.12449@asgard.lang.hm>
	<AANLkTi=q2cjizeMtDwyQO8af3jdy6w+v=B79qXiGmMM4@mail.gmail.com>
	<alpine.DEB.2.00.1008301713290.12449@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA71DD289@GRFEXC.intern.adiscon.com>
Message-ID: <AANLkTin76EymCdPc5t2yk1BWBAtreg9xR1Y98GFr+Sjj@mail.gmail.com>

On Tue, Aug 31, 2010 at 1:58 AM, Rainer Gerhards
<rgerhards at hq.adiscon.com> wrote:
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Tuesday, August 31, 2010 2:18 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] $template to create filename, ownership, and
>> other file-related questions
>>
>> On Mon, 30 Aug 2010, Mauricio Tavares wrote:
>>
>> > On Mon, Aug 30, 2010 at 7:02 PM, ?<david at lang.hm> wrote:
>> >> On Mon, 30 Aug 2010, Mauricio Tavares wrote:
>> >>
>> >>> On Mon, Aug 30, 2010 at 6:43 PM, ?<david at lang.hm> wrote:
>> >>>>
>> >>>> On Mon, 30 Aug 2010, Mauricio Tavares wrote:
>> >>>>
>> >>>>> In my ongoing quest to understand rsyslog, I have a few log
>> >>>>> file-related questions:
>> >>>>>
>> >>>>> 1) What does the "-" in
>> >>>>>
>> >>>>> news.err ? ? ? ? ? ? ? ? ? ? ? ?/var/log/news/news.err
>> >>>>> news.notice ? ? ? ? ? ? ? ? ? ? -/var/log/news/news.notice
>> >>>>>
>> >>>>> stand for? I probably passed through its explanation a few times
>> but
>> >>>>> did not see it.
>> >>>>
>> >>>> actually, in rsyslog the - has no effect (and can even cause
>> problems)
>> >>>>
>> >>> ? ? How could it cause problems?
>> >>
>> >> in sysklogd syslog.conf - at ip is valid, in rsyslog this isn't valid
>> and
>> >> causes ?a problem. I don't remember if the problem is that rsyslog
>> doesn't
>> >> send the logs, or that it has some other problem.
>> >>
>> > ? ? ?Ok. But would that be a problem if instead of being forwarded to
>> > another machine it is just being saved to a local file? I ask since
>> > the ubuntu config file for rsyslog only has entries to save to log
>> > file with and without the "-".
>>
>> In that case I believe that the '-' does nothing and causes no harm.
>
> What it does depends on startup switches (see v3 compatibility doc:
> http://www.rsyslog.com/doc/v3compatibility.html ) as well as config. By
> default, it does nothing (see v3 comp. doc to learn why this is good). With
> "$ActionFileEnableSync on" you can enable old-style behavior.

Very nice reading! I do not know how much I have digested yet but it
is pretty nice. Now, I will need to keep it as close to
backwards-compatibility mode as I can for now... at least until we
move out of syslog-ng and I can make sure all machines we want to
monitor can do rsyslog (I did find out sunfreeware has a rsyslog
package, which is very good to know).

>>
>> >>>> in other syslog implementations the - tells syslog that it doesn't
>> have
>> >>>> to
>> >>>> do a fsync after writing each message to disk, it can just keep
>> writing
>> >>>> and let the OS buffer them and write them to disk.
>> >>>>
>> >>>> This is the default in rsyslog.
>> >>>>
>> >>> ? ? Now it makes sense. ?Thanks!
>> >>>
>> >>>>> 2) Let's say I want to generate dynamic file names. Based on the
>> man
>> >>>>> page, I create the following template:
>> >>>>>
>> >>>>> $template AuthFile,"/var/log/auth/auth-%$YEAR%-%$MONTH%-
>> %$DAY%.log"
>> >>>>>
>> >>>>> And apply it to, say, auth:
>> >>>>>
>> >>>>> # auth,authpriv.* ? ? ? ? ? ? ? ? ? ? ? /var/log/auth.log
>> >>>>> # auth,authpriv.* ? ? ? ? ? ? ? ? -?AuthFile
>> >>>>> auth,authpriv.* ? ? ? ? ? ? ? ? ?AuthFile
>> >>>>>
>> >>>>> What I noticed is that instead of the log file be owned by
>> syslog:adm,
>> >>>>> it is owned by syslog:syslog. Did I miss anything?
>> >>>>
>> >>>> I believe that there are options to define what the file ownership
>> is.
>> >>>>
>> >>> ? ? in /etc/rsyslog.conf I have
>> >>>
>> >>> #
>> >>> # Set the default permissions for all log files.
>> >>> #
>> >>> $FileOwner syslog
>> >>> $FileGroup adm
>> >>> $FileCreateMode 0640
>> >>> $DirCreateMode 0755
>> >>> $Umask 0022
>> >>> $PrivDropToUser syslog
>> >>> $PrivDropToGroup syslog
>> >>>
>> >>> If I tell it to write to /var/log/auth.log, rsyslog seems to honor
>> >>> $FileOwner and $FileGroup. But when I tell it to create a dynamic
>> >>> filename log file, it seems to be using the user:group it dropped
>> to
>> >>> once it started, namely syslog:syslog:
>> >>
>> >> that makes sense, once it drops privilages it may not have the
>> ability to
>> >> use different credentials.
>> >>
>> >> the privilage dropping feature of rsyslog was a quick-and-dirty
>> >> modification, there are many things that do not work with it because
>> after
>> >> the privilages are dropped the process doesn't have the rights to do
>> what
>> >> it's trying to do.
>> >>
>> > ? ? ?That said, in examples like
>> > http://wiki.rsyslog.com/index.php/DailyLogRotation, they are using
>> > dynamic filenames. I would think those examples reflect used
>> > configurations. Could it be they are older/newer examples, so that
>> > either the issue I am having did not exist or was solved? Or perhaps
>> > it is just me who did not set it up right.
>>
>> but that example says nothing about dropping privilage.
>>
>> There are a lot of things in rsyslog that work perfectly without
>> dropping
>> privilage, but don't work once privilages have been dropped.
>> Unfortunantly
>> such things have not been documented as such (mostly they've been
>> identified as not working, questions are asked here or on the message
>> board and the response is very similar to what I posted, just with the
>> authority of the developers saying "yeah, that won't work with reduced
>> privilages")
>
> David is right on privilege drop. Most of these things are simply impossible
> after you have dropped privileges. I am not sure if this is the case in your
> scenario. Please remove the privilege drop for a test and let us know if the
> files then have proper user and group.
>
      That seems to have done the trick:

raub at ubuntu1004-x64:/etc/rsyslog.d$ ls -lh /var/log/auth/
total 4.0K
-rw-r----- 1 syslog syslog   0 2010-08-30 17:33 auth-2010-08-30.log
-rw-r----- 1 syslog adm    135 2010-08-31 09:30 auth-2010-08-31.log
raub at ubuntu1004-x64:/etc/rsyslog.d$

Thanks!

Ok, I have yet another answer: the template I created,

$template AuthFile,"/var/log/auth/auth-%$YEAR%-%$MONTH%-%$DAY%.log"

is nice but i was wondering if there is a way to pass an argument to
it from the selector line. This way, I could do, say,

$template LogFile,"/var/log/$Moose/$Moose-%$YEAR%-%$MONTH%-%$DAY%.log"

and then pass the value of $Moose when calling that template

mail.info    $Moose="mail", ?LogFile

I know the syntax is completely wrong but you can see where I am
getting at. Is there a way to do that?

> Rainer
>>
>> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>