From rgerhards at hq.adiscon.com  Tue Dec  1 10:52:25 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 1 Dec 2009 10:52:25 +0100
Subject: [rsyslog] bug reporting & bugzilla
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034ED@GRFEXC.intern.adiscon.com>

Hi all,

I just wanted to send a friendly reminder about rsyslog's bugzilla, available
at

http://bugzilla.adiscon.com/

It is very useful in keeping track what is open. I just browsed through the
mailing list as I thought there were some more issues open, but either I
didn't find them or all bugs mentioned are now fixed. But I guess I have
overlooked some. It would be good if you could open up a bug tracker once we
know that something is a bug. Of course, that is not strictly necessary, but
it can help getting things done as quickly as possible (usually, I look at
the bug tracker and mailing list each time I am done with a major milestone).

I just wanted to remind everyone of that resource. Currently, I am looking
into a problem with transaction batch procesing and ompgsql.

Rainer

From rgerhards at hq.adiscon.com  Tue Dec  1 18:05:50 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 1 Dec 2009 18:05:50 +0100
Subject: [rsyslog] using arbitrary facility id
References: <87a1ae540911300820s3ce60cafq5c258b355ed3e887@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA71034E7@GRFEXC.intern.adiscon.com><87a1ae540911300851r1f0bab68ldf7f995ed0a094d8@mail.gmail.com>
	<87a1ae540911300858v1d4cda71k787a53158e162b7d@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034F7@GRFEXC.intern.adiscon.com>

Oh, that was a long debate... I intended to permit 2^32-1 facilities, but
many others insisted on backward compatibility. If you search the ietf
syslog-sec mailing list archive, you'll find a heated debate ;) So I really
don't like to argue about that again. But the fact is that it doesn't help if
everyone ignores the standard. There are ample ways of filtering different
messages, especially if using RFC5424 format...

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury
> Sent: Monday, November 30, 2009 5:59 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] using arbitrary facility id
> 
> BTW, just out of interest, why is this number restricted to 23 in the
> rfc?
> also each facility other than local0-7 seems to be rigidly defined.
> Just
> wanted to know why is it so ..shouldn't they have  left scope for
> extension
> on this?
> 
> 
> On Mon, Nov 30, 2009 at 11:51 AM, Sayan Chowdhury
> <iamsayan at gmail.com>wrote:
> 
> > Hello Rainer,
> > Thanks... Yes I agree what I am doing is invalid.
> >  I was just trying it out, my idea was if I can use numbers greater
> than
> > 23, then I can maybe use it as my own customized logging system, and
> I would
> > use numbers greater than 23 for my application and use different ids
> (>23)
> > for different kind of application level log messages. I would try to
> look
> > into the code as well, is this something you thing may be useful in
> general?
> > Regards,
> > Sayan
> >
> >
> >
> > On Mon, Nov 30, 2009 at 11:40 AM, Rainer Gerhards <
> > rgerhards at hq.adiscon.com> wrote:
> >
> >> You can use facilities other than local0..local7, but you cannot use
> a
> >> facility with a numerical value greater than 23, because the
> relevant
> >> standards do not permit this (see RFC5424, Table 1).
> >>
> >> It may be that rsyslog does not properly prevent this, maybe it uses
> >> modulo
> >> 24 in this case. Will check that. But it is invalid in any case (not
> only
> >> with rsyslog but with any syslogd).
> >>
> >> Rainer
> >>
> >> > -----Original Message-----
> >> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury
> >> > Sent: Monday, November 30, 2009 5:20 PM
> >> > To: rsyslog-users
> >> > Subject: [rsyslog] using arbitrary facility id
> >> >
> >> > Hello All,
> >> > Is it possible to use a facility id other than local0-local7?
> >> > I was using a facility id of 50 in some of the messages , and I
> had
> >> > written
> >> > a selector line in my rsyslog file as well to log messages with
> >> > facility id
> >> > of 50 into a seperate file.
> >> > However, I see that sometimes the messages are being written into
> all
> >> > the
> >> > files(/var/log/messages, boot.log,/var/log/secure etc) instead of
> the
> >> > one I
> >> > specified in the rsyslog.conf. If I restart rsyslog the problem
> goes
> >> > away.
> >> > I am using rsyslog version 4.2.0.
> >> >
> >> > Here is the selector line in my config
> >> >
> >> >
> >> > if $fromhost-ip == '127.0.0.1' and $syslogfacility == '50' and
> >> > $syslogseverity <= '6' then $log_rotation_50
> >> >
> >> > where $log_rotation_50 is an outchannel which is configured to
> rotate
> >> > the
> >> > file when it reaches the size of 2MB.
> >> > Regards,
> >> > Sayan
> >> > _______________________________________________
> >> > rsyslog mailing list
> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> > http://www.rsyslog.com
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >>
> >
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Tue Dec  1 18:07:19 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 1 Dec 2009 18:07:19 +0100
Subject: [rsyslog] using arbitrary facility id
References: <87a1ae540911300820s3ce60cafq5c258b355ed3e887@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA71034E7@GRFEXC.intern.adiscon.com>
	<87a1ae540911300851r1f0bab68ldf7f995ed0a094d8@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034F8@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury
> Sent: Monday, November 30, 2009 5:52 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] using arbitrary facility id
> 
> Hello Rainer,
> Thanks... Yes I agree what I am doing is invalid.
>  I was just trying it out, my idea was if I can use numbers greater
> than 23,
> then I can maybe use it as my own customized logging system, and I
> would use
> numbers greater than 23 for my application and use different ids (>23)
> for
> different kind of application level log messages. I would try to look
> into
> the code as well, is this something you thing may be useful in general?
> Regards,
> Sayan


You can definitely modify rsyslogd to support this, but you must be very
careful that you catch all the loose ends. Be especially careful with the
lookup table e.g. in msg.c.

I think the overall usefulness is limited, as no well-behaved sender would
emit such facility IDs (please read my other mail from a minute ago as well).

Rainer
 
> 
> 
> On Mon, Nov 30, 2009 at 11:40 AM, Rainer Gerhards
> <rgerhards at hq.adiscon.com>wrote:
> 
> > You can use facilities other than local0..local7, but you cannot use
> a
> > facility with a numerical value greater than 23, because the relevant
> > standards do not permit this (see RFC5424, Table 1).
> >
> > It may be that rsyslog does not properly prevent this, maybe it uses
> modulo
> > 24 in this case. Will check that. But it is invalid in any case (not
> only
> > with rsyslog but with any syslogd).
> >
> > Rainer
> >
> > > -----Original Message-----
> > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury
> > > Sent: Monday, November 30, 2009 5:20 PM
> > > To: rsyslog-users
> > > Subject: [rsyslog] using arbitrary facility id
> > >
> > > Hello All,
> > > Is it possible to use a facility id other than local0-local7?
> > > I was using a facility id of 50 in some of the messages , and I had
> > > written
> > > a selector line in my rsyslog file as well to log messages with
> > > facility id
> > > of 50 into a seperate file.
> > > However, I see that sometimes the messages are being written into
> all
> > > the
> > > files(/var/log/messages, boot.log,/var/log/secure etc) instead of
> the
> > > one I
> > > specified in the rsyslog.conf. If I restart rsyslog the problem
> goes
> > > away.
> > > I am using rsyslog version 4.2.0.
> > >
> > > Here is the selector line in my config
> > >
> > >
> > > if $fromhost-ip == '127.0.0.1' and $syslogfacility == '50' and
> > > $syslogseverity <= '6' then $log_rotation_50
> > >
> > > where $log_rotation_50 is an outchannel which is configured to
> rotate
> > > the
> > > file when it reaches the size of 2MB.
> > > Regards,
> > > Sayan
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Tue Dec  1 18:08:27 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 1 Dec 2009 18:08:27 +0100
Subject: [rsyslog] using arbitrary facility id
References: <87a1ae540911300820s3ce60cafq5c258b355ed3e887@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA71034E7@GRFEXC.intern.adiscon.com><87a1ae540911300851r1f0bab68ldf7f995ed0a094d8@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA71034F8@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034F9@GRFEXC.intern.adiscon.com>

oh, and I forgot to mention that you will run into serious trouble with those
non-standard facility IDs if you ever intend to use it with the upcoming
syslog-sign standard (for end-to-end message verfification). They simply
won't play together.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Tuesday, December 01, 2009 6:07 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] using arbitrary facility id
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury
> > Sent: Monday, November 30, 2009 5:52 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] using arbitrary facility id
> >
> > Hello Rainer,
> > Thanks... Yes I agree what I am doing is invalid.
> >  I was just trying it out, my idea was if I can use numbers greater
> > than 23,
> > then I can maybe use it as my own customized logging system, and I
> > would use
> > numbers greater than 23 for my application and use different ids
> (>23)
> > for
> > different kind of application level log messages. I would try to look
> > into
> > the code as well, is this something you thing may be useful in
> general?
> > Regards,
> > Sayan
> 
> 
> You can definitely modify rsyslogd to support this, but you must be
> very
> careful that you catch all the loose ends. Be especially careful with
> the
> lookup table e.g. in msg.c.
> 
> I think the overall usefulness is limited, as no well-behaved sender
> would
> emit such facility IDs (please read my other mail from a minute ago as
> well).
> 
> Rainer
> 
> >
> >
> > On Mon, Nov 30, 2009 at 11:40 AM, Rainer Gerhards
> > <rgerhards at hq.adiscon.com>wrote:
> >
> > > You can use facilities other than local0..local7, but you cannot
> use
> > a
> > > facility with a numerical value greater than 23, because the
> relevant
> > > standards do not permit this (see RFC5424, Table 1).
> > >
> > > It may be that rsyslog does not properly prevent this, maybe it
> uses
> > modulo
> > > 24 in this case. Will check that. But it is invalid in any case
> (not
> > only
> > > with rsyslog but with any syslogd).
> > >
> > > Rainer
> > >
> > > > -----Original Message-----
> > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury
> > > > Sent: Monday, November 30, 2009 5:20 PM
> > > > To: rsyslog-users
> > > > Subject: [rsyslog] using arbitrary facility id
> > > >
> > > > Hello All,
> > > > Is it possible to use a facility id other than local0-local7?
> > > > I was using a facility id of 50 in some of the messages , and I
> had
> > > > written
> > > > a selector line in my rsyslog file as well to log messages with
> > > > facility id
> > > > of 50 into a seperate file.
> > > > However, I see that sometimes the messages are being written into
> > all
> > > > the
> > > > files(/var/log/messages, boot.log,/var/log/secure etc) instead of
> > the
> > > > one I
> > > > specified in the rsyslog.conf. If I restart rsyslog the problem
> > goes
> > > > away.
> > > > I am using rsyslog version 4.2.0.
> > > >
> > > > Here is the selector line in my config
> > > >
> > > >
> > > > if $fromhost-ip == '127.0.0.1' and $syslogfacility == '50' and
> > > > $syslogseverity <= '6' then $log_rotation_50
> > > >
> > > > where $log_rotation_50 is an outchannel which is configured to
> > rotate
> > > > the
> > > > file when it reaches the size of 2MB.
> > > > Regards,
> > > > Sayan
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> > >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From dorourke at qualcomm.com  Tue Dec  1 21:50:48 2009
From: dorourke at qualcomm.com (O'Rourke, Don)
Date: Tue, 1 Dec 2009 12:50:48 -0800
Subject: [rsyslog] rsyslog V2: Applications block when rsyslog has remote
 messages queued
Message-ID: <FB6406AAF9E49944AB1E95DA5823A64C015F332991@NALASEXMB08.na.qualcomm.com>

I'm running rsyslog 2.0.6 and ran into a problem in which an application blocked when the remote rsyslog server became unavailable.  I noticed this problem reported in a post to this message board in August:

http://lists.adiscon.net/pipermail/rsyslog/2009-August/002576.html

Is it possible that this problem will eventually be fixed in V2?


Thanks,
Don


From rgerhards at hq.adiscon.com  Wed Dec  2 07:27:31 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 2 Dec 2009 07:27:31 +0100
Subject: [rsyslog] rsyslog V2: Applications block when rsyslog has
	remote messages queued
References: <FB6406AAF9E49944AB1E95DA5823A64C015F332991@NALASEXMB08.na.qualcomm.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034FC@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of O'Rourke, Don
> Sent: Tuesday, December 01, 2009 9:51 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] rsyslog V2: Applications block when 
> rsyslog has remote messages queued
> 
> I'm running rsyslog 2.0.6 and ran into a problem in which an 
> application blocked when the remote rsyslog server became 
> unavailable.  I noticed this problem reported in a post to 
> this message board in August:
> 
> http://lists.adiscon.net/pipermail/rsyslog/2009-August/002576.html
> 
> Is it possible that this problem will eventually be fixed in V2?

It's very unlikely. V2 is a *very* old release (we are at v5 now), that I
only work on under paid assignments. Even then, I am not sure if it can be
fixed without pulling most of the core v3 code. But I haven't looked at the
issue, so it may be possible...

Rainer

From Helwing at distec.de  Wed Dec  2 09:22:37 2009
From: Helwing at distec.de (DISTEC Helwing Lutz)
Date: Wed, 2 Dec 2009 09:22:37 +0100
Subject: [rsyslog] Cross compiling rsyslogd...
Message-ID: <BFE810BB1BD20E47876F1BB1EA7BF7303D1DC5@srv-exchange.datadisplay.de>

Hi folks,

I'm trying to cross compile rsyslog for an embedded system running with
ARM v5t processor. While doing this I ran into some troubles:
(I've considered some hints from this site:
http://cross-lfs.org/view/clfs-2.0/arm/final-system/rsyslog.html)
 

1. Configure seems to not determine correct values for the following: 
- ac_cv_func_malloc_0_nonnull
- ac_cv_func_realloc_0_nonnull
After setting these to yes the list of errors shrinks dramatically but
there are still some left...


2. There seems to be a problem with atomic operations. AFAIK my
processor supports atomic operations (please correct me if I'm wrong).
Configure instead determines that atomic support should not be enabled
(ap_cv_atomic_builtins=no). When running make afterwards i get some
errors like "glbl.c:134: undefined reference to
'ATOMIC_STORE_1_TO_INT'". When looking into atomic.h it becomes clear
what causes this error: some macros are not defined when
HAVE_ATOMIC_BUILTINS is not defined but are obviously used nevertheless.

I get rid of this problem by setting ap_cv_atomic_builtins=yes
explicitly. When combining the solutions of 1. and 2. make builds
successfully.
 

3. When starting rsyslogd an the ARM machine I get a segmentation fault
(unfortunately i have no stack trace, but i could send debug output from
rsyslogd and rsyslog.conf if neccessary).


4. When commenting out the two lines for module load i get an alignment
trap: Alignment trap: rs:main Q:Reg (12879) PC=0x0004ff8c
Instr=0xe5932014 Address=0x20766f62 FSR 0x001 Bus error 
(debug output could be sent if neccessary)


Is it possible that issues 3. and 4. have something to do with the
atomics issue? 
Maybe anyone has an idea what else could be wrong here?
Thanks a lot.
 
Cheers,
Lutz

From rgerhards at hq.adiscon.com  Wed Dec  2 10:04:05 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 2 Dec 2009 10:04:05 +0100
Subject: [rsyslog] Cross compiling rsyslogd...
References: <BFE810BB1BD20E47876F1BB1EA7BF7303D1DC5@srv-exchange.datadisplay.de>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034FF@GRFEXC.intern.adiscon.com>

Hi Lutz,

interesting project :)

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of DISTEC Helwing Lutz
> Sent: Wednesday, December 02, 2009 9:23 AM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] Cross compiling rsyslogd...
> 
> Hi folks,
> 
> I'm trying to cross compile rsyslog for an embedded system running with
> ARM v5t processor. While doing this I ran into some troubles:
> (I've considered some hints from this site:
> http://cross-lfs.org/view/clfs-2.0/arm/final-system/rsyslog.html )

mhhh - the link doesn't work for me. Some typo?

> 1. Configure seems to not determine correct values for the following:
> - ac_cv_func_malloc_0_nonnull
> - ac_cv_func_realloc_0_nonnull
> After setting these to yes the list of errors shrinks dramatically but
> there are still some left...

I am primarily a user of autotools and I have to admit that I do not know
what these settings do or cause. Can someone jump in?

> 2. There seems to be a problem with atomic operations. AFAIK my
> processor supports atomic operations (please correct me if I'm wrong).

I have no idea if it supports them.

> Configure instead determines that atomic support should not be enabled
> (ap_cv_atomic_builtins=no). 

That sounds like a strong indication they are actually not supported. What
the configure test tries is compile a program using them and that obviously
fails...

> When running make afterwards i get some
> errors like "glbl.c:134: undefined reference to
> 'ATOMIC_STORE_1_TO_INT'". When looking into atomic.h it becomes clear
> what causes this error: some macros are not defined when
> HAVE_ATOMIC_BUILTINS is not defined but are obviously used
> nevertheless.

That is because recent versions of rsyslog can simply not work without
atomics. So far, it was not worth the effort implementing that. 

> I get rid of this problem by setting ap_cv_atomic_builtins=yes
> explicitly. When combining the solutions of 1. and 2. make builds
> successfully.

This sounds dangerous, but when it does not result in compile errors, it may
work. 

> 3. When starting rsyslogd an the ARM machine I get a segmentation fault
> (unfortunately i have no stack trace, but i could send debug output
> from
> rsyslogd and rsyslog.conf if neccessary).
> 
> 
> 4. When commenting out the two lines for module load i get an alignment

which lines do you mean?

> trap: Alignment trap: rs:main Q:Reg (12879) PC=0x0004ff8c
> Instr=0xe5932014 Address=0x20766f62 FSR 0x001 Bus error
> (debug output could be sent if neccessary)
> 
> 
> Is it possible that issues 3. and 4. have something to do with the
> atomics issue?
> Maybe anyone has an idea what else could be wrong here?
> Thanks a lot.

To approach this problem, I think it is critical to get a clear understanding
of the availability of atomic instructions on your platform. If they are not
actually available but you use dummies, all kinds of strange things may
happen (including #3 and #4).

So the first thing to do is find a definite reference if your
processor/compiler combination supports them.

Depending on the outcome the next step is:

If atomics are supported, find out why configure claims they are not and how
to fix that (this ensures that everything actually needed is applied during
the build process).

If atomics are not supported, you need to replace the atomic instructions
with carfully-crafted mutex calls inside the code. This is the only safe way
to do it. It's quite possible, but I have not done it because I have no need
for it.

After these steps, we should have a platform that we know to work correctly
in regard to the atomics, so then we can look at debugging any remaining
issues.

If you modify some code, please contribute the patches back, I will then be
able to include them (and also do some checks if they fit into the overall
picture etc).

HTH
Rainer
> 
> Cheers,
> Lutz
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From sebastien.bonnegent at insa-rouen.fr  Wed Dec  2 10:46:27 2009
From: sebastien.bonnegent at insa-rouen.fr (Sebastien Bonnegent)
Date: Wed, 02 Dec 2009 10:46:27 +0100
Subject: [rsyslog] rsyslog v3 / v4 and Ubuntu Karmic
Message-ID: <4B163773.1080309@insa-rouen.fr>

Hello,

I have a rsyslog server and I use gtls (Debian GNU/Linux 5.0 / rsyslog 3.18.6-4). I have no
problem with a client with Ubuntu 9.04 and rsyslog 3.18.6-3, but I have some troubles with
the same configuration on a client with Ubuntu 9.10 / rsyslog 4.2.0-2. With this client, I
obtain on the server this line (bkpasi is the name of my client):

Dec  2 10:38:12 bkpasi #026#003#002#000C#001#000#000?#003#002K#0265????6?4*?*^???tN/)
?dy?9?#000#000#030#0003#0009#000#026#0002#0008#000#023#000f#000/#0005

It seems to be a SSL problem, but this config works well with rsyslog v3 client.

Are they a problem between v3 and v4 ? Maybe a problem with the ubuntu package ?
What do you think ?

Client with problem: Ubuntu 9.10 / rsyslog 4.2.0-2 (package rsyslog and rsyslog-gnutls)
Client ok: Ubuntu 9.04 / rsyslog 3.18.6-3

Client configuration:
> cat /etc/rsyslog.conf /etc/rsyslog.d/*.conf | egrep -v '^$|^#'
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$FileOwner root
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$IncludeConfig /etc/rsyslog.d/*.conf
*.emerg                         *
$DefaultNetstreamDriver gtls
$DefaultNetstreamDriverCAFile /root/pki/ca.crt
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode anon
*.* @@(o)172.29.0.5:9999

Server: Debian GNU/Linux 5.0 / rsyslog 3.18.6-4

Server configuration:
> cat /etc/rsyslog.conf /etc/rsyslog.d/*.conf | egrep -v '^$|^#'
$ModLoad imuxsock # provides support for local system logging
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$FileOwner root
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$IncludeConfig /etc/rsyslog.d/*.conf
auth,authpriv.*                 /var/log/auth.log
cron.*                          /var/log/cron.log
daemon.*                        -/var/log/daemon.log
kern.*                          -/var/log/kern.log
lpr.*                           -/var/log/lpr.log
mail.*                          -/var/log/mail.log
user.*                          -/var/log/user.log
mail.info                       -/var/log/mail.info
mail.warn                       -/var/log/mail.warn
mail.err                        /var/log/mail.err
news.crit                       /var/log/news/news.crit
news.err                        /var/log/news/news.err
news.notice                     -/var/log/news/news.notice
local1.info                     /var/log/apache2/access.log
local7.*                        /var/log/apache2/error.log
*.emerg                         *
$DefaultNetstreamDriver gtls
$DefaultNetstreamDriverCAFile /root/pki/ca.crt
$DefaultNetstreamDriverCertFile /root/pki/log.crt
$DefaultNetstreamDriverKeyFile /root/pki/log.key
$ModLoad /usr/lib/rsyslog/imtcp
$InputTCPServerStreamDriverMode 1 # TLS only mode
$InputTCPServerStreamDriverAuthMode anon # client not authenticated
$InputTCPServerRun 9999



Thank you.
-- 
Cordialement,

Sebastien Bonnegent
--
          "GNU/Linux, il y a moins bien mais c'est plus cher."
  -----------------------------------------------------------------------
  | http://www.insa-rouen.fr/ | Tel: 02 32 95 98 61 | GnuPG: 0x669176B0 |
  -----------------------------------------------------------------------

From ktm at rice.edu  Wed Dec  2 15:03:07 2009
From: ktm at rice.edu (Kenneth Marshall)
Date: Wed, 2 Dec 2009 08:03:07 -0600
Subject: [rsyslog] omfile does not compile on 32-bit platforms in 5.3.5
Message-ID: <20091202140307.GI3237@it.is.rice.edu>

Hi Rainier,

The version of omfile.c does not compile/run on 32-bit
systems anymore. Here is the problem function:

static uint64 clockFileAccess = 0;
/* and the "tick" function */
static inline uint64
getClockFileAccess(void)
{
        return ATOMIC_INC_AND_FETCH(clockFileAccess);
}

You cannot perform an atomic operation on an 8 byte value
on a 32-bit system. Would it be possible to use the atomic
operations on two 4 byte values to allow this code to work
on 32-bit systems as well?

Regards,
Ken

From rgerhards at hq.adiscon.com  Wed Dec  2 15:45:24 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 2 Dec 2009 15:45:24 +0100
Subject: [rsyslog] omfile does not compile on 32-bit platforms in 5.3.5
Message-ID: <000401ca735e$27b5fd7d$100013ac@intern.adiscon.com>

Well, we could use a single 32 bit value without much problem, but the gcc doc claims gcc will replace the call with a helper function (Using a mutex, it can be implemented on any platform).

rainer

----- Urspr?ngliche Nachricht -----
Von: "Kenneth Marshall" <ktm at rice.edu>
An: "rsyslog at lists.adiscon.com" <rsyslog at lists.adiscon.com>
Gesendet: 02.12.09 15:22
Betreff: [rsyslog] omfile does not compile on 32-bit platforms in 5.3.5

Hi Rainier,

The version of omfile.c does not compile/run on 32-bit
systems anymore. Here is the problem function:

static uint64 clockFileAccess = 0;
/* and the "tick" function */
static inline uint64
getClockFileAccess(void)
{
        return ATOMIC_INC_AND_FETCH(clockFileAccess);
}

You cannot perform an atomic operation on an 8 byte value
on a 32-bit system. Would it be possible to use the atomic
operations on two 4 byte values to allow this code to work
on 32-bit systems as well?

Regards,
Ken
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

From ktm at rice.edu  Wed Dec  2 15:55:31 2009
From: ktm at rice.edu (Kenneth Marshall)
Date: Wed, 2 Dec 2009 08:55:31 -0600
Subject: [rsyslog] omfile does not compile on 32-bit platforms in 5.3.5
In-Reply-To: <000401ca735e$27b5fd7d$100013ac@intern.adiscon.com>
References: <000401ca735e$27b5fd7d$100013ac@intern.adiscon.com>
Message-ID: <20091202145531.GJ3237@it.is.rice.edu>

On Wed, Dec 02, 2009 at 03:45:24PM +0100, Rainer Gerhards wrote:
> Well, we could use a single 32 bit value without much problem, but the gcc doc claims gcc will replace the call with a helper function (Using a mutex, it can be implemented on any platform).
> 
> rainer

Yes, it does put a call to a function __sync_fetch_and_add_8()
stub which is why the link fails. It just seemed that it would
be easier to use the 4 byte counter and have one method that
would work across 32-bit and 64-bit systems, instead of needing
to support the missing function which could be implemented
using the same call with a 4 byte value. It would also simplify
code maintenance.

Regards,
Ken

> 
> ----- Urspr??ngliche Nachricht -----
> Von: "Kenneth Marshall" <ktm at rice.edu>
> An: "rsyslog at lists.adiscon.com" <rsyslog at lists.adiscon.com>
> Gesendet: 02.12.09 15:22
> Betreff: [rsyslog] omfile does not compile on 32-bit platforms in 5.3.5
> 
> Hi Rainier,
> 
> The version of omfile.c does not compile/run on 32-bit
> systems anymore. Here is the problem function:
> 
> static uint64 clockFileAccess = 0;
> /* and the "tick" function */
> static inline uint64
> getClockFileAccess(void)
> {
>         return ATOMIC_INC_AND_FETCH(clockFileAccess);
> }
> 
> You cannot perform an atomic operation on an 8 byte value
> on a 32-bit system. Would it be possible to use the atomic
> operations on two 4 byte values to allow this code to work
> on 32-bit systems as well?
> 
> Regards,
> Ken
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Wed Dec  2 16:01:57 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 2 Dec 2009 16:01:57 +0100
Subject: [rsyslog] omfile does not compile on 32-bit platforms in 5.3.5
Message-ID: <000501ca7360$798449f7$100013ac@intern.adiscon.com>

Well, gcc itself should provide the helper - at least this is how I read the doc. But, well, it looks more appropriate to change the variable definition for 32 bit systems. I'll add a --enable-32-bit-atomics-where-possible configure switch. If someone could contribute an automatic check for 32bit, that would be appreciated. 

I would like to retain 64 bits on platforms that support it, because this is the cleanest solution.

rainer

----- Urspr?ngliche Nachricht -----
Von: "Kenneth Marshall" <ktm at rice.edu>
An: "rsyslog-users" <rsyslog at lists.adiscon.com>
Gesendet: 02.12.09 15:56
Betreff: Re: [rsyslog] omfile does not compile on 32-bit platforms in 5.3.5

On Wed, Dec 02, 2009 at 03:45:24PM +0100, Rainer Gerhards wrote:
> Well, we could use a single 32 bit value without much problem, but the gcc doc claims gcc will replace the call with a helper function (Using a mutex, it can be implemented on any platform).
> 
> rainer

Yes, it does put a call to a function __sync_fetch_and_add_8()
stub which is why the link fails. It just seemed that it would
be easier to use the 4 byte counter and have one method that
would work across 32-bit and 64-bit systems, instead of needing
to support the missing function which could be implemented
using the same call with a 4 byte value. It would also simplify
code maintenance.

Regards,
Ken

> 
> ----- Urspr??ngliche Nachricht -----
> Von: "Kenneth Marshall" <ktm at rice.edu>
> An: "rsyslog at lists.adiscon.com" <rsyslog at lists.adiscon.com>
> Gesendet: 02.12.09 15:22
> Betreff: [rsyslog] omfile does not compile on 32-bit platforms in 5.3.5
> 
> Hi Rainier,
> 
> The version of omfile.c does not compile/run on 32-bit
> systems anymore. Here is the problem function:
> 
> static uint64 clockFileAccess = 0;
> /* and the "tick" function */
> static inline uint64
> getClockFileAccess(void)
> {
>         return ATOMIC_INC_AND_FETCH(clockFileAccess);
> }
> 
> You cannot perform an atomic operation on an 8 byte value
> on a 32-bit system. Would it be possible to use the atomic
> operations on two 4 byte values to allow this code to work
> on 32-bit systems as well?
> 
> Regards,
> Ken
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

From ktm at rice.edu  Wed Dec  2 16:35:26 2009
From: ktm at rice.edu (Kenneth Marshall)
Date: Wed, 2 Dec 2009 09:35:26 -0600
Subject: [rsyslog] omfile does not compile on 32-bit platforms in 5.3.5
In-Reply-To: <000501ca7360$798449f7$100013ac@intern.adiscon.com>
References: <000501ca7360$798449f7$100013ac@intern.adiscon.com>
Message-ID: <20091202153526.GK3237@it.is.rice.edu>

On Wed, Dec 02, 2009 at 04:01:57PM +0100, Rainer Gerhards wrote:
> Well, gcc itself should provide the helper - at least this is how I read the doc. But, well, it looks more appropriate to change the variable definition for 32 bit systems. I'll add a --enable-32-bit-atomics-where-possible configure switch. If someone could contribute an automatic check for 32bit, that would be appreciated. 
> 
> I would like to retain 64 bits on platforms that support it, because this is the cleanest solution.
> 
> rainer
> 

If you just change the type used in the current check function
to "long long", it will work on 64-bit systems and fail on 32-bit
systems. Then if the existing function with the "long" definition
work, the system supports 32-bit atomics. Or am I missing something?

Regards,
Ken

> ----- Urspr??ngliche Nachricht -----
> Von: "Kenneth Marshall" <ktm at rice.edu>
> An: "rsyslog-users" <rsyslog at lists.adiscon.com>
> Gesendet: 02.12.09 15:56
> Betreff: Re: [rsyslog] omfile does not compile on 32-bit platforms in 5.3.5
> 
> On Wed, Dec 02, 2009 at 03:45:24PM +0100, Rainer Gerhards wrote:
> > Well, we could use a single 32 bit value without much problem, but the gcc doc claims gcc will replace the call with a helper function (Using a mutex, it can be implemented on any platform).
> > 
> > rainer
> 
> Yes, it does put a call to a function __sync_fetch_and_add_8()
> stub which is why the link fails. It just seemed that it would
> be easier to use the 4 byte counter and have one method that
> would work across 32-bit and 64-bit systems, instead of needing
> to support the missing function which could be implemented
> using the same call with a 4 byte value. It would also simplify
> code maintenance.
> 
> Regards,
> Ken
> 
> > 
> > ----- Urspr??ngliche Nachricht -----
> > Von: "Kenneth Marshall" <ktm at rice.edu>
> > An: "rsyslog at lists.adiscon.com" <rsyslog at lists.adiscon.com>
> > Gesendet: 02.12.09 15:22
> > Betreff: [rsyslog] omfile does not compile on 32-bit platforms in 5.3.5
> > 
> > Hi Rainier,
> > 
> > The version of omfile.c does not compile/run on 32-bit
> > systems anymore. Here is the problem function:
> > 
> > static uint64 clockFileAccess = 0;
> > /* and the "tick" function */
> > static inline uint64
> > getClockFileAccess(void)
> > {
> >         return ATOMIC_INC_AND_FETCH(clockFileAccess);
> > }
> > 
> > You cannot perform an atomic operation on an 8 byte value
> > on a 32-bit system. Would it be possible to use the atomic
> > operations on two 4 byte values to allow this code to work
> > on 32-bit systems as well?
> > 
> > Regards,
> > Ken
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From Helwing at distec.de  Wed Dec  2 17:25:38 2009
From: Helwing at distec.de (DISTEC Helwing Lutz)
Date: Wed, 2 Dec 2009 17:25:38 +0100
Subject: [rsyslog] Cross compiling rsyslogd...
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71034FF@GRFEXC.intern.adiscon.com>
References: <BFE810BB1BD20E47876F1BB1EA7BF7303D1DC5@srv-exchange.datadisplay.de>
	<9B6E2A8877C38245BFB15CC491A11DA71034FF@GRFEXC.intern.adiscon.com>
Message-ID: <BFE810BB1BD20E47876F1BB1EA7BF7303D1F68@srv-exchange.datadisplay.de>

Hi Rainer,

> > (I've considered some hints from this site:
> > http://cross-lfs.org/view/clfs-2.0/arm/final-system/rsyslog.html )
> 
> mhhh - the link doesn't work for me. Some typo?

Unfortunately something was restructured at their server and i don't
have any clue where that particular is now. The time i posted this
message it was still functional :(
It is still in the Google cache if you are interested:
http://209.85.129.132/search?q=cache:E_ieUu502DQJ:cross-lfs.org/view/clf
s-2.0/arm/final-system/rsyslog.html

> > 1. Configure seems to not determine correct values for the 
> following:
> > - ac_cv_func_malloc_0_nonnull
> > - ac_cv_func_realloc_0_nonnull
> > After setting these to yes the list of errors shrinks 
> dramatically but 
> > there are still some left...
> 
> I am primarily a user of autotools and I have to admit that I 
> do not know what these settings do or cause. Can someone jump in?

The only thing i know is, when cross compiling, configure is not able to
find malloc for some reason and so tries to use rpl_malloc instead.
Since this is not available compilation fails. Setting these two values
to yes explicitly tells the compiler to use malloc. I also don't have
that much insights into autotools to tell you more...

> > Configure instead determines that atomic support should not 
> be enabled 
> > (ap_cv_atomic_builtins=no).
> 
> That sounds like a strong indication they are actually not 
> supported. What the configure test tries is compile a program 
> using them and that obviously fails...

When looking into the configure script at the lines where the test is
run it looks like atomics are always assumed as not working when cross
compiling:
  
configure: line 16918
	if test "$cross_compiling" = yes; then
  	ap_cv_atomic_builtins=no

The test program would only be run in the else case, so it isn't
actually executed. When running it manually on the target platform it
seems to work (i.e. no error messages). Running it on the host platform
must fail because it is built for ARM ;)
I hope i don't give any wrong info here, as i said, my knowledge of
autotools ist very limited...
As a next step I will look how good atomics work with the ARM processor
and how others build rsyslogd for their platform (e.g. there exists an
ARM port of Debian).

> > 4. When commenting out the two lines for module load i get an 
> > alignment
> 
> which lines do you mean?

Oh sorry, i've forgot to tell them (they can be found at the link i've
posted at the beginning). It's two lines for loading modules:
	# Support for Local System Logging
	$ModLoad imuxsock.so

	# Support for Kernel Logging
	$ModLoad imklog.so


Cheers,
Lutz

From rgerhards at hq.adiscon.com  Wed Dec  2 18:00:33 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 2 Dec 2009 18:00:33 +0100
Subject: [rsyslog] omfile does not compile on 32-bit platforms in 5.3.5
References: <000501ca7360$798449f7$100013ac@intern.adiscon.com>
	<20091202153526.GK3237@it.is.rice.edu>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103510@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Kenneth Marshall
> Sent: Wednesday, December 02, 2009 4:35 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] omfile does not compile on 32-bit platforms in
> 5.3.5
> 
> On Wed, Dec 02, 2009 at 04:01:57PM +0100, Rainer Gerhards wrote:
> > Well, gcc itself should provide the helper - at least this is how I
> read the doc. But, well, it looks more appropriate to change the
> variable definition for 32 bit systems. I'll add a --enable-32-bit-
> atomics-where-possible configure switch. If someone could contribute an
> automatic check for 32bit, that would be appreciated.
> >
> > I would like to retain 64 bits on platforms that support it, because
> this is the cleanest solution.
> >
> > rainer
> >
> 
> If you just change the type used in the current check function
> to "long long", it will work on 64-bit systems and fail on 32-bit
> systems. Then if the existing function with the "long" definition
> work, the system supports 32-bit atomics. Or am I missing something?

I missed something ;) Patch is now available in public git:

http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=113915eb9dfddf5a04f8116b
01b71e591e75f193

Thanks for the hint. 

Rainer

> 
> Regards,
> Ken
> 
> > ----- Urspr??ngliche Nachricht -----
> > Von: "Kenneth Marshall" <ktm at rice.edu>
> > An: "rsyslog-users" <rsyslog at lists.adiscon.com>
> > Gesendet: 02.12.09 15:56
> > Betreff: Re: [rsyslog] omfile does not compile on 32-bit platforms in
> 5.3.5
> >
> > On Wed, Dec 02, 2009 at 03:45:24PM +0100, Rainer Gerhards wrote:
> > > Well, we could use a single 32 bit value without much problem, but
> the gcc doc claims gcc will replace the call with a helper function
> (Using a mutex, it can be implemented on any platform).
> > >
> > > rainer
> >
> > Yes, it does put a call to a function __sync_fetch_and_add_8()
> > stub which is why the link fails. It just seemed that it would
> > be easier to use the 4 byte counter and have one method that
> > would work across 32-bit and 64-bit systems, instead of needing
> > to support the missing function which could be implemented
> > using the same call with a 4 byte value. It would also simplify
> > code maintenance.
> >
> > Regards,
> > Ken
> >
> > >
> > > ----- Urspr??ngliche Nachricht -----
> > > Von: "Kenneth Marshall" <ktm at rice.edu>
> > > An: "rsyslog at lists.adiscon.com" <rsyslog at lists.adiscon.com>
> > > Gesendet: 02.12.09 15:22
> > > Betreff: [rsyslog] omfile does not compile on 32-bit platforms in
> 5.3.5
> > >
> > > Hi Rainier,
> > >
> > > The version of omfile.c does not compile/run on 32-bit
> > > systems anymore. Here is the problem function:
> > >
> > > static uint64 clockFileAccess = 0;
> > > /* and the "tick" function */
> > > static inline uint64
> > > getClockFileAccess(void)
> > > {
> > >         return ATOMIC_INC_AND_FETCH(clockFileAccess);
> > > }
> > >
> > > You cannot perform an atomic operation on an 8 byte value
> > > on a 32-bit system. Would it be possible to use the atomic
> > > operations on two 4 byte values to allow this code to work
> > > on 32-bit systems as well?
> > >
> > > Regards,
> > > Ken
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rory at ooma.com  Wed Dec  2 22:11:33 2009
From: rory at ooma.com (Rory Toma)
Date: Wed, 02 Dec 2009 13:11:33 -0800
Subject: [rsyslog] rsyslog crash with tls
Message-ID: <4B16D805.3000803@ooma.com>

rsyslog is crashing on me at some point. I'm getting ready to run it 
under gdb to see where, but I thought I'd report this here first. It 
just stop s running, and, afaik, doesn't drop a core. Basically, I just 
need a front-end rsyslog server to decrypt the incoming data and pass it 
on. If I do unencrypted, I don't believe it crashes. I've seen some 
"overflow" messages shortly before crashing, so perhaps we're 
overflowing somewhere. I'll probably also do a valgrind pass on it. I'll 
let you know what I find, or, if someone sees something wrong with my 
conf file that would cause a problem, please let me know.

I'm also simply invoking rsyslog as

rsyslogd -c4

thx


I've tried:

5.3.5
5.2.0
4.4.2
4.5.7

with the following conf file:

$DefaultNetStreamDriver gtls

$DefaultNetStreamDriverCAFile /export/rsyslog/certs/ca.pem
$DefaultNetStreamDriverCertFile /export/rsyslog/certs/cert.pem
$DefaultNetStreamDriverKeyFile /export/rsyslog/certs/key.pem

$WorkDirectory /export/rsyslog/spool

$ActionQueueType LinkedList
$ActionQueueFileName rsyslog-fwd
$ActionResumeRetryCount -1
$ActionQueueSaveOnShutdown on

$ModLoad imtcp
$ModLoad imuxsock
$ModLoad lmnsd_gtls

$InputTCPServerStreamDriverMode 1
$InputTCPServerStreamDriverAuthMode anon
$InputTCPServerRun 80

*.* @syslog.ooma.com:514

From rgerhards at hq.adiscon.com  Thu Dec  3 07:29:49 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 3 Dec 2009 07:29:49 +0100
Subject: [rsyslog] rsyslog crash with tls
References: <4B16D805.3000803@ooma.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103515@GRFEXC.intern.adiscon.com>

Hi, 

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Rory Toma
> Sent: Wednesday, December 02, 2009 10:12 PM
> To: rsyslog-users
> Subject: [rsyslog] rsyslog crash with tls
> 
> rsyslog is crashing on me at some point. I'm getting ready to run it 
> under gdb to see where, but I thought I'd report this here first. It 
> just stop s running, and, afaik, doesn't drop a core. 
> Basically, I just 
> need a front-end rsyslog server to decrypt the incoming data 
> and pass it 
> on. If I do unencrypted, I don't believe it crashes. I've seen some 
> "overflow" messages shortly before crashing, so perhaps we're 
> overflowing somewhere. 

Can you post some example of these? Do you mean they are too long?

> I'll probably also do a valgrind pass 
> on it.

I would appreciate if you could do that. A major pain is to reproduce these
kind of things, so if you have the environment and the know how to do that,
it is a great help :)

> I'll 
> let you know what I find, or, if someone sees something wrong with my 
> conf file that would cause a problem, please let me know.
> 
> I'm also simply invoking rsyslog as
> 
> rsyslogd -c4
> 
> thx
> 
> 
> I've tried:
> 
> 5.3.5
> 5.2.0
> 4.4.2
> 4.5.7

I suggest that you also give 5.5.1 a try, there was a TLS-patch that, I
think, is not yet included in any versions you mention.

> 
> with the following conf file:
> 
> $DefaultNetStreamDriver gtls
> 
> $DefaultNetStreamDriverCAFile /export/rsyslog/certs/ca.pem
> $DefaultNetStreamDriverCertFile /export/rsyslog/certs/cert.pem
> $DefaultNetStreamDriverKeyFile /export/rsyslog/certs/key.pem
> 
> $WorkDirectory /export/rsyslog/spool
> 
> $ActionQueueType LinkedList
> $ActionQueueFileName rsyslog-fwd
> $ActionResumeRetryCount -1
> $ActionQueueSaveOnShutdown on
> 
> $ModLoad imtcp
> $ModLoad imuxsock
> $ModLoad lmnsd_gtls
> 
> $InputTCPServerStreamDriverMode 1
> $InputTCPServerStreamDriverAuthMode anon
> $InputTCPServerRun 80
> 
> *.* @syslog.ooma.com:514

This - unfortunately ;) - looks good to me. But I will try to run a similar
config a bit later today in my lab and see if it gives me problems. However,
being pretty basic, it resembles what I myself usually use during testing (I
guess it is even similar to one of the automatted tests...).

Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 

From jacopo89 at gmail.com  Fri Dec  4 11:17:05 2009
From: jacopo89 at gmail.com (Jacopo Cappelli)
Date: Fri, 4 Dec 2009 11:17:05 +0100
Subject: [rsyslog] [rSyslog] About mysql compression
Message-ID: <f1274dd40912040217u83909d9reee6053098abca28@mail.gmail.com>

It's possible to activate mysql compression/uncompression for storage
the log and see with phplogcon?
Because my problem is the space, i can sacrifice some cpu for make the
operation of compress/decompress...

Thanks,
Jacopo

-- 
Linux, Windows Xp ed MS-DOS
(anche conosciuti come il Bello, il Brutto ed il Cattivo).
-- Matt Welsh

From rgerhards at hq.adiscon.com  Fri Dec  4 12:37:12 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 4 Dec 2009 12:37:12 +0100
Subject: [rsyslog] [rSyslog] About mysql compression
References: <f1274dd40912040217u83909d9reee6053098abca28@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103528@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Jacopo Cappelli
> Sent: Friday, December 04, 2009 11:17 AM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] [rSyslog] About mysql compression
> 
> It's possible to activate mysql compression/uncompression for storage
> the log and see with phplogcon?
> Because my problem is the space, i can sacrifice some cpu for make the
> operation of compress/decompress...


I simply don't know. What do you need to do in order to enable compression?
Is there any change to application programs required?

Rainer

> 
> Thanks,
> Jacopo
> 
> --
> Linux, Windows Xp ed MS-DOS
> (anche conosciuti come il Bello, il Brutto ed il Cattivo).
> -- Matt Welsh
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rory at ooma.com  Fri Dec  4 20:02:50 2009
From: rory at ooma.com (Rory Toma)
Date: Fri, 04 Dec 2009 11:02:50 -0800
Subject: [rsyslog] /dev/log on read only filesystem
Message-ID: <4B195CDA.90906@ooma.com>

I notice rsyslog clobbers the /dev/log symlink to /tmp/log. I do this 
because /dev/ is ro on my system. Is there a known way around this, or 
will I have to change the code?

thx

From rory at ooma.com  Fri Dec  4 21:10:25 2009
From: rory at ooma.com (Rory Toma)
Date: Fri, 04 Dec 2009 12:10:25 -0800
Subject: [rsyslog] /dev/log on read only filesystem
In-Reply-To: <4B195CDA.90906@ooma.com>
References: <4B195CDA.90906@ooma.com>
Message-ID: <4B196CB1.5080003@ooma.com>

Rory Toma wrote:
> I notice rsyslog clobbers the /dev/log symlink to /tmp/log. I do this 
> because /dev/ is ro on my system. Is there a known way around this, or 
> will I have to change the code?
>
> thx
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>   
Nevermind... missed this the first time thru the docs:

$SystemLogSocketName /var/log/log


From sjain at silverspringnet.com  Fri Dec  4 22:26:27 2009
From: sjain at silverspringnet.com (Siddhartha Jain)
Date: Fri, 4 Dec 2009 13:26:27 -0800
Subject: [rsyslog] Noob log filtering question
Message-ID: <3A240503F9F2194780469F072D9A705410FB92C8@m342.silverspringnet.com>

I am setting up a relay such that the relay should:
- Log locally generated messages locally (as syslog did)
- Log all, received over the network and local logs, to a remote server.

How do I create a filter/template such that logs originating from
imuxsock, imklog and src-ip:127.0.0.1 are sent to local files to match
these:
*.info;mail.none;authpriv.none;cron.none
/var/log/messages
authpriv.*                                              /var/log/secure
mail.*
-/var/log/maillog
cron.*                                                  /var/log/cron
*.emerg                                                 *
local7.*
/var/log/boot.log
kern.debug
/var/adm/syslog.dated/kern.log
user.debug
/var/adm/syslog.dated/user.log
daemon.debug
/var/adm/syslog.dated/daemon.log
auth.crit;syslog.debug
/var/adm/syslog.dated/syslog.log
kern.debug
/var/adm/messages
kern.debug
/dev/console
*.emerg
*

Something like 

"If (inputmodule == imuxsock || inputmodule == imklog || src-ip ==
127.0.0.1) && (facility == cron) && (severity == *) 
then log-to file /var/log/cron.log"


Thanks,

Siddhartha

From rory at ooma.com  Sat Dec  5 00:02:52 2009
From: rory at ooma.com (Rory Toma)
Date: Fri, 04 Dec 2009 15:02:52 -0800
Subject: [rsyslog] tls crash
Message-ID: <4B19951C.3060707@ooma.com>

I've worked around my issue for the moment, syslog-ng seems to use less 
CPU time and so far, hasn't crashed and is generally more responsive.

Because this works, I don't know how much time I'll have to debug the 
issue on rsyslog.

I still have the setup, so, hopefully, at some point I can at least get 
the valgrind run done.

thx

From sjain at silverspringnet.com  Sat Dec  5 00:33:46 2009
From: sjain at silverspringnet.com (Siddhartha Jain)
Date: Fri, 4 Dec 2009 15:33:46 -0800
Subject: [rsyslog] Logfile/directory dynamic names and errant clients
Message-ID: <3A240503F9F2194780469F072D9A705410FB9636@m342.silverspringnet.com>

Hi,

 

I am running 4.2.2 on CentOS 5.4 x64.

 

I have one relay and one central server. Clients run stock syslogd/klogd
and send logs over UDP to the relay. The relay is configured to relay
over TLS/TCP to the central server. The communication so far works ok
between all the components but I am still grappling with file/directory
naming issues.

 

On the relay, I do a very simply forwarding, without any explicit
templates:

---xxxx---

*.*       @@(z9)logmaster:10514

---xxxx---

 

On the central server, the messages are caught this way:

---xxxx---

$template DynFile,"/var/rsyslog/logs/%hostname%/%programname%.log"

*.*     -?DynFile

---xxxx---

 

This work ok for most apps except that a goof-up in syslogd on the
client generates these directories on the central server:

[root at logmaster logs]# ls -lR exiting/

exiting/:

total 4

-rw-r--r-- 1 root root 47 Dec  4 15:12 on.log

 

[root at logmaster logs]# ls -lR syslogd/

syslogd/:

total 4

-rw-r--r-- 1 root root 69 Dec  4 15:12 1.4.1.log

 

How do I fix this?

 

Second, I want to segregate the logs per site. I read this doc but it
wasn't clear how do handle different sites A, B, C ....Z. How do I group
hosts into a site-X?

http://wiki.rsyslog.com/index.php/Splitting_messages_based_on_a_site_ID

 

 

Thanks,

 

Siddhartha


From raheel213 at yahoo.com  Mon Dec  7 11:34:25 2009
From: raheel213 at yahoo.com (Raheel Hassan)
Date: Mon, 7 Dec 2009 02:34:25 -0800 (PST)
Subject: [rsyslog] Unable to sned and receive the syslogs.
Message-ID: <541835.83243.qm@web112402.mail.gq1.yahoo.com>

Hi,

I have three machines and i want to forward the logs to the main server. The networks details are,

1- Windows XP running Snare for sending logs to Fedora Core 11.
2- Ubuntu 9.10 using syslog to send the logs to Fedora Core 11.
3- Fedora Core 11 using rsyslog to send the logs to Fedora Core 11.

I have problems in configuring the syslogs for client and server and i am unable to send or receive the logs at the server. Please suggest me any article/resource where i can read about configuring them correctly. In addition do i need to configure only the syslogs files or i need to do any changes in operating systems configurations.

Regards,
RAHEEL



      

From joe at joetify.com  Mon Dec  7 20:17:42 2009
From: joe at joetify.com (Joe Williams)
Date: Mon, 07 Dec 2009 11:17:42 -0800
Subject: [rsyslog] rsyslog stops logging
Message-ID: <4B1D54D6.7070302@joetify.com>


I have a reoccurring issue with rsyslog where it stops logging. From the 
logs it seems like it is related to a restart that happens around the 
same time:

root at HOST:~# rsyslogd -v
rsyslogd 3.18.1, compiled with:
     FEATURE_REGEXP:                Yes
     FEATURE_LARGEFILE:            Yes
     FEATURE_NETZIP (message compression):    Yes
     GSSAPI Kerberos 5 support:        No
     FEATURE_DEBUG (debug build, slow code):    No
     Runtime Instrumentation (slow code):    No



Dec  5 07:06:39 HOST kernel: Kernel logging (proc) stopped.
Dec  5 07:06:39 HOST kernel: imklog 3.18.1, log source = /proc/kmsg started.
Dec  5 07:06:39 HOST rsyslogd: [origin software="rsyslogd" 
swVersion="3.18.1" x-pid="13376" x-info="http://www.rsyslog.com"] restart

I am unsure of the reason for the restart in the first place, in 
addition the kernel logging stopping. Although I assume it is likely 
when rsyslog rotates the logs, but it doesn't happen consistently nor at 
the same time each time I've experienced it. Could this be related to 
the known startup race condition bug that I've seen stuff about?

Thanks!

-Joe



-- 
Name: Joseph A. Williams
Email: joe at joetify.com
Blog: http://www.joeandmotorboat.com/


From ryan.b.lynch at gmail.com  Tue Dec  8 03:42:10 2009
From: ryan.b.lynch at gmail.com (Ryan Lynch)
Date: Mon, 7 Dec 2009 21:42:10 -0500
Subject: [rsyslog] Auto-creating directories,
	when using templates and the 	property replacer.
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71034DC@GRFEXC.intern.adiscon.com>
References: <000101ca7008$230eb68e$100013ac@intern.adiscon.com> 
	<115906d10911280208m3ee48754i1c755ebaee67a1d2@mail.gmail.com> 
	<9B6E2A8877C38245BFB15CC491A11DA71034DB@GRFEXC.intern.adiscon.com> 
	<9B6E2A8877C38245BFB15CC491A11DA71034DC@GRFEXC.intern.adiscon.com>
Message-ID: <115906d10912071842p2b38383bpfb837e6767cc4a3b@mail.gmail.com>

On Mon, Nov 30, 2009 at 06:08, Rainer Gerhards <rgerhards at hq.adiscon.com> wrote:
> http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=7b40604e9ae8a0948f17eafd
> 4299eeb7fb3356c2
> I would appreciate if you let me know if it works for you.

Rainer,

I finally got a chance to rebuild rsyslog with that patch. Works fine
for me. Thank you for taking the time to fix that bug--it helped me
out, a lot.

-Ryan

From ryan.b.lynch at gmail.com  Tue Dec  8 03:48:12 2009
From: ryan.b.lynch at gmail.com (Ryan Lynch)
Date: Mon, 7 Dec 2009 21:48:12 -0500
Subject: [rsyslog] Is '--enable-unlimited-select' in 5.5.1 considered usable?
Message-ID: <115906d10912071848m6e2b7535p490852b2014ac8e9@mail.gmail.com>

While playing around with some of the less-documented modules in the
5.5.1 source, I noticed the 'unlimited Select()' ./configure option,
but it breaks compilation on my CentOS 5.4 machine.

I did notice this thread on the forum, and it sounds to me like this
feature isn't 100% finished yet:

 * http://www.gossamer-threads.com/lists/rsyslog/users/3029?page=last

Is that right?

-Ryan

From rgerhards at hq.adiscon.com  Tue Dec  8 08:23:49 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 8 Dec 2009 08:23:49 +0100
Subject: [rsyslog] Is '--enable-unlimited-select' in 5.5.1 considered
	usable?
References: <115906d10912071848m6e2b7535p490852b2014ac8e9@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103544@GRFEXC.intern.adiscon.com>

Hi Ryan,

As you can see in the thread you quoted, this is contributed code. I could
not check it out but thought it is a good addition. Maybe Tomas can jump in
here...

Please note that in the recent build, rsyslog has native support for epoll()
in both imudp and plain-tcp based imtcp. That should be a good solution for a
large number of cases.

Rainer 

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Ryan Lynch
> Sent: Tuesday, December 08, 2009 3:48 AM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] Is '--enable-unlimited-select' in 5.5.1 
> considered usable?
> 
> While playing around with some of the less-documented modules in the
> 5.5.1 source, I noticed the 'unlimited Select()' ./configure option,
> but it breaks compilation on my CentOS 5.4 machine.
> 
> I did notice this thread on the forum, and it sounds to me like this
> feature isn't 100% finished yet:
> 
>  * http://www.gossamer-threads.com/lists/rsyslog/users/3029?page=last
> 
> Is that right?
> 
> -Ryan
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 

From rgerhards at hq.adiscon.com  Tue Dec  8 14:05:44 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 8 Dec 2009 14:05:44 +0100
Subject: [rsyslog] Auto-creating directories,
	when using templates and the 	property replacer.
References: <000101ca7008$230eb68e$100013ac@intern.adiscon.com>
	<115906d10911280208m3ee48754i1c755ebaee67a1d2@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA71034DB@GRFEXC.intern.adiscon.com>
	<9B6E2A8877C38245BFB15CC491A11DA71034DC@GRFEXC.intern.adiscon.com>
	<115906d10912071842p2b38383bpfb837e6767cc4a3b@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103551@GRFEXC.intern.adiscon.com>

Ryan,

thanks for reporting back, it is always good to know that a fix really works
:)

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Ryan Lynch
> Sent: Tuesday, December 08, 2009 3:42 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Auto-creating directories,when using templates
> and the property replacer.
> 
> On Mon, Nov 30, 2009 at 06:08, Rainer Gerhards
> <rgerhards at hq.adiscon.com> wrote:
> >
> http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=7b40604e9ae8a0948f
> 17eafd
> > 4299eeb7fb3356c2
> > I would appreciate if you let me know if it works for you.
> 
> Rainer,
> 
> I finally got a chance to rebuild rsyslog with that patch. Works fine
> for me. Thank you for taking the time to fix that bug--it helped me
> out, a lot.
> 
> -Ryan
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Tue Dec  8 14:29:08 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 8 Dec 2009 14:29:08 +0100
Subject: [rsyslog] rsyslog stops logging
References: <4B1D54D6.7070302@joetify.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103552@GRFEXC.intern.adiscon.com>

Hmmm... This looks like being related to log rotation and thus HUP
processing. Could you please try and see if the issue persists with the
current v3-stable (3.22.1) as this may be a problem that already has been
fixed.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Joe Williams
> Sent: Monday, December 07, 2009 8:18 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] rsyslog stops logging
> 
> 
> I have a reoccurring issue with rsyslog where it stops logging. From
> the
> logs it seems like it is related to a restart that happens around the
> same time:
> 
> root at HOST:~# rsyslogd -v
> rsyslogd 3.18.1, compiled with:
>      FEATURE_REGEXP:                Yes
>      FEATURE_LARGEFILE:            Yes
>      FEATURE_NETZIP (message compression):    Yes
>      GSSAPI Kerberos 5 support:        No
>      FEATURE_DEBUG (debug build, slow code):    No
>      Runtime Instrumentation (slow code):    No
> 
> 
> 
> Dec  5 07:06:39 HOST kernel: Kernel logging (proc) stopped.
> Dec  5 07:06:39 HOST kernel: imklog 3.18.1, log source = /proc/kmsg
> started.
> Dec  5 07:06:39 HOST rsyslogd: [origin software="rsyslogd"
> swVersion="3.18.1" x-pid="13376" x-info="http://www.rsyslog.com"]
> restart
> 
> I am unsure of the reason for the restart in the first place, in
> addition the kernel logging stopping. Although I assume it is likely
> when rsyslog rotates the logs, but it doesn't happen consistently nor
> at
> the same time each time I've experienced it. Could this be related to
> the known startup race condition bug that I've seen stuff about?
> 
> Thanks!
> 
> -Joe
> 
> 
> 
> --
> Name: Joseph A. Williams
> Email: joe at joetify.com
> Blog: http://www.joeandmotorboat.com/
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From theinric at redhat.com  Tue Dec  8 16:15:08 2009
From: theinric at redhat.com (Tomas Heinrich)
Date: Tue, 08 Dec 2009 16:15:08 +0100
Subject: [rsyslog] Is '--enable-unlimited-select' in 5.5.1 considered
 usable?
In-Reply-To: <115906d10912071848m6e2b7535p490852b2014ac8e9@mail.gmail.com>
References: <115906d10912071848m6e2b7535p490852b2014ac8e9@mail.gmail.com>
Message-ID: <4B1E6D7C.1050006@redhat.com>

Hi Ryan,

this mechanism is intended to provide for higher number of open files 
and network connections in branches that do not support epoll() (it is 
present in v4-devel, can be backported to v3 with ease). It is obsoleted 
in the most recent v5 builds.

Tomas

On 12/08/2009 03:48 AM, Ryan Lynch wrote:
> While playing around with some of the less-documented modules in the
> 5.5.1 source, I noticed the 'unlimited Select()' ./configure option,
> but it breaks compilation on my CentOS 5.4 machine.
> 
> I did notice this thread on the forum, and it sounds to me like this
> feature isn't 100% finished yet:
> 
>  * http://www.gossamer-threads.com/lists/rsyslog/users/3029?page=last
> 
> Is that right?
> 
> -Ryan
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Tue Dec  8 16:36:35 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 8 Dec 2009 16:36:35 +0100
Subject: [rsyslog] Is '--enable-unlimited-select' in 5.5.1 considered
	usable?
References: <115906d10912071848m6e2b7535p490852b2014ac8e9@mail.gmail.com>
	<4B1E6D7C.1050006@redhat.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103554@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Tomas Heinrich
> Sent: Tuesday, December 08, 2009 4:15 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Is '--enable-unlimited-select' in 5.5.1
> considered usable?
> 
> Hi Ryan,
> 
> this mechanism is intended to provide for higher number of open files
> and network connections in branches that do not support epoll() (it is
> present in v4-devel, can be backported to v3 with ease). It is
> obsoleted
> in the most recent v5 builds.

good point: it is not really obsoleted for *all* sources (unfortunately). So
far, I have only moved imudp and the plain tcp netstream driver to epoll. The
others use your patch, if enabled. Also, on systems where epoll is not
available, the patch is included for plain tcp as well (I will keep the
select() code because it is more portable, but disable it if I find epoll).

Rainer

> 
> Tomas
> 
> On 12/08/2009 03:48 AM, Ryan Lynch wrote:
> > While playing around with some of the less-documented modules in the
> > 5.5.1 source, I noticed the 'unlimited Select()' ./configure option,
> > but it breaks compilation on my CentOS 5.4 machine.
> >
> > I did notice this thread on the forum, and it sounds to me like this
> > feature isn't 100% finished yet:
> >
> >  * http://www.gossamer-threads.com/lists/rsyslog/users/3029?page=last
> >
> > Is that right?
> >
> > -Ryan
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From ryan.b.lynch at gmail.com  Tue Dec  8 16:42:03 2009
From: ryan.b.lynch at gmail.com (Ryan Lynch)
Date: Tue, 8 Dec 2009 10:42:03 -0500
Subject: [rsyslog] Is '--enable-unlimited-select' in 5.5.1 considered
	usable?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103554@GRFEXC.intern.adiscon.com>
References: <115906d10912071848m6e2b7535p490852b2014ac8e9@mail.gmail.com> 
	<4B1E6D7C.1050006@redhat.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103554@GRFEXC.intern.adiscon.com>
Message-ID: <115906d10912080742s63ee1b8va11434bd33c81ad3@mail.gmail.com>

On Tue, Dec 8, 2009 at 10:36, Rainer Gerhards <rgerhards at hq.adiscon.com> wrote:
>> this mechanism is intended to provide for higher number of open files
>> and network connections in branches that do not support epoll() (it is
>> present in v4-devel, can be backported to v3 with ease). It is
>> obsoleted
>> in the most recent v5 builds.
>
> good point: it is not really obsoleted for *all* sources (unfortunately). So
> far, I have only moved imudp and the plain tcp netstream driver to epoll. The
> others use your patch, if enabled. Also, on systems where epoll is not
> available, the patch is included for plain tcp as well (I will keep the
> select() code because it is more portable, but disable it if I find epoll).

So, if I understand you, correctly, epoll() is not supported for RELP,
GSSAPI, and/or TLS connections? Do you plan to support epoll() for
those types of connections, in the future?

-Ryan

From rgerhards at hq.adiscon.com  Tue Dec  8 16:44:42 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 8 Dec 2009 16:44:42 +0100
Subject: [rsyslog] Is '--enable-unlimited-select' in 5.5.1
	consideredusable?
References: <115906d10912071848m6e2b7535p490852b2014ac8e9@mail.gmail.com>
	<4B1E6D7C.1050006@redhat.com><9B6E2A8877C38245BFB15CC491A11DA7103554@GRFEXC.intern.adiscon.com>
	<115906d10912080742s63ee1b8va11434bd33c81ad3@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103559@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Ryan Lynch
> Sent: Tuesday, December 08, 2009 4:42 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Is '--enable-unlimited-select' in 5.5.1
> consideredusable?
> 
> On Tue, Dec 8, 2009 at 10:36, Rainer Gerhards
> <rgerhards at hq.adiscon.com> wrote:
> >> this mechanism is intended to provide for higher number of open
> files
> >> and network connections in branches that do not support epoll() (it
> is
> >> present in v4-devel, can be backported to v3 with ease). It is
> >> obsoleted
> >> in the most recent v5 builds.
> >
> > good point: it is not really obsoleted for *all* sources
> (unfortunately). So
> > far, I have only moved imudp and the plain tcp netstream driver to
> epoll. The
> > others use your patch, if enabled. Also, on systems where epoll is
> not
> > available, the patch is included for plain tcp as well (I will keep
> the
> > select() code because it is more portable, but disable it if I find
> epoll).
> 
> So, if I understand you, correctly, epoll() is not supported for RELP,
> GSSAPI, and/or TLS connections? 

correct

> Do you plan to support epoll() for
> those types of connections, in the future?

Yes, but I want to gain some experience with the current implementation
first. I needed to make quite some changes to the driver layer and would like
to see that used in practice first. That, btw, was a primary reason why I
selected plain tcp as the initial target -it is the most often used transport
of all these.

Rainer
> 
> -Ryan
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From ryan.b.lynch at gmail.com  Tue Dec  8 16:46:48 2009
From: ryan.b.lynch at gmail.com (Ryan Lynch)
Date: Tue, 8 Dec 2009 10:46:48 -0500
Subject: [rsyslog] Is '--enable-unlimited-select' in 5.5.1
	consideredusable?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103559@GRFEXC.intern.adiscon.com>
References: <115906d10912071848m6e2b7535p490852b2014ac8e9@mail.gmail.com> 
	<4B1E6D7C.1050006@redhat.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103554@GRFEXC.intern.adiscon.com>
	<115906d10912080742s63ee1b8va11434bd33c81ad3@mail.gmail.com> 
	<9B6E2A8877C38245BFB15CC491A11DA7103559@GRFEXC.intern.adiscon.com>
Message-ID: <115906d10912080746m1d6eea90s74430ea1760eab93@mail.gmail.com>

On Tue, Dec 8, 2009 at 10:44, Rainer Gerhards <rgerhards at hq.adiscon.com> wrote:
>> So, if I understand you, correctly, epoll() is not supported for RELP,
>> GSSAPI, and/or TLS connections?
>
> correct
>
>> Do you plan to support epoll() for
>> those types of connections, in the future?
>
> Yes, but I want to gain some experience with the current implementation
> first. I needed to make quite some changes to the driver layer and would like
> to see that used in practice first. That, btw, was a primary reason why I
> selected plain tcp as the initial target -it is the most often used transport
> of all these.

Good to know, and thanks for the answers.

-Ryan

From rory at ooma.com  Tue Dec  8 20:38:35 2009
From: rory at ooma.com (Rory Toma)
Date: Tue, 08 Dec 2009 11:38:35 -0800
Subject: [rsyslog] Help with client config
Message-ID: <4B1EAB3B.2070003@ooma.com>

I have a client config as follows, running on armv4, rsyslog 4.4.2 (I 
can't get anything newer to compile due to the atomic kernel support not 
wanting to compile) The problem I'm having is:

1) (this is probably an rtfm, but I'll ask it anyway) It seems that my 
logs being sent over the network are being batched up in huge packets, I 
see one transaction for a bunch of data. Sometimes this means my logs 
are delayed quite a bit.

2) Sometimes, the client just doesn't send out over the network at all 
(verified with a tcpdump), but continues logging.

Any suggestions or config file tweaks welcome.

thx

Here is my config:

$DefaultNetStreamDriverCAFile /etc/ca.pem

$DefaultNetStreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode anon

$WorkDirectory /var/log

$ActionQueueType LinkedList
$ActionQueueFileName rsyslog-fwd
$ActionResumeRetryCount -1
$ActionQueueSaveOnShutdown on
$ActionQueueMaxDiskSpace 256k

$ModLoad imuxsock
$SystemLogSocketName /var/log/log
$OptimizeForUniprocessor on

$outchannel locallog,/var/log/locallog,262144,/usr/bin/rotate_locallog

*.* @@foo.bar.com:80
*.* $locallog


From rory at ooma.com  Tue Dec  8 20:41:45 2009
From: rory at ooma.com (Rory Toma)
Date: Tue, 08 Dec 2009 11:41:45 -0800
Subject: [rsyslog] Help with client config
In-Reply-To: <4B1EAB3B.2070003@ooma.com>
References: <4B1EAB3B.2070003@ooma.com>
Message-ID: <4B1EABF9.70603@ooma.com>

That is, it continues logging to the local file.
>
> 2) Sometimes, the client just doesn't send out over the network at all 
> (verified with a tcpdump), but continues logging.
>
>   


From rgerhards at hq.adiscon.com  Tue Dec  8 20:47:41 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 8 Dec 2009 20:47:41 +0100
Subject: [rsyslog] Help with client config
Message-ID: <001301ca783f$691b9aa5$100013ac@intern.adiscon.com>

Can 1 be a tls (lib) artifact? Can you try with plain tcp?

----- Urspr?ngliche Nachricht -----
Von: "Rory Toma" <rory at ooma.com>
An: "rsyslog-users" <rsyslog at lists.adiscon.com>
Gesendet: 08.12.09 20:39
Betreff: [rsyslog] Help with client config

I have a client config as follows, running on armv4, rsyslog 4.4.2 (I 
can't get anything newer to compile due to the atomic kernel support not 
wanting to compile) The problem I'm having is:

1) (this is probably an rtfm, but I'll ask it anyway) It seems that my 
logs being sent over the network are being batched up in huge packets, I 
see one transaction for a bunch of data. Sometimes this means my logs 
are delayed quite a bit.

2) Sometimes, the client just doesn't send out over the network at all 
(verified with a tcpdump), but continues logging.

Any suggestions or config file tweaks welcome.

thx

Here is my config:

$DefaultNetStreamDriverCAFile /etc/ca.pem

$DefaultNetStreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode anon

$WorkDirectory /var/log

$ActionQueueType LinkedList
$ActionQueueFileName rsyslog-fwd
$ActionResumeRetryCount -1
$ActionQueueSaveOnShutdown on
$ActionQueueMaxDiskSpace 256k

$ModLoad imuxsock
$SystemLogSocketName /var/log/log
$OptimizeForUniprocessor on

$outchannel locallog,/var/log/locallog,262144,/usr/bin/rotate_locallog

*.* @@foo.bar.com:80
*.* $locallog

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

From ryan.b.lynch at gmail.com  Tue Dec  8 21:41:52 2009
From: ryan.b.lynch at gmail.com (Ryan Lynch)
Date: Tue, 8 Dec 2009 15:41:52 -0500
Subject: [rsyslog] Any roadmap for RELP over GSSAPI and/or TLS?
Message-ID: <115906d10912081241md55d83y9a4af29abb25581c@mail.gmail.com>

Rainer,

Sorry to bug you with all of these crystal-ball roadmap questions, but
I'm curious about your plans for Rsyslog.

The doc page for the TLS module
(http://www.rsyslog.com/doc-rsyslog_tls.html) mentions that joint
RELP/TLS connections aren't currently supported, and that the
implementation is planned for the future. That page is dated May of
2008. The GSSAPI module doc page doesn't mention RELP, at all, or
future development plans.

Two questions:
 1) Do you have any updates to your plans to support RELP w/ TLS?
 2) Do you have any plans to support RELP w/ GSSAPI, at all?

If the answer is hazy, right now, I totally understand, and I
apologize for bugging you, again.

Ryan B. Lynch
ryan.b.lynch at gmail.com

From rory at ooma.com  Tue Dec  8 22:36:42 2009
From: rory at ooma.com (Rory Toma)
Date: Tue, 08 Dec 2009 13:36:42 -0800
Subject: [rsyslog] Help with client config
In-Reply-To: <001301ca783f$691b9aa5$100013ac@intern.adiscon.com>
References: <001301ca783f$691b9aa5$100013ac@intern.adiscon.com>
Message-ID: <4B1EC6EA.6090508@ooma.com>

It works just fine without tls. I do not know much about tls, is there a 
default size somewhere that I can swizzle?

Rainer Gerhards wrote:
> Can 1 be a tls (lib) artifact? Can you try with plain tcp?
>
> ----- Urspr?ngliche Nachricht -----
> Von: "Rory Toma" <rory at ooma.com>
> An: "rsyslog-users" <rsyslog at lists.adiscon.com>
> Gesendet: 08.12.09 20:39
> Betreff: [rsyslog] Help with client config
>
> I have a client config as follows, running on armv4, rsyslog 4.4.2 (I 
> can't get anything newer to compile due to the atomic kernel support not 
> wanting to compile) The problem I'm having is:
>
> 1) (this is probably an rtfm, but I'll ask it anyway) It seems that my 
> logs being sent over the network are being batched up in huge packets, I 
> see one transaction for a bunch of data. Sometimes this means my logs 
> are delayed quite a bit.
>
> 2) Sometimes, the client just doesn't send out over the network at all 
> (verified with a tcpdump), but continues logging.
>
> Any suggestions or config file tweaks welcome.
>
> thx
>
> Here is my config:
>
> $DefaultNetStreamDriverCAFile /etc/ca.pem
>
> $DefaultNetStreamDriver gtls
> $ActionSendStreamDriverMode 1
> $ActionSendStreamDriverAuthMode anon
>
> $WorkDirectory /var/log
>
> $ActionQueueType LinkedList
> $ActionQueueFileName rsyslog-fwd
> $ActionResumeRetryCount -1
> $ActionQueueSaveOnShutdown on
> $ActionQueueMaxDiskSpace 256k
>
> $ModLoad imuxsock
> $SystemLogSocketName /var/log/log
> $OptimizeForUniprocessor on
>
> $outchannel locallog,/var/log/locallog,262144,/usr/bin/rotate_locallog
>
> *.* @@foo.bar.com:80
> *.* $locallog
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>   


From rory at ooma.com  Tue Dec  8 23:10:03 2009
From: rory at ooma.com (Rory Toma)
Date: Tue, 08 Dec 2009 14:10:03 -0800
Subject: [rsyslog] Help with client config
In-Reply-To: <4B1EC6EA.6090508@ooma.com>
References: <001301ca783f$691b9aa5$100013ac@intern.adiscon.com>
	<4B1EC6EA.6090508@ooma.com>
Message-ID: <4B1ECEBB.4030905@ooma.com>

And, actually, this only happens some of the time, where some of the 
time means never on my dev systems and always on remote ones that I 
can't easily debug. 8-)


Rory Toma wrote:
> It works just fine without tls. I do not know much about tls, is there a 
> default size somewhere that I can swizzle?
>
> Rainer Gerhards wrote:
>   
>> Can 1 be a tls (lib) artifact? Can you try with plain tcp?
>>
>> ----- Urspr?ngliche Nachricht -----
>> Von: "Rory Toma" <rory at ooma.com>
>> An: "rsyslog-users" <rsyslog at lists.adiscon.com>
>> Gesendet: 08.12.09 20:39
>> Betreff: [rsyslog] Help with client config
>>
>> I have a client config as follows, running on armv4, rsyslog 4.4.2 (I 
>> can't get anything newer to compile due to the atomic kernel support not 
>> wanting to compile) The problem I'm having is:
>>
>> 1) (this is probably an rtfm, but I'll ask it anyway) It seems that my 
>> logs being sent over the network are being batched up in huge packets, I 
>> see one transaction for a bunch of data. Sometimes this means my logs 
>> are delayed quite a bit.
>>
>> 2) Sometimes, the client just doesn't send out over the network at all 
>> (verified with a tcpdump), but continues logging.
>>
>> Any suggestions or config file tweaks welcome.
>>
>> thx
>>
>> Here is my config:
>>
>> $DefaultNetStreamDriverCAFile /etc/ca.pem
>>
>> $DefaultNetStreamDriver gtls
>> $ActionSendStreamDriverMode 1
>> $ActionSendStreamDriverAuthMode anon
>>
>> $WorkDirectory /var/log
>>
>> $ActionQueueType LinkedList
>> $ActionQueueFileName rsyslog-fwd
>> $ActionResumeRetryCount -1
>> $ActionQueueSaveOnShutdown on
>> $ActionQueueMaxDiskSpace 256k
>>
>> $ModLoad imuxsock
>> $SystemLogSocketName /var/log/log
>> $OptimizeForUniprocessor on
>>
>> $outchannel locallog,/var/log/locallog,262144,/usr/bin/rotate_locallog
>>
>> *.* @@foo.bar.com:80
>> *.* $locallog
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>   
>>     
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>   


From ryan.b.lynch at gmail.com  Tue Dec  8 23:28:06 2009
From: ryan.b.lynch at gmail.com (Ryan Lynch)
Date: Tue, 8 Dec 2009 17:28:06 -0500
Subject: [rsyslog] Feature suggestion: Query rsyslog daemon for a list of
	currently-open log files.
Message-ID: <115906d10912081428t263114afv3dcc5fc8228ecfe5@mail.gmail.com>

Obviously this isn't a low priority suggestion, but I think it could
be a pretty useful in managing large/complex Rsyslog configurations.

On some of my Linux hosts, I have Rsyslog configured such that it can
easily hold 20-30 open files at any given time. (I'm using the utility
'lsof', BTW, to list open files, like so: `lsof | gawk '$1 ~
"^rsyslogd$" && $9 ~ "^/var/log/"'`) Since I'm using dynamic templates
with date/time elements, the exact list of currently-open files
changes from minute to minute.

It would be helpful for the Rsyslog daemon to periodically output a
list of all the log files (by full path) that it currently holds open.
Primarily, I'd use this for log file rotation and similar maintenance
tasks, but I could imagine others would find other uses, especially if
the output mechanism were generic enough to dump other runtime
statistics, too. If Rsyslog kept a count of the number of bytes
written to each open file, and included those per-byte counts in the
output, another program could monitor log-output rates, to watch out
for dangerously high disk I/O loads.

I don't really have a fixed idea for the communication mechanism. It
could be as simple as having Rsyslog dump the stats to a text file,
and add a configuration knob to tell it how many seconds to wait
between dumps. The output format wouldn't need to be simple, maybe
just a newline-delimited list of file names/stats.

If I get some free time, and nobody objects, I may try to implement
this. But I'm pretty rusty with C, so I figured I should ask whether
the idea floats, first.

Ryan B. Lynch
ryan.b.lynch at gmail.com

From ryan.b.lynch at gmail.com  Tue Dec  8 23:29:03 2009
From: ryan.b.lynch at gmail.com (Ryan Lynch)
Date: Tue, 8 Dec 2009 17:29:03 -0500
Subject: [rsyslog] Feature suggestion: Query rsyslog daemon for a list
	of currently-open log files.
In-Reply-To: <115906d10912081428t263114afv3dcc5fc8228ecfe5@mail.gmail.com>
References: <115906d10912081428t263114afv3dcc5fc8228ecfe5@mail.gmail.com>
Message-ID: <115906d10912081429y7b936821s345c07f4ff3c8c59@mail.gmail.com>

(Whoops--that first line should read 'this IS a low priority
suggestion... Sorry!)

On Tue, Dec 8, 2009 at 17:28, Ryan Lynch <ryan.b.lynch at gmail.com> wrote:
> Obviously this isn't a low priority suggestion, but I think it could
> be a pretty useful in managing large/complex Rsyslog configurations.
>
> On some of my Linux hosts, I have Rsyslog configured such that it can
> easily hold 20-30 open files at any given time. (I'm using the utility
> 'lsof', BTW, to list open files, like so: `lsof | gawk '$1 ~
> "^rsyslogd$" && $9 ~ "^/var/log/"'`) Since I'm using dynamic templates
> with date/time elements, the exact list of currently-open files
> changes from minute to minute.
>
> It would be helpful for the Rsyslog daemon to periodically output a
> list of all the log files (by full path) that it currently holds open.
> Primarily, I'd use this for log file rotation and similar maintenance
> tasks, but I could imagine others would find other uses, especially if
> the output mechanism were generic enough to dump other runtime
> statistics, too. If Rsyslog kept a count of the number of bytes
> written to each open file, and included those per-byte counts in the
> output, another program could monitor log-output rates, to watch out
> for dangerously high disk I/O loads.
>
> I don't really have a fixed idea for the communication mechanism. It
> could be as simple as having Rsyslog dump the stats to a text file,
> and add a configuration knob to tell it how many seconds to wait
> between dumps. The output format wouldn't need to be simple, maybe
> just a newline-delimited list of file names/stats.
>
> If I get some free time, and nobody objects, I may try to implement
> this. But I'm pretty rusty with C, so I figured I should ask whether
> the idea floats, first.
>
> Ryan B. Lynch
> ryan.b.lynch at gmail.com
>

From ryan.b.lynch at gmail.com  Wed Dec  9 02:18:30 2009
From: ryan.b.lynch at gmail.com (Ryan Lynch)
Date: Tue, 8 Dec 2009 20:18:30 -0500
Subject: [rsyslog] Multiple-server failover questions (and round-robin DNS
	resolution, too)
Message-ID: <115906d10912081718u2fad4334i7426310ea01a95a@mail.gmail.com>

In the doc example, the server "primary-syslog.example.com" is always
tried first, unless it's down, followed by
"secondary-1-syslog.example.com", and then finally
"secondary-2-syslog.example.com" only if both of the others are down.
Here's the doc in question, BTW:

 * http://wiki.rsyslog.com/index.php/FailoverSyslogServer

1.  Is it possible to configure Rsyslog to pick randomly from a list
of remote syslog destinations instead of following the listed order?
If I have a large population of log generating hosts, I'd like to be
able to distribute the total load amongst two or more "central" log
receivers, with failover if a receiver dies. Assuming that we have a
sound random-selection implementation, and a large enough population
of hosts generating approximately equal amounts of log data, I should
get an approximately even distribution across my receiver hosts. In
the event that a receiver dies, its clients would randomly try another
receiver, keeping the load close to even. Is this possible, with the
existing failover mechanism?

2.  How does Rsyslog handle a round-robin DNS hostname (i.e., a single
A record resolves to multiple IP addresses), if that hostname is a
remote log destination? Does it just call gethostbyname()?

3.  If Rsyslog does just call gethostbyname(), how does it handle
multiple occurrences of the same hostname in the config file? Does it
call gethostbyname() once for each instance of a duplicate name, or
does it perform a single lookup and use the same result for all?

4.  When and how does the failover code check for remote destination
failures? Does it re-check failed servers after they're declared
"failed"? If so, does Rsyslog re-check a failed server every time it
sends a log message, or does it have some kind of polling timeout, or
is the check interval determined by something else entirely? When a
server fails, how long does it take Rsyslog to notice the failure and
start re-directing messages to the next host? When a server recovers,
does Rsyslog automatically notice the recovery, and if so, how long
does it take to start re-directing messages to the primary host?

In case anybody's wondering, I've include some examples of what I'm
thinking of trying, here. Assume that there are three (3) central log
receiver hosts, with each with a unique DNS name ('log-alfa',
'log-bravo', and 'log-charlie') that points to its own IP. Also,
assume that the round-robin DNS name 'log' points to all three hosts'
IP addresses, and that all of my log-generating hosts (the clients)
are configured to pick a random IP when calling gethostbyname() on a
round-robin name.

    *.* @@log
    $ActionExecOnlyWhenPreviousIsSuspended on
    & @@log
    & @@log
    & /var/log/localbuffer
    $ActionExecOnlyWhenPreviousIsSuspended off

But if Rsyslog resolves all three instances of the name 'log' with a
single call to gethostbyname(), this won't work. I could probably hack
around it with some additional round robin entries similar to 'log',
but named 'log-all-1', 'log-all-2', 'log-all-3', etc., to force an
independent lookup attempt for each:

    *.* @@log
    $ActionExecOnlyWhenPreviousIsSuspended on
    & @@log-all-1
    & @@log-all-2
    & @@log-all-3
    & @@log-all-4
    & @@log-all-5
    & @@log-all-6
    ...
    & /var/log/localbuffer
    $ActionExecOnlyWhenPreviousIsSuspended off

But this is pretty messy. First, there's a lot of extra DNS records to
maintain, which is a real pain. Second, I have to use a lot of extra
round-robin names (definitely more than 3) to maximize the probability
that gethostbyname() will return at least one working server before
reaching the end of the list. (gethostbyname() isn't aware of downed
hosts, nor does it enforce balance in its responses, so it might
return the same bad server several times in a row.)

It would nice to have a random variant of the
'$ActionExecOnlyWhenPreviousIsSuspended' functionality. As a
hypothetical example I just cooked up off the top of my head:

    *.* @@log-alfa
    $ActionExecPickRandom on
    $ActionExecPickRandomRetryWait 1
    $ActionExecPickRandomRetryLimit -1
    & @@log-bravo
    & @@log-charlie
    $ActionExecPickRandom off

where Rsyslog randomly selects one of log-alfa, log-bravo, and
log-charlie, initially, and then makes another random pick at
failover, etc. I can think of all sorts of nifty options and
configurable knobs that would come in handy, here, too.

I'm planning to try out my first two examples, tomorrow, to see what
works. If anybody has any comments, I'd love to hear them.

Ryan B. Lynch
ryan.b.lynch at gmail.com

From ryan.b.lynch at gmail.com  Wed Dec  9 06:58:07 2009
From: ryan.b.lynch at gmail.com (Ryan Lynch)
Date: Wed, 9 Dec 2009 00:58:07 -0500
Subject: [rsyslog] No DNS resolution for the property '$fromhost',
	it's the local host.
Message-ID: <115906d10912082158v50ff93f8y843470ae106bdd5d@mail.gmail.com>

I've configured syslog forwarding via RELP, and it's mostly working
properly. The only problem is that DNS resolution doesn't seem to be
enabled for the '$fromhost' property, if the log message is coming
from a remote machine via RELP. If the log destination is local, it
works fine: $fromhost resolves to the simple (no FQDN) hostname, as
does $hostname. But if the log destination is a remote machine via
RELP, I '$fromhost' doesn't resolves to a hostname, it's always an IP
address.

I found a  similar issue on the forum, here:

 * http://kb.monitorware.com/reverse-dns-resolution-why-t10040.html

Like that poster, my forward and reverse DNS lookups work just fine,
on all of the hosts involved.

Is this a known limitation of RELP, or remote destinations in general,
or should I file a bug report?

-Ryan

Ryan B. Lynch
ryan.b.lynch at gmail.com

From rgerhards at hq.adiscon.com  Wed Dec  9 10:53:10 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 9 Dec 2009 10:53:10 +0100
Subject: [rsyslog] Multiple-server failover questions (and round-robin
	DNSresolution, too)
References: <115906d10912081718u2fad4334i7426310ea01a95a@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103560@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Ryan Lynch
> Sent: Wednesday, December 09, 2009 2:19 AM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] Multiple-server failover questions (and round-robin
> DNSresolution, too)
> 
> In the doc example, the server "primary-syslog.example.com" is always
> tried first, unless it's down, followed by
> "secondary-1-syslog.example.com", and then finally
> "secondary-2-syslog.example.com" only if both of the others are down.
> Here's the doc in question, BTW:
> 
>  * http://wiki.rsyslog.com/index.php/FailoverSyslogServer
> 
> 1.  Is it possible to configure Rsyslog to pick randomly from a list
> of remote syslog destinations instead of following the listed order?
> If I have a large population of log generating hosts, I'd like to be
> able to distribute the total load amongst two or more "central" log
> receivers, with failover if a receiver dies. Assuming that we have a
> sound random-selection implementation, and a large enough population
> of hosts generating approximately equal amounts of log data, I should
> get an approximately even distribution across my receiver hosts. In
> the event that a receiver dies, its clients would randomly try another
> receiver, keeping the load close to even. Is this possible, with the
> existing failover mechanism?

No. It would not be too hard to add, but it is currently not supported.

> 2.  How does Rsyslog handle a round-robin DNS hostname (i.e., a single
> A record resolves to multiple IP addresses), if that hostname is a
> remote log destination? Does it just call gethostbyname()?

It just calls gethostbyname() and this as infrequently as possible. So round
robin is not handled well.

> 
> 3.  If Rsyslog does just call gethostbyname(), how does it handle
> multiple occurrences of the same hostname in the config file? Does it
> call gethostbyname() once for each instance of a duplicate name,

yes

> or
> does it perform a single lookup and use the same result for all?
> 
> 4.  When and how does the failover code check for remote destination
> failures? 
 
API return code

> Does it re-check failed servers after they're declared
> "failed"? 
ye
s
> If so, does Rsyslog re-check a failed server every time it
> sends a log message, or does it have some kind of polling timeout, or
> is the check interval determined by something else entirely? 

It's an increasing interval (the longer it is down, the less frequently it is
probed with some upper bound). I think (but not sure) you can configure the
timers.

> When a
> server fails, how long does it take Rsyslog to notice the failure and
> start re-directing messages to the next host? 

As soon as the socket call returns an error, what may not be very soon.

> When a server recovers,
> does Rsyslog automatically notice the recovery, and if so, how long
> does it take to start re-directing messages to the primary host?

see above - it depends on how long it was down. The lower bound is, I think,
30 seconds. (Except that when a failure happens, one retry is done
immediately). 

> 
> In case anybody's wondering, I've include some examples of what I'm
> thinking of trying, here. Assume that there are three (3) central log
> receiver hosts, with each with a unique DNS name ('log-alfa',
> 'log-bravo', and 'log-charlie') that points to its own IP. Also,
> assume that the round-robin DNS name 'log' points to all three hosts'
> IP addresses, and that all of my log-generating hosts (the clients)
> are configured to pick a random IP when calling gethostbyname() on a
> round-robin name.
> 
>     *.* @@log
>     $ActionExecOnlyWhenPreviousIsSuspended on
>     & @@log
>     & @@log
>     & /var/log/localbuffer
>     $ActionExecOnlyWhenPreviousIsSuspended off
> 
> But if Rsyslog resolves all three instances of the name 'log' with a
> single call to gethostbyname(), this won't work. 

Not with a single call, but you never know who else queries DNS in the mean
time. So I'd say that configuration is at least unreliable. I strongly
recommend against it.

> I could probably hack
> around it with some additional round robin entries similar to 'log',
> but named 'log-all-1', 'log-all-2', 'log-all-3', etc., to force an
> independent lookup attempt for each:
> 
>     *.* @@log
>     $ActionExecOnlyWhenPreviousIsSuspended on
>     & @@log-all-1
>     & @@log-all-2
>     & @@log-all-3
>     & @@log-all-4
>     & @@log-all-5
>     & @@log-all-6
>     ...
>     & /var/log/localbuffer
>     $ActionExecOnlyWhenPreviousIsSuspended off
> 
> But this is pretty messy. First, there's a lot of extra DNS records to
> maintain, which is a real pain. Second, I have to use a lot of extra
> round-robin names (definitely more than 3) to maximize the probability
> that gethostbyname() will return at least one working server before
> reaching the end of the list. (gethostbyname() isn't aware of downed
> hosts, nor does it enforce balance in its responses, so it might
> return the same bad server several times in a row.)
> 
> It would nice to have a random variant of the
> '$ActionExecOnlyWhenPreviousIsSuspended' functionality. As a
> hypothetical example I just cooked up off the top of my head:
> 
>     *.* @@log-alfa
>     $ActionExecPickRandom on
>     $ActionExecPickRandomRetryWait 1
>     $ActionExecPickRandomRetryLimit -1
>     & @@log-bravo
>     & @@log-charlie
>     $ActionExecPickRandom off
> 
> where Rsyslog randomly selects one of log-alfa, log-bravo, and
> log-charlie, initially, and then makes another random pick at
> failover, etc. I can think of all sorts of nifty options and
> configurable knobs that would come in handy, here, too.
> 

That's pretty interesting, but I think community demand is very low for this
feature. So it is unlikely that I will find time to implement it in the
foreseeable future (if others like that feature, too, please make yourself
heard! It may influence priorities, what means that something else does not
get done ;)).

I'll provide more info on my schedule with a separate mail.

Rainer

> I'm planning to try out my first two examples, tomorrow, to see what
> works. If anybody has any comments, I'd love to hear them.
> Ryan B. Lynch
> ryan.b.lynch at gmail.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Wed Dec  9 10:56:45 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 9 Dec 2009 10:56:45 +0100
Subject: [rsyslog] No DNS resolution for the property '$fromhost',
	it's the local host.
References: <115906d10912082158v50ff93f8y843470ae106bdd5d@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103561@GRFEXC.intern.adiscon.com>

the currently released version of librelp simply does not provide that
information. However, I have a version 1.0 available, which does. I'll see
that I manage to release it today. You need to use a recent version of
rsyslog to use the new librelp 1.0 functionality.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Ryan Lynch
> Sent: Wednesday, December 09, 2009 6:58 AM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] No DNS resolution for the property '$fromhost',it's
> the local host.
> 
> I've configured syslog forwarding via RELP, and it's mostly working
> properly. The only problem is that DNS resolution doesn't seem to be
> enabled for the '$fromhost' property, if the log message is coming
> from a remote machine via RELP. If the log destination is local, it
> works fine: $fromhost resolves to the simple (no FQDN) hostname, as
> does $hostname. But if the log destination is a remote machine via
> RELP, I '$fromhost' doesn't resolves to a hostname, it's always an IP
> address.
> 
> I found a  similar issue on the forum, here:
> 
>  * http://kb.monitorware.com/reverse-dns-resolution-why-t10040.html
> 
> Like that poster, my forward and reverse DNS lookups work just fine,
> on all of the hosts involved.
> 
> Is this a known limitation of RELP, or remote destinations in general,
> or should I file a bug report?
> 
> -Ryan
> 
> Ryan B. Lynch
> ryan.b.lynch at gmail.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Wed Dec  9 10:59:04 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 9 Dec 2009 10:59:04 +0100
Subject: [rsyslog] Help with client config
References: <001301ca783f$691b9aa5$100013ac@intern.adiscon.com><4B1EC6EA.6090508@ooma.com>
	<4B1ECEBB.4030905@ooma.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103562@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rory Toma
> Sent: Tuesday, December 08, 2009 11:10 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Help with client config
> 
> And, actually, this only happens some of the time, where some of the
> time means never on my dev systems and always on remote ones that I
> can't easily debug. 8-)

Now you feel a bit like me ;)

> 
> 
> Rory Toma wrote:
> > It works just fine without tls. I do not know much about tls, is
> there a
> > default size somewhere that I can swizzle?

TLS is a stream cipher, so it needs some minimal block size to be secure.
However, that minimum is pretty low. Rsyslog does not do any batching by
itself for outbound connections. But it could be that GnuTLS can be tweaked
one way or the other. I'd look into that direction.

Rainer

> >
> > Rainer Gerhards wrote:
> >
> >> Can 1 be a tls (lib) artifact? Can you try with plain tcp?
> >>
> >> ----- Urspr?ngliche Nachricht -----
> >> Von: "Rory Toma" <rory at ooma.com>
> >> An: "rsyslog-users" <rsyslog at lists.adiscon.com>
> >> Gesendet: 08.12.09 20:39
> >> Betreff: [rsyslog] Help with client config
> >>
> >> I have a client config as follows, running on armv4, rsyslog 4.4.2
> (I
> >> can't get anything newer to compile due to the atomic kernel support
> not
> >> wanting to compile) The problem I'm having is:
> >>
> >> 1) (this is probably an rtfm, but I'll ask it anyway) It seems that
> my
> >> logs being sent over the network are being batched up in huge
> packets, I
> >> see one transaction for a bunch of data. Sometimes this means my
> logs
> >> are delayed quite a bit.
> >>
> >> 2) Sometimes, the client just doesn't send out over the network at
> all
> >> (verified with a tcpdump), but continues logging.
> >>
> >> Any suggestions or config file tweaks welcome.
> >>
> >> thx
> >>
> >> Here is my config:
> >>
> >> $DefaultNetStreamDriverCAFile /etc/ca.pem
> >>
> >> $DefaultNetStreamDriver gtls
> >> $ActionSendStreamDriverMode 1
> >> $ActionSendStreamDriverAuthMode anon
> >>
> >> $WorkDirectory /var/log
> >>
> >> $ActionQueueType LinkedList
> >> $ActionQueueFileName rsyslog-fwd
> >> $ActionResumeRetryCount -1
> >> $ActionQueueSaveOnShutdown on
> >> $ActionQueueMaxDiskSpace 256k
> >>
> >> $ModLoad imuxsock
> >> $SystemLogSocketName /var/log/log
> >> $OptimizeForUniprocessor on
> >>
> >> $outchannel
> locallog,/var/log/locallog,262144,/usr/bin/rotate_locallog
> >>
> >> *.* @@foo.bar.com:80
> >> *.* $locallog
> >>
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >>
> >>
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From mrdemeanour at jackpot.uk.net  Wed Dec  9 12:23:10 2009
From: mrdemeanour at jackpot.uk.net (Mr. Demeanour)
Date: Wed, 09 Dec 2009 11:23:10 +0000
Subject: [rsyslog] TLS: memory leaks
Message-ID: <4B1F889E.8020409@jackpot.uk.net>

Hi,

I've been trying for some weeks to construct a reproducible scenario for
the memory leaks I've been observing with gnutls. In summary, what I
have been seeing is that rsyslog eventually consumes all the system
memory and hangs the machine; this only happens if gnutls is configured.

This was happening pretty quickly (and consistently) with 4.4.2. As
suggested, I upgraded to a locally-built 4.5.6; around the same time, I
checked (and corrected) certain errors in the certificate setup. And
incidentally I also tightened-up the firewall, so that random
port-scanners and general internet pollution was less likely to be
making TCP connections on the syslog server's port.

The result is that the OOM condition *has* recurred once or twice; but
not consistently, and nothing like as often or as quickly as it was
initially with 4.4.2. However I'm not sure the version of rsyslog is the
only software-version factor involved, as I have also upgraded the
version of the gnutls library on both the clients and the server.

So with the 4.5.6 configuration, I have now managed to demonstrate a
reproducible leak scenario (reproducible on my setup, anyway). I simply
have a PHP script open then close 100 non-TLS TCP connections (from
another computer) to the TLS syslog port. No data is sent. Connections
are opened then closed individually - the script has no more than one
socket open at one time. The aim was to simulate the effect of some
port-scanner bot hitting the Gnutls port repeatedly (i.e. what I have
configured my firewall to prevent). E.g.:

# valgrind --log-file=valgrind.log --leak-check=full
/usr/local/sbin/rsyslogd -c4 -f /etc/rsyslog.conf -d >rsyslog.log

Before:
# ps -p `cat /var/run/rsyslogd.pid` -o args -o pid=pid -o sz=sz -o vsz=vsz
COMMAND                       pid    sz    vsz
/usr/bin/valgrind.bin --log 11249 31142 124568

After making 100 TCP client socket connections:
# ps -p `cat /var/run/rsyslogd.pid` -o args -o pid=pid -o sz=sz -o vsz=vsz
COMMAND                       pid    sz    vsz
/usr/bin/valgrind.bin --log 11249 32214 128856

The way I read that, the amount of virtual memory (vsz) consumed by
rsyslog has increased by 4288 x 1024 bytes between these two ps
invocations, and the size of the core image (sz - I guess that means
excluding heap memory - I dunno) by 1072 pages (without valgrind, the
numbers are much smaller - the increase is smaller by a factor of about
10). It's unlikely that this memory has been consumed in the course of
normal handling of syslog messages, as the two ps runs are just about
10s apart, and the normal flow of messages on this system would normally
include only a few tens of messages maximum during such an interval.

Tests run separately, and without valgrind, indicate that memory
consumed as a result of the "unencrypted connections" test is not
returned to the system within ten minutes.

It seems that this configuration of rsyslog does not show observable
memory draining in normal operation when run for several days. With the
test, the draining that is observed is much greater than the typical
fluctuation in memory usage observed during normal operation.

Incidentally: after running the test, "netstat -ant" shows a large
number of connections to the TLS syslog port in a CLOSE_WAIT state; as I
understand it, this means the connection has been closed by the client,
but the TCP stack is waiting for the server to complete the close. These
dangling connections disappear when rsyslog is terminated.

I'm rather new at this kind of memory-tracking work on Linux; so
apologies for the low quality.

rsyslog.log and valgrind.log are being sent to Rainer under separate
cover. I haven't been able to match-up the results from ps with the data
in the valgrind logfile, but I assume that is just further evidence of
my ignorance :-)

-- 
Jack.

From xkubina at fi.muni.cz  Wed Dec  9 13:39:35 2009
From: xkubina at fi.muni.cz (Tomas Kubina)
Date: Wed, 09 Dec 2009 13:39:35 +0100
Subject: [rsyslog] TLS/GSSAPI client message lost
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71034DF@GRFEXC.intern.adiscon.com>
References: <4B13CEEB.4040504@fi.muni.cz>
	<9B6E2A8877C38245BFB15CC491A11DA71034DF@GRFEXC.intern.adiscon.com>
Message-ID: <4B1F9A87.7050605@fi.muni.cz>

Rainer Gerhards wrote:
> I unfortunately do not have enough insight into GSSAPI to provide a real
> answer. My guess is that this happens for the same reason it can happen with
> any ack-less transport. In the plain tcp driver, I have done quite some work
> to try to limit loss potential. I have also some ideas of how to further try
> to prevent message loss, but you cannot totally aovid it.
>
> some background:
>
> http://blog.gerhards.net/2008/04/on-unreliability-of-plain-tcp-syslog.html
>
> Rainer
>
>   
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Tomas Kubina
>> Sent: Monday, November 30, 2009 2:56 PM
>> To: rsyslog at lists.adiscon.com
>> Subject: [rsyslog] TLS/GSSAPI client message lost
>>
>> Hi,
>>
>> I want to use TLS or GSS for message delivering to central rsyslog
>> server.
>> The problem is that the first message logged after server's shutdown is
>> lost,
>> but when I use plain TCP this issue doesn't happen. Is it a feature or
>> mistake
>> in my config?
>>
>> Tomas
>
I have done a patch to GSSAPI based on your code from nsd_ptcp.c.
It works and can be added into svn if you want.

Regards,

Tomas Kubina

From xkubina at fi.muni.cz  Wed Dec  9 13:47:36 2009
From: xkubina at fi.muni.cz (Tomas Kubina)
Date: Wed, 09 Dec 2009 13:47:36 +0100
Subject: [rsyslog] TLS/GSSAPI client message lost
In-Reply-To: <4B1F9A87.7050605@fi.muni.cz>
References: <4B13CEEB.4040504@fi.muni.cz>	<9B6E2A8877C38245BFB15CC491A11DA71034DF@GRFEXC.intern.adiscon.com>
	<4B1F9A87.7050605@fi.muni.cz>
Message-ID: <4B1F9C68.6000703@fi.muni.cz>

Tomas Kubina wrote:
> Rainer Gerhards wrote:
>> I unfortunately do not have enough insight into GSSAPI to provide a real
>> answer. My guess is that this happens for the same reason it can 
>> happen with
>> any ack-less transport. In the plain tcp driver, I have done quite 
>> some work
>> to try to limit loss potential. I have also some ideas of how to 
>> further try
>> to prevent message loss, but you cannot totally aovid it.
>>
>> some background:
>>
>> http://blog.gerhards.net/2008/04/on-unreliability-of-plain-tcp-syslog.html 
>>
>>
>> Rainer
>>
>>  
>>> -----Original Message-----
>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>> bounces at lists.adiscon.com] On Behalf Of Tomas Kubina
>>> Sent: Monday, November 30, 2009 2:56 PM
>>> To: rsyslog at lists.adiscon.com
>>> Subject: [rsyslog] TLS/GSSAPI client message lost
>>>
>>> Hi,
>>>
>>> I want to use TLS or GSS for message delivering to central rsyslog
>>> server.
>>> The problem is that the first message logged after server's shutdown is
>>> lost,
>>> but when I use plain TCP this issue doesn't happen. Is it a feature or
>>> mistake
>>> in my config?
>>>
>>> Tomas
>>
> I have done a patch to GSSAPI based on your code from nsd_ptcp.c.
> It works and can be added into svn if you want.
>
> Regards,
>
> Tomas Kubina
Sorry, I'm not sure if I the previous mail contained right attachment, 
so here is the patch:

--- rsyslog-5.5.1/gss-misc.c    2009-11-27 10:40:51.000000000 +0100
+++ rsyslog-5.5.1-tom/gss-misc.c    2009-12-09 13:26:47.000000000 +0100
@@ -152,8 +152,15 @@
 {
     int     ret;
     char   *ptr;
+    char    msgbuf[1]; /* dummy */
 
     for (ptr = buf; nbyte; ptr += ret, nbyte -= ret) {
+    ret = recv(fd, msgbuf, 1, MSG_DONTWAIT | MSG_PEEK);
+    if(ret == 0) {
+        dbgprintf("write_all function detected broken connection\n");
+        /* in this case, the remote peer had shut down the connection */
+        return -1;
+    }
     ret = send(fd, ptr, nbyte, 0);
     if (ret < 0) {
         if (errno == EINTR)

From rgerhards at hq.adiscon.com  Wed Dec  9 13:59:22 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 9 Dec 2009 13:59:22 +0100
Subject: [rsyslog] TLS/GSSAPI client message lost
References: <4B13CEEB.4040504@fi.muni.cz><9B6E2A8877C38245BFB15CC491A11DA71034DF@GRFEXC.intern.adiscon.com>
	<4B1F9A87.7050605@fi.muni.cz>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103579@GRFEXC.intern.adiscon.com>

any patches are most welcome! :-)

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Tomas Kubina
> Sent: Wednesday, December 09, 2009 1:40 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] TLS/GSSAPI client message lost
> 
> Rainer Gerhards wrote:
> > I unfortunately do not have enough insight into GSSAPI to provide a
> real
> > answer. My guess is that this happens for the same reason it can
> happen with
> > any ack-less transport. In the plain tcp driver, I have done quite
> some work
> > to try to limit loss potential. I have also some ideas of how to
> further try
> > to prevent message loss, but you cannot totally aovid it.
> >
> > some background:
> >
> > http://blog.gerhards.net/2008/04/on-unreliability-of-plain-tcp-
> syslog.html
> >
> > Rainer
> >
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Tomas Kubina
> >> Sent: Monday, November 30, 2009 2:56 PM
> >> To: rsyslog at lists.adiscon.com
> >> Subject: [rsyslog] TLS/GSSAPI client message lost
> >>
> >> Hi,
> >>
> >> I want to use TLS or GSS for message delivering to central rsyslog
> >> server.
> >> The problem is that the first message logged after server's shutdown
> is
> >> lost,
> >> but when I use plain TCP this issue doesn't happen. Is it a feature
> or
> >> mistake
> >> in my config?
> >>
> >> Tomas
> >
> I have done a patch to GSSAPI based on your code from nsd_ptcp.c.
> It works and can be added into svn if you want.
> 
> Regards,
> 
> Tomas Kubina

From rgerhards at hq.adiscon.com  Wed Dec  9 13:59:55 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 9 Dec 2009 13:59:55 +0100
Subject: [rsyslog] TLS/GSSAPI client message lost
References: <4B13CEEB.4040504@fi.muni.cz>	<9B6E2A8877C38245BFB15CC491A11DA71034DF@GRFEXC.intern.adiscon.com><4B1F9A87.7050605@fi.muni.cz>
	<4B1F9C68.6000703@fi.muni.cz>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710357A@GRFEXC.intern.adiscon.com>

oh, there it is - thanks, will see that I integrate it ASAP, what most
probably means tomorrow morning :)

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Tomas Kubina
> Sent: Wednesday, December 09, 2009 1:48 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] TLS/GSSAPI client message lost
> 
> Tomas Kubina wrote:
> > Rainer Gerhards wrote:
> >> I unfortunately do not have enough insight into GSSAPI to provide a
> real
> >> answer. My guess is that this happens for the same reason it can
> >> happen with
> >> any ack-less transport. In the plain tcp driver, I have done quite
> >> some work
> >> to try to limit loss potential. I have also some ideas of how to
> >> further try
> >> to prevent message loss, but you cannot totally aovid it.
> >>
> >> some background:
> >>
> >> http://blog.gerhards.net/2008/04/on-unreliability-of-plain-tcp-
> syslog.html
> >>
> >>
> >> Rainer
> >>
> >>
> >>> -----Original Message-----
> >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >>> bounces at lists.adiscon.com] On Behalf Of Tomas Kubina
> >>> Sent: Monday, November 30, 2009 2:56 PM
> >>> To: rsyslog at lists.adiscon.com
> >>> Subject: [rsyslog] TLS/GSSAPI client message lost
> >>>
> >>> Hi,
> >>>
> >>> I want to use TLS or GSS for message delivering to central rsyslog
> >>> server.
> >>> The problem is that the first message logged after server's
> shutdown is
> >>> lost,
> >>> but when I use plain TCP this issue doesn't happen. Is it a feature
> or
> >>> mistake
> >>> in my config?
> >>>
> >>> Tomas
> >>
> > I have done a patch to GSSAPI based on your code from nsd_ptcp.c.
> > It works and can be added into svn if you want.
> >
> > Regards,
> >
> > Tomas Kubina
> Sorry, I'm not sure if I the previous mail contained right attachment,
> so here is the patch:
> 
> --- rsyslog-5.5.1/gss-misc.c    2009-11-27 10:40:51.000000000 +0100
> +++ rsyslog-5.5.1-tom/gss-misc.c    2009-12-09 13:26:47.000000000 +0100
> @@ -152,8 +152,15 @@
>  {
>      int     ret;
>      char   *ptr;
> +    char    msgbuf[1]; /* dummy */
> 
>      for (ptr = buf; nbyte; ptr += ret, nbyte -= ret) {
> +    ret = recv(fd, msgbuf, 1, MSG_DONTWAIT | MSG_PEEK);
> +    if(ret == 0) {
> +        dbgprintf("write_all function detected broken connection\n");
> +        /* in this case, the remote peer had shut down the connection
> */
> +        return -1;
> +    }
>      ret = send(fd, ptr, nbyte, 0);
>      if (ret < 0) {
>          if (errno == EINTR)
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From marc.schiffbauer at mightycare.de  Wed Dec  9 14:40:05 2009
From: marc.schiffbauer at mightycare.de (Marc Schiffbauer)
Date: Wed, 9 Dec 2009 14:40:05 +0100
Subject: [rsyslog] rsyslog 4.5.x queue file cleanup?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710335F@GRFEXC.intern.adiscon.com>
References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com>
	<200911031158.02666.marc.schiffbauer@mightycare.de>
	<9B6E2A8877C38245BFB15CC491A11DA710335F@GRFEXC.intern.adiscon.com>
Message-ID: <200912091440.05811.marc.schiffbauer@mightycare.de>

Hallo Rainer,

I am back now. And as promised I wanted to test 5.3.4 now... I saw 5.3.5 is 
out so I wanted to give that version a try.

But unfortunately it does not build. Any hint?



In file included from omfile.c:67:
../runtime/atomic.h:64:3: warning: #warning "atomic builtins not available, 
using nul operations - rsyslogd will probably be racy!"
omfile.c: In function ?getClockFileAccess?:
omfile.c:91: warning: implicit declaration of function ?ATOMIC_INC_AND_FETCH?
  CC     rsyslogd-omdiscard.o
  CC     rsyslogd-pmrfc5424.o
  CC     rsyslogd-pmrfc3164.o
  CC     rsyslogd-iminternal.o
  CC     rsyslogd-pidfile.o
  CCLD   rsyslogd
rsyslogd-omfile.o: In function `getClockFileAccess':
/root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to 
`ATOMIC_INC_AND_FETCH'
/root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to 
`ATOMIC_INC_AND_FETCH'
/root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to 
`ATOMIC_INC_AND_FETCH'
/root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to 
`ATOMIC_INC_AND_FETCH'
../runtime/.libs/librsyslog.a(librsyslog_la-glbl.o): In function 
`SetGlobalInputTermination':
/root/src/rsyslog-5.3.5/runtime/glbl.c:136: undefined reference to 
`ATOMIC_STORE_1_TO_INT'
../runtime/.libs/librsyslog.a(librsyslog_la-msg.o): In function `msgDestruct':
/root/src/rsyslog-5.3.5/runtime/msg.c:802: undefined reference to 
`ATOMIC_INC_AND_FETCH'
../runtime/.libs/librsyslog.a(librsyslog_la-wti.o): In function `wtiSetState':
/root/src/rsyslog-5.3.5/runtime/wti.c:111: undefined reference to 
`ATOMIC_STORE_0_TO_INT'
/root/src/rsyslog-5.3.5/runtime/wti.c:109: undefined reference to 
`ATOMIC_STORE_1_TO_INT'
../runtime/.libs/librsyslog.a(librsyslog_la-queue.o): In function 
`DoDeleteBatchFromQStore':
/root/src/rsyslog-5.3.5/runtime/queue.c:1291: undefined reference to 
`ATOMIC_SUB'
/root/src/rsyslog-5.3.5/runtime/queue.c:1292: undefined reference to 
`ATOMIC_SUB'
/root/src/rsyslog-5.3.5/runtime/queue.c:1291: undefined reference to 
`ATOMIC_SUB'
/root/src/rsyslog-5.3.5/runtime/queue.c:1292: undefined reference to 
`ATOMIC_SUB'
/root/src/rsyslog-5.3.5/runtime/queue.c:1291: undefined reference to 
`ATOMIC_SUB'
../runtime/.libs/librsyslog.a(librsyslog_la-
queue.o):/root/src/rsyslog-5.3.5/runtime/queue.c:1292: more undefined 
references to `ATOMIC_SUB' follow
collect2: ld returned 1 exit status
make[2]: *** [rsyslogd] Error 1
make[2]: Leaving directory `/root/src/rsyslog-5.3.5/tools'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/src/rsyslog-5.3.5'
make: *** [all] Error 2




Am Donnerstag, 5. November 2009 17:42:01 schrieb Rainer Gerhards:
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer
> > Sent: Tuesday, November 03, 2009 11:58 AM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup?
> >
> > Hello Rainer,
> >
> > is there some news about this issue?
> 
> Unfortuantely not at the moment (I guess you see it is a busy time). Could
> you give 5.3.4 a try? It would be very interesting (even for a v4-fix) to
>  see if the problem persists or not...
> 
> Rainer
> 
> > TIA
> > -Marc
> >
> > Am Dienstag, 20. Oktober 2009 18:38:16 schrieb Rainer Gerhards:
> > > thanks!
> > >
> > > Interesting, I see that only one of the messages that should be in
> >
> > the .qi
> >
> > > are actually present. I wonder if this is related to the fact that
> >
> > ompgsql
> >
> > > emits an error message itself while being down (the others don't do
> >
> > this
> >
> > > AFIK). Will try to dig down to this (but I have to do so as time
> >
> > permits).
> >
> > > Rainer
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 

From rgerhards at hq.adiscon.com  Wed Dec  9 14:52:15 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 9 Dec 2009 14:52:15 +0100
Subject: [rsyslog] rsyslog 4.5.x queue file cleanup?
References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com><200911031158.02666.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA710335F@GRFEXC.intern.adiscon.com>
	<200912091440.05811.marc.schiffbauer@mightycare.de>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710357F@GRFEXC.intern.adiscon.com>

Marc,

I guess you use a 32-bit platform. Please pull the latest version from the
git master branch, it contains a fix for this type of build error.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer
> Sent: Wednesday, December 09, 2009 2:40 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup?
> 
> Hallo Rainer,
> 
> I am back now. And as promised I wanted to test 5.3.4 now... I saw
> 5.3.5 is
> out so I wanted to give that version a try.
> 
> But unfortunately it does not build. Any hint?
> 
> 
> 
> In file included from omfile.c:67:
> ../runtime/atomic.h:64:3: warning: #warning "atomic builtins not
> available,
> using nul operations - rsyslogd will probably be racy!"
> omfile.c: In function ?getClockFileAccess?:
> omfile.c:91: warning: implicit declaration of function
> ?ATOMIC_INC_AND_FETCH?
>   CC     rsyslogd-omdiscard.o
>   CC     rsyslogd-pmrfc5424.o
>   CC     rsyslogd-pmrfc3164.o
>   CC     rsyslogd-iminternal.o
>   CC     rsyslogd-pidfile.o
>   CCLD   rsyslogd
> rsyslogd-omfile.o: In function `getClockFileAccess':
> /root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to
> `ATOMIC_INC_AND_FETCH'
> /root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to
> `ATOMIC_INC_AND_FETCH'
> /root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to
> `ATOMIC_INC_AND_FETCH'
> /root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to
> `ATOMIC_INC_AND_FETCH'
> ../runtime/.libs/librsyslog.a(librsyslog_la-glbl.o): In function
> `SetGlobalInputTermination':
> /root/src/rsyslog-5.3.5/runtime/glbl.c:136: undefined reference to
> `ATOMIC_STORE_1_TO_INT'
> ../runtime/.libs/librsyslog.a(librsyslog_la-msg.o): In function
> `msgDestruct':
> /root/src/rsyslog-5.3.5/runtime/msg.c:802: undefined reference to
> `ATOMIC_INC_AND_FETCH'
> ../runtime/.libs/librsyslog.a(librsyslog_la-wti.o): In function
> `wtiSetState':
> /root/src/rsyslog-5.3.5/runtime/wti.c:111: undefined reference to
> `ATOMIC_STORE_0_TO_INT'
> /root/src/rsyslog-5.3.5/runtime/wti.c:109: undefined reference to
> `ATOMIC_STORE_1_TO_INT'
> ../runtime/.libs/librsyslog.a(librsyslog_la-queue.o): In function
> `DoDeleteBatchFromQStore':
> /root/src/rsyslog-5.3.5/runtime/queue.c:1291: undefined reference to
> `ATOMIC_SUB'
> /root/src/rsyslog-5.3.5/runtime/queue.c:1292: undefined reference to
> `ATOMIC_SUB'
> /root/src/rsyslog-5.3.5/runtime/queue.c:1291: undefined reference to
> `ATOMIC_SUB'
> /root/src/rsyslog-5.3.5/runtime/queue.c:1292: undefined reference to
> `ATOMIC_SUB'
> /root/src/rsyslog-5.3.5/runtime/queue.c:1291: undefined reference to
> `ATOMIC_SUB'
> ../runtime/.libs/librsyslog.a(librsyslog_la-
> queue.o):/root/src/rsyslog-5.3.5/runtime/queue.c:1292: more undefined
> references to `ATOMIC_SUB' follow
> collect2: ld returned 1 exit status
> make[2]: *** [rsyslogd] Error 1
> make[2]: Leaving directory `/root/src/rsyslog-5.3.5/tools'
> make[1]: *** [all-recursive] Error 1
> make[1]: Leaving directory `/root/src/rsyslog-5.3.5'
> make: *** [all] Error 2
> 
> 
> 
> 
> Am Donnerstag, 5. November 2009 17:42:01 schrieb Rainer Gerhards:
> > > -----Original Message-----
> > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer
> > > Sent: Tuesday, November 03, 2009 11:58 AM
> > > To: rsyslog-users
> > > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup?
> > >
> > > Hello Rainer,
> > >
> > > is there some news about this issue?
> >
> > Unfortuantely not at the moment (I guess you see it is a busy time).
> Could
> > you give 5.3.4 a try? It would be very interesting (even for a v4-
> fix) to
> >  see if the problem persists or not...
> >
> > Rainer
> >
> > > TIA
> > > -Marc
> > >
> > > Am Dienstag, 20. Oktober 2009 18:38:16 schrieb Rainer Gerhards:
> > > > thanks!
> > > >
> > > > Interesting, I see that only one of the messages that should be
> in
> > >
> > > the .qi
> > >
> > > > are actually present. I wonder if this is related to the fact
> that
> > >
> > > ompgsql
> > >
> > > > emits an error message itself while being down (the others don't
> do
> > >
> > > this
> > >
> > > > AFIK). Will try to dig down to this (but I have to do so as time
> > >
> > > permits).
> > >
> > > > Rainer
> > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From marc.schiffbauer at mightycare.de  Wed Dec  9 15:16:15 2009
From: marc.schiffbauer at mightycare.de (Marc Schiffbauer)
Date: Wed, 9 Dec 2009 15:16:15 +0100
Subject: [rsyslog] rsyslog 4.5.x queue file cleanup?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710357F@GRFEXC.intern.adiscon.com>
References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com>
	<200912091440.05811.marc.schiffbauer@mightycare.de>
	<9B6E2A8877C38245BFB15CC491A11DA710357F@GRFEXC.intern.adiscon.com>
Message-ID: <200912091516.15909.marc.schiffbauer@mightycare.de>


Sorry, but I am bit puzzled from all the branches. Which one should I take 
now? Really the head master? I think I will get some 5.5.x version then, not?

What do you mean by "latest version" Head of a 5.3-Branch? Or checkout the 
5.3.4 tag from git?

This has the same error:
http://git.adiscon.com/?p=rsyslog.git;a=snapshot;h=c051786ea6318f9a9a5e531c49512e4ae19f249b;sf=tgz

-Marc



Am Mittwoch, 9. Dezember 2009 14:52:15 schrieb Rainer Gerhards:
> Marc,
> 
> I guess you use a 32-bit platform. Please pull the latest version from the
> git master branch, it contains a fix for this type of build error.
> 
> Rainer
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer
> > Sent: Wednesday, December 09, 2009 2:40 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup?
> >
> > Hallo Rainer,
> >
> > I am back now. And as promised I wanted to test 5.3.4 now... I saw
> > 5.3.5 is
> > out so I wanted to give that version a try.
> >
> > But unfortunately it does not build. Any hint?
> >
> >
> >
> > In file included from omfile.c:67:
> > ../runtime/atomic.h:64:3: warning: #warning "atomic builtins not
> > available,
> > using nul operations - rsyslogd will probably be racy!"
> > omfile.c: In function ?getClockFileAccess?:
> > omfile.c:91: warning: implicit declaration of function
> > ?ATOMIC_INC_AND_FETCH?
> >   CC     rsyslogd-omdiscard.o
> >   CC     rsyslogd-pmrfc5424.o
> >   CC     rsyslogd-pmrfc3164.o
> >   CC     rsyslogd-iminternal.o
> >   CC     rsyslogd-pidfile.o
> >   CCLD   rsyslogd
> > rsyslogd-omfile.o: In function `getClockFileAccess':
> > /root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to
> > `ATOMIC_INC_AND_FETCH'
> > /root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to
> > `ATOMIC_INC_AND_FETCH'
> > /root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to
> > `ATOMIC_INC_AND_FETCH'
> > /root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to
> > `ATOMIC_INC_AND_FETCH'
> > ../runtime/.libs/librsyslog.a(librsyslog_la-glbl.o): In function
> > `SetGlobalInputTermination':
> > /root/src/rsyslog-5.3.5/runtime/glbl.c:136: undefined reference to
> > `ATOMIC_STORE_1_TO_INT'
> > ../runtime/.libs/librsyslog.a(librsyslog_la-msg.o): In function
> > `msgDestruct':
> > /root/src/rsyslog-5.3.5/runtime/msg.c:802: undefined reference to
> > `ATOMIC_INC_AND_FETCH'
> > ../runtime/.libs/librsyslog.a(librsyslog_la-wti.o): In function
> > `wtiSetState':
> > /root/src/rsyslog-5.3.5/runtime/wti.c:111: undefined reference to
> > `ATOMIC_STORE_0_TO_INT'
> > /root/src/rsyslog-5.3.5/runtime/wti.c:109: undefined reference to
> > `ATOMIC_STORE_1_TO_INT'
> > ../runtime/.libs/librsyslog.a(librsyslog_la-queue.o): In function
> > `DoDeleteBatchFromQStore':
> > /root/src/rsyslog-5.3.5/runtime/queue.c:1291: undefined reference to
> > `ATOMIC_SUB'
> > /root/src/rsyslog-5.3.5/runtime/queue.c:1292: undefined reference to
> > `ATOMIC_SUB'
> > /root/src/rsyslog-5.3.5/runtime/queue.c:1291: undefined reference to
> > `ATOMIC_SUB'
> > /root/src/rsyslog-5.3.5/runtime/queue.c:1292: undefined reference to
> > `ATOMIC_SUB'
> > /root/src/rsyslog-5.3.5/runtime/queue.c:1291: undefined reference to
> > `ATOMIC_SUB'
> > ../runtime/.libs/librsyslog.a(librsyslog_la-
> > queue.o):/root/src/rsyslog-5.3.5/runtime/queue.c:1292: more undefined
> > references to `ATOMIC_SUB' follow
> > collect2: ld returned 1 exit status
> > make[2]: *** [rsyslogd] Error 1
> > make[2]: Leaving directory `/root/src/rsyslog-5.3.5/tools'
> > make[1]: *** [all-recursive] Error 1
> > make[1]: Leaving directory `/root/src/rsyslog-5.3.5'
> > make: *** [all] Error 2
> >
> > Am Donnerstag, 5. November 2009 17:42:01 schrieb Rainer Gerhards:
> > > > -----Original Message-----
> > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > > > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer
> > > > Sent: Tuesday, November 03, 2009 11:58 AM
> > > > To: rsyslog-users
> > > > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup?
> > > >
> > > > Hello Rainer,
> > > >
> > > > is there some news about this issue?
> > >
> > > Unfortuantely not at the moment (I guess you see it is a busy time).
> >
> > Could
> >
> > > you give 5.3.4 a try? It would be very interesting (even for a v4-
> >
> > fix) to
> >
> > >  see if the problem persists or not...
> > >
> > > Rainer
> > >
> > > > TIA
> > > > -Marc
> > > >
> > > > Am Dienstag, 20. Oktober 2009 18:38:16 schrieb Rainer Gerhards:
> > > > > thanks!
> > > > >
> > > > > Interesting, I see that only one of the messages that should be
> >
> > in
> >
> > > > the .qi
> > > >
> > > > > are actually present. I wonder if this is related to the fact
> >
> > that
> >
> > > > ompgsql
> > > >
> > > > > emits an error message itself while being down (the others don't
> >
> > do
> >
> > > > this
> > > >
> > > > > AFIK). Will try to dig down to this (but I have to do so as time
> > > >
> > > > permits).
> > > >
> > > > > Rainer
> > > >
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com
> > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 

From marc.schiffbauer at mightycare.de  Wed Dec  9 15:23:00 2009
From: marc.schiffbauer at mightycare.de (Marc Schiffbauer)
Date: Wed, 9 Dec 2009 15:23:00 +0100
Subject: [rsyslog] rsyslog 4.5.x queue file cleanup?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710357F@GRFEXC.intern.adiscon.com>
References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com>
	<200912091440.05811.marc.schiffbauer@mightycare.de>
	<9B6E2A8877C38245BFB15CC491A11DA710357F@GRFEXC.intern.adiscon.com>
Message-ID: <200912091523.00555.marc.schiffbauer@mightycare.de>


Same error with this:

http://git.adiscon.com/?p=rsyslog.git;a=snapshot;h=refs/heads/master;sf=tgz


Am Mittwoch, 9. Dezember 2009 14:52:15 schrieb Rainer Gerhards:
> Marc,
> 
> I guess you use a 32-bit platform. Please pull the latest version from the
> git master branch, it contains a fix for this type of build error.
> 
> Rainer
> 

From ktm at rice.edu  Wed Dec  9 15:35:44 2009
From: ktm at rice.edu (Kenneth Marshall)
Date: Wed, 9 Dec 2009 08:35:44 -0600
Subject: [rsyslog] rsyslog 4.5.x queue file cleanup?
In-Reply-To: <200912091516.15909.marc.schiffbauer@mightycare.de>
References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com>
	<200912091440.05811.marc.schiffbauer@mightycare.de>
	<9B6E2A8877C38245BFB15CC491A11DA710357F@GRFEXC.intern.adiscon.com>
	<200912091516.15909.marc.schiffbauer@mightycare.de>
Message-ID: <20091209143544.GL3237@it.is.rice.edu>

Hi Marc,

You need to have the atomic operation support to use omfile in
later rsyslog releases. Please check your gcc documentation and/or
your config.log to see if the check for atomic operations is failing.
You also need to use GCC versions 4.1 or higher. Earlier versions
are missing the support.

Regards,
Ken

On Wed, Dec 09, 2009 at 03:16:15PM +0100, Marc Schiffbauer wrote:
> 
> Sorry, but I am bit puzzled from all the branches. Which one should I take 
> now? Really the head master? I think I will get some 5.5.x version then, not?
> 
> What do you mean by "latest version" Head of a 5.3-Branch? Or checkout the 
> 5.3.4 tag from git?
> 
> This has the same error:
> http://git.adiscon.com/?p=rsyslog.git;a=snapshot;h=c051786ea6318f9a9a5e531c49512e4ae19f249b;sf=tgz
> 
> -Marc
> 
> 
> 
> Am Mittwoch, 9. Dezember 2009 14:52:15 schrieb Rainer Gerhards:
> > Marc,
> > 
> > I guess you use a 32-bit platform. Please pull the latest version from the
> > git master branch, it contains a fix for this type of build error.
> > 
> > Rainer
> > 
> > > -----Original Message-----
> > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer
> > > Sent: Wednesday, December 09, 2009 2:40 PM
> > > To: rsyslog-users
> > > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup?
> > >
> > > Hallo Rainer,
> > >
> > > I am back now. And as promised I wanted to test 5.3.4 now... I saw
> > > 5.3.5 is
> > > out so I wanted to give that version a try.
> > >
> > > But unfortunately it does not build. Any hint?
> > >
> > >
> > >
> > > In file included from omfile.c:67:
> > > ../runtime/atomic.h:64:3: warning: #warning "atomic builtins not
> > > available,
> > > using nul operations - rsyslogd will probably be racy!"
> > > omfile.c: In function ?getClockFileAccess?:
> > > omfile.c:91: warning: implicit declaration of function
> > > ?ATOMIC_INC_AND_FETCH?
> > >   CC     rsyslogd-omdiscard.o
> > >   CC     rsyslogd-pmrfc5424.o
> > >   CC     rsyslogd-pmrfc3164.o
> > >   CC     rsyslogd-iminternal.o
> > >   CC     rsyslogd-pidfile.o
> > >   CCLD   rsyslogd
> > > rsyslogd-omfile.o: In function `getClockFileAccess':
> > > /root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to
> > > `ATOMIC_INC_AND_FETCH'
> > > /root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to
> > > `ATOMIC_INC_AND_FETCH'
> > > /root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to
> > > `ATOMIC_INC_AND_FETCH'
> > > /root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to
> > > `ATOMIC_INC_AND_FETCH'
> > > ../runtime/.libs/librsyslog.a(librsyslog_la-glbl.o): In function
> > > `SetGlobalInputTermination':
> > > /root/src/rsyslog-5.3.5/runtime/glbl.c:136: undefined reference to
> > > `ATOMIC_STORE_1_TO_INT'
> > > ../runtime/.libs/librsyslog.a(librsyslog_la-msg.o): In function
> > > `msgDestruct':
> > > /root/src/rsyslog-5.3.5/runtime/msg.c:802: undefined reference to
> > > `ATOMIC_INC_AND_FETCH'
> > > ../runtime/.libs/librsyslog.a(librsyslog_la-wti.o): In function
> > > `wtiSetState':
> > > /root/src/rsyslog-5.3.5/runtime/wti.c:111: undefined reference to
> > > `ATOMIC_STORE_0_TO_INT'
> > > /root/src/rsyslog-5.3.5/runtime/wti.c:109: undefined reference to
> > > `ATOMIC_STORE_1_TO_INT'
> > > ../runtime/.libs/librsyslog.a(librsyslog_la-queue.o): In function
> > > `DoDeleteBatchFromQStore':
> > > /root/src/rsyslog-5.3.5/runtime/queue.c:1291: undefined reference to
> > > `ATOMIC_SUB'
> > > /root/src/rsyslog-5.3.5/runtime/queue.c:1292: undefined reference to
> > > `ATOMIC_SUB'
> > > /root/src/rsyslog-5.3.5/runtime/queue.c:1291: undefined reference to
> > > `ATOMIC_SUB'
> > > /root/src/rsyslog-5.3.5/runtime/queue.c:1292: undefined reference to
> > > `ATOMIC_SUB'
> > > /root/src/rsyslog-5.3.5/runtime/queue.c:1291: undefined reference to
> > > `ATOMIC_SUB'
> > > ../runtime/.libs/librsyslog.a(librsyslog_la-
> > > queue.o):/root/src/rsyslog-5.3.5/runtime/queue.c:1292: more undefined
> > > references to `ATOMIC_SUB' follow
> > > collect2: ld returned 1 exit status
> > > make[2]: *** [rsyslogd] Error 1
> > > make[2]: Leaving directory `/root/src/rsyslog-5.3.5/tools'
> > > make[1]: *** [all-recursive] Error 1
> > > make[1]: Leaving directory `/root/src/rsyslog-5.3.5'
> > > make: *** [all] Error 2
> > >
> > > Am Donnerstag, 5. November 2009 17:42:01 schrieb Rainer Gerhards:
> > > > > -----Original Message-----
> > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > > > > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer
> > > > > Sent: Tuesday, November 03, 2009 11:58 AM
> > > > > To: rsyslog-users
> > > > > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup?
> > > > >
> > > > > Hello Rainer,
> > > > >
> > > > > is there some news about this issue?
> > > >
> > > > Unfortuantely not at the moment (I guess you see it is a busy time).
> > >
> > > Could
> > >
> > > > you give 5.3.4 a try? It would be very interesting (even for a v4-
> > >
> > > fix) to
> > >
> > > >  see if the problem persists or not...
> > > >
> > > > Rainer
> > > >
> > > > > TIA
> > > > > -Marc
> > > > >
> > > > > Am Dienstag, 20. Oktober 2009 18:38:16 schrieb Rainer Gerhards:
> > > > > > thanks!
> > > > > >
> > > > > > Interesting, I see that only one of the messages that should be
> > >
> > > in
> > >
> > > > > the .qi
> > > > >
> > > > > > are actually present. I wonder if this is related to the fact
> > >
> > > that
> > >
> > > > > ompgsql
> > > > >
> > > > > > emits an error message itself while being down (the others don't
> > >
> > > do
> > >
> > > > > this
> > > > >
> > > > > > AFIK). Will try to dig down to this (but I have to do so as time
> > > > >
> > > > > permits).
> > > > >
> > > > > > Rainer
> > > > >
> > > > > _______________________________________________
> > > > > rsyslog mailing list
> > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > > http://www.rsyslog.com
> > > >
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com
> > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> > 
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> > 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 

From marc.schiffbauer at mightycare.de  Wed Dec  9 15:43:18 2009
From: marc.schiffbauer at mightycare.de (Marc Schiffbauer)
Date: Wed, 9 Dec 2009 15:43:18 +0100
Subject: [rsyslog] rsyslog 4.5.x queue file cleanup?
In-Reply-To: <20091209143544.GL3237@it.is.rice.edu>
References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com>
	<200912091516.15909.marc.schiffbauer@mightycare.de>
	<20091209143544.GL3237@it.is.rice.edu>
Message-ID: <200912091543.19168.marc.schiffbauer@mightycare.de>


Hi Ken,

the test was failing, yes. 
This is with gcc version 4.3.2 on a SLES11.

I have to leave now to get my train... will be back later.

-Marc


Am Mittwoch, 9. Dezember 2009 15:35:44 schrieb Kenneth Marshall:
> Hi Marc,
> 
> You need to have the atomic operation support to use omfile in
> later rsyslog releases. Please check your gcc documentation and/or
> your config.log to see if the check for atomic operations is failing.
> You also need to use GCC versions 4.1 or higher. Earlier versions
> are missing the support.
> 
> Regards,
> Ken
> 
> On Wed, Dec 09, 2009 at 03:16:15PM +0100, Marc Schiffbauer wrote:
> > Sorry, but I am bit puzzled from all the branches. Which one should I
> > take now? Really the head master? I think I will get some 5.5.x version
> > then, not?
> >
> > What do you mean by "latest version" Head of a 5.3-Branch? Or checkout
> > the 5.3.4 tag from git?
> >
> > This has the same error:
> > http://git.adiscon.com/?p=rsyslog.git;a=snapshot;h=c051786ea6318f9a9a5e53
> >1c49512e4ae19f249b;sf=tgz
> >
> > -Marc
> >
> > Am Mittwoch, 9. Dezember 2009 14:52:15 schrieb Rainer Gerhards:
> > > Marc,
> > >
> > > I guess you use a 32-bit platform. Please pull the latest version from
> > > the git master branch, it contains a fix for this type of build error.
> > >
> > > Rainer
> > >
> > > > -----Original Message-----
> > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > > > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer
> > > > Sent: Wednesday, December 09, 2009 2:40 PM
> > > > To: rsyslog-users
> > > > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup?
> > > >
> > > > Hallo Rainer,
> > > >
> > > > I am back now. And as promised I wanted to test 5.3.4 now... I saw
> > > > 5.3.5 is
> > > > out so I wanted to give that version a try.
> > > >
> > > > But unfortunately it does not build. Any hint?
> > > >
> > > >
> > > >
> > > > In file included from omfile.c:67:
> > > > ../runtime/atomic.h:64:3: warning: #warning "atomic builtins not
> > > > available,
> > > > using nul operations - rsyslogd will probably be racy!"
> > > > omfile.c: In function ?getClockFileAccess?:
> > > > omfile.c:91: warning: implicit declaration of function
> > > > ?ATOMIC_INC_AND_FETCH?
> > > >   CC     rsyslogd-omdiscard.o
> > > >   CC     rsyslogd-pmrfc5424.o
> > > >   CC     rsyslogd-pmrfc3164.o
> > > >   CC     rsyslogd-iminternal.o
> > > >   CC     rsyslogd-pidfile.o
> > > >   CCLD   rsyslogd
> > > > rsyslogd-omfile.o: In function `getClockFileAccess':
> > > > /root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to
> > > > `ATOMIC_INC_AND_FETCH'
> > > > /root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to
> > > > `ATOMIC_INC_AND_FETCH'
> > > > /root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to
> > > > `ATOMIC_INC_AND_FETCH'
> > > > /root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to
> > > > `ATOMIC_INC_AND_FETCH'
> > > > ../runtime/.libs/librsyslog.a(librsyslog_la-glbl.o): In function
> > > > `SetGlobalInputTermination':
> > > > /root/src/rsyslog-5.3.5/runtime/glbl.c:136: undefined reference to
> > > > `ATOMIC_STORE_1_TO_INT'
> > > > ../runtime/.libs/librsyslog.a(librsyslog_la-msg.o): In function
> > > > `msgDestruct':
> > > > /root/src/rsyslog-5.3.5/runtime/msg.c:802: undefined reference to
> > > > `ATOMIC_INC_AND_FETCH'
> > > > ../runtime/.libs/librsyslog.a(librsyslog_la-wti.o): In function
> > > > `wtiSetState':
> > > > /root/src/rsyslog-5.3.5/runtime/wti.c:111: undefined reference to
> > > > `ATOMIC_STORE_0_TO_INT'
> > > > /root/src/rsyslog-5.3.5/runtime/wti.c:109: undefined reference to
> > > > `ATOMIC_STORE_1_TO_INT'
> > > > ../runtime/.libs/librsyslog.a(librsyslog_la-queue.o): In function
> > > > `DoDeleteBatchFromQStore':
> > > > /root/src/rsyslog-5.3.5/runtime/queue.c:1291: undefined reference to
> > > > `ATOMIC_SUB'
> > > > /root/src/rsyslog-5.3.5/runtime/queue.c:1292: undefined reference to
> > > > `ATOMIC_SUB'
> > > > /root/src/rsyslog-5.3.5/runtime/queue.c:1291: undefined reference to
> > > > `ATOMIC_SUB'
> > > > /root/src/rsyslog-5.3.5/runtime/queue.c:1292: undefined reference to
> > > > `ATOMIC_SUB'
> > > > /root/src/rsyslog-5.3.5/runtime/queue.c:1291: undefined reference to
> > > > `ATOMIC_SUB'
> > > > ../runtime/.libs/librsyslog.a(librsyslog_la-
> > > > queue.o):/root/src/rsyslog-5.3.5/runtime/queue.c:1292: more undefined
> > > > references to `ATOMIC_SUB' follow
> > > > collect2: ld returned 1 exit status
> > > > make[2]: *** [rsyslogd] Error 1
> > > > make[2]: Leaving directory `/root/src/rsyslog-5.3.5/tools'
> > > > make[1]: *** [all-recursive] Error 1
> > > > make[1]: Leaving directory `/root/src/rsyslog-5.3.5'
> > > > make: *** [all] Error 2
> > > >
> > > > Am Donnerstag, 5. November 2009 17:42:01 schrieb Rainer Gerhards:
> > > > > > -----Original Message-----
> > > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > > > > > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer
> > > > > > Sent: Tuesday, November 03, 2009 11:58 AM
> > > > > > To: rsyslog-users
> > > > > > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup?
> > > > > >
> > > > > > Hello Rainer,
> > > > > >
> > > > > > is there some news about this issue?
> > > > >
> > > > > Unfortuantely not at the moment (I guess you see it is a busy
> > > > > time).
> > > >
> > > > Could
> > > >
> > > > > you give 5.3.4 a try? It would be very interesting (even for a v4-
> > > >
> > > > fix) to
> > > >
> > > > >  see if the problem persists or not...
> > > > >
> > > > > Rainer
> > > > >
> > > > > > TIA
> > > > > > -Marc
> > > > > >
> > > > > > Am Dienstag, 20. Oktober 2009 18:38:16 schrieb Rainer Gerhards:
> > > > > > > thanks!
> > > > > > >
> > > > > > > Interesting, I see that only one of the messages that should be
> > > >
> > > > in
> > > >
> > > > > > the .qi
> > > > > >
> > > > > > > are actually present. I wonder if this is related to the fact
> > > >
> > > > that
> > > >
> > > > > > ompgsql
> > > > > >
> > > > > > > emits an error message itself while being down (the others
> > > > > > > don't
> > > >
> > > > do
> > > >
> > > > > > this
> > > > > >
> > > > > > > AFIK). Will try to dig down to this (but I have to do so as
> > > > > > > time
> > > > > >
> > > > > > permits).
> > > > > >
> > > > > > > Rainer
> > > > > >
> > > > > > _______________________________________________
> > > > > > rsyslog mailing list
> > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > > > http://www.rsyslog.com
> > > > >
> > > > > _______________________________________________
> > > > > rsyslog mailing list
> > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > > http://www.rsyslog.com
> > > >
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com
> > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 

From ryan.b.lynch at gmail.com  Wed Dec  9 15:06:27 2009
From: ryan.b.lynch at gmail.com (Ryan Lynch)
Date: Wed, 9 Dec 2009 09:06:27 -0500
Subject: [rsyslog] Multiple-server failover questions (and round-robin
	DNSresolution, too)
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103560@GRFEXC.intern.adiscon.com>
References: <115906d10912081718u2fad4334i7426310ea01a95a@mail.gmail.com> 
	<9B6E2A8877C38245BFB15CC491A11DA7103560@GRFEXC.intern.adiscon.com>
Message-ID: <115906d10912090606t4cbd87eft9f6b142e3e985953@mail.gmail.com>

Rainer,

Thanks for the fast responses. I know a lot of the is speculative, and
therefore low-priority, so I really appreciate the attention to such a
niche item.

On Wed, Dec 9, 2009 at 04:53, Rainer Gerhards <rgerhards at hq.adiscon.com> wrote:
>> If so, does Rsyslog re-check a failed server every time it
>> sends a log message, or does it have some kind of polling timeout, or
>> is the check interval determined by something else entirely?
>
> It's an increasing interval (the longer it is down, the less frequently it is
> probed with some upper bound). I think (but not sure) you can configure the
> timers.

Is the timer configuration option documented somewhere? I don't
remember seeing it on the Wiki config sample page, but I might have
missed it, in the docs. (If you don't know off the top of your head,
don't worry about it, I'll go check the source code.)


>> When a
>> server fails, how long does it take Rsyslog to notice the failure and
>> start re-directing messages to the next host?
>
> As soon as the socket call returns an error, what may not be very soon.
>
>> When a server recovers,
>> does Rsyslog automatically notice the recovery, and if so, how long
>> does it take to start re-directing messages to the primary host?
>
> see above - it depends on how long it was down. The lower bound is, I think,
> 30 seconds. (Except that when a failure happens, one retry is done
> immediately).

By the way, does this apply to RELP connections, or just plain TCP syslog?


>> In case anybody's wondering, I've include some examples of what I'm
>> thinking of trying, here. Assume that there are three (3) central log
>> receiver hosts, with each with a unique DNS name ('log-alfa',
>> 'log-bravo', and 'log-charlie') that points to its own IP. Also,
>> assume that the round-robin DNS name 'log' points to all three hosts'
>> IP addresses, and that all of my log-generating hosts (the clients)
>> are configured to pick a random IP when calling gethostbyname() on a
>> round-robin name.
>>
>> ? ? *.* @@log
>> ? ? $ActionExecOnlyWhenPreviousIsSuspended on
>> ? ? & @@log
>> ? ? & @@log
>> ? ? & /var/log/localbuffer
>> ? ? $ActionExecOnlyWhenPreviousIsSuspended off
>>
>> But if Rsyslog resolves all three instances of the name 'log' with a
>> single call to gethostbyname(), this won't work.
>
> Not with a single call, but you never know who else queries DNS in the mean
> time. So I'd say that configuration is at least unreliable. I strongly
> recommend against it.

Your recommendation is well taken--round-robin DNS comes with a lot of
"gotchas". This is a pretty special case, though, since I can verify
that the resolver actually caches all of the round-robin records, not
just a random pick. So each individual host that calls calls
gethostbyname() gets an a random selection, in response.

But that cache behavior is not generally guaranteed, at all sites,
which might bite somebody else who wants to try this failover
strategy. Round-robin isn't a very well-implemented feature,
especially on older platforms, and in my experience, it's not terribly
reliable unless you have verified the operations of the client
resolver, the resolver cache server, and probably the authoritative
server, too. (Which is usually not the case, but I got lucky, here.)

In any event, I'm planning to monitor the client behavior pretty
closely, and to keep a pretty close eye on the distribution of clients
across the Rsyslog servers. If it won't stay in balance, I may have to
go back to the drawing board, anyway. I'll post about my experiences,
when I have some data.


>> It would nice to have a random variant of the
>> '$ActionExecOnlyWhenPreviousIsSuspended' functionality. As a
>> hypothetical example I just cooked up off the top of my head:
>>
>> ? ? *.* @@log-alfa
>> ? ? $ActionExecPickRandom on
>> ? ? $ActionExecPickRandomRetryWait 1
>> ? ? $ActionExecPickRandomRetryLimit -1
>> ? ? & @@log-bravo
>> ? ? & @@log-charlie
>> ? ? $ActionExecPickRandom off
>>
>> where Rsyslog randomly selects one of log-alfa, log-bravo, and
>> log-charlie, initially, and then makes another random pick at
>> failover, etc. I can think of all sorts of nifty options and
>> configurable knobs that would come in handy, here, too.
>>
>
> That's pretty interesting, but I think community demand is very low for this
> feature. So it is unlikely that I will find time to implement it in the
> foreseeable future (if others like that feature, too, please make yourself
> heard! It may influence priorities, what means that something else does not
> get done ;)).
>
> I'll provide more info on my schedule with a separate mail.

I'm curious about the level of demand, but I suspect you're probably
right. As far as I can figure, this kind of
"clustered-failover-with-load-distribution" would only be worth a lot
of effort when you have (A) a large enough group of high-volume
log-generating hosts to overwhelm a single log-receiver (hence the
need for load-distribution), and (B) a strict no-loss requirement for
the log data (hence the need for failover). Most sites probably don't
have both those requirements.

But, then again, most of the sysadmin objections I hear to purely
centralized logging are vague, uninformed variations on the theme of
"That's not how I learned it, and it makes me uncomfortable".
Historically, there were good reasons not to trust networked syslog,
even over TCP. But Rsyslog is systematically dismantling those
barriers, with RELP, queues, and failover. I can imagine that as
first-hand sysadmin knowledge of these techniques spreads, we will see
an inflection point in the adoption rate curve, and it will eventually
become a standard practice.

But that's enough speculation. Hopefully, some other Rsyslog users
will be able to weigh in.

-Ryan

From rgerhards at hq.adiscon.com  Wed Dec  9 16:37:24 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 9 Dec 2009 16:37:24 +0100
Subject: [rsyslog] rsyslog 4.5.x queue file cleanup?
References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com><200912091440.05811.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA710357F@GRFEXC.intern.adiscon.com>
	<200912091516.15909.marc.schiffbauer@mightycare.de>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103583@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer
> Sent: Wednesday, December 09, 2009 3:16 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup?
> 
> 
> Sorry, but I am bit puzzled from all the branches. Which one should I
> take
> now? Really the head master? I think I will get some 5.5.x version
> then, not?

Yes, but that was what I thought. I missed the fact that 5.3 experiences the
same problem. I'll see that I backport that patch (should have done that in
the first place). You then need to pull the "beta" branch. Tagged versions
are the actual releases and none of them contain the patch.

Rainer
> 
> What do you mean by "latest version" Head of a 5.3-Branch? Or checkout
> the
> 5.3.4 tag from git?
> 
> This has the same error:
> http://git.adiscon.com/?p=rsyslog.git;a=snapshot;h=c051786ea6318f9a9a5e
> 531c49512e4ae19f249b;sf=tgz
> 
> -Marc
> 
> 
> 
> Am Mittwoch, 9. Dezember 2009 14:52:15 schrieb Rainer Gerhards:
> > Marc,
> >
> > I guess you use a 32-bit platform. Please pull the latest version
> from the
> > git master branch, it contains a fix for this type of build error.
> >
> > Rainer
> >
> > > -----Original Message-----
> > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer
> > > Sent: Wednesday, December 09, 2009 2:40 PM
> > > To: rsyslog-users
> > > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup?
> > >
> > > Hallo Rainer,
> > >
> > > I am back now. And as promised I wanted to test 5.3.4 now... I saw
> > > 5.3.5 is
> > > out so I wanted to give that version a try.
> > >
> > > But unfortunately it does not build. Any hint?
> > >
> > >
> > >
> > > In file included from omfile.c:67:
> > > ../runtime/atomic.h:64:3: warning: #warning "atomic builtins not
> > > available,
> > > using nul operations - rsyslogd will probably be racy!"
> > > omfile.c: In function ?getClockFileAccess?:
> > > omfile.c:91: warning: implicit declaration of function
> > > ?ATOMIC_INC_AND_FETCH?
> > >   CC     rsyslogd-omdiscard.o
> > >   CC     rsyslogd-pmrfc5424.o
> > >   CC     rsyslogd-pmrfc3164.o
> > >   CC     rsyslogd-iminternal.o
> > >   CC     rsyslogd-pidfile.o
> > >   CCLD   rsyslogd
> > > rsyslogd-omfile.o: In function `getClockFileAccess':
> > > /root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to
> > > `ATOMIC_INC_AND_FETCH'
> > > /root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to
> > > `ATOMIC_INC_AND_FETCH'
> > > /root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to
> > > `ATOMIC_INC_AND_FETCH'
> > > /root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to
> > > `ATOMIC_INC_AND_FETCH'
> > > ../runtime/.libs/librsyslog.a(librsyslog_la-glbl.o): In function
> > > `SetGlobalInputTermination':
> > > /root/src/rsyslog-5.3.5/runtime/glbl.c:136: undefined reference to
> > > `ATOMIC_STORE_1_TO_INT'
> > > ../runtime/.libs/librsyslog.a(librsyslog_la-msg.o): In function
> > > `msgDestruct':
> > > /root/src/rsyslog-5.3.5/runtime/msg.c:802: undefined reference to
> > > `ATOMIC_INC_AND_FETCH'
> > > ../runtime/.libs/librsyslog.a(librsyslog_la-wti.o): In function
> > > `wtiSetState':
> > > /root/src/rsyslog-5.3.5/runtime/wti.c:111: undefined reference to
> > > `ATOMIC_STORE_0_TO_INT'
> > > /root/src/rsyslog-5.3.5/runtime/wti.c:109: undefined reference to
> > > `ATOMIC_STORE_1_TO_INT'
> > > ../runtime/.libs/librsyslog.a(librsyslog_la-queue.o): In function
> > > `DoDeleteBatchFromQStore':
> > > /root/src/rsyslog-5.3.5/runtime/queue.c:1291: undefined reference
> to
> > > `ATOMIC_SUB'
> > > /root/src/rsyslog-5.3.5/runtime/queue.c:1292: undefined reference
> to
> > > `ATOMIC_SUB'
> > > /root/src/rsyslog-5.3.5/runtime/queue.c:1291: undefined reference
> to
> > > `ATOMIC_SUB'
> > > /root/src/rsyslog-5.3.5/runtime/queue.c:1292: undefined reference
> to
> > > `ATOMIC_SUB'
> > > /root/src/rsyslog-5.3.5/runtime/queue.c:1291: undefined reference
> to
> > > `ATOMIC_SUB'
> > > ../runtime/.libs/librsyslog.a(librsyslog_la-
> > > queue.o):/root/src/rsyslog-5.3.5/runtime/queue.c:1292: more
> undefined
> > > references to `ATOMIC_SUB' follow
> > > collect2: ld returned 1 exit status
> > > make[2]: *** [rsyslogd] Error 1
> > > make[2]: Leaving directory `/root/src/rsyslog-5.3.5/tools'
> > > make[1]: *** [all-recursive] Error 1
> > > make[1]: Leaving directory `/root/src/rsyslog-5.3.5'
> > > make: *** [all] Error 2
> > >
> > > Am Donnerstag, 5. November 2009 17:42:01 schrieb Rainer Gerhards:
> > > > > -----Original Message-----
> > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > > > > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer
> > > > > Sent: Tuesday, November 03, 2009 11:58 AM
> > > > > To: rsyslog-users
> > > > > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup?
> > > > >
> > > > > Hello Rainer,
> > > > >
> > > > > is there some news about this issue?
> > > >
> > > > Unfortuantely not at the moment (I guess you see it is a busy
> time).
> > >
> > > Could
> > >
> > > > you give 5.3.4 a try? It would be very interesting (even for a
> v4-
> > >
> > > fix) to
> > >
> > > >  see if the problem persists or not...
> > > >
> > > > Rainer
> > > >
> > > > > TIA
> > > > > -Marc
> > > > >
> > > > > Am Dienstag, 20. Oktober 2009 18:38:16 schrieb Rainer Gerhards:
> > > > > > thanks!
> > > > > >
> > > > > > Interesting, I see that only one of the messages that should
> be
> > >
> > > in
> > >
> > > > > the .qi
> > > > >
> > > > > > are actually present. I wonder if this is related to the fact
> > >
> > > that
> > >
> > > > > ompgsql
> > > > >
> > > > > > emits an error message itself while being down (the others
> don't
> > >
> > > do
> > >
> > > > > this
> > > > >
> > > > > > AFIK). Will try to dig down to this (but I have to do so as
> time
> > > > >
> > > > > permits).
> > > > >
> > > > > > Rainer
> > > > >
> > > > > _______________________________________________
> > > > > rsyslog mailing list
> > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > > http://www.rsyslog.com
> > > >
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com
> > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Wed Dec  9 16:51:13 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 9 Dec 2009 16:51:13 +0100
Subject: [rsyslog] rsyslog 4.5.x queue file cleanup?
References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com><200912091440.05811.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA710357F@GRFEXC.intern.adiscon.com>
	<200912091516.15909.marc.schiffbauer@mightycare.de>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103585@GRFEXC.intern.adiscon.com>

I have now backported the patch to the beta branch.

BTW, branch namings: master - recent devel, beta current beta
v<x>-beta - older version beta (when version still receives feature updates)
v<x>-stable - older stable version

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer
> Sent: Wednesday, December 09, 2009 3:16 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup?
> 
> 
> Sorry, but I am bit puzzled from all the branches. Which one should I
> take
> now? Really the head master? I think I will get some 5.5.x version
> then, not?
> 
> What do you mean by "latest version" Head of a 5.3-Branch? Or checkout
> the
> 5.3.4 tag from git?
> 
> This has the same error:
> http://git.adiscon.com/?p=rsyslog.git;a=snapshot;h=c051786ea6318f9a9a5e
> 531c49512e4ae19f249b;sf=tgz
> 
> -Marc
> 
> 
> 
> Am Mittwoch, 9. Dezember 2009 14:52:15 schrieb Rainer Gerhards:
> > Marc,
> >
> > I guess you use a 32-bit platform. Please pull the latest version
> from the
> > git master branch, it contains a fix for this type of build error.
> >
> > Rainer
> >
> > > -----Original Message-----
> > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer
> > > Sent: Wednesday, December 09, 2009 2:40 PM
> > > To: rsyslog-users
> > > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup?
> > >
> > > Hallo Rainer,
> > >
> > > I am back now. And as promised I wanted to test 5.3.4 now... I saw
> > > 5.3.5 is
> > > out so I wanted to give that version a try.
> > >
> > > But unfortunately it does not build. Any hint?
> > >
> > >
> > >
> > > In file included from omfile.c:67:
> > > ../runtime/atomic.h:64:3: warning: #warning "atomic builtins not
> > > available,
> > > using nul operations - rsyslogd will probably be racy!"
> > > omfile.c: In function ?getClockFileAccess?:
> > > omfile.c:91: warning: implicit declaration of function
> > > ?ATOMIC_INC_AND_FETCH?
> > >   CC     rsyslogd-omdiscard.o
> > >   CC     rsyslogd-pmrfc5424.o
> > >   CC     rsyslogd-pmrfc3164.o
> > >   CC     rsyslogd-iminternal.o
> > >   CC     rsyslogd-pidfile.o
> > >   CCLD   rsyslogd
> > > rsyslogd-omfile.o: In function `getClockFileAccess':
> > > /root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to
> > > `ATOMIC_INC_AND_FETCH'
> > > /root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to
> > > `ATOMIC_INC_AND_FETCH'
> > > /root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to
> > > `ATOMIC_INC_AND_FETCH'
> > > /root/src/rsyslog-5.3.5/tools/omfile.c:91: undefined reference to
> > > `ATOMIC_INC_AND_FETCH'
> > > ../runtime/.libs/librsyslog.a(librsyslog_la-glbl.o): In function
> > > `SetGlobalInputTermination':
> > > /root/src/rsyslog-5.3.5/runtime/glbl.c:136: undefined reference to
> > > `ATOMIC_STORE_1_TO_INT'
> > > ../runtime/.libs/librsyslog.a(librsyslog_la-msg.o): In function
> > > `msgDestruct':
> > > /root/src/rsyslog-5.3.5/runtime/msg.c:802: undefined reference to
> > > `ATOMIC_INC_AND_FETCH'
> > > ../runtime/.libs/librsyslog.a(librsyslog_la-wti.o): In function
> > > `wtiSetState':
> > > /root/src/rsyslog-5.3.5/runtime/wti.c:111: undefined reference to
> > > `ATOMIC_STORE_0_TO_INT'
> > > /root/src/rsyslog-5.3.5/runtime/wti.c:109: undefined reference to
> > > `ATOMIC_STORE_1_TO_INT'
> > > ../runtime/.libs/librsyslog.a(librsyslog_la-queue.o): In function
> > > `DoDeleteBatchFromQStore':
> > > /root/src/rsyslog-5.3.5/runtime/queue.c:1291: undefined reference
> to
> > > `ATOMIC_SUB'
> > > /root/src/rsyslog-5.3.5/runtime/queue.c:1292: undefined reference
> to
> > > `ATOMIC_SUB'
> > > /root/src/rsyslog-5.3.5/runtime/queue.c:1291: undefined reference
> to
> > > `ATOMIC_SUB'
> > > /root/src/rsyslog-5.3.5/runtime/queue.c:1292: undefined reference
> to
> > > `ATOMIC_SUB'
> > > /root/src/rsyslog-5.3.5/runtime/queue.c:1291: undefined reference
> to
> > > `ATOMIC_SUB'
> > > ../runtime/.libs/librsyslog.a(librsyslog_la-
> > > queue.o):/root/src/rsyslog-5.3.5/runtime/queue.c:1292: more
> undefined
> > > references to `ATOMIC_SUB' follow
> > > collect2: ld returned 1 exit status
> > > make[2]: *** [rsyslogd] Error 1
> > > make[2]: Leaving directory `/root/src/rsyslog-5.3.5/tools'
> > > make[1]: *** [all-recursive] Error 1
> > > make[1]: Leaving directory `/root/src/rsyslog-5.3.5'
> > > make: *** [all] Error 2
> > >
> > > Am Donnerstag, 5. November 2009 17:42:01 schrieb Rainer Gerhards:
> > > > > -----Original Message-----
> > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > > > > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer
> > > > > Sent: Tuesday, November 03, 2009 11:58 AM
> > > > > To: rsyslog-users
> > > > > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup?
> > > > >
> > > > > Hello Rainer,
> > > > >
> > > > > is there some news about this issue?
> > > >
> > > > Unfortuantely not at the moment (I guess you see it is a busy
> time).
> > >
> > > Could
> > >
> > > > you give 5.3.4 a try? It would be very interesting (even for a
> v4-
> > >
> > > fix) to
> > >
> > > >  see if the problem persists or not...
> > > >
> > > > Rainer
> > > >
> > > > > TIA
> > > > > -Marc
> > > > >
> > > > > Am Dienstag, 20. Oktober 2009 18:38:16 schrieb Rainer Gerhards:
> > > > > > thanks!
> > > > > >
> > > > > > Interesting, I see that only one of the messages that should
> be
> > >
> > > in
> > >
> > > > > the .qi
> > > > >
> > > > > > are actually present. I wonder if this is related to the fact
> > >
> > > that
> > >
> > > > > ompgsql
> > > > >
> > > > > > emits an error message itself while being down (the others
> don't
> > >
> > > do
> > >
> > > > > this
> > > > >
> > > > > > AFIK). Will try to dig down to this (but I have to do so as
> time
> > > > >
> > > > > permits).
> > > > >
> > > > > > Rainer
> > > > >
> > > > > _______________________________________________
> > > > > rsyslog mailing list
> > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > > http://www.rsyslog.com
> > > >
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com
> > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From david at lang.hm  Wed Dec  9 18:15:02 2009
From: david at lang.hm (david at lang.hm)
Date: Wed, 9 Dec 2009 09:15:02 -0800 (PST)
Subject: [rsyslog] using arbitrary facility id
In-Reply-To: <87a1ae540911300858v1d4cda71k787a53158e162b7d@mail.gmail.com>
References: <87a1ae540911300820s3ce60cafq5c258b355ed3e887@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA71034E7@GRFEXC.intern.adiscon.com>
	<87a1ae540911300851r1f0bab68ldf7f995ed0a094d8@mail.gmail.com>
	<87a1ae540911300858v1d4cda71k787a53158e162b7d@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.0912090912490.10307@asgard.lang.hm>

On Mon, 30 Nov 2009, Sayan Chowdhury wrote:

> BTW, just out of interest, why is this number restricted to 23 in the rfc?
> also each facility other than local0-7 seems to be rigidly defined. Just
> wanted to know why is it so ..shouldn't they have  left scope for extension
> on this?

they were packing facility and severity into one 8-bit value. why they 
picked 24 instead of 32 I don't know, but that was a decision made a few 
decades ago.

David Lang

>
> On Mon, Nov 30, 2009 at 11:51 AM, Sayan Chowdhury <iamsayan at gmail.com>wrote:
>
>> Hello Rainer,
>> Thanks... Yes I agree what I am doing is invalid.
>>  I was just trying it out, my idea was if I can use numbers greater than
>> 23, then I can maybe use it as my own customized logging system, and I would
>> use numbers greater than 23 for my application and use different ids (>23)
>> for different kind of application level log messages. I would try to look
>> into the code as well, is this something you thing may be useful in general?
>> Regards,
>> Sayan
>>
>>
>>
>> On Mon, Nov 30, 2009 at 11:40 AM, Rainer Gerhards <
>> rgerhards at hq.adiscon.com> wrote:
>>
>>> You can use facilities other than local0..local7, but you cannot use a
>>> facility with a numerical value greater than 23, because the relevant
>>> standards do not permit this (see RFC5424, Table 1).
>>>
>>> It may be that rsyslog does not properly prevent this, maybe it uses
>>> modulo
>>> 24 in this case. Will check that. But it is invalid in any case (not only
>>> with rsyslog but with any syslogd).
>>>
>>> Rainer
>>>
>>>> -----Original Message-----
>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>> bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury
>>>> Sent: Monday, November 30, 2009 5:20 PM
>>>> To: rsyslog-users
>>>> Subject: [rsyslog] using arbitrary facility id
>>>>
>>>> Hello All,
>>>> Is it possible to use a facility id other than local0-local7?
>>>> I was using a facility id of 50 in some of the messages , and I had
>>>> written
>>>> a selector line in my rsyslog file as well to log messages with
>>>> facility id
>>>> of 50 into a seperate file.
>>>> However, I see that sometimes the messages are being written into all
>>>> the
>>>> files(/var/log/messages, boot.log,/var/log/secure etc) instead of the
>>>> one I
>>>> specified in the rsyslog.conf. If I restart rsyslog the problem goes
>>>> away.
>>>> I am using rsyslog version 4.2.0.
>>>>
>>>> Here is the selector line in my config
>>>>
>>>>
>>>> if $fromhost-ip == '127.0.0.1' and $syslogfacility == '50' and
>>>> $syslogseverity <= '6' then $log_rotation_50
>>>>
>>>> where $log_rotation_50 is an outchannel which is configured to rotate
>>>> the
>>>> file when it reaches the size of 2MB.
>>>> Regards,
>>>> Sayan
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>>
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From rgerhards at hq.adiscon.com  Wed Dec  9 18:18:59 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 9 Dec 2009 18:18:59 +0100
Subject: [rsyslog] using arbitrary facility id
References: <87a1ae540911300820s3ce60cafq5c258b355ed3e887@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA71034E7@GRFEXC.intern.adiscon.com><87a1ae540911300851r1f0bab68ldf7f995ed0a094d8@mail.gmail.com><87a1ae540911300858v1d4cda71k787a53158e162b7d@mail.gmail.com>
	<alpine.DEB.2.00.0912090912490.10307@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103586@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Wednesday, December 09, 2009 6:15 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] using arbitrary facility id
> 
> On Mon, 30 Nov 2009, Sayan Chowdhury wrote:
> 
> > BTW, just out of interest, why is this number restricted to 23 in the
> rfc?
> > also each facility other than local0-7 seems to be rigidly defined.
> Just
> > wanted to know why is it so ..shouldn't they have  left scope for
> extension
> > on this?
> 
> they were packing facility and severity into one 8-bit value. why they
> picked 24 instead of 32 I don't know, but that was a decision made a
> few
> decades ago.

You need to keep on your mind that syslog was once a debug system for
sendmail. I guess Eric simply meant that 24 different values were far enough
(and I tend to agree thinking about that use case ;)).

Rainer

> 
> David Lang
> 
> >
> > On Mon, Nov 30, 2009 at 11:51 AM, Sayan Chowdhury
> <iamsayan at gmail.com>wrote:
> >
> >> Hello Rainer,
> >> Thanks... Yes I agree what I am doing is invalid.
> >>  I was just trying it out, my idea was if I can use numbers greater
> than
> >> 23, then I can maybe use it as my own customized logging system, and
> I would
> >> use numbers greater than 23 for my application and use different ids
> (>23)
> >> for different kind of application level log messages. I would try to
> look
> >> into the code as well, is this something you thing may be useful in
> general?
> >> Regards,
> >> Sayan
> >>
> >>
> >>
> >> On Mon, Nov 30, 2009 at 11:40 AM, Rainer Gerhards <
> >> rgerhards at hq.adiscon.com> wrote:
> >>
> >>> You can use facilities other than local0..local7, but you cannot
> use a
> >>> facility with a numerical value greater than 23, because the
> relevant
> >>> standards do not permit this (see RFC5424, Table 1).
> >>>
> >>> It may be that rsyslog does not properly prevent this, maybe it
> uses
> >>> modulo
> >>> 24 in this case. Will check that. But it is invalid in any case
> (not only
> >>> with rsyslog but with any syslogd).
> >>>
> >>> Rainer
> >>>
> >>>> -----Original Message-----
> >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >>>> bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury
> >>>> Sent: Monday, November 30, 2009 5:20 PM
> >>>> To: rsyslog-users
> >>>> Subject: [rsyslog] using arbitrary facility id
> >>>>
> >>>> Hello All,
> >>>> Is it possible to use a facility id other than local0-local7?
> >>>> I was using a facility id of 50 in some of the messages , and I
> had
> >>>> written
> >>>> a selector line in my rsyslog file as well to log messages with
> >>>> facility id
> >>>> of 50 into a seperate file.
> >>>> However, I see that sometimes the messages are being written into
> all
> >>>> the
> >>>> files(/var/log/messages, boot.log,/var/log/secure etc) instead of
> the
> >>>> one I
> >>>> specified in the rsyslog.conf. If I restart rsyslog the problem
> goes
> >>>> away.
> >>>> I am using rsyslog version 4.2.0.
> >>>>
> >>>> Here is the selector line in my config
> >>>>
> >>>>
> >>>> if $fromhost-ip == '127.0.0.1' and $syslogfacility == '50' and
> >>>> $syslogseverity <= '6' then $log_rotation_50
> >>>>
> >>>> where $log_rotation_50 is an outchannel which is configured to
> rotate
> >>>> the
> >>>> file when it reaches the size of 2MB.
> >>>> Regards,
> >>>> Sayan
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.com
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >>>
> >>
> >>
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From david at lang.hm  Wed Dec  9 18:24:42 2009
From: david at lang.hm (david at lang.hm)
Date: Wed, 9 Dec 2009 09:24:42 -0800 (PST)
Subject: [rsyslog] Feature suggestion: Query rsyslog daemon for a list
 of	currently-open log files.
In-Reply-To: <115906d10912081428t263114afv3dcc5fc8228ecfe5@mail.gmail.com>
References: <115906d10912081428t263114afv3dcc5fc8228ecfe5@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.0912090923520.10307@asgard.lang.hm>

On Tue, 8 Dec 2009, Ryan Lynch wrote:

> Obviously this isn't a low priority suggestion, but I think it could
> be a pretty useful in managing large/complex Rsyslog configurations.
>
> On some of my Linux hosts, I have Rsyslog configured such that it can
> easily hold 20-30 open files at any given time. (I'm using the utility
> 'lsof', BTW, to list open files, like so: `lsof | gawk '$1 ~
> "^rsyslogd$" && $9 ~ "^/var/log/"'`) Since I'm using dynamic templates
> with date/time elements, the exact list of currently-open files
> changes from minute to minute.
>
> It would be helpful for the Rsyslog daemon to periodically output a
> list of all the log files (by full path) that it currently holds open.
> Primarily, I'd use this for log file rotation and similar maintenance
> tasks, but I could imagine others would find other uses, especially if
> the output mechanism were generic enough to dump other runtime
> statistics, too. If Rsyslog kept a count of the number of bytes
> written to each open file, and included those per-byte counts in the
> output, another program could monitor log-output rates, to watch out
> for dangerously high disk I/O loads.
>
> I don't really have a fixed idea for the communication mechanism. It
> could be as simple as having Rsyslog dump the stats to a text file,
> and add a configuration knob to tell it how many seconds to wait
> between dumps. The output format wouldn't need to be simple, maybe
> just a newline-delimited list of file names/stats.
>
> If I get some free time, and nobody objects, I may try to implement
> this. But I'm pretty rusty with C, so I figured I should ask whether
> the idea floats, first.

you can find this information via lsof (or via /proc)

this doesn't mean that querying rsyslog isn't better in the long run, just 
ways to deal with the problem in the mantime.

David Lang

From david at lang.hm  Wed Dec  9 18:30:53 2009
From: david at lang.hm (david at lang.hm)
Date: Wed, 9 Dec 2009 09:30:53 -0800 (PST)
Subject: [rsyslog] Multiple-server failover questions (and round-robin
 DNS	resolution, too)
In-Reply-To: <115906d10912081718u2fad4334i7426310ea01a95a@mail.gmail.com>
References: <115906d10912081718u2fad4334i7426310ea01a95a@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.0912090925090.10307@asgard.lang.hm>

On Tue, 8 Dec 2009, Ryan Lynch wrote:

> In the doc example, the server "primary-syslog.example.com" is always
> tried first, unless it's down, followed by
> "secondary-1-syslog.example.com", and then finally
> "secondary-2-syslog.example.com" only if both of the others are down.
> Here's the doc in question, BTW:
>
> * http://wiki.rsyslog.com/index.php/FailoverSyslogServer
>
> 1.  Is it possible to configure Rsyslog to pick randomly from a list
> of remote syslog destinations instead of following the listed order?
> If I have a large population of log generating hosts, I'd like to be
> able to distribute the total load amongst two or more "central" log
> receivers, with failover if a receiver dies. Assuming that we have a
> sound random-selection implementation, and a large enough population
> of hosts generating approximately equal amounts of log data, I should
> get an approximately even distribution across my receiver hosts. In
> the event that a receiver dies, its clients would randomly try another
> receiver, keeping the load close to even. Is this possible, with the
> existing failover mechanism?

this sort of thing is possible today, without using round-robin DNS

on linux there is an iptables option called CLUSTERIP that lets you create 
a single IP address on multiple machines, and each machine processes a 
portion of the traffic (based on a hash of one or more of dest port, dest 
IP, source port, source IP)

heartbeat (http://linux-ha.org) will manage a cluster of machines, 
including ones sharing an IP address like this.

if you have many systems sending to one central server they will be spread 
roughly evenly across the servers. if this is not even enough for you, 
version 5 has an option to close and re-open connections to the server 
every X messages, and since doing so changes the source port of the 
connection, this will spread the connections around the cluster more 
evenly.


David Lang

From ryan.b.lynch at gmail.com  Wed Dec  9 18:41:23 2009
From: ryan.b.lynch at gmail.com (Ryan Lynch)
Date: Wed, 9 Dec 2009 12:41:23 -0500
Subject: [rsyslog] Feature suggestion: Query rsyslog daemon for a list
	of currently-open log files.
In-Reply-To: <alpine.DEB.2.00.0912090923520.10307@asgard.lang.hm>
References: <115906d10912081428t263114afv3dcc5fc8228ecfe5@mail.gmail.com> 
	<alpine.DEB.2.00.0912090923520.10307@asgard.lang.hm>
Message-ID: <115906d10912090941o7cb2867aq9d674c80b0bc3f71@mail.gmail.com>

On Wed, Dec 9, 2009 at 12:24,  <david at lang.hm> wrote:
> On Tue, 8 Dec 2009, Ryan Lynch wrote:
>
>> Obviously this isn't a low priority suggestion, but I think it could
>> be a pretty useful in managing large/complex Rsyslog configurations.
>>
>> On some of my Linux hosts, I have Rsyslog configured such that it can
>> easily hold 20-30 open files at any given time. (I'm using the utility
>> 'lsof', BTW, to list open files, like so: `lsof | gawk '$1 ~
>> "^rsyslogd$" && $9 ~ "^/var/log/"'`) Since I'm using dynamic templates
>> with date/time elements, the exact list of currently-open files
>> changes from minute to minute.
>
> you can find this information via lsof (or via /proc)
>
> this doesn't mean that querying rsyslog isn't better in the long run, just
> ways to deal with the problem in the mantime.

In theory, sure. In practice, well...

A single 'lsof' command is a pretty expensive operation--try running
`watch -n 1 'lsof >/dev/null'` for a minute, and then check your load
average. It has to do an enormous amount of work every time it runs.
Also, it will only provide a limited amount of information to an
unprivileged user. It'll tell you certain into, but it won't give you
the paths of the files that 'rsyslog' has open.

So 'lsof' can work for diagnostic purposes, but it isn't really
designed for this sort of thing.

-Ryan

From rgerhards at hq.adiscon.com  Wed Dec  9 18:58:22 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 9 Dec 2009 18:58:22 +0100
Subject: [rsyslog] rsyslog "feature schedule"
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103587@GRFEXC.intern.adiscon.com>

Hi all,

I've finally managed to write up my "feature schedule" (read it to understand
the quotes if you do not already know why they are there ;)) as requested:

http://blog.gerhards.net/2009/12/rsyslog-feature-schedule.html

Rainer

From ryan.b.lynch at gmail.com  Wed Dec  9 19:08:27 2009
From: ryan.b.lynch at gmail.com (Ryan Lynch)
Date: Wed, 9 Dec 2009 13:08:27 -0500
Subject: [rsyslog] Multiple-server failover questions (and round-robin
	DNS resolution, too)
In-Reply-To: <alpine.DEB.2.00.0912090925090.10307@asgard.lang.hm>
References: <115906d10912081718u2fad4334i7426310ea01a95a@mail.gmail.com> 
	<alpine.DEB.2.00.0912090925090.10307@asgard.lang.hm>
Message-ID: <115906d10912091008q43930462t4f3bf7b313e6a088@mail.gmail.com>

Hi, David,

On Wed, Dec 9, 2009 at 12:30,  <david at lang.hm> wrote:
> this sort of thing is possible today, without using round-robin DNS
>
> on linux there is an iptables option called CLUSTERIP that lets you create
> a single IP address on multiple machines, and each machine processes a
> portion of the traffic (based on a hash of one or more of dest port, dest
> IP, source port, source IP)

ClusterIP is a heck of a neat technology, and it would definitely work
for a lot of load-balancing cases.

But it does require that all of the participating hosts share a single
LAN segment. In my case, the "central" log-receiving hosts are are
different physical sites, and are separated by routers.

I haven't thought this out, completely, but depending on the load
characteristics and proximity-distribution requirements, it might be
possible to use a combination of both: A per-site ClusterIP address
for the primary host, with various clever round-robin DNS records for
failover.

Either way, thanks for the suggestion.


> if you have many systems sending to one central server they will be spread
> roughly evenly across the servers. if this is not even enough for you,
> version 5 has an option to close and re-open connections to the server
> every X messages, and since doing so changes the source port of the
> connection, this will spread the connections around the cluster more
> evenly.

Do you happen to know what the name of that option is? That will come
in handy no matter how I end up implementing this.

-Ryan

From david at lang.hm  Wed Dec  9 19:35:14 2009
From: david at lang.hm (david at lang.hm)
Date: Wed, 9 Dec 2009 10:35:14 -0800 (PST)
Subject: [rsyslog] Multiple-server failover questions (and round-robin
 DNS resolution, too)
In-Reply-To: <115906d10912091008q43930462t4f3bf7b313e6a088@mail.gmail.com>
References: <115906d10912081718u2fad4334i7426310ea01a95a@mail.gmail.com>
	<alpine.DEB.2.00.0912090925090.10307@asgard.lang.hm>
	<115906d10912091008q43930462t4f3bf7b313e6a088@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.0912091033060.10307@asgard.lang.hm>

On Wed, 9 Dec 2009, Ryan Lynch wrote:

> Hi, David,
>
> On Wed, Dec 9, 2009 at 12:30,  <david at lang.hm> wrote:
>> this sort of thing is possible today, without using round-robin DNS
>>
>> on linux there is an iptables option called CLUSTERIP that lets you create
>> a single IP address on multiple machines, and each machine processes a
>> portion of the traffic (based on a hash of one or more of dest port, dest
>> IP, source port, source IP)
>
> ClusterIP is a heck of a neat technology, and it would definitely work
> for a lot of load-balancing cases.
>
> But it does require that all of the participating hosts share a single
> LAN segment. In my case, the "central" log-receiving hosts are are
> different physical sites, and are separated by routers.
>
> I haven't thought this out, completely, but depending on the load
> characteristics and proximity-distribution requirements, it might be
> possible to use a combination of both: A per-site ClusterIP address
> for the primary host, with various clever round-robin DNS records for
> failover.

remember that even with TCP data can be lost in transit, so doing logging 
to remote sites is risky without using RELP.

> Either way, thanks for the suggestion.
>
>
>> if you have many systems sending to one central server they will be spread
>> roughly evenly across the servers. if this is not even enough for you,
>> version 5 has an option to close and re-open connections to the server
>> every X messages, and since doing so changes the source port of the
>> connection, this will spread the connections around the cluster more
>> evenly.
>
> Do you happen to know what the name of that option is? That will come
> in handy no matter how I end up implementing this.

there are two seperate options, one for TCP, one for UDP (I don't know if 
the TCP option also affects other transports that use TCP like RELP, TLS, 
etc)

# $ActionSendTCPRebindInterval nbr- [available since 4.5.1] - instructs 
the TCP send action to close and re-open the connection to the remote host 
every nbr of messages sent. Zero, the default, means that no such 
processing is done. This directive is useful for use with load-balancers. 
Note that there is some performance overhead associated with it, so it is 
advisable to not too often "rebind" the connection (what "too often" 
actually means depends on your configuration, a rule of thumb is that it 
should be not be much more often than once per second).

# $ActionSendUDPRebindInterval nbr- [available since 4.3.2] - instructs 
the UDP send action to rebind the send socket every nbr of messages sent. 
Zero, the default, means that no rebind is done. This directive is useful 
for use with load-balancers.

David Lang

From david at lang.hm  Wed Dec  9 19:36:19 2009
From: david at lang.hm (david at lang.hm)
Date: Wed, 9 Dec 2009 10:36:19 -0800 (PST)
Subject: [rsyslog] Multiple-server failover questions (and round-robin
 DNS resolution, too)
In-Reply-To: <115906d10912091008q43930462t4f3bf7b313e6a088@mail.gmail.com>
References: <115906d10912081718u2fad4334i7426310ea01a95a@mail.gmail.com>
	<alpine.DEB.2.00.0912090925090.10307@asgard.lang.hm>
	<115906d10912091008q43930462t4f3bf7b313e6a088@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.0912091035270.10307@asgard.lang.hm>

On Wed, 9 Dec 2009, Ryan Lynch wrote:

> On Wed, Dec 9, 2009 at 12:30,  <david at lang.hm> wrote:
>> this sort of thing is possible today, without using round-robin DNS
>>
>> on linux there is an iptables option called CLUSTERIP that lets you create
>> a single IP address on multiple machines, and each machine processes a
>> portion of the traffic (based on a hash of one or more of dest port, dest
>> IP, source port, source IP)
>
> ClusterIP is a heck of a neat technology, and it would definitely work
> for a lot of load-balancing cases.
>
> But it does require that all of the participating hosts share a single
> LAN segment. In my case, the "central" log-receiving hosts are are
> different physical sites, and are separated by routers.

if all your central hosts are in one place you can use ClusterIP even from 
remote sites. it's just that the machines in a cluster need to be on the 
same lan segment.

David Lang

> I haven't thought this out, completely, but depending on the load
> characteristics and proximity-distribution requirements, it might be
> possible to use a combination of both: A per-site ClusterIP address
> for the primary host, with various clever round-robin DNS records for
> failover.
>
> Either way, thanks for the suggestion.
>
>
>> if you have many systems sending to one central server they will be spread
>> roughly evenly across the servers. if this is not even enough for you,
>> version 5 has an option to close and re-open connections to the server
>> every X messages, and since doing so changes the source port of the
>> connection, this will spread the connections around the cluster more
>> evenly.
>
> Do you happen to know what the name of that option is? That will come
> in handy no matter how I end up implementing this.
>
> -Ryan
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From aoz.syn at gmail.com  Wed Dec  9 19:37:02 2009
From: aoz.syn at gmail.com (RB)
Date: Wed, 9 Dec 2009 11:37:02 -0700
Subject: [rsyslog] Multiple-server failover questions (and round-robin
	DNS resolution, too)
In-Reply-To: <115906d10912091008q43930462t4f3bf7b313e6a088@mail.gmail.com>
References: <115906d10912081718u2fad4334i7426310ea01a95a@mail.gmail.com>
	<alpine.DEB.2.00.0912090925090.10307@asgard.lang.hm>
	<115906d10912091008q43930462t4f3bf7b313e6a088@mail.gmail.com>
Message-ID: <4255c2570912091037v7a5f5c3ci3d6301c524177aeb@mail.gmail.com>

On Wed, Dec 9, 2009 at 11:08, Ryan Lynch <ryan.b.lynch at gmail.com> wrote:
> ClusterIP is a heck of a neat technology, and it would definitely work
> for a lot of load-balancing cases.
>
> But it does require that all of the participating hosts share a single
> LAN segment. In my case, the "central" log-receiving hosts are are
> different physical sites, and are separated by routers.

That's definitely the case for CLUSTERIP, but have you looked at LVS
and the associated ipvsadm tool?  It's been in the Linux kernel for a
while, and provides the L3-L4 load balancing you seem to be looking
for.  To make the balancer itself dispersed you'd need to use other
clustering tools to complete the picture, but it's a decent tool
nonetheless.  If you're not married to Linux, I can highly recommend
CARP + slbd (or just use pfSense) as a nicely powerful balancer.

From ryan.b.lynch at gmail.com  Wed Dec  9 23:05:18 2009
From: ryan.b.lynch at gmail.com (Ryan Lynch)
Date: Wed, 9 Dec 2009 17:05:18 -0500
Subject: [rsyslog] Multiple-server failover questions (and round-robin
	DNS resolution, too)
In-Reply-To: <4255c2570912091037v7a5f5c3ci3d6301c524177aeb@mail.gmail.com>
References: <115906d10912081718u2fad4334i7426310ea01a95a@mail.gmail.com> 
	<alpine.DEB.2.00.0912090925090.10307@asgard.lang.hm>
	<115906d10912091008q43930462t4f3bf7b313e6a088@mail.gmail.com> 
	<4255c2570912091037v7a5f5c3ci3d6301c524177aeb@mail.gmail.com>
Message-ID: <115906d10912091405m63edeef6hf643a7a219cd86c4@mail.gmail.com>

David and RB:

I've learned something about DNS round-robin, in the past 24 hours:
Apparently, everybody hates it!

Just kidding--I already knew that. It's very hate-able. Forgive me if
it makes me chuckle that after I mentioned it, and three separate
people in a row responded with some variation on "Ewww... DNS round
robin is gross."

In my initial response to David, I was a little ambigious as to what
"central" log servers means: The hosts RECEIVING syslog messages are
all on different subnets, at different sites. I quoted the word
"central" because the logs are centralized, not the servers
themselves. Ideally, the servers are dispersed far and wide, for
better survivability. So ClusterIP might still play a role, but it
wouldn't be a drop-in replacement for a round-robin. I'm not sure,
yet, how much of the initial, shared-nothing design I want to
compromise--it's going to be a judgement call.

RB suggested LVS as another possible alternative. I actually have some
firsthand experience with LVS, having built some WWW server farms on
it, and I love it--truly a great piece of software. But in this
situation, I wonder whether LVS really adds anything. I'd have to
dedicate at least 2 extra machines to load-balancing duties, and the
added complexity of the ipvsadm, CARP, etc. configs. The
dedicated/clustered LBs would have to reside on a single subnet,
though we could direct traffic to syslog receivers anywhere.

The problem is, I feel like those dedicated LBs could create new
problems: The LB mechanism is now both a potential bottleneck and a
potential point of failure, neither of which existed before. Also,
we'd have to route all of our traffic through the designated LBs,
which would increase latency and load the network more than is
strictly necessary. (I'm not sure if we really care about either of
those issues, though--the effects probably won't be very big.) On the
other hand, DNS round-robin has zero ability to control the traffic
distribution, or to actively balance the load. I'd couldn't
efficiently use differently-sized receiving servers. Plus, we really
have to count the DNS as a potential point of failure, too.

Hmmm... Decisions, decisions.

-Ryan

From rory at ooma.com  Wed Dec  9 23:08:51 2009
From: rory at ooma.com (Rory Toma)
Date: Wed, 09 Dec 2009 14:08:51 -0800
Subject: [rsyslog] Help with client config
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103562@GRFEXC.intern.adiscon.com>
References: <001301ca783f$691b9aa5$100013ac@intern.adiscon.com><4B1EC6EA.6090508@ooma.com>	<4B1ECEBB.4030905@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7103562@GRFEXC.intern.adiscon.com>
Message-ID: <4B201FF3.8040505@ooma.com>

OK, thx, I'll check into gnutls.

On 12/9/09 1:59 AM, Rainer Gerhards wrote:
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Rory Toma
>> Sent: Tuesday, December 08, 2009 11:10 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] Help with client config
>>
>> And, actually, this only happens some of the time, where some of the
>> time means never on my dev systems and always on remote ones that I
>> can't easily debug. 8-)
>>      
> Now you feel a bit like me ;)
>
>    
>>
>> Rory Toma wrote:
>>      
>>> It works just fine without tls. I do not know much about tls, is
>>>        
>> there a
>>      
>>> default size somewhere that I can swizzle?
>>>        
> TLS is a stream cipher, so it needs some minimal block size to be secure.
> However, that minimum is pretty low. Rsyslog does not do any batching by
> itself for outbound connections. But it could be that GnuTLS can be tweaked
> one way or the other. I'd look into that direction.
>
> Rainer
>
>    
>>> Rainer Gerhards wrote:
>>>
>>>        
>>>> Can 1 be a tls (lib) artifact? Can you try with plain tcp?
>>>>
>>>> ----- Urspr?ngliche Nachricht -----
>>>> Von: "Rory Toma"<rory at ooma.com>
>>>> An: "rsyslog-users"<rsyslog at lists.adiscon.com>
>>>> Gesendet: 08.12.09 20:39
>>>> Betreff: [rsyslog] Help with client config
>>>>
>>>> I have a client config as follows, running on armv4, rsyslog 4.4.2
>>>>          
>> (I
>>      
>>>> can't get anything newer to compile due to the atomic kernel support
>>>>          
>> not
>>      
>>>> wanting to compile) The problem I'm having is:
>>>>
>>>> 1) (this is probably an rtfm, but I'll ask it anyway) It seems that
>>>>          
>> my
>>      
>>>> logs being sent over the network are being batched up in huge
>>>>          
>> packets, I
>>      
>>>> see one transaction for a bunch of data. Sometimes this means my
>>>>          
>> logs
>>      
>>>> are delayed quite a bit.
>>>>
>>>> 2) Sometimes, the client just doesn't send out over the network at
>>>>          
>> all
>>      
>>>> (verified with a tcpdump), but continues logging.
>>>>
>>>> Any suggestions or config file tweaks welcome.
>>>>
>>>> thx
>>>>
>>>> Here is my config:
>>>>
>>>> $DefaultNetStreamDriverCAFile /etc/ca.pem
>>>>
>>>> $DefaultNetStreamDriver gtls
>>>> $ActionSendStreamDriverMode 1
>>>> $ActionSendStreamDriverAuthMode anon
>>>>
>>>> $WorkDirectory /var/log
>>>>
>>>> $ActionQueueType LinkedList
>>>> $ActionQueueFileName rsyslog-fwd
>>>> $ActionResumeRetryCount -1
>>>> $ActionQueueSaveOnShutdown on
>>>> $ActionQueueMaxDiskSpace 256k
>>>>
>>>> $ModLoad imuxsock
>>>> $SystemLogSocketName /var/log/log
>>>> $OptimizeForUniprocessor on
>>>>
>>>> $outchannel
>>>>          
>> locallog,/var/log/locallog,262144,/usr/bin/rotate_locallog
>>      
>>>> *.* @@foo.bar.com:80
>>>> *.* $locallog
>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>>
>>>>
>>>>          
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>>>        
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>      
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>    


From rgerhards at hq.adiscon.com  Thu Dec 10 11:54:28 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 10 Dec 2009 11:54:28 +0100
Subject: [rsyslog] librelp 1.0.0 available
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103591@GRFEXC.intern.adiscon.com>

Hi all,

I have released librelp 1.0.0 today, please see the (spartanic) librelp
website at

http://www.librelp.com

for the download. This release primarily adds support to pass the remote
system's IP address to rsyslog. Note that contrary to what I said, the new
version does NOT enhance name resolution (I was simply wrong when I said
that). So if you have name resolution problems, we may need to look into them
seperately.

Note that the librelp version has been promoted to 1.0.0, as it has proven
quite stable in practice. The new version number hopefully conveys that fact.

Rainer

From david at lang.hm  Thu Dec 10 14:25:04 2009
From: david at lang.hm (david at lang.hm)
Date: Thu, 10 Dec 2009 05:25:04 -0800 (PST)
Subject: [rsyslog] Multiple-server failover questions (and round-robin
 DNS resolution, too)
In-Reply-To: <115906d10912091405m63edeef6hf643a7a219cd86c4@mail.gmail.com>
References: <115906d10912081718u2fad4334i7426310ea01a95a@mail.gmail.com>
	<alpine.DEB.2.00.0912090925090.10307@asgard.lang.hm>
	<115906d10912091008q43930462t4f3bf7b313e6a088@mail.gmail.com>
	<4255c2570912091037v7a5f5c3ci3d6301c524177aeb@mail.gmail.com>
	<115906d10912091405m63edeef6hf643a7a219cd86c4@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.0912100517430.3370@asgard.lang.hm>

On Wed, 9 Dec 2009, Ryan Lynch wrote:

> Date: Wed, 9 Dec 2009 17:05:18 -0500
> From: Ryan Lynch <ryan.b.lynch at gmail.com>
> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> To: rsyslog-users <rsyslog at lists.adiscon.com>
> Subject: Re: [rsyslog] Multiple-server failover questions (and round-robin	DNS
>     resolution, too)
> 
> David and RB:
>
> I've learned something about DNS round-robin, in the past 24 hours:
> Apparently, everybody hates it!
>
> Just kidding--I already knew that. It's very hate-able. Forgive me if
> it makes me chuckle that after I mentioned it, and three separate
> people in a row responded with some variation on "Ewww... DNS round
> robin is gross."

actually, in my case it's more a matter that doing a DNS lookup is a 
fairly slow operation, so it's something to avoid if possible.

> In my initial response to David, I was a little ambigious as to what
> "central" log servers means: The hosts RECEIVING syslog messages are
> all on different subnets, at different sites. I quoted the word
> "central" because the logs are centralized, not the servers
> themselves. Ideally, the servers are dispersed far and wide, for
> better survivability. So ClusterIP might still play a role, but it
> wouldn't be a drop-in replacement for a round-robin. I'm not sure,
> yet, how much of the initial, shared-nothing design I want to
> compromise--it's going to be a judgement call.

unless you send copies of all your logs to all your servers do you really 
get the survivability that you need? (I don't know what you are trying 
for)

> RB suggested LVS as another possible alternative. I actually have some
> firsthand experience with LVS, having built some WWW server farms on
> it, and I love it--truly a great piece of software. But in this
> situation, I wonder whether LVS really adds anything. I'd have to
> dedicate at least 2 extra machines to load-balancing duties, and the
> added complexity of the ipvsadm, CARP, etc. configs. The
> dedicated/clustered LBs would have to reside on a single subnet,
> though we could direct traffic to syslog receivers anywhere.
>
> The problem is, I feel like those dedicated LBs could create new
> problems: The LB mechanism is now both a potential bottleneck and a
> potential point of failure, neither of which existed before. Also,
> we'd have to route all of our traffic through the designated LBs,
> which would increase latency and load the network more than is
> strictly necessary. (I'm not sure if we really care about either of
> those issues, though--the effects probably won't be very big.) On the
> other hand, DNS round-robin has zero ability to control the traffic
> distribution, or to actively balance the load. I'd couldn't
> efficiently use differently-sized receiving servers. Plus, we really
> have to count the DNS as a potential point of failure, too.

the load balancers themselves would need to be in one place, but could 
then direct the traffic to multiple sites.

would it make sense to have fairly local relay boxes (possibly in a HA 
pair) that the local servers send their logs to, then these relay boxes 
forward the logs to more central servers (possibly multiple such servers 
in multiple locations for survivability)?

David Lang

From ktm at rice.edu  Thu Dec 10 15:19:57 2009
From: ktm at rice.edu (Kenneth Marshall)
Date: Thu, 10 Dec 2009 08:19:57 -0600
Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71033F1@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA71033F1@GRFEXC.intern.adiscon.com>
Message-ID: <20091210141957.GC23781@it.is.rice.edu>

Hi,

Just an update on the 5.3.5 release. It successfully compiled on
my Redhat 4 box after the latest patch to omfile and the configure
process. I ran a check of the current, very basic configuration,
with:

rsyslog -c5 -f/etc/opt/rsyslog/rsyslog.conf

and there were no errors reported. After I shutdown rsyslog 4.2.0 
and tried to restart the server, it started fine with no errors
logged, but the PostgreSQL 8.4.1 backend started spewing these
error messages over and over:

...
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
WARNING:  there is already a transaction in progress
...

Here is my configuration:

$ModLoad immark # provides --MARK-- message capability
$ModLoad imtcp # provides TCP syslog reception and GSS-API (if compiled to support it)
$ModLoad imuxsock
$InputTCPServerRun 601 # listen on this port
$WorkDirectory /var/spool/rsyslog # default location for work (spool) files
$MainMsgQueueFileName mainq  # set file name, also enables disk mode
$ActionResumeRetryCount -1   # infinite retries on insert failure
$ModLoad ompgsql # provides postgres support
# Everything we receive goes in the database
*.*       :ompgsql:127.0.0.1,Syslog,database,xxxxxxxx

Any ideas about what could be going on? I put back the
version 4.2.0 for now, but I would like to be able to
take advantage of the parallel input processing that is
available in the 5.3.5 release. Let me know if I can
do anything to help debug the problem.

Regards,
Ken

From ktm at rice.edu  Thu Dec 10 15:22:28 2009
From: ktm at rice.edu (Kenneth Marshall)
Date: Thu, 10 Dec 2009 08:22:28 -0600
Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released
In-Reply-To: <20091210141957.GC23781@it.is.rice.edu>
References: <9B6E2A8877C38245BFB15CC491A11DA71033F1@GRFEXC.intern.adiscon.com>
	<20091210141957.GC23781@it.is.rice.edu>
Message-ID: <20091210142228.GD23781@it.is.rice.edu>

On Thu, Dec 10, 2009 at 08:19:57AM -0600, Kenneth Marshall wrote:
> Hi,
> 
> Just an update on the 5.3.5 release. It successfully compiled on
> my Redhat 4 box after the latest patch to omfile and the configure
> process. I ran a check of the current, very basic configuration,
> with:
> 
> rsyslog -c5 -f/etc/opt/rsyslog/rsyslog.conf
> 
> and there were no errors reported. After I shutdown rsyslog 4.2.0 
> and tried to restart the server, it started fine with no errors
> logged, but the PostgreSQL 8.4.1 backend started spewing these
> error messages over and over:
> 
> ...
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> WARNING:  there is already a transaction in progress
> ...
> 
> Here is my configuration:
> 
> $ModLoad immark # provides --MARK-- message capability
> $ModLoad imtcp # provides TCP syslog reception and GSS-API (if compiled to support it)
> $ModLoad imuxsock
> $InputTCPServerRun 601 # listen on this port
> $WorkDirectory /var/spool/rsyslog # default location for work (spool) files
> $MainMsgQueueFileName mainq  # set file name, also enables disk mode
> $ActionResumeRetryCount -1   # infinite retries on insert failure
> $ModLoad ompgsql # provides postgres support
> # Everything we receive goes in the database
> *.*       :ompgsql:127.0.0.1,Syslog,database,xxxxxxxx
> 
> Any ideas about what could be going on? I put back the
> version 4.2.0 for now, but I would like to be able to
> take advantage of the parallel input processing that is
> available in the 5.3.5 release. Let me know if I can
> do anything to help debug the problem.
> 
> Regards,
> Ken

Sorry, I forgot to mention that no data is ever written to the
database.

Cheers,
Ken

From tronyx at tronyx.networkgimps.com  Thu Dec 10 15:02:58 2009
From: tronyx at tronyx.networkgimps.com (tronyx at tronyx.networkgimps.com)
Date: Thu, 10 Dec 2009 08:02:58 -0600
Subject: [rsyslog] Duplicate log entries with Apache+rsyslog
Message-ID: <20091210080258.i7lymnkrkkk0og48@tronyx.networkgimps.com>

Greetings List,

I have configured Apache to log remotely to an rsyslog machine and  
while it works perfectly, I am having a strange issue that I am not  
sure how to get around.  The problem is that the access log from the  
Apache machine are creating log entries in the  
/var/log/%HOSTNAME%/messages file in addition to the domain logs.

The web server is running a custom built GRSec kernel and while I am  
pretty sure I _see_ what the problem is, I am not sure of the best way  
to correct it.

On the web server, the log directives are set as below:

ErrorLog "|/usr/bin/logger -p local7.err -t error_domain.com"
CustomLog "|/usr/bin/logger -p local6.info -t domain.com " "combined"

Rsyslog.conf on the web server contains the below:

local6.info @@127.0.0.1:61514
local7.err @@127.0.0.1:61514
*.* @@127.0.0.1:61514
(using stunnel for the connection)

Below is the server's rsyslog.conf

$AllowedSender UDP, 127.0.0.1, 10.32.1.81/29
$AllowedSender TCP, 127.0.0.1, 10.32.1.81/29
# The authpriv file has restricted access.
$template DynAuth, "/var/log/%HOSTNAME%/secure.log"

# Log anything (except mail and cron) of level info or higher.
$template DynMSG, "/var/log/%HOSTNAME%/messages"

# Log all the mail messages in one place.
$template Dynmail, "/var/log/%HOSTNAME%/maillog"

# Log cron stuff
$template Dyncron,"/var/log/%HOSTNAME%/cron"

# Save news errors of level crit and higher in a special file.
$template Dynspool, "/var/log/%HOSTNAME%/spooler"

# Save boot messages also to boot.log
$template Dynboot, "/var/log/%HOSTNAME%/boot.log"
$template ApacheRemoteCustom, "/var/log/%HOSTNAME%/Apache/access.log"
local6.info -?ApacheRemoteCustom
$template ApacheRemoteErr, "/var/log/%HOSTNAME%/Apache/error.log"
local7.err -?ApacheRemoteErr
authpriv.* ?DynAuth
*.info,mail.none,authpriv.none,cron.none ?DynMSG
#mail.none,authpriv.none,cron.none ?DynMSG

mail.* -?Dynmail
cron.* ?Dyncron
news.crit ?Dynspool


Now, as noted, the logging works perfectly fine, but the page accesses  
are creating 2 entries.  One is in the messages file, one is in the  
intended log file.

Based on the configuration file, this looks to be due to the line:
*.info,mail.none,authpriv.none,cron.none ?DynMSG

But the problem is that if I comment that line out or remove *.info, I  
no longer get the GRSec messages which I very much need for debugging  
purposes.

Any advice on this would be a great deal of assistance as rsyslog is  
completely new to me.  Thank you!


From david at lang.hm  Thu Dec 10 16:28:19 2009
From: david at lang.hm (david at lang.hm)
Date: Thu, 10 Dec 2009 07:28:19 -0800 (PST)
Subject: [rsyslog] Duplicate log entries with Apache+rsyslog
In-Reply-To: <20091210080258.i7lymnkrkkk0og48@tronyx.networkgimps.com>
References: <20091210080258.i7lymnkrkkk0og48@tronyx.networkgimps.com>
Message-ID: <alpine.DEB.2.00.0912100722590.3370@asgard.lang.hm>

On Thu, 10 Dec 2009, tronyx at tronyx.networkgimps.com wrote:

> Greetings List,
>
> I have configured Apache to log remotely to an rsyslog machine and
> while it works perfectly, I am having a strange issue that I am not
> sure how to get around.  The problem is that the access log from the
> Apache machine are creating log entries in the
> /var/log/%HOSTNAME%/messages file in addition to the domain logs.
>
> The web server is running a custom built GRSec kernel and while I am
> pretty sure I _see_ what the problem is, I am not sure of the best way
> to correct it.
>
> On the web server, the log directives are set as below:
>
> ErrorLog "|/usr/bin/logger -p local7.err -t error_domain.com"
> CustomLog "|/usr/bin/logger -p local6.info -t domain.com " "combined"
>
> Rsyslog.conf on the web server contains the below:
>
> local6.info @@127.0.0.1:61514
> local7.err @@127.0.0.1:61514

here you are saying to send the logs of this type to the local server port 
61514

> *.* @@127.0.0.1:61514

here you are saying to send everything to the local server port 61514, 
this includes sending an additional copy of the logs that you sent in the 
lines above.

you don't need the first two lines, they are causing duplicate entries.


> (using stunnel for the connection)
>
> Below is the server's rsyslog.conf
>
> $AllowedSender UDP, 127.0.0.1, 10.32.1.81/29
> $AllowedSender TCP, 127.0.0.1, 10.32.1.81/29
> # The authpriv file has restricted access.
> $template DynAuth, "/var/log/%HOSTNAME%/secure.log"
>
> # Log anything (except mail and cron) of level info or higher.
> $template DynMSG, "/var/log/%HOSTNAME%/messages"
>
> # Log all the mail messages in one place.
> $template Dynmail, "/var/log/%HOSTNAME%/maillog"
>
> # Log cron stuff
> $template Dyncron,"/var/log/%HOSTNAME%/cron"
>
> # Save news errors of level crit and higher in a special file.
> $template Dynspool, "/var/log/%HOSTNAME%/spooler"
>
> # Save boot messages also to boot.log
> $template Dynboot, "/var/log/%HOSTNAME%/boot.log"
> $template ApacheRemoteCustom, "/var/log/%HOSTNAME%/Apache/access.log"
> local6.info -?ApacheRemoteCustom

here you say send local6.ifo to apacheremotecustom

> $template ApacheRemoteErr, "/var/log/%HOSTNAME%/Apache/error.log"
> local7.err -?ApacheRemoteErr
> authpriv.* ?DynAuth
> *.info,mail.none,authpriv.none,cron.none ?DynMSG

but here you say send *.info to DynMSG

if you don't want to have the logs processed more than once, you need to 
tell rsyslog to drop them after they have been processed

so you would do

local6.info -?ApacheRemoteCustom
local6.info ~
*.info,mail.none,authpriv.none,cron.none ?DynMSG

the ~ tells rsyslog to stop processing logs like this.

I believe that you can shorten this to

local6.info -?ApacheRemoteCustom
~
*.info,mail.none,authpriv.none,cron.none ?DynMSG

because rsyslog will continue to use the prior filter rules until it sees 
new ones (I know you can do this with more complex filters, I haven't done 
it with the simple syslog-style filters)

David Lang

> #mail.none,authpriv.none,cron.none ?DynMSG
>
> mail.* -?Dynmail
> cron.* ?Dyncron
> news.crit ?Dynspool
>
>
> Now, as noted, the logging works perfectly fine, but the page accesses
> are creating 2 entries.  One is in the messages file, one is in the
> intended log file.
>
> Based on the configuration file, this looks to be due to the line:
> *.info,mail.none,authpriv.none,cron.none ?DynMSG
>
> But the problem is that if I comment that line out or remove *.info, I
> no longer get the GRSec messages which I very much need for debugging
> purposes.
>
> Any advice on this would be a great deal of assistance as rsyslog is
> completely new to me.  Thank you!
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From rgerhards at hq.adiscon.com  Thu Dec 10 16:36:01 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 10 Dec 2009 16:36:01 +0100
Subject: [rsyslog] Duplicate log entries with Apache+rsyslog
References: <20091210080258.i7lymnkrkkk0og48@tronyx.networkgimps.com>
	<alpine.DEB.2.00.0912100722590.3370@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103594@GRFEXC.intern.adiscon.com>

David, 

small correction:

> I believe that you can shorten this to

local6.info -?ApacheRemoteCustom
& ~                                             # note the "&"!
*.info,mail.none,authpriv.none,cron.none ?DynMSG


> because rsyslog will continue to use the prior filter rules until it
> sees
> new ones (I know you can do this with more complex filters, I haven't
> done
> it with the simple syslog-style filters)

slightly different. "&" chains together actions. A rule consist of a filter
and actions. There is one filter, but there can be many actions. The "&"
tells that an action is to be added to the previous rule.

Rainer



From rgerhards at hq.adiscon.com  Thu Dec 10 16:56:15 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 10 Dec 2009 16:56:15 +0100
Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released
References: <9B6E2A8877C38245BFB15CC491A11DA71033F1@GRFEXC.intern.adiscon.com><20091210141957.GC23781@it.is.rice.edu>
	<20091210142228.GD23781@it.is.rice.edu>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103596@GRFEXC.intern.adiscon.com>

Hi Ken,

this is a known issue, please see:

http://bugzilla.adiscon.com/show_bug.cgi?id=162

If you follow the forum thread in the bug tracker, you'll find some request
for debug log together with instructions (page 2). If you could send me such
a log, that would be most helpful. I had begun to set up a postgres system
myself, but ran out of time...

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Kenneth Marshall
> Sent: Thursday, December 10, 2009 3:22 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 5.3.5 (v5-beta) released
> 
> On Thu, Dec 10, 2009 at 08:19:57AM -0600, Kenneth Marshall wrote:
> > Hi,
> >
> > Just an update on the 5.3.5 release. It successfully compiled on
> > my Redhat 4 box after the latest patch to omfile and the configure
> > process. I ran a check of the current, very basic configuration,
> > with:
> >
> > rsyslog -c5 -f/etc/opt/rsyslog/rsyslog.conf
> >
> > and there were no errors reported. After I shutdown rsyslog 4.2.0
> > and tried to restart the server, it started fine with no errors
> > logged, but the PostgreSQL 8.4.1 backend started spewing these
> > error messages over and over:
> >
> > ...
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > WARNING:  there is already a transaction in progress
> > ...
> >
> > Here is my configuration:
> >
> > $ModLoad immark # provides --MARK-- message capability
> > $ModLoad imtcp # provides TCP syslog reception and GSS-API (if
> compiled to support it)
> > $ModLoad imuxsock
> > $InputTCPServerRun 601 # listen on this port
> > $WorkDirectory /var/spool/rsyslog # default location for work (spool)
> files
> > $MainMsgQueueFileName mainq  # set file name, also enables disk mode
> > $ActionResumeRetryCount -1   # infinite retries on insert failure
> > $ModLoad ompgsql # provides postgres support
> > # Everything we receive goes in the database
> > *.*       :ompgsql:127.0.0.1,Syslog,database,xxxxxxxx
> >
> > Any ideas about what could be going on? I put back the
> > version 4.2.0 for now, but I would like to be able to
> > take advantage of the parallel input processing that is
> > available in the 5.3.5 release. Let me know if I can
> > do anything to help debug the problem.
> >
> > Regards,
> > Ken
> 
> Sorry, I forgot to mention that no data is ever written to the
> database.
> 
> Cheers,
> Ken
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From ktm at rice.edu  Thu Dec 10 18:42:50 2009
From: ktm at rice.edu (Kenneth Marshall)
Date: Thu, 10 Dec 2009 11:42:50 -0600
Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103596@GRFEXC.intern.adiscon.com>
References: <20091210142228.GD23781@it.is.rice.edu>
	<9B6E2A8877C38245BFB15CC491A11DA7103596@GRFEXC.intern.adiscon.com>
Message-ID: <20091210174250.GF23781@it.is.rice.edu>

On Thu, Dec 10, 2009 at 04:56:15PM +0100, Rainer Gerhards wrote:
> Hi Ken,
> 
> this is a known issue, please see:
> 
> http://bugzilla.adiscon.com/show_bug.cgi?id=162
> 
> If you follow the forum thread in the bug tracker, you'll find some request
> for debug log together with instructions (page 2). If you could send me such
> a log, that would be most helpful. I had begun to set up a postgres system
> myself, but ran out of time...
> 
> Rainer
> 
Hi Rainer,

I just read the thread and that is what I am seeing. The error
that was reported at the end of the thread is actually coming
from the PostgreSQL DB and not from rsyslog, although the lack
of commits is triggering the error. I suspect that the tester
start the database from the same terminal as the rsyslog test
which is why the error showed up there.

Regards,
Ken

From rgerhards at hq.adiscon.com  Thu Dec 10 18:52:11 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 10 Dec 2009 18:52:11 +0100
Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released
References: <20091210142228.GD23781@it.is.rice.edu><9B6E2A8877C38245BFB15CC491A11DA7103596@GRFEXC.intern.adiscon.com>
	<20091210174250.GF23781@it.is.rice.edu>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710359A@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Kenneth Marshall
> Sent: Thursday, December 10, 2009 6:43 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 5.3.5 (v5-beta) released
> 
> On Thu, Dec 10, 2009 at 04:56:15PM +0100, Rainer Gerhards wrote:
> > Hi Ken,
> >
> > this is a known issue, please see:
> >
> > http://bugzilla.adiscon.com/show_bug.cgi?id=162
> >
> > If you follow the forum thread in the bug tracker, you'll find some
> request
> > for debug log together with instructions (page 2). If you could send
> me such
> > a log, that would be most helpful. I had begun to set up a postgres
> system
> > myself, but ran out of time...
> >
> > Rainer
> >
> Hi Rainer,
> 
> I just read the thread and that is what I am seeing. The error
> that was reported at the end of the thread is actually coming
> from the PostgreSQL DB and not from rsyslog, although the lack
> of commits is triggering the error. I suspect that the tester
> start the database from the same terminal as the rsyslog test
> which is why the error showed up there.

ah! that explains a lot! I am now waiting for some in-depth debug log. My
hope is that I can quickly fix the situation once I see the control flow.

Rainer

> 
> Regards,
> Ken
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From ryan.b.lynch at gmail.com  Thu Dec 10 19:01:58 2009
From: ryan.b.lynch at gmail.com (Ryan Lynch)
Date: Thu, 10 Dec 2009 13:01:58 -0500
Subject: [rsyslog] Multiple-server failover questions (and round-robin
	DNS resolution, too)
In-Reply-To: <alpine.DEB.2.00.0912100517430.3370@asgard.lang.hm>
References: <115906d10912081718u2fad4334i7426310ea01a95a@mail.gmail.com> 
	<alpine.DEB.2.00.0912090925090.10307@asgard.lang.hm>
	<115906d10912091008q43930462t4f3bf7b313e6a088@mail.gmail.com> 
	<4255c2570912091037v7a5f5c3ci3d6301c524177aeb@mail.gmail.com> 
	<115906d10912091405m63edeef6hf643a7a219cd86c4@mail.gmail.com> 
	<alpine.DEB.2.00.0912100517430.3370@asgard.lang.hm>
Message-ID: <115906d10912101001n3bf9dfebke35dc8d97f5385f0@mail.gmail.com>

On Thu, Dec 10, 2009 at 08:25,  <david at lang.hm> wrote:
> On Wed, 9 Dec 2009, Ryan Lynch wrote:
>> I've learned something about DNS round-robin, in the past 24 hours:
>> Apparently, everybody hates it!
>
> actually, in my case it's more a matter that doing a DNS lookup is a
> fairly slow operation, so it's something to avoid if possible.

Makes sense to me.


>> In my initial response to David, I was a little ambigious as to what
>> "central" log servers means: The hosts RECEIVING syslog messages are
>> all on different subnets, at different sites. I quoted the word
>> "central" because the logs are centralized, not the servers
>> themselves. Ideally, the servers are dispersed far and wide, for
>> better survivability. So ClusterIP might still play a role, but it
>> wouldn't be a drop-in replacement for a round-robin. I'm not sure,
>> yet, how much of the initial, shared-nothing design I want to
>> compromise--it's going to be a judgement call.
>
> unless you send copies of all your logs to all your servers do you really
> get the survivability that you need? (I don't know what you are trying
> for)

That's a very good question. If we divide the log data across a pool
of N log-receiving hosts, a catastrophic failure of one receiver would
blow away (1/N) of our logs. And like RAID striping, the probability
of a single-element failure increases as we add more receivers. I can
mitigate that risk by building the receivers on redundant RAID arrays,
using battery-backed cache and reliable journaling filesystems, but
the risk of a single receiver losing data is still there. So it's not
as simple as saying "let's not put all our eggs in one basket", if
losing even a single egg is a problem. (Our logs aren't all quite that
important, but we don't want to ignore the issue completely.) I have a
couple of ideas, to start with.

My first idea is probably overkil: Secondary relaying between the
log-receiving hosts. I could configure the receivers' Rsyslog daemons
to write edge server log data to local file, but then to also send a
copy to another receiver. The distribution of secondary relay targets
could be statically assigned, or I could imagine using another
clustering mechanism to make it handle failover conditions
automatically. We'd have to be careful of logging "loops", where two
servers keep sending the same log data back and forth to each other,
but that's pretty easy if you know to watch out for it. The real
problem, I think, is doubling the amount of storage, network
bandwidth, disk bandwidth, and CPU time that we would otherwise need
to handle a single copy of the edge logs in real time.

A second possibility is to periodically batch compress/copy each
receiver's local files to another receiver, or to some other backup
location. We still use some extra CPU, network bandwidth, and storage,
but batch processing and compression will probably increase the
per-message efficiency. I think we might actually use more disk IO,
though, because the batch processing needs a separate read operation
to grab the already-written files on disk (whereas real time
processing already has the message in memory when it makes the 2nd
copy). We could still lose some data to a receiver failure, from the
interval between batches, so it would be important to consider the
costs of data loss versus the lower efficiency with shorter intervals.
(It's worth noting, here, that this method could become *less*
efficient than real-time processing, if the interval gets too short.)

It might make sense to do both, in parallel, for different types of
data. Some of the logs (audit, auth, error) are low-volume, while
others (WWW access, mail info) generate stupid numbers of messages.
And our tolerance for data loss varies considerably: Nobody really
cares if we accidentally lose a fraction of our WWW access logs for
some random half-hour period, but some of the audit and auth logs need
to be guaranteed a little better. With the right Rsyslog
configurations, we could double-relay the low-volume and high-value
data, and either batch-backup the rest, or even just leave it be, for
the most trivial data.


>> RB suggested LVS as another possible alternative. I actually have some
>> firsthand experience with LVS, having built some WWW server farms on
>> it, and I love it--truly a great piece of software. But in this
>> situation, I wonder whether LVS really adds anything. I'd have to
>> dedicate at least 2 extra machines to load-balancing duties, and the
>> added complexity of the ipvsadm, CARP, etc. configs. The
>> dedicated/clustered LBs would have to reside on a single subnet,
>> though we could direct traffic to syslog receivers anywhere.
>
> the load balancers themselves would need to be in one place, but could
> then direct the traffic to multiple sites.
>
> would it make sense to have fairly local relay boxes (possibly in a HA
> pair) that the local servers send their logs to, then these relay boxes
> forward the logs to more central servers (possibly multiple such servers
> in multiple locations for survivability)?

I actually like the local relay idea, and we could probably configure
things to address the log backup copy issue. After all, we ought to
have at least one (hopefully two) receivers at each site (in case we
lose a site, or an inter-site link). It would make sense for clients
to default to using the closest (local) log receivers, if they're
available.

I don't think there's a really simple "best" answer, here, and just
figuring out the optimal corner cases is going to take some thought.
There are a lot of possible combinations of LVS, ClusterIP, DNS
round-robin, and Rsyslog's own built-in failover mechanism, and right
now my brain feels like a tumble-dryer full of possibilities.

-Ryan

From ktm at rice.edu  Thu Dec 10 19:02:55 2009
From: ktm at rice.edu (Kenneth Marshall)
Date: Thu, 10 Dec 2009 12:02:55 -0600
Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710359A@GRFEXC.intern.adiscon.com>
References: <20091210174250.GF23781@it.is.rice.edu>
	<9B6E2A8877C38245BFB15CC491A11DA710359A@GRFEXC.intern.adiscon.com>
Message-ID: <20091210180255.GG23781@it.is.rice.edu>

On Thu, Dec 10, 2009 at 06:52:11PM +0100, Rainer Gerhards wrote:
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Kenneth Marshall
> > Sent: Thursday, December 10, 2009 6:43 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] rsyslog 5.3.5 (v5-beta) released
> > 
> > On Thu, Dec 10, 2009 at 04:56:15PM +0100, Rainer Gerhards wrote:
> > > Hi Ken,
> > >
> > > this is a known issue, please see:
> > >
> > > http://bugzilla.adiscon.com/show_bug.cgi?id=162
> > >
> > > If you follow the forum thread in the bug tracker, you'll find some
> > request
> > > for debug log together with instructions (page 2). If you could send
> > me such
> > > a log, that would be most helpful. I had begun to set up a postgres
> > system
> > > myself, but ran out of time...
> > >
> > > Rainer
> > >
> > Hi Rainer,
> > 
> > I just read the thread and that is what I am seeing. The error
> > that was reported at the end of the thread is actually coming
> > from the PostgreSQL DB and not from rsyslog, although the lack
> > of commits is triggering the error. I suspect that the tester
> > start the database from the same terminal as the rsyslog test
> > which is why the error showed up there.
> 
> ah! that explains a lot! I am now waiting for some in-depth debug log. My
> hope is that I can quickly fix the situation once I see the control flow.
> 
> Rainer
> 

I will try and get the debug information between my afternoon meetings.

Ken

From tronyx at tronyx.networkgimps.com  Thu Dec 10 19:35:44 2009
From: tronyx at tronyx.networkgimps.com (tronyx at tronyx.networkgimps.com)
Date: Thu, 10 Dec 2009 12:35:44 -0600
Subject: [rsyslog] Duplicate log entries with Apache+rsyslog
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103594@GRFEXC.intern.adiscon.com>
References: <20091210080258.i7lymnkrkkk0og48@tronyx.networkgimps.com>
	<alpine.DEB.2.00.0912100722590.3370@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7103594@GRFEXC.intern.adiscon.com>
Message-ID: <20091210123544.uy4o9f604kggcc88@tronyx.networkgimps.com>

Quoting Rainer Gerhards <rgerhards at hq.adiscon.com>:

> David,
>
> small correction:
>
>> I believe that you can shorten this to
>
> local6.info -?ApacheRemoteCustom
> & ~                                             # note the "&"!
> *.info,mail.none,authpriv.none,cron.none ?DynMSG
>
>
>> because rsyslog will continue to use the prior filter rules until it
>> sees
>> new ones (I know you can do this with more complex filters, I haven't
>> done
>> it with the simple syslog-style filters)
>
> slightly different. "&" chains together actions. A rule consist of a filter
> and actions. There is one filter, but there can be many actions. The "&"
> tells that an action is to be added to the previous rule.
>
> Rainer
>
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

All of the above has worked perfectly.  Thank you SO much!


From ryan.b.lynch at gmail.com  Thu Dec 10 20:07:48 2009
From: ryan.b.lynch at gmail.com (Ryan Lynch)
Date: Thu, 10 Dec 2009 14:07:48 -0500
Subject: [rsyslog] Duplicate log entries with Apache+rsyslog
In-Reply-To: <20091210123544.uy4o9f604kggcc88@tronyx.networkgimps.com>
References: <20091210080258.i7lymnkrkkk0og48@tronyx.networkgimps.com> 
	<alpine.DEB.2.00.0912100722590.3370@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7103594@GRFEXC.intern.adiscon.com>
	<20091210123544.uy4o9f604kggcc88@tronyx.networkgimps.com>
Message-ID: <115906d10912101107j24f38b8j2b2b7aaf6a3033e0@mail.gmail.com>

On Thu, Dec 10, 2009 at 09:02,  <tronyx at tronyx.networkgimps.com> wrote:
> On the web server, the log directives are set as below:
>
> ErrorLog "|/usr/bin/logger -p local7.err -t error_domain.com"
> CustomLog "|/usr/bin/logger -p local6.info -t domain.com " "combined"

This is totally unrelated to your question, and I'm sure you have
reasons for it, but I'm curious about something: Why are you piping
ErrorLog to 'logger'?

Apache's CustomLog doesn't support writing directly to syslog, so you
do need to pipe it to 'logger'. But ErrorLog can do it: 'ErrorLog
syslog:local7'. Apache doesn't call an external program, it just makes
the syslog calls, itself. (No idea why Apache supports one and not the
other.)

I can think of two practical differences with your method, versus the
built-in Apache support:

 - You can use the syslog tag 'error_domain.com'. I don't think Apache
can specify a custom tag, like that, can it?
 - Syslog can't see the real severity levels that Apache associates
with its messages. ErrorLog messages can be err, crit, warn, info,
debug, etc., but I think 'logger' forces everything into a single
severity level (err, in this case).

I hope you don't mind my curiousity. I do something similar with my
Apache logs, and I'm always looking for outside input on alternatices
and good practices.

-Ryan

From david at lang.hm  Thu Dec 10 21:09:40 2009
From: david at lang.hm (david at lang.hm)
Date: Thu, 10 Dec 2009 12:09:40 -0800 (PST)
Subject: [rsyslog] Multiple-server failover questions (and round-robin
 DNS resolution, too)
In-Reply-To: <115906d10912101001n3bf9dfebke35dc8d97f5385f0@mail.gmail.com>
References: <115906d10912081718u2fad4334i7426310ea01a95a@mail.gmail.com>
	<alpine.DEB.2.00.0912090925090.10307@asgard.lang.hm>
	<115906d10912091008q43930462t4f3bf7b313e6a088@mail.gmail.com>
	<4255c2570912091037v7a5f5c3ci3d6301c524177aeb@mail.gmail.com> 
	<115906d10912091405m63edeef6hf643a7a219cd86c4@mail.gmail.com>
	<alpine.DEB.2.00.0912100517430.3370@asgard.lang.hm>
	<115906d10912101001n3bf9dfebke35dc8d97f5385f0@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.0912101158590.10307@asgard.lang.hm>

On Thu, 10 Dec 2009, Ryan Lynch wrote:

> On Thu, Dec 10, 2009 at 08:25,  <david at lang.hm> wrote:
>> On Wed, 9 Dec 2009, Ryan Lynch wrote:
>
>>> In my initial response to David, I was a little ambigious as to what
>>> "central" log servers means: The hosts RECEIVING syslog messages are
>>> all on different subnets, at different sites. I quoted the word
>>> "central" because the logs are centralized, not the servers
>>> themselves. Ideally, the servers are dispersed far and wide, for
>>> better survivability. So ClusterIP might still play a role, but it
>>> wouldn't be a drop-in replacement for a round-robin. I'm not sure,
>>> yet, how much of the initial, shared-nothing design I want to
>>> compromise--it's going to be a judgement call.
>>
>> unless you send copies of all your logs to all your servers do you really
>> get the survivability that you need? (I don't know what you are trying
>> for)
>
> That's a very good question. If we divide the log data across a pool
> of N log-receiving hosts, a catastrophic failure of one receiver would
> blow away (1/N) of our logs. And like RAID striping, the probability
> of a single-element failure increases as we add more receivers. I can
> mitigate that risk by building the receivers on redundant RAID arrays,
> using battery-backed cache and reliable journaling filesystems, but
> the risk of a single receiver losing data is still there. So it's not
> as simple as saying "let's not put all our eggs in one basket", if
> losing even a single egg is a problem. (Our logs aren't all quite that
> important, but we don't want to ignore the issue completely.) I have a
> couple of ideas, to start with.

one really interesting trick that you can pull with clusterIP is that you 
can use the same IP on more than one cluster with UDP.

so assuming that you don't need load balancing and can use UDP for the 
final hop, you could have two machines, both set to use clusterIP with a 
node count of 1 and a node id of 1. both machines would see and process 
all the logs. if one machine dies the other still has the logs.

as one machine dies and is fixed and later the other does the same end up 
with all your logs there, but neither machine having all of them.

I am doing something along these lines now.

I have several clusters setup with each cluster using the same IP

on one of these clusters I have a log searching applicationa (splunk), 
which needs many machines. so I have the 20 machines in this cluster setup 
as 10 pairs of 2 machines, each pair has both machines configured to 
receive the same 1/10 of the log data, which rsyslog then writes to disk. 
when the logs roll, I configure the 'standby' box of each set to throw 
away it's logs, and the 'active' box of each set moves it's logs both to 
long-term storage in itself, and to long-term storage on the standby box.

when a failover happens I may loose or duplicate a small amount of logs 
(the log rotation isn't going to be at exactly the same instant on the two 
machines), but this is close enough for my situation.

David Lang

From ryan.b.lynch at gmail.com  Thu Dec 10 21:41:29 2009
From: ryan.b.lynch at gmail.com (Ryan Lynch)
Date: Thu, 10 Dec 2009 15:41:29 -0500
Subject: [rsyslog] rsyslog "feature schedule"
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103587@GRFEXC.intern.adiscon.com>
References: <Acp4+TLYJxhhtw1XQhu8KR901LML1Q==>
	<9B6E2A8877C38245BFB15CC491A11DA7103587@GRFEXC.intern.adiscon.com>
Message-ID: <115906d10912101241u21b6dc39uc2caa353b94ca9ca@mail.gmail.com>

On Wed, Dec 9, 2009 at 12:58, Rainer Gerhards <rgerhards at hq.adiscon.com> wrote:
> I've finally managed to write up my "feature schedule" (read it to understand
> the quotes if you do not already know why they are there ;)) as requested:

Thanks for the news, very informative.

I'm excited to hear that Rsyslog will be getting a runtime stats
engine--that will be incredibly useful.

-Ryan

From rgerhards at hq.adiscon.com  Fri Dec 11 13:04:25 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 11 Dec 2009 13:04:25 +0100
Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released
References: <20091210174250.GF23781@it.is.rice.edu><9B6E2A8877C38245BFB15CC491A11DA710359A@GRFEXC.intern.adiscon.com>
	<20091210180255.GG23781@it.is.rice.edu>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71035AD@GRFEXC.intern.adiscon.com>

if not subscribed to it, revisit bug tracker or forum post, there now exist a
patch! :)

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Kenneth Marshall
> Sent: Thursday, December 10, 2009 7:03 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 5.3.5 (v5-beta) released
> 
> On Thu, Dec 10, 2009 at 06:52:11PM +0100, Rainer Gerhards wrote:
> > > -----Original Message-----
> > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > > bounces at lists.adiscon.com] On Behalf Of Kenneth Marshall
> > > Sent: Thursday, December 10, 2009 6:43 PM
> > > To: rsyslog-users
> > > Subject: Re: [rsyslog] rsyslog 5.3.5 (v5-beta) released
> > >
> > > On Thu, Dec 10, 2009 at 04:56:15PM +0100, Rainer Gerhards wrote:
> > > > Hi Ken,
> > > >
> > > > this is a known issue, please see:
> > > >
> > > > http://bugzilla.adiscon.com/show_bug.cgi?id=162
> > > >
> > > > If you follow the forum thread in the bug tracker, you'll find
> some
> > > request
> > > > for debug log together with instructions (page 2). If you could
> send
> > > me such
> > > > a log, that would be most helpful. I had begun to set up a
> postgres
> > > system
> > > > myself, but ran out of time...
> > > >
> > > > Rainer
> > > >
> > > Hi Rainer,
> > >
> > > I just read the thread and that is what I am seeing. The error
> > > that was reported at the end of the thread is actually coming
> > > from the PostgreSQL DB and not from rsyslog, although the lack
> > > of commits is triggering the error. I suspect that the tester
> > > start the database from the same terminal as the rsyslog test
> > > which is why the error showed up there.
> >
> > ah! that explains a lot! I am now waiting for some in-depth debug
> log. My
> > hope is that I can quickly fix the situation once I see the control
> flow.
> >
> > Rainer
> >
> 
> I will try and get the debug information between my afternoon meetings.
> 
> Ken
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From ktm at rice.edu  Fri Dec 11 15:06:03 2009
From: ktm at rice.edu (Kenneth Marshall)
Date: Fri, 11 Dec 2009 08:06:03 -0600
Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71035AD@GRFEXC.intern.adiscon.com>
References: <20091210180255.GG23781@it.is.rice.edu>
	<9B6E2A8877C38245BFB15CC491A11DA71035AD@GRFEXC.intern.adiscon.com>
Message-ID: <20091211140603.GK23781@it.is.rice.edu>

On Fri, Dec 11, 2009 at 01:04:25PM +0100, Rainer Gerhards wrote:
> if not subscribed to it, revisit bug tracker or forum post, there now exist a
> patch! :)
> 
> Rainer
> 
Thank you Rainer. It now logs once more. Unfortunately, I am having
trouble getting the INSERT batching to work. Here is my current
rsyslog.conf:

$ModLoad immark # provides --MARK-- message capability
$ModLoad imtcp # provides TCP syslog reception and GSS-API (if compiled to support it)
$ModLoad imuxsock
$InputTCPServerRun 601 # listen on this port
$WorkDirectory /var/spool/rsyslog # default location for work (spool) files
#$MainMsgQueueFileName mainq  # set file name, also enables disk mode
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueDequeueBatchSize 16
$ActionQueueFileName dbq    # set file name, also enables disk mode
$ActionResumeRetryCount -1   # infinite retries on insert failure
$ModLoad ompgsql # provides postgres support
# Everything we receive goes in the database
*.*       :ompgsql:127.0.0.1,Syslog,dbuser,xxxxxx

The $ActionQueueDequeueBatchSize option does not appear to have
any effect. The inserts are still performed one at a time. Any
ideas?

Regards,
Ken


From rgerhards at hq.adiscon.com  Fri Dec 11 15:25:32 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 11 Dec 2009 15:25:32 +0100
Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released
References: <20091210180255.GG23781@it.is.rice.edu><9B6E2A8877C38245BFB15CC491A11DA71035AD@GRFEXC.intern.adiscon.com>
	<20091211140603.GK23781@it.is.rice.edu>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71035AF@GRFEXC.intern.adiscon.com>

I am not sure if you judge the right thing. The way ompgsql is programmed is
that it does a begin transaction ... insert (n times) ... end transaction.
However, each of these statements is sent via a separate call to the postgres
lib. May it be that you are tracing the library calls and not the
transactions?

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Kenneth Marshall
> Sent: Friday, December 11, 2009 3:06 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 5.3.5 (v5-beta) released
> 
> On Fri, Dec 11, 2009 at 01:04:25PM +0100, Rainer Gerhards wrote:
> > if not subscribed to it, revisit bug tracker or forum post, there now
> exist a
> > patch! :)
> >
> > Rainer
> >
> Thank you Rainer. It now logs once more. Unfortunately, I am having
> trouble getting the INSERT batching to work. Here is my current
> rsyslog.conf:
> 
> $ModLoad immark # provides --MARK-- message capability
> $ModLoad imtcp # provides TCP syslog reception and GSS-API (if compiled
> to support it)
> $ModLoad imuxsock
> $InputTCPServerRun 601 # listen on this port
> $WorkDirectory /var/spool/rsyslog # default location for work (spool)
> files
> #$MainMsgQueueFileName mainq  # set file name, also enables disk mode
> $ActionQueueType LinkedList # use asynchronous processing
> $ActionQueueDequeueBatchSize 16
> $ActionQueueFileName dbq    # set file name, also enables disk mode
> $ActionResumeRetryCount -1   # infinite retries on insert failure
> $ModLoad ompgsql # provides postgres support
> # Everything we receive goes in the database
> *.*       :ompgsql:127.0.0.1,Syslog,dbuser,xxxxxx
> 
> The $ActionQueueDequeueBatchSize option does not appear to have
> any effect. The inserts are still performed one at a time. Any
> ideas?
> 
> Regards,
> Ken
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From david at lang.hm  Fri Dec 11 16:24:00 2009
From: david at lang.hm (david at lang.hm)
Date: Fri, 11 Dec 2009 07:24:00 -0800 (PST)
Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released
In-Reply-To: <20091211140603.GK23781@it.is.rice.edu>
References: <20091210180255.GG23781@it.is.rice.edu>
	<9B6E2A8877C38245BFB15CC491A11DA71035AD@GRFEXC.intern.adiscon.com>
	<20091211140603.GK23781@it.is.rice.edu>
Message-ID: <alpine.DEB.2.00.0912110721500.3370@asgard.lang.hm>

On Fri, 11 Dec 2009, Kenneth Marshall wrote:

> On Fri, Dec 11, 2009 at 01:04:25PM +0100, Rainer Gerhards wrote:
>> if not subscribed to it, revisit bug tracker or forum post, there now exist a
>> patch! :)
>>
>> Rainer
>>
> Thank you Rainer. It now logs once more. Unfortunately, I am having
> trouble getting the INSERT batching to work. Here is my current
> rsyslog.conf:

note that rsyslog will only batch if it falls behind. the output module 
basicly does a tight loop:

any messages to process?
yes, grab up to batch size of them, process them and repeat.

if there is only one message in the queue it will process that and loop 
back.

David Lang

> $ModLoad immark # provides --MARK-- message capability
> $ModLoad imtcp # provides TCP syslog reception and GSS-API (if compiled to support it)
> $ModLoad imuxsock
> $InputTCPServerRun 601 # listen on this port
> $WorkDirectory /var/spool/rsyslog # default location for work (spool) files
> #$MainMsgQueueFileName mainq  # set file name, also enables disk mode
> $ActionQueueType LinkedList # use asynchronous processing
> $ActionQueueDequeueBatchSize 16
> $ActionQueueFileName dbq    # set file name, also enables disk mode
> $ActionResumeRetryCount -1   # infinite retries on insert failure
> $ModLoad ompgsql # provides postgres support
> # Everything we receive goes in the database
> *.*       :ompgsql:127.0.0.1,Syslog,dbuser,xxxxxx
>
> The $ActionQueueDequeueBatchSize option does not appear to have
> any effect. The inserts are still performed one at a time. Any
> ideas?
>
> Regards,
> Ken
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From josesan311 at yahoo.com  Fri Dec 11 16:41:15 2009
From: josesan311 at yahoo.com (Jose Sanchez)
Date: Fri, 11 Dec 2009 07:41:15 -0800 (PST)
Subject: [rsyslog] filter logger tags from syslog
In-Reply-To: <alpine.DEB.2.00.0911280141060.18067@asgard.lang.hm>
Message-ID: <723027.49088.qm@web35601.mail.mud.yahoo.com>

Hello again,

Everything is running pretty smooth after the solution applied, I have noted just one thing, my /var/log/messages is caughting all the logs (including the statistics log from apache) on each server and due this the file goes up to 5-6GB every few days, is there anyway to prevent the apache statistics to go to this log file?

This is my syslog.conf

-------------
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*							/dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none		/var/log/messages

# The authpriv file has restricted access.
authpriv.*						/var/log/secure

# Log all the mail messages in one place.
mail.*							-/var/log/maillog


# Log cron stuff
cron.*							/var/log/cron

# Everybody gets emergency messages
*.emerg							*

# Save news errors of level crit and higher in a special file.
uucp,news.crit						/var/log/spooler

# Save boot messages also to boot.log
local7.*						/var/log/boot.log

local1.*    @rsyslog-server.domain.com
-------------

Thank you in advance.


--- On Sat, 11/28/09, david at lang.hm <david at lang.hm> wrote:

> From: david at lang.hm <david at lang.hm>
> Subject: Re: [rsyslog] filter logger tags from syslog
> To: "rsyslog-users" <rsyslog at lists.adiscon.com>
> Date: Saturday, November 28, 2009, 3:43 AM
> On Fri, 27 Nov 2009, Jose Sanchez
> wrote:
> 
> > Hello David and Reiner,
> >
> > First I would like to thank you for all the help
> offered, I was able to setup almost everything because of
> you guys.
> >
> > I had some issues today, though. I found that rsyslog
> was removing the "logger" properly but it was adding an
> extra empty space not sure why so I had to cut if off (by
> watching how to do it on video tutorial first!) by modifying
> the template that David gave me, I currently have it like
> this,
> >
> > $template line,"%msg:2:1000%\n"
> >
> > The thing here is Im not sure if this is a reliable
> solution, I couldnt find if there is any setting that will
> tell rsyslog to simply remove the empty space or to get
> everything until the last letter so I configured a very long
> (1000) number in case rsyslog cuts some part of the text.
> Not sure if there is any negative impact on doing it this
> way, if there is any other better way, please let me know.
> 
> if you use '$' instead of '1000' it will go to the end of
> the message, no 
> matter how long it is (1000 is not long enough for some
> messages)
> 
> I think that what you are doing is probably the best way to
> deal with 
> this space.
> 
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 

From ktm at rice.edu  Fri Dec 11 16:44:39 2009
From: ktm at rice.edu (Kenneth Marshall)
Date: Fri, 11 Dec 2009 09:44:39 -0600
Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released
In-Reply-To: <alpine.DEB.2.00.0912110721500.3370@asgard.lang.hm>
References: <20091210180255.GG23781@it.is.rice.edu>
	<9B6E2A8877C38245BFB15CC491A11DA71035AD@GRFEXC.intern.adiscon.com>
	<20091211140603.GK23781@it.is.rice.edu>
	<alpine.DEB.2.00.0912110721500.3370@asgard.lang.hm>
Message-ID: <20091211154439.GL23781@it.is.rice.edu>

On Fri, Dec 11, 2009 at 07:24:00AM -0800, david at lang.hm wrote:
> On Fri, 11 Dec 2009, Kenneth Marshall wrote:
> 
> > On Fri, Dec 11, 2009 at 01:04:25PM +0100, Rainer Gerhards wrote:
> >> if not subscribed to it, revisit bug tracker or forum post, there now exist a
> >> patch! :)
> >>
> >> Rainer
> >>
> > Thank you Rainer. It now logs once more. Unfortunately, I am having
> > trouble getting the INSERT batching to work. Here is my current
> > rsyslog.conf:
> 
> note that rsyslog will only batch if it falls behind. the output module 
> basicly does a tight loop:
> 
> any messages to process?
> yes, grab up to batch size of them, process them and repeat.
> 
> if there is only one message in the queue it will process that and loop 
> back.
> 
> David Lang
> 

David,

Thank you. That makes sense. I will do some load testing now.

Regards,
Ken


From rgerhards at hq.adiscon.com  Fri Dec 11 16:54:32 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 11 Dec 2009 16:54:32 +0100
Subject: [rsyslog] filter logger tags from syslog
References: <723027.49088.qm@web35601.mail.mud.yahoo.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71035B1@GRFEXC.intern.adiscon.com>

See this, especially "splitting local and remote logging" (especially the
traditional approach)

http://www.rsyslog.com/doc-multi_ruleset.html

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Jose Sanchez
> Sent: Friday, December 11, 2009 4:41 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] filter logger tags from syslog
> 
> Hello again,
> 
> Everything is running pretty smooth after the solution applied, I have
> noted just one thing, my /var/log/messages is caughting all the logs
> (including the statistics log from apache) on each server and due this
> the file goes up to 5-6GB every few days, is there anyway to prevent
> the apache statistics to go to this log file?
> 
> This is my syslog.conf
> 
> -------------
> # Log all kernel messages to the console.
> # Logging much else clutters up the screen.
> #kern.*							/dev/console
> 
> # Log anything (except mail) of level info or higher.
> # Don't log private authentication messages!
> *.info;mail.none;authpriv.none;cron.none		/var/log/messages
> 
> # The authpriv file has restricted access.
> authpriv.*						/var/log/secure
> 
> # Log all the mail messages in one place.
> mail.*
-/var/log/maillog
> 
> 
> # Log cron stuff
> cron.*							/var/log/cron
> 
> # Everybody gets emergency messages
> *.emerg							*
> 
> # Save news errors of level crit and higher in a special file.
> uucp,news.crit
/var/log/spooler
> 
> # Save boot messages also to boot.log
> local7.*						/var/log/boot.log
> 
> local1.*    @rsyslog-server.domain.com
> -------------
> 
> Thank you in advance.
> 
> 
> --- On Sat, 11/28/09, david at lang.hm <david at lang.hm> wrote:
> 
> > From: david at lang.hm <david at lang.hm>
> > Subject: Re: [rsyslog] filter logger tags from syslog
> > To: "rsyslog-users" <rsyslog at lists.adiscon.com>
> > Date: Saturday, November 28, 2009, 3:43 AM
> > On Fri, 27 Nov 2009, Jose Sanchez
> > wrote:
> >
> > > Hello David and Reiner,
> > >
> > > First I would like to thank you for all the help
> > offered, I was able to setup almost everything because of
> > you guys.
> > >
> > > I had some issues today, though. I found that rsyslog
> > was removing the "logger" properly but it was adding an
> > extra empty space not sure why so I had to cut if off (by
> > watching how to do it on video tutorial first!) by modifying
> > the template that David gave me, I currently have it like
> > this,
> > >
> > > $template line,"%msg:2:1000%\n"
> > >
> > > The thing here is Im not sure if this is a reliable
> > solution, I couldnt find if there is any setting that will
> > tell rsyslog to simply remove the empty space or to get
> > everything until the last letter so I configured a very long
> > (1000) number in case rsyslog cuts some part of the text.
> > Not sure if there is any negative impact on doing it this
> > way, if there is any other better way, please let me know.
> >
> > if you use '$' instead of '1000' it will go to the end of
> > the message, no
> > matter how long it is (1000 is not long enough for some
> > messages)
> >
> > I think that what you are doing is probably the best way to
> > deal with
> > this space.
> >
> > David Lang
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From josesan311 at yahoo.com  Fri Dec 11 21:45:34 2009
From: josesan311 at yahoo.com (Jose Sanchez)
Date: Fri, 11 Dec 2009 12:45:34 -0800 (PST)
Subject: [rsyslog] filter logger tags from syslog
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71035B1@GRFEXC.intern.adiscon.com>
Message-ID: <430555.13804.qm@web35607.mail.mud.yahoo.com>

Hello Rainer,

Thank you for the prompt response, the thing is Im running rsyslog on the log server, the other servers where apache is running has the standard syslog daemon running.

Any other hint?

--- On Fri, 12/11/09, Rainer Gerhards <rgerhards at hq.adiscon.com> wrote:

> From: Rainer Gerhards <rgerhards at hq.adiscon.com>
> Subject: Re: [rsyslog] filter logger tags from syslog
> To: "rsyslog-users" <rsyslog at lists.adiscon.com>
> Date: Friday, December 11, 2009, 9:54 AM
> See this, especially "splitting local
> and remote logging" (especially the
> traditional approach)
> 
> http://www.rsyslog.com/doc-multi_ruleset.html
> 
> Rainer
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com
> [mailto:rsyslog-
> > bounces at lists.adiscon.com]
> On Behalf Of Jose Sanchez
> > Sent: Friday, December 11, 2009 4:41 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] filter logger tags from syslog
> > 
> > Hello again,
> > 
> > Everything is running pretty smooth after the solution
> applied, I have
> > noted just one thing, my /var/log/messages is
> caughting all the logs
> > (including the statistics log from apache) on each
> server and due this
> > the file goes up to 5-6GB every few days, is there
> anyway to prevent
> > the apache statistics to go to this log file?
> > 
> > This is my syslog.conf
> > 
> > -------------
> > # Log all kernel messages to the console.
> > # Logging much else clutters up the screen.
> > #kern.*??? ???
> ??? ??? ???
> ??? ??? /dev/console
> > 
> > # Log anything (except mail) of level info or higher.
> > # Don't log private authentication messages!
> >
> *.info;mail.none;authpriv.none;cron.none???
> ??? /var/log/messages
> > 
> > # The authpriv file has restricted access.
> > authpriv.*??? ???
> ??? ??? ???
> ??? /var/log/secure
> > 
> > # Log all the mail messages in one place.
> > mail.*
> -/var/log/maillog
> > 
> > 
> > # Log cron stuff
> > cron.*??? ???
> ??? ??? ???
> ??? ??? /var/log/cron
> > 
> > # Everybody gets emergency messages
> > *.emerg??? ???
> ??? ??? ???
> ??? ??? *
> > 
> > # Save news errors of level crit and higher in a
> special file.
> > uucp,news.crit
> /var/log/spooler
> > 
> > # Save boot messages also to boot.log
> > local7.*??? ???
> ??? ??? ???
> ??? /var/log/boot.log
> > 
> > local1.*? ? @rsyslog-server.domain.com
> > -------------
> > 
> > Thank you in advance.
> > 
> > 
> > --- On Sat, 11/28/09, david at lang.hm <david at lang.hm>
> wrote:
> > 
> > > From: david at lang.hm <david at lang.hm>
> > > Subject: Re: [rsyslog] filter logger tags from
> syslog
> > > To: "rsyslog-users" <rsyslog at lists.adiscon.com>
> > > Date: Saturday, November 28, 2009, 3:43 AM
> > > On Fri, 27 Nov 2009, Jose Sanchez
> > > wrote:
> > >
> > > > Hello David and Reiner,
> > > >
> > > > First I would like to thank you for all the
> help
> > > offered, I was able to setup almost everything
> because of
> > > you guys.
> > > >
> > > > I had some issues today, though. I found
> that rsyslog
> > > was removing the "logger" properly but it was
> adding an
> > > extra empty space not sure why so I had to cut if
> off (by
> > > watching how to do it on video tutorial first!)
> by modifying
> > > the template that David gave me, I currently have
> it like
> > > this,
> > > >
> > > > $template line,"%msg:2:1000%\n"
> > > >
> > > > The thing here is Im not sure if this is a
> reliable
> > > solution, I couldnt find if there is any setting
> that will
> > > tell rsyslog to simply remove the empty space or
> to get
> > > everything until the last letter so I configured
> a very long
> > > (1000) number in case rsyslog cuts some part of
> the text.
> > > Not sure if there is any negative impact on doing
> it this
> > > way, if there is any other better way, please let
> me know.
> > >
> > > if you use '$' instead of '1000' it will go to
> the end of
> > > the message, no
> > > matter how long it is (1000 is not long enough
> for some
> > > messages)
> > >
> > > I think that what you are doing is probably the
> best way to
> > > deal with
> > > this space.
> > >
> > > David Lang
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> > >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 

From josesan311 at yahoo.com  Fri Dec 11 21:47:56 2009
From: josesan311 at yahoo.com (Jose Sanchez)
Date: Fri, 11 Dec 2009 12:47:56 -0800 (PST)
Subject: [rsyslog] filter logger tags from syslog
In-Reply-To: <430555.13804.qm@web35607.mail.mud.yahoo.com>
Message-ID: <619488.48770.qm@web35606.mail.mud.yahoo.com>

Hello,

Disregard my previous email, I got confused with another server. The one having the message file so big is the one that runs rsyslog.

I will look at the suggestion and see how does it run.

Thanks!

--- On Fri, 12/11/09, Jose Sanchez <josesan311 at yahoo.com> wrote:

> From: Jose Sanchez <josesan311 at yahoo.com>
> Subject: Re: [rsyslog] filter logger tags from syslog
> To: "rsyslog-users" <rsyslog at lists.adiscon.com>
> Date: Friday, December 11, 2009, 2:45 PM
> Hello Rainer,
> 
> Thank you for the prompt response, the thing is Im running
> rsyslog on the log server, the other servers where apache is
> running has the standard syslog daemon running.
> 
> Any other hint?
> 
> --- On Fri, 12/11/09, Rainer Gerhards <rgerhards at hq.adiscon.com>
> wrote:
> 
> > From: Rainer Gerhards <rgerhards at hq.adiscon.com>
> > Subject: Re: [rsyslog] filter logger tags from syslog
> > To: "rsyslog-users" <rsyslog at lists.adiscon.com>
> > Date: Friday, December 11, 2009, 9:54 AM
> > See this, especially "splitting local
> > and remote logging" (especially the
> > traditional approach)
> > 
> > http://www.rsyslog.com/doc-multi_ruleset.html
> > 
> > Rainer
> > 
> > > -----Original Message-----
> > > From: rsyslog-bounces at lists.adiscon.com
> > [mailto:rsyslog-
> > > bounces at lists.adiscon.com]
> > On Behalf Of Jose Sanchez
> > > Sent: Friday, December 11, 2009 4:41 PM
> > > To: rsyslog-users
> > > Subject: Re: [rsyslog] filter logger tags from
> syslog
> > > 
> > > Hello again,
> > > 
> > > Everything is running pretty smooth after the
> solution
> > applied, I have
> > > noted just one thing, my /var/log/messages is
> > caughting all the logs
> > > (including the statistics log from apache) on
> each
> > server and due this
> > > the file goes up to 5-6GB every few days, is
> there
> > anyway to prevent
> > > the apache statistics to go to this log file?
> > > 
> > > This is my syslog.conf
> > > 
> > > -------------
> > > # Log all kernel messages to the console.
> > > # Logging much else clutters up the screen.
> > > #kern.*??? ???
> > ??? ??? ???
> > ??? ??? /dev/console
> > > 
> > > # Log anything (except mail) of level info or
> higher.
> > > # Don't log private authentication messages!
> > >
> > *.info;mail.none;authpriv.none;cron.none???
> > ??? /var/log/messages
> > > 
> > > # The authpriv file has restricted access.
> > > authpriv.*??? ???
> > ??? ??? ???
> > ??? /var/log/secure
> > > 
> > > # Log all the mail messages in one place.
> > > mail.*
> > -/var/log/maillog
> > > 
> > > 
> > > # Log cron stuff
> > > cron.*??? ???
> > ??? ??? ???
> > ??? ??? /var/log/cron
> > > 
> > > # Everybody gets emergency messages
> > > *.emerg??? ???
> > ??? ??? ???
> > ??? ??? *
> > > 
> > > # Save news errors of level crit and higher in a
> > special file.
> > > uucp,news.crit
> > /var/log/spooler
> > > 
> > > # Save boot messages also to boot.log
> > > local7.*??? ???
> > ??? ??? ???
> > ??? /var/log/boot.log
> > > 
> > > local1.*? ? @rsyslog-server.domain.com
> > > -------------
> > > 
> > > Thank you in advance.
> > > 
> > > 
> > > --- On Sat, 11/28/09, david at lang.hm <david at lang.hm>
> > wrote:
> > > 
> > > > From: david at lang.hm <david at lang.hm>
> > > > Subject: Re: [rsyslog] filter logger tags
> from
> > syslog
> > > > To: "rsyslog-users" <rsyslog at lists.adiscon.com>
> > > > Date: Saturday, November 28, 2009, 3:43 AM
> > > > On Fri, 27 Nov 2009, Jose Sanchez
> > > > wrote:
> > > >
> > > > > Hello David and Reiner,
> > > > >
> > > > > First I would like to thank you for all
> the
> > help
> > > > offered, I was able to setup almost
> everything
> > because of
> > > > you guys.
> > > > >
> > > > > I had some issues today, though. I
> found
> > that rsyslog
> > > > was removing the "logger" properly but it
> was
> > adding an
> > > > extra empty space not sure why so I had to
> cut if
> > off (by
> > > > watching how to do it on video tutorial
> first!)
> > by modifying
> > > > the template that David gave me, I currently
> have
> > it like
> > > > this,
> > > > >
> > > > > $template line,"%msg:2:1000%\n"
> > > > >
> > > > > The thing here is Im not sure if this
> is a
> > reliable
> > > > solution, I couldnt find if there is any
> setting
> > that will
> > > > tell rsyslog to simply remove the empty
> space or
> > to get
> > > > everything until the last letter so I
> configured
> > a very long
> > > > (1000) number in case rsyslog cuts some part
> of
> > the text.
> > > > Not sure if there is any negative impact on
> doing
> > it this
> > > > way, if there is any other better way,
> please let
> > me know.
> > > >
> > > > if you use '$' instead of '1000' it will go
> to
> > the end of
> > > > the message, no
> > > > matter how long it is (1000 is not long
> enough
> > for some
> > > > messages)
> > > >
> > > > I think that what you are doing is probably
> the
> > best way to
> > > > deal with
> > > > this space.
> > > >
> > > > David Lang
> > > >
> _______________________________________________
> > > > rsyslog mailing list
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com
> > > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> > 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 

From david at lang.hm  Fri Dec 11 21:49:56 2009
From: david at lang.hm (david at lang.hm)
Date: Fri, 11 Dec 2009 12:49:56 -0800 (PST)
Subject: [rsyslog] filter logger tags from syslog
In-Reply-To: <430555.13804.qm@web35607.mail.mud.yahoo.com>
References: <430555.13804.qm@web35607.mail.mud.yahoo.com>
Message-ID: <alpine.DEB.2.00.0912111248540.10307@asgard.lang.hm>

On Fri, 11 Dec 2009, Jose Sanchez wrote:

> Hello Rainer,
>
> Thank you for the prompt response, the thing is Im running rsyslog on 
> the log server, the other servers where apache is running has the 
> standard syslog daemon running.
>
> Any other hint?

you would have to configure it so that the facility you are useing for 
your apache logs does not get logged anywere (eliminate all *. rules and 
replace them with explit listings of every other facility)

David Lang

> --- On Fri, 12/11/09, Rainer Gerhards <rgerhards at hq.adiscon.com> wrote:
>
>> From: Rainer Gerhards <rgerhards at hq.adiscon.com>
>> Subject: Re: [rsyslog] filter logger tags from syslog
>> To: "rsyslog-users" <rsyslog at lists.adiscon.com>
>> Date: Friday, December 11, 2009, 9:54 AM
>> See this, especially "splitting local
>> and remote logging" (especially the
>> traditional approach)
>>
>> http://www.rsyslog.com/doc-multi_ruleset.html
>>
>> Rainer
>>
>>> -----Original Message-----
>>> From: rsyslog-bounces at lists.adiscon.com
>> [mailto:rsyslog-
>>> bounces at lists.adiscon.com]
>> On Behalf Of Jose Sanchez
>>> Sent: Friday, December 11, 2009 4:41 PM
>>> To: rsyslog-users
>>> Subject: Re: [rsyslog] filter logger tags from syslog
>>>
>>> Hello again,
>>>
>>> Everything is running pretty smooth after the solution
>> applied, I have
>>> noted just one thing, my /var/log/messages is
>> caughting all the logs
>>> (including the statistics log from apache) on each
>> server and due this
>>> the file goes up to 5-6GB every few days, is there
>> anyway to prevent
>>> the apache statistics to go to this log file?
>>>
>>> This is my syslog.conf
>>>
>>> -------------
>>> # Log all kernel messages to the console.
>>> # Logging much else clutters up the screen.
>>> #kern.*??? ???
>> ??? ??? ???
>> ??? ??? /dev/console
>>>
>>> # Log anything (except mail) of level info or higher.
>>> # Don't log private authentication messages!
>>>
>> *.info;mail.none;authpriv.none;cron.none???
>> ??? /var/log/messages
>>>
>>> # The authpriv file has restricted access.
>>> authpriv.*??? ???
>> ??? ??? ???
>> ??? /var/log/secure
>>>
>>> # Log all the mail messages in one place.
>>> mail.*
>> -/var/log/maillog
>>>
>>>
>>> # Log cron stuff
>>> cron.*??? ???
>> ??? ??? ???
>> ??? ??? /var/log/cron
>>>
>>> # Everybody gets emergency messages
>>> *.emerg??? ???
>> ??? ??? ???
>> ??? ??? *
>>>
>>> # Save news errors of level crit and higher in a
>> special file.
>>> uucp,news.crit
>> /var/log/spooler
>>>
>>> # Save boot messages also to boot.log
>>> local7.*??? ???
>> ??? ??? ???
>> ??? /var/log/boot.log
>>>
>>> local1.*? ? @rsyslog-server.domain.com
>>> -------------
>>>
>>> Thank you in advance.
>>>
>>>
>>> --- On Sat, 11/28/09, david at lang.hm <david at lang.hm>
>> wrote:
>>>
>>>> From: david at lang.hm <david at lang.hm>
>>>> Subject: Re: [rsyslog] filter logger tags from
>> syslog
>>>> To: "rsyslog-users" <rsyslog at lists.adiscon.com>
>>>> Date: Saturday, November 28, 2009, 3:43 AM
>>>> On Fri, 27 Nov 2009, Jose Sanchez
>>>> wrote:
>>>>
>>>>> Hello David and Reiner,
>>>>>
>>>>> First I would like to thank you for all the
>> help
>>>> offered, I was able to setup almost everything
>> because of
>>>> you guys.
>>>>>
>>>>> I had some issues today, though. I found
>> that rsyslog
>>>> was removing the "logger" properly but it was
>> adding an
>>>> extra empty space not sure why so I had to cut if
>> off (by
>>>> watching how to do it on video tutorial first!)
>> by modifying
>>>> the template that David gave me, I currently have
>> it like
>>>> this,
>>>>>
>>>>> $template line,"%msg:2:1000%\n"
>>>>>
>>>>> The thing here is Im not sure if this is a
>> reliable
>>>> solution, I couldnt find if there is any setting
>> that will
>>>> tell rsyslog to simply remove the empty space or
>> to get
>>>> everything until the last letter so I configured
>> a very long
>>>> (1000) number in case rsyslog cuts some part of
>> the text.
>>>> Not sure if there is any negative impact on doing
>> it this
>>>> way, if there is any other better way, please let
>> me know.
>>>>
>>>> if you use '$' instead of '1000' it will go to
>> the end of
>>>> the message, no
>>>> matter how long it is (1000 is not long enough
>> for some
>>>> messages)
>>>>
>>>> I think that what you are doing is probably the
>> best way to
>>>> deal with
>>>> this space.
>>>>
>>>> David Lang
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From josesan311 at yahoo.com  Tue Dec 15 02:11:51 2009
From: josesan311 at yahoo.com (Jose Sanchez)
Date: Mon, 14 Dec 2009 17:11:51 -0800 (PST)
Subject: [rsyslog] filter logger tags from syslog
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71035B1@GRFEXC.intern.adiscon.com>
Message-ID: <150551.24393.qm@web35608.mail.mud.yahoo.com>

Hello,

Thank you for the resolution, is there anyway I can specify just one rule for doing this? The thing is I have more than 50 lines specifying ":programname, isequal, .." so I would like to know if there is other better solution rather than adding "& ~" below every line.
I tried adding "& ~" at the very botton but it didnt work.

Many thanks.

--- On Fri, 12/11/09, Rainer Gerhards <rgerhards at hq.adiscon.com> wrote:

> From: Rainer Gerhards <rgerhards at hq.adiscon.com>
> Subject: Re: [rsyslog] filter logger tags from syslog
> To: "rsyslog-users" <rsyslog at lists.adiscon.com>
> Date: Friday, December 11, 2009, 9:54 AM
> See this, especially "splitting local
> and remote logging" (especially the
> traditional approach)
> 
> http://www.rsyslog.com/doc-multi_ruleset.html
> 
> Rainer
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com
> [mailto:rsyslog-
> > bounces at lists.adiscon.com]
> On Behalf Of Jose Sanchez
> > Sent: Friday, December 11, 2009 4:41 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] filter logger tags from syslog
> > 
> > Hello again,
> > 
> > Everything is running pretty smooth after the solution
> applied, I have
> > noted just one thing, my /var/log/messages is
> caughting all the logs
> > (including the statistics log from apache) on each
> server and due this
> > the file goes up to 5-6GB every few days, is there
> anyway to prevent
> > the apache statistics to go to this log file?
> > 
> > This is my syslog.conf
> > 
> > -------------
> > # Log all kernel messages to the console.
> > # Logging much else clutters up the screen.
> > #kern.*??? ???
> ??? ??? ???
> ??? ??? /dev/console
> > 
> > # Log anything (except mail) of level info or higher.
> > # Don't log private authentication messages!
> >
> *.info;mail.none;authpriv.none;cron.none???
> ??? /var/log/messages
> > 
> > # The authpriv file has restricted access.
> > authpriv.*??? ???
> ??? ??? ???
> ??? /var/log/secure
> > 
> > # Log all the mail messages in one place.
> > mail.*
> -/var/log/maillog
> > 
> > 
> > # Log cron stuff
> > cron.*??? ???
> ??? ??? ???
> ??? ??? /var/log/cron
> > 
> > # Everybody gets emergency messages
> > *.emerg??? ???
> ??? ??? ???
> ??? ??? *
> > 
> > # Save news errors of level crit and higher in a
> special file.
> > uucp,news.crit
> /var/log/spooler
> > 
> > # Save boot messages also to boot.log
> > local7.*??? ???
> ??? ??? ???
> ??? /var/log/boot.log
> > 
> > local1.*? ? @rsyslog-server.domain.com
> > -------------
> > 
> > Thank you in advance.
> > 
> > 
> > --- On Sat, 11/28/09, david at lang.hm <david at lang.hm>
> wrote:
> > 
> > > From: david at lang.hm <david at lang.hm>
> > > Subject: Re: [rsyslog] filter logger tags from
> syslog
> > > To: "rsyslog-users" <rsyslog at lists.adiscon.com>
> > > Date: Saturday, November 28, 2009, 3:43 AM
> > > On Fri, 27 Nov 2009, Jose Sanchez
> > > wrote:
> > >
> > > > Hello David and Reiner,
> > > >
> > > > First I would like to thank you for all the
> help
> > > offered, I was able to setup almost everything
> because of
> > > you guys.
> > > >
> > > > I had some issues today, though. I found
> that rsyslog
> > > was removing the "logger" properly but it was
> adding an
> > > extra empty space not sure why so I had to cut if
> off (by
> > > watching how to do it on video tutorial first!)
> by modifying
> > > the template that David gave me, I currently have
> it like
> > > this,
> > > >
> > > > $template line,"%msg:2:1000%\n"
> > > >
> > > > The thing here is Im not sure if this is a
> reliable
> > > solution, I couldnt find if there is any setting
> that will
> > > tell rsyslog to simply remove the empty space or
> to get
> > > everything until the last letter so I configured
> a very long
> > > (1000) number in case rsyslog cuts some part of
> the text.
> > > Not sure if there is any negative impact on doing
> it this
> > > way, if there is any other better way, please let
> me know.
> > >
> > > if you use '$' instead of '1000' it will go to
> the end of
> > > the message, no
> > > matter how long it is (1000 is not long enough
> for some
> > > messages)
> > >
> > > I think that what you are doing is probably the
> best way to
> > > deal with
> > > this space.
> > >
> > > David Lang
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> > >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 

From oleg.myrk at gmail.com  Fri Dec 18 16:38:37 2009
From: oleg.myrk at gmail.com (=?ISO-8859-1?Q?Oleg_M=FCrk?=)
Date: Fri, 18 Dec 2009 17:38:37 +0200
Subject: [rsyslog] Help needed: rsyslog with ommail hang on network outage
Message-ID: <1dd3af250912180738j55b64cc3m622bb54faa50c0c3@mail.gmail.com>

Hello,

I am running rsyslog 3.18.0 provided with Debian 5.0 (lenny/stable)
using the default configuration + ommail configuration (see below).
When SMTP server is not available (either dns name mistyped or network
outage), running command
	logger Pattern.something
that should result in email being sent, hangs and almost everything
else becomes unresponsive (cron, sshd).

I do realize that it has something to do with lack of async queueing
by default, but I don't understand why logging commands just hang
seemingly forever.
The configuration itself is as follows:
	$ModLoad ommail
	$template mailSubject,"RSYSLOG: %msg%"
	$ActionMailSMTPServer smtp.server.com
	$ActionMailFrom something at gmail.com
	$ActionMailSubject mailSubject
	$ActionMailEnableBody off
	$ActionExecOnlyOnceEveryInterval 900

	$ActionMailTo somebody at gmail.com
	if $msg contains 'Pattern.' then :ommail:
	.... # Quite many more such rules

My aim is to have rsyslog configured so that when SMTP server is
unavailable, corresponding mails just don't get delivered, and first
of all to make sure the whole system does not hang/slow down because
of this. As I have quite many actions sending emails I'd rather not
configure per-action queue.

For now I have resolved the problem by adding:
	$WorkDirectory /var/log/queue
	$MainMsgQueueType LinkedList
	$MainMsgQueueFileName Main
	$MainMsgQueueMaxDiskSpace 1g
	$MainMsgQueueSaveOnShutdown on
But I am not sure if this is the best solution and why there is a
problem in the first place.

Thanks!
Oleg M?rk

From david at lang.hm  Sun Dec 20 00:10:39 2009
From: david at lang.hm (david at lang.hm)
Date: Sat, 19 Dec 2009 15:10:39 -0800 (PST)
Subject: [rsyslog] Help needed: rsyslog with ommail hang on network
 outage
In-Reply-To: <1dd3af250912180738j55b64cc3m622bb54faa50c0c3@mail.gmail.com>
References: <1dd3af250912180738j55b64cc3m622bb54faa50c0c3@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.0912191454211.5587@asgard.lang.hm>

On Fri, 18 Dec 2009, Oleg M?rk wrote:

> Hello,
>
> I am running rsyslog 3.18.0 provided with Debian 5.0 (lenny/stable)
> using the default configuration + ommail configuration (see below).
> When SMTP server is not available (either dns name mistyped or network
> outage), running command
> 	logger Pattern.something
> that should result in email being sent, hangs and almost everything
> else becomes unresponsive (cron, sshd).

rsyslog does do async queuing by default, but only up until the point 
where the queue is full. At that point it defaults to blocking until there 
is more space in the queue.

see 'Discarding Messages' on http://www.rsyslog.com/doc-queues.html for 
infor on how to set rsyslog to throw away messages when it runs out of 
space instead of blocking.

David Lang


> I do realize that it has something to do with lack of async queueing
> by default, but I don't understand why logging commands just hang
> seemingly forever.
> The configuration itself is as follows:
> 	$ModLoad ommail
> 	$template mailSubject,"RSYSLOG: %msg%"
> 	$ActionMailSMTPServer smtp.server.com
> 	$ActionMailFrom something at gmail.com
> 	$ActionMailSubject mailSubject
> 	$ActionMailEnableBody off
> 	$ActionExecOnlyOnceEveryInterval 900
>
> 	$ActionMailTo somebody at gmail.com
> 	if $msg contains 'Pattern.' then :ommail:
> 	.... # Quite many more such rules
>
> My aim is to have rsyslog configured so that when SMTP server is
> unavailable, corresponding mails just don't get delivered, and first
> of all to make sure the whole system does not hang/slow down because
> of this. As I have quite many actions sending emails I'd rather not
> configure per-action queue.
>
> For now I have resolved the problem by adding:
> 	$WorkDirectory /var/log/queue
> 	$MainMsgQueueType LinkedList
> 	$MainMsgQueueFileName Main
> 	$MainMsgQueueMaxDiskSpace 1g
> 	$MainMsgQueueSaveOnShutdown on
> But I am not sure if this is the best solution and why there is a
> problem in the first place.
>
> Thanks!
> Oleg M?rk
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From oleg.myrk at gmail.com  Sun Dec 20 01:04:58 2009
From: oleg.myrk at gmail.com (=?ISO-8859-1?Q?Oleg_M=FCrk?=)
Date: Sun, 20 Dec 2009 02:04:58 +0200
Subject: [rsyslog] Help needed: rsyslog with ommail hang on network
	outage
In-Reply-To: <alpine.DEB.2.00.0912191454211.5587@asgard.lang.hm>
References: <1dd3af250912180738j55b64cc3m622bb54faa50c0c3@mail.gmail.com>
	<alpine.DEB.2.00.0912191454211.5587@asgard.lang.hm>
Message-ID: <1dd3af250912191604t7b1601fct6e50bb6012ddff91@mail.gmail.com>

Hello,

On Sun, Dec 20, 2009 at 1:10 AM,  <david at lang.hm> wrote:
> rsyslog does do async queuing by default, but only up until the point
> where the queue is full. At that point it defaults to blocking until there
> is more space in the queue.

Thanks for the answer. But why does ommail hang when SMTP server is
unavailable? If I understand correclty with ActionResumeRetryCount
equal by default to 0 it should timeout and then give up sending the
message?

Thanks,
Oleg

From david at lang.hm  Sun Dec 20 01:08:54 2009
From: david at lang.hm (david at lang.hm)
Date: Sat, 19 Dec 2009 16:08:54 -0800 (PST)
Subject: [rsyslog] Help needed: rsyslog with ommail hang on network
 outage
In-Reply-To: <1dd3af250912191604t7b1601fct6e50bb6012ddff91@mail.gmail.com>
References: <1dd3af250912180738j55b64cc3m622bb54faa50c0c3@mail.gmail.com>
	<alpine.DEB.2.00.0912191454211.5587@asgard.lang.hm>
	<1dd3af250912191604t7b1601fct6e50bb6012ddff91@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.0912191607520.5587@asgard.lang.hm>

On Sun, 20 Dec 2009, Oleg M?rk wrote:

> Hello,
>
> On Sun, Dec 20, 2009 at 1:10 AM,  <david at lang.hm> wrote:
>> rsyslog does do async queuing by default, but only up until the point
>> where the queue is full. At that point it defaults to blocking until there
>> is more space in the queue.
>
> Thanks for the answer. But why does ommail hang when SMTP server is
> unavailable? If I understand correclty with ActionResumeRetryCount
> equal by default to 0 it should timeout and then give up sending the
> message?

I don't know. I haven't messed with the retry/fail mechanisms.

however, remember that it takes time to fail, during that time more 
messages arrive, if some of those try to send messages they then take time 
to fail......

David Lang

From rgerhards at hq.adiscon.com  Sun Dec 20 12:45:52 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Sun, 20 Dec 2009 12:45:52 +0100
Subject: [rsyslog] Help needed: rsyslog with ommail hang on network
	outage
References: <1dd3af250912180738j55b64cc3m622bb54faa50c0c3@mail.gmail.com><alpine.DEB.2.00.0912191454211.5587@asgard.lang.hm><1dd3af250912191604t7b1601fct6e50bb6012ddff91@mail.gmail.com>
	<alpine.DEB.2.00.0912191607520.5587@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103605@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Sunday, December 20, 2009 1:09 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Help needed: rsyslog with ommail hang on network
> outage
> 
> On Sun, 20 Dec 2009, Oleg M?rk wrote:
> 
> > Hello,
> >
> > On Sun, Dec 20, 2009 at 1:10 AM,  <david at lang.hm> wrote:
> >> rsyslog does do async queuing by default, but only up until the
> point
> >> where the queue is full. At that point it defaults to blocking until
> there
> >> is more space in the queue.
> >
> > Thanks for the answer. But why does ommail hang when SMTP server is
> > unavailable? If I understand correclty with ActionResumeRetryCount
> > equal by default to 0 it should timeout and then give up sending the
> > message?
> 
> I don't know. I haven't messed with the retry/fail mechanisms.
> 
> however, remember that it takes time to fail, during that time more
> messages arrive, if some of those try to send messages they then take
> time
> to fail......

I will soon be out for xmas break (until early January) and have some
assignments before I go, so I can unfortunately not look into this case.
However, I guess that the connection to the SMTP server hangs, that is the
connection setup does neither abort nor succeed so that the connect() hangs -
which is beyond the control of rsyslog core's retry mechanism. Other than
that, I have no good clue so far. A debug log may help, but I can't promise
to look at it before mid-january.

Rainer

> 
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From oleg.myrk at gmail.com  Mon Dec 21 13:54:27 2009
From: oleg.myrk at gmail.com (=?ISO-8859-1?Q?Oleg_M=FCrk?=)
Date: Mon, 21 Dec 2009 14:54:27 +0200
Subject: [rsyslog] Help needed: rsyslog with ommail hang on network
	outage
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103605@GRFEXC.intern.adiscon.com>
References: <1dd3af250912180738j55b64cc3m622bb54faa50c0c3@mail.gmail.com>
	<alpine.DEB.2.00.0912191454211.5587@asgard.lang.hm>
	<1dd3af250912191604t7b1601fct6e50bb6012ddff91@mail.gmail.com>
	<alpine.DEB.2.00.0912191607520.5587@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7103605@GRFEXC.intern.adiscon.com>
Message-ID: <1dd3af250912210454r1380c63fuc21e4ae26b6c77f6@mail.gmail.com>

Hello,

Thank You all for the answers. For now I have resolved the problem by
adding:
       $WorkDirectory /var/log/queue
       $MainMsgQueueType LinkedList
       $MainMsgQueueFileName Main
       $MainMsgQueueMaxDiskSpace 1g
       $MainMsgQueueSaveOnShutdown on

Now I would also like to add
       $MainQueueTimeoutEnqueue 0
but pages
   http://www.rsyslog.com/doc-rsyslog_conf_global.html
   http://www.rsyslog.com/doc-queues.html
give contradictory meaning for value 0. Does this setting mean indefinite or
immediate timeout?

Best,
Oleg

On Sun, Dec 20, 2009 at 1:45 PM, Rainer Gerhards
<rgerhards at hq.adiscon.com>wrote:

> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> > Sent: Sunday, December 20, 2009 1:09 AM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] Help needed: rsyslog with ommail hang on network
> > outage
> >
> > On Sun, 20 Dec 2009, Oleg M?rk wrote:
> >
> > > Hello,
> > >
> > > On Sun, Dec 20, 2009 at 1:10 AM,  <david at lang.hm> wrote:
> > >> rsyslog does do async queuing by default, but only up until the
> > point
> > >> where the queue is full. At that point it defaults to blocking until
> > there
> > >> is more space in the queue.
> > >
> > > Thanks for the answer. But why does ommail hang when SMTP server is
> > > unavailable? If I understand correclty with ActionResumeRetryCount
> > > equal by default to 0 it should timeout and then give up sending the
> > > message?
> >
> > I don't know. I haven't messed with the retry/fail mechanisms.
> >
> > however, remember that it takes time to fail, during that time more
> > messages arrive, if some of those try to send messages they then take
> > time
> > to fail......
>
> I will soon be out for xmas break (until early January) and have some
> assignments before I go, so I can unfortunately not look into this case.
> However, I guess that the connection to the SMTP server hangs, that is the
> connection setup does neither abort nor succeed so that the connect() hangs
> -
> which is beyond the control of rsyslog core's retry mechanism. Other than
> that, I have no good clue so far. A debug log may help, but I can't promise
> to look at it before mid-january.
>
> Rainer
>
> >
> > David Lang
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From rory at ooma.com  Mon Dec 21 21:01:01 2009
From: rory at ooma.com (Rory Toma)
Date: Mon, 21 Dec 2009 12:01:01 -0800
Subject: [rsyslog] problems with tls
Message-ID: <4B2FD3FD.3090100@ooma.com>

We're still having end to end problems with tls. It got better when I 
switch from rsyslog to syslog-ng as the backend, but the clients still 
seem to have issues.

Do you know if anyone offers professional consulting services on this 
that we can throw money at?

thx

From rgerhards at hq.adiscon.com  Tue Dec 22 07:47:32 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 22 Dec 2009 07:47:32 +0100
Subject: [rsyslog] problems with tls
References: <4B2FD3FD.3090100@ooma.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103619@GRFEXC.intern.adiscon.com>

We offer professional services ourselves:

http://www.rsyslog.com/Article412.phtml

Just please note that I will be off for winter break starting tomorrow (for
this issue, it would probably make sense if I looked into it myself).

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rory Toma
> Sent: Monday, December 21, 2009 9:01 PM
> To: rsyslog-users
> Subject: [rsyslog] problems with tls
> 
> We're still having end to end problems with tls. It got better when I
> switch from rsyslog to syslog-ng as the backend, but the clients still
> seem to have issues.
> 
> Do you know if anyone offers professional consulting services on this
> that we can throw money at?
> 
> thx
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Tue Dec 22 18:44:52 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 22 Dec 2009 18:44:52 +0100
Subject: [rsyslog] seasons greetings to everyone
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710363F@GRFEXC.intern.adiscon.com>

Hi all,

I would like to use the opportunity and wish all of you the very best for the
holiday season and the new year!

Rsyslog has seen an area of tremendous growth in 2009. Not only could we add
many enterprise (and other) features, we also saw more and more Linux
distributions switching to rsyslog as the default syslogd. All large
distributions (except Suse) either today use rsyslog or will do so in the
next future!  This is very good news.

Also good news is that the community is growing. There are more people than
ever that help others on the mailing list, and some begin to regularly show
up in the support forums on the web. We had some request for help with new
features from "third parties", which will hopefully manifest in some good
announcements in 2010. Also, we saw our first commercial sponsorships, which
was a great relief to me, showing that the enterprise space is also moving
into our directions.

Of course, there is lots left to do. With growing community help, I am very
positive 2010 will be a great year as well. A stats engine plus - finally - a
new config format is on my "schedule" for 2010 and I guess a lot of new
features will also come in. I promise to keep up maintaining and enhancing
the project. Feature-wise, I guess, 2010 will have fewer additions than we
saw in 2009, simply because there are already so many features and getting
them used turns out to be a more important focus than adding "just that extra
bell and whistle". Getting them used means more testing (thanks to all who
reported bugs!), better doc and tutorials and, to re-iterate, a more usable
config syntax. But, as usual, I may end up many more new features than I now
envision ;)

I am sure that 2010 will be as exciting for the rsyslog community as 2009
was. My sincere thanks to everyone who helped make that happen. Writing some
software is one thing - thorough testing, bug reporting, and help to users is
another three ;) The community help is very important to get that part done,
so that I can focus on enhancing and improving the code itself. Once again
for all the help in this regard! 

Oh, yes, in case you wonder why I write all this on December, 22nd: I'll be
heading to my holiday vacation tomorrow morning and will probably not be
online until early January. So I need to be a bit early ;)

My warmest regards and thanks,
Rainer 

From sjain at silverspringnet.com  Wed Dec 23 04:53:34 2009
From: sjain at silverspringnet.com (Siddhartha Jain)
Date: Tue, 22 Dec 2009 19:53:34 -0800
Subject: [rsyslog] "Field not Found" crash
Message-ID: <3A240503F9F2194780469F072D9A70541162F7CD@m342.silverspringnet.com>

I am running two machines, a relay and a collecter on CentOS 5.2 x64
with 5.2.0 code.

The relay sends logs with this formatting:
$template tplSiteID,"<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME%
%syslogtag:1:32%,mxxx-relay,%msg%"

The collectors parses it with this expression:
$template
percustID,"/var/rsyslog/logs/%msg:F,44:2:%/%hostname%-%programname%.log"

Due to bad formatting by client machines running sysklogd, the collector
crashes at "repeated messages" lines with this output:
-----xxxxxxxxxxxx---------
7569.669488000:426f7940: msg parser: flags 30, from 'mxxx.abc.corp.com',
msg '<30>2009-12-22T16:19:29.668288-08:00 last message,mxxx-relay,
repeated 24 times'
7569.669611000:426f7940: Message has legacy syslog format.
7569.669717000:43af9940: hasRcvInBuffer on nsd 0x2aaaac039910: pszRcvBuf
(nil), lenRcvBuf 0
7569.669819000:426f7940: Called action, logging to builtin-file
7569.669970000:426f7940: submitBatch: i:0, batch size 1, to process 1,
pMsg: 0x2aaaac047070, state 0
7569.670071000:426f7940: ../action.c:736: actionProcessMessage: inside
actionProcessMsg()
7569.670152000:426f7940: Action 0x1b378cb0 transitioned to state: itx
7569.670316000:426f7940: entering actionCalldoAction(), state: itx
7569.670414000:426f7940: field requested 2, field found 1
7569.670524000:426f7940: file to log to: percustID
7569.670582000:426f7940: Added new entry 5 for file cache, file
'/var/rsyslog/logs/**FIELD NOT FOUND*/last-message,mxxx-relay,.log'.
7569.670628000:426f7940: doWrite, pData->pStrm 0x1b3a5450, lenBuf 76
7569.670733000:426f7940: strm 0x1b3a5450: file -1 flush, buflen 76
7569.670816000:43af9940: hasRcvInBuffer on nsd 0x2aaaac03cbf0: pszRcvBuf
(nil), lenRcvBuf 0
7569.670944000:426f7940: strm 0x1b3a5450: open error 2, file
'/var/rsyslog/logs/**FIELD NOT FOUND*/last-message,mxxx-relay,.log'
7569.671107000:426f7940: action call returned -2040
7569.671170000:426f7940: Action 0x1b378cb0 transitioned to state: rdy
7569.671277000:43af9940: Segmentation fault (core dumped)
-----xxxxxxxxxxxx---------

To patch the situation, I tried to replace "HOSTNAME" with "fromhost"
but that also caused crashes. Eventually, I replaced it with
"fromhost-ip" as a temporary fix but is there a more elegant solution to
take care of errant clients?


Happy Holidays and thanks for all the great work, Rainer!



Thanks,

Siddhartha

From david at lang.hm  Wed Dec 23 06:38:18 2009
From: david at lang.hm (david at lang.hm)
Date: Tue, 22 Dec 2009 21:38:18 -0800 (PST)
Subject: [rsyslog] "Field not Found" crash
In-Reply-To: <3A240503F9F2194780469F072D9A70541162F7CD@m342.silverspringnet.com>
References: <3A240503F9F2194780469F072D9A70541162F7CD@m342.silverspringnet.com>
Message-ID: <alpine.DEB.2.00.0912222132370.24717@asgard.lang.hm>

On Tue, 22 Dec 2009, Siddhartha Jain wrote:

> I am running two machines, a relay and a collecter on CentOS 5.2 x64
> with 5.2.0 code.

5.2.0 has many known (and fixed in later versions) problems. it really 
should not be used at this point (can someone from adiscon make a note on 
the download page for it?)

unfortunantly none of the later 5.x versions have been promoted to stable, 
but you really are better off with the latest devel (5.5.1 at this point) 
rather than 5.2.0

David Lang

> The relay sends logs with this formatting:
> $template tplSiteID,"<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME%
> %syslogtag:1:32%,mxxx-relay,%msg%"
>
> The collectors parses it with this expression:
> $template
> percustID,"/var/rsyslog/logs/%msg:F,44:2:%/%hostname%-%programname%.log"
>
> Due to bad formatting by client machines running sysklogd, the collector
> crashes at "repeated messages" lines with this output:
> -----xxxxxxxxxxxx---------
> 7569.669488000:426f7940: msg parser: flags 30, from 'mxxx.abc.corp.com',
> msg '<30>2009-12-22T16:19:29.668288-08:00 last message,mxxx-relay,
> repeated 24 times'
> 7569.669611000:426f7940: Message has legacy syslog format.
> 7569.669717000:43af9940: hasRcvInBuffer on nsd 0x2aaaac039910: pszRcvBuf
> (nil), lenRcvBuf 0
> 7569.669819000:426f7940: Called action, logging to builtin-file
> 7569.669970000:426f7940: submitBatch: i:0, batch size 1, to process 1,
> pMsg: 0x2aaaac047070, state 0
> 7569.670071000:426f7940: ../action.c:736: actionProcessMessage: inside
> actionProcessMsg()
> 7569.670152000:426f7940: Action 0x1b378cb0 transitioned to state: itx
> 7569.670316000:426f7940: entering actionCalldoAction(), state: itx
> 7569.670414000:426f7940: field requested 2, field found 1
> 7569.670524000:426f7940: file to log to: percustID
> 7569.670582000:426f7940: Added new entry 5 for file cache, file
> '/var/rsyslog/logs/**FIELD NOT FOUND*/last-message,mxxx-relay,.log'.
> 7569.670628000:426f7940: doWrite, pData->pStrm 0x1b3a5450, lenBuf 76
> 7569.670733000:426f7940: strm 0x1b3a5450: file -1 flush, buflen 76
> 7569.670816000:43af9940: hasRcvInBuffer on nsd 0x2aaaac03cbf0: pszRcvBuf
> (nil), lenRcvBuf 0
> 7569.670944000:426f7940: strm 0x1b3a5450: open error 2, file
> '/var/rsyslog/logs/**FIELD NOT FOUND*/last-message,mxxx-relay,.log'
> 7569.671107000:426f7940: action call returned -2040
> 7569.671170000:426f7940: Action 0x1b378cb0 transitioned to state: rdy
> 7569.671277000:43af9940: Segmentation fault (core dumped)
> -----xxxxxxxxxxxx---------
>
> To patch the situation, I tried to replace "HOSTNAME" with "fromhost"
> but that also caused crashes. Eventually, I replaced it with
> "fromhost-ip" as a temporary fix but is there a more elegant solution to
> take care of errant clients?
>
>
> Happy Holidays and thanks for all the great work, Rainer!
>
>
>
> Thanks,
>
> Siddhartha
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From kenneho.ndu at gmail.com  Wed Dec 23 09:12:40 2009
From: kenneho.ndu at gmail.com (Kenneth Holter)
Date: Wed, 23 Dec 2009 09:12:40 +0100
Subject: [rsyslog] rsyslog+stunnel works only when running "rsyslogd" from
	the shell
Message-ID: <c25f25140912230012s5570db5r9c1959ab0f2d5b5e@mail.gmail.com>

Hi.


I'm running rsyslog v2.0.6 provided with my RHEL 5 installation. For some
time now I've had rsyslog issues with some of my RHEL 5 servers, and I've
not been able to figure out the problems, and would like to hear from others
that may have experienced the same problem. I've been in contact with Red
Hat support, but they've not been able to reproduce this problem, so we'be
not succeeded in resolving the issue.

First, let me describe my setup: My RHEL 5 servers have set up a TLS tunnel
(using stunnel) between themselves and the log host. This works perfectly.
I've configured rsyslog to forward messages to this tunnel by adding a "
*.*  @@127.0.0.1:61514 " line to the bottom of /etc/rsyslog.conf file. The
stunnel is listening on port 61514.

On almost all my servers, this works as planned. But for some reason, a few
servers are having problems forwarding messages to their stunnel connection.
By running "tcpdump -i lo" I can see that these servers are not transmitting
anything on the loopback interface, and are thus not forwarding anything to
the stunnel port. One of my theories was that the line above simply wasn't
picked up by rsyslog daemon. So I stopped the daemon, ran "rsyslogd -d" to
view the debug output, and everthing works fine.

For some reason, when I run rsyslog like this (i.e by issuing "rsyslogd" in
the command prompt) instead of issuing "/etc/init.d/rsyslog start",
everything work fine. I'm really puzzled as to why this is so. Does anyone
know why this is so? I have the exact same setup one all my servers, but one
a small number of them have this problem.


Best regards,
Kenneth Holter

From r.bhatia at ipax.at  Wed Dec 23 09:58:08 2009
From: r.bhatia at ipax.at (Raoul Bhatia [IPAX])
Date: Wed, 23 Dec 2009 09:58:08 +0100
Subject: [rsyslog] seasons greetings to everyone
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710363F@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA710363F@GRFEXC.intern.adiscon.com>
Message-ID: <4B31DBA0.6030509@ipax.at>

On 12/22/2009 06:44 PM, Rainer Gerhards wrote:
> Hi all,
> 
> I would like to use the opportunity and wish all of you the very best for the
> holiday season and the new year!
> 
> Rsyslog has seen an area of tremendous growth in 2009. Not only could we add
> many enterprise (and other) features, we also saw more and more Linux
> distributions switching to rsyslog as the default syslogd. All large
> distributions (except Suse) either today use rsyslog or will do so in the
> next future!  This is very good news.

[...]

Dear Rainer,

thank you for all your hard work you have put into this wonderful
product! Without your everlasting efforts, rsyslog wouldn't be near
it's current position!

Enjoy your vacation,
Raoul
-- 
____________________________________________________________________
DI (FH) Raoul Bhatia M.Sc.          email.          r.bhatia at ipax.at
Technischer Leiter

IPAX - Aloy Bhatia Hava OEG         web.          http://www.ipax.at
Barawitzkagasse 10/2/2/11           email.            office at ipax.at
1190 Wien                           tel.               +43 1 3670030
FN 277995t HG Wien                  fax.            +43 1 3670030 15
____________________________________________________________________

From sjain at silverspringnet.com  Wed Dec 23 21:59:24 2009
From: sjain at silverspringnet.com (Siddhartha Jain)
Date: Wed, 23 Dec 2009 12:59:24 -0800
Subject: [rsyslog] rsyslog+stunnel works only when running "rsyslogd"
	fromthe shell
In-Reply-To: <c25f25140912230012s5570db5r9c1959ab0f2d5b5e@mail.gmail.com>
References: <c25f25140912230012s5570db5r9c1959ab0f2d5b5e@mail.gmail.com>
Message-ID: <3A240503F9F2194780469F072D9A70541162FF2B@m342.silverspringnet.com>

Kenneth,

Not sure why RedHat/CentOS continue to bundle rsyslog 2.0.6. This
version is ancient. Since 2.x, rsyslog has gone through 2.x, 4.x and now
the current, 5.x.

I would highly recommend rolling your own RPM from recent 5.x or 4.x
code.

- Siddhartha


> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Kenneth Holter
> Sent: Wednesday, December 23, 2009 12:13 AM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] rsyslog+stunnel works only when running "rsyslogd"
> fromthe shell
> 
> Hi.
> 
> 
> I'm running rsyslog v2.0.6 provided with my RHEL 5 installation. For
> some
> time now I've had rsyslog issues with some of my RHEL 5 servers, and
> I've
> not been able to figure out the problems, and would like to hear from
> others
> that may have experienced the same problem. I've been in contact with
> Red
> Hat support, but they've not been able to reproduce this problem, so
> we'be
> not succeeded in resolving the issue.
> 
> First, let me describe my setup: My RHEL 5 servers have set up a TLS
> tunnel
> (using stunnel) between themselves and the log host. This works
> perfectly.
> I've configured rsyslog to forward messages to this tunnel by adding a
> "
> *.*  @@127.0.0.1:61514 " line to the bottom of /etc/rsyslog.conf file.
> The
> stunnel is listening on port 61514.
> 
> On almost all my servers, this works as planned. But for some reason,
a
> few
> servers are having problems forwarding messages to their stunnel
> connection.
> By running "tcpdump -i lo" I can see that these servers are not
> transmitting
> anything on the loopback interface, and are thus not forwarding
> anything to
> the stunnel port. One of my theories was that the line above simply
> wasn't
> picked up by rsyslog daemon. So I stopped the daemon, ran "rsyslogd
-d"
> to
> view the debug output, and everthing works fine.
> 
> For some reason, when I run rsyslog like this (i.e by issuing
> "rsyslogd" in
> the command prompt) instead of issuing "/etc/init.d/rsyslog start",
> everything work fine. I'm really puzzled as to why this is so. Does
> anyone
> know why this is so? I have the exact same setup one all my servers,
> but one
> a small number of them have this problem.
> 
> 
> Best regards,
> Kenneth Holter
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From sjain at silverspringnet.com  Wed Dec 23 22:01:16 2009
From: sjain at silverspringnet.com (Siddhartha Jain)
Date: Wed, 23 Dec 2009 13:01:16 -0800
Subject: [rsyslog] "Field not Found" crash
In-Reply-To: <alpine.DEB.2.00.0912222132370.24717@asgard.lang.hm>
References: <3A240503F9F2194780469F072D9A70541162F7CD@m342.silverspringnet.com>
	<alpine.DEB.2.00.0912222132370.24717@asgard.lang.hm>
Message-ID: <3A240503F9F2194780469F072D9A70541162FF34@m342.silverspringnet.com>

Thanks for that note, David. I will put 5.5.1 through our test
environment.

- Siddhartha


> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, December 22, 2009 9:38 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] "Field not Found" crash
> 
> On Tue, 22 Dec 2009, Siddhartha Jain wrote:
> 
> > I am running two machines, a relay and a collecter on CentOS 5.2 x64
> > with 5.2.0 code.
> 
> 5.2.0 has many known (and fixed in later versions) problems. it really
> should not be used at this point (can someone from adiscon make a note
> on
> the download page for it?)
> 
> unfortunantly none of the later 5.x versions have been promoted to
> stable,
> but you really are better off with the latest devel (5.5.1 at this
> point)
> rather than 5.2.0
> 
> David Lang
> 
> > The relay sends logs with this formatting:
> > $template tplSiteID,"<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME%
> > %syslogtag:1:32%,mxxx-relay,%msg%"
> >
> > The collectors parses it with this expression:
> > $template
> > percustID,"/var/rsyslog/logs/%msg:F,44:2:%/%hostname%-
> %programname%.log"
> >
> > Due to bad formatting by client machines running sysklogd, the
> collector
> > crashes at "repeated messages" lines with this output:
> > -----xxxxxxxxxxxx---------
> > 7569.669488000:426f7940: msg parser: flags 30, from
> 'mxxx.abc.corp.com',
> > msg '<30>2009-12-22T16:19:29.668288-08:00 last message,mxxx-relay,
> > repeated 24 times'
> > 7569.669611000:426f7940: Message has legacy syslog format.
> > 7569.669717000:43af9940: hasRcvInBuffer on nsd 0x2aaaac039910:
> pszRcvBuf
> > (nil), lenRcvBuf 0
> > 7569.669819000:426f7940: Called action, logging to builtin-file
> > 7569.669970000:426f7940: submitBatch: i:0, batch size 1, to process
> 1,
> > pMsg: 0x2aaaac047070, state 0
> > 7569.670071000:426f7940: ../action.c:736: actionProcessMessage:
> inside
> > actionProcessMsg()
> > 7569.670152000:426f7940: Action 0x1b378cb0 transitioned to state:
itx
> > 7569.670316000:426f7940: entering actionCalldoAction(), state: itx
> > 7569.670414000:426f7940: field requested 2, field found 1
> > 7569.670524000:426f7940: file to log to: percustID
> > 7569.670582000:426f7940: Added new entry 5 for file cache, file
> > '/var/rsyslog/logs/**FIELD NOT FOUND*/last-message,mxxx-relay,.log'.
> > 7569.670628000:426f7940: doWrite, pData->pStrm 0x1b3a5450, lenBuf 76
> > 7569.670733000:426f7940: strm 0x1b3a5450: file -1 flush, buflen 76
> > 7569.670816000:43af9940: hasRcvInBuffer on nsd 0x2aaaac03cbf0:
> pszRcvBuf
> > (nil), lenRcvBuf 0
> > 7569.670944000:426f7940: strm 0x1b3a5450: open error 2, file
> > '/var/rsyslog/logs/**FIELD NOT FOUND*/last-message,mxxx-relay,.log'
> > 7569.671107000:426f7940: action call returned -2040
> > 7569.671170000:426f7940: Action 0x1b378cb0 transitioned to state:
rdy
> > 7569.671277000:43af9940: Segmentation fault (core dumped)
> > -----xxxxxxxxxxxx---------
> >
> > To patch the situation, I tried to replace "HOSTNAME" with
"fromhost"
> > but that also caused crashes. Eventually, I replaced it with
> > "fromhost-ip" as a temporary fix but is there a more elegant
solution
> to
> > take care of errant clients?
> >
> >
> > Happy Holidays and thanks for all the great work, Rainer!
> >
> >
> >
> > Thanks,
> >
> > Siddhartha
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From mbiebl at gmail.com  Thu Dec 24 12:12:05 2009
From: mbiebl at gmail.com (Michael Biebl)
Date: Thu, 24 Dec 2009 12:12:05 +0100
Subject: [rsyslog] seasons greetings to everyone
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710363F@GRFEXC.intern.adiscon.com>
References: <AcqDLncwhK8PpGj4RZ2ew8uC4ED9JA==>
	<9B6E2A8877C38245BFB15CC491A11DA710363F@GRFEXC.intern.adiscon.com>
Message-ID: <aa76cdac0912240312r55f1124fle5711dabbb540986@mail.gmail.com>

2009/12/22 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Hi all,
>
> I would like to use the opportunity and wish all of you the very best for the
> holiday season and the new year!

Dear Rainer and everyone reading the list,

have a merry christmas and a happy new year.

You eared your days off :-)

Michael


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

From joe at joetify.com  Thu Dec 31 00:43:57 2009
From: joe at joetify.com (Joe Williams)
Date: Wed, 30 Dec 2009 15:43:57 -0800
Subject: [rsyslog] shell execute parameters
Message-ID: <4B3BE5BD.8080904@joetify.com>


I am using shell execute 
(http://www.rsyslog.com/doc-rsyslog_conf_actions.html) to send emails 
from rsyslog. Other than the template string is there any way to send 
other stuff to the script?

Here's how I'm using it:

:msg,contains,"has no server available" 
^/usr/local/bin/rsyslog_mailer;PerHostHaproxyError

If possible I would like to send the entire :msg (line in the log) where 
the text is found to the script as well.

Thanks.

-Joe

-- 
Name: Joseph A. Williams
Email: joe at joetify.com
Blog: http://www.joeandmotorboat.com/


From david at lang.hm  Thu Dec 31 00:53:10 2009
From: david at lang.hm (david at lang.hm)
Date: Wed, 30 Dec 2009 15:53:10 -0800 (PST)
Subject: [rsyslog] shell execute parameters
In-Reply-To: <4B3BE5BD.8080904@joetify.com>
References: <4B3BE5BD.8080904@joetify.com>
Message-ID: <alpine.DEB.2.00.0912301550160.24130@asgard.lang.hm>

On Wed, 30 Dec 2009, Joe Williams wrote:

> I am using shell execute
> (http://www.rsyslog.com/doc-rsyslog_conf_actions.html) to send emails
> from rsyslog. Other than the template string is there any way to send
> other stuff to the script?
>
> Here's how I'm using it:
>
> :msg,contains,"has no server available"
> ^/usr/local/bin/rsyslog_mailer;PerHostHaproxyError
>
> If possible I would like to send the entire :msg (line in the log) where
> the text is found to the script as well.

no, you would need to do this with the template string

however, you can do LOTS of very interesting things with the template 
string

for example, when I was having trouble figuring out why a log was parsing 
inforrectly I had a format string something like

"\n%raw%\n%fromhost-ip%\n%hostname%\n%fromhost%"

which would output a blank line, followed by a line containing the entire 
message, followed by three lines showing the values of those three fields

with multi-line output like this you can easily send lots of different 
things to your script and have it handle it easily.

David Lang

From joe at joetify.com  Thu Dec 31 01:19:30 2009
From: joe at joetify.com (Joe Williams)
Date: Wed, 30 Dec 2009 16:19:30 -0800
Subject: [rsyslog] shell execute parameters
In-Reply-To: <alpine.DEB.2.00.0912301550160.24130@asgard.lang.hm>
References: <4B3BE5BD.8080904@joetify.com>
	<alpine.DEB.2.00.0912301550160.24130@asgard.lang.hm>
Message-ID: <4B3BEE12.2010800@joetify.com>

Ah, very cool. So would I need to do something like the following?

$template HaproxyMailer,"\n%raw\n"
local1.err -?HaproxyMailer
:msg,contains,"has no server available" 
^/usr/local/bin/rsyslog_mailer;HaproxyMailer

Thanks.

-Joe



On 12/30/09 3:53 PM, david at lang.hm wrote:
> On Wed, 30 Dec 2009, Joe Williams wrote:
>
>    
>> I am using shell execute
>> (http://www.rsyslog.com/doc-rsyslog_conf_actions.html) to send emails
>> from rsyslog. Other than the template string is there any way to send
>> other stuff to the script?
>>
>> Here's how I'm using it:
>>
>> :msg,contains,"has no server available"
>> ^/usr/local/bin/rsyslog_mailer;PerHostHaproxyError
>>
>> If possible I would like to send the entire :msg (line in the log) where
>> the text is found to the script as well.
>>      
> no, you would need to do this with the template string
>
> however, you can do LOTS of very interesting things with the template
> string
>
> for example, when I was having trouble figuring out why a log was parsing
> inforrectly I had a format string something like
>
> "\n%raw%\n%fromhost-ip%\n%hostname%\n%fromhost%"
>
> which would output a blank line, followed by a line containing the entire
> message, followed by three lines showing the values of those three fields
>
> with multi-line output like this you can easily send lots of different
> things to your script and have it handle it easily.
>
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>    

-- 
Name: Joseph A. Williams
Email: joe at joetify.com
Blog: http://www.joeandmotorboat.com/