From mbiebl at gmail.com  Sat Dec  1 00:14:35 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Sat, 1 Dec 2007 00:14:35 +0100
Subject: [rsyslog] gss-api support for rsyslog
In-Reply-To: <473B2961.9010705@redhat.com>
References: <473B2961.9010705@redhat.com>
Message-ID: <aa76cdac0711301514x711721a7i2767e8a99e175f79@mail.gmail.com>

2007/11/14, theinric at redhat.com <theinric at redhat.com>:
> Hi,
>
> here is a patch that enables rsyslog to use gss-api for authentication
> and encryption.

Hi,

ist seems not all of the patch has made it into CVS. The part in
Makefile.am is missing, so the compilation fails. Please apply the
(inlined) patch ;-)

Index: Makefile.am
===================================================================
RCS file: /cvsroot/rsyslog/rsyslog/Makefile.am,v
retrieving revision 1.36
diff -u -3 -p -r1.36 Makefile.am
--- Makefile.am 30 Nov 2007 13:43:12 -0000      1.36
+++ Makefile.am 30 Nov 2007 23:10:27 -0000
@@ -63,7 +63,7 @@ rsyslogd_SOURCES = \
        gss-misc.h

 rsyslogd_CPPFLAGS = -D_PATH_MODDIR=\"$(pkglibdir)/\" $(pthreads_cflags)
-rsyslogd_LDADD = $(zlib_libs) $(pthreads_libs) -ldl
+rsyslogd_LDADD = $(zlib_libs) $(pthreads_libs) $(gss_libs) -ldl
 rsyslogd_LDFLAGS = -export-dynamic

 man_MANS = rfc3195d.8 rklogd.8 rsyslogd.8 rsyslog.conf.5



Cheers,
Michael
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From sur5r at sur5r.net  Sat Dec  1 22:36:44 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Sat, 1 Dec 2007 22:36:44 +0100
Subject: [rsyslog] [patch] output module for PostgreSQL
Message-ID: <20071201223644.036dccd6@marathon.haufe>

Hi all!

As already mentioned in the forum: I've created an output module for
PostgreSQL.

The attached patch does the following:

* add plugins/ompgsql/*
* extend configure.ac and Makefile.am
* msg.h
** struct msg_t: add pszTIMESTAMP_PgSQL, add pszRcvdAt_PgSQL, fix typo
* msg.c
** getTimeGenerated(): add handling of tplFmtPgSQLDate
** getTimeReported(): add handling of tplFmtPgSQLDate
* syslogd.h:
** add definition of formatTimestampToPgSQL()
* syslogd.c:
** add StdPgSQLFmt
** mainThread(): include StdPgSQLFmt
* template.h:
** add tplFmtPgSQLDate to tplFormatTypes
* template.c:
** doOptions(): date-pgsql -> tplFmtPgSQLDate
** tplPrintList(): add tplFmtPgSQLDate to switch()

Please note that the patch has not yet been tested thoroughly, so until
that has been done, it is to be considered alpha quality.

If anyone wants to try it: any feedback is appreciated.

Regards,
sur5r

P.S.: I hope the patch survives mailman... ;)
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: rsyslog_ompgsql.diff
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20071201/3543bb53/attachment.txt>

From sur5r at sur5r.net  Sat Dec  1 23:01:26 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Sat, 1 Dec 2007 23:01:26 +0100
Subject: [rsyslog] [patch] output module for PostgreSQL
References: <20071201223644.036dccd6@marathon.haufe>
Message-ID: <20071201230126.3537f120@marathon.haufe>

Jakob Haufe <sur5r at sur5r.net> wrote on Sat, 1 Dec 2007 22:36:44 +0100
in 20071201223644.036dccd6 at marathon.haufe:

> The attached patch does the following:
> 
> * add plugins/ompgsql/*

Erm, yes. Maybe i should have noticed, that cvs doesn't recurse into
new subdirectories... so here are the files for the plugins/ompgsql
directory.

From sur5r at sur5r.net  Sat Dec  1 23:10:11 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Sat, 1 Dec 2007 23:10:11 +0100
Subject: [rsyslog] [patch] output module for PostgreSQL
References: <20071201223644.036dccd6@marathon.haufe>
	<20071201230126.3537f120@marathon.haufe>
Message-ID: <20071201231011.5a27a361@marathon.haufe>

Jakob Haufe <sur5r at sur5r.net> wrote on Sat, 1 Dec 2007 23:01:26 +0100
in 20071201230126.3537f120 at marathon.haufe:

> Jakob Haufe <sur5r at sur5r.net> wrote on Sat, 1 Dec 2007 22:36:44 +0100
> in 20071201223644.036dccd6 at marathon.haufe:
> 
> > The attached patch does the following:
> > 
> > * add plugins/ompgsql/*
> 
> Erm, yes. Maybe i should have noticed, that cvs doesn't recurse into
> new subdirectories... so here are the files for the plugins/ompgsql
> directory.
Ok, mailman doesn't like any mimetypes but text/plain... next try *grml*
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: Makefile.am
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20071201/0f6fc89b/attachment.asc>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: createDB.sql
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20071201/0f6fc89b/attachment.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: ompgsql.c
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20071201/0f6fc89b/attachment-0001.asc>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: ompgsql.h
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20071201/0f6fc89b/attachment-0001.txt>

From mbiebl at gmail.com  Sun Dec  2 23:10:07 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Sun, 2 Dec 2007 23:10:07 +0100
Subject: [rsyslog] [patch] output module for PostgreSQL
In-Reply-To: <20071201231011.5a27a361@marathon.haufe>
References: <20071201223644.036dccd6@marathon.haufe>
	<20071201230126.3537f120@marathon.haufe>
	<20071201231011.5a27a361@marathon.haufe>
Message-ID: <aa76cdac0712021410l2a218cftb5a6883c38bacd32@mail.gmail.com>

2007/12/1, Jakob Haufe <sur5r at sur5r.net>:
> Jakob Haufe <sur5r at sur5r.net> wrote on Sat, 1 Dec 2007 23:01:26 +0100
> in 20071201230126.3537f120 at marathon.haufe:
>
> > Jakob Haufe <sur5r at sur5r.net> wrote on Sat, 1 Dec 2007 22:36:44 +0100
> > in 20071201223644.036dccd6 at marathon.haufe:
> >
> > > The attached patch does the following:
> > >
> > > * add plugins/ompgsql/*
> >
> > Erm, yes. Maybe i should have noticed, that cvs doesn't recurse into
> > new subdirectories... so here are the files for the plugins/ompgsql
> > directory.
> Ok, mailman doesn't like any mimetypes but text/plain... next try *grml*

Hi,

just a very minor nitpick in Makefile.am:
pkglibdir is by default defined to be $(libdir)/@PACKAGE@/ [1], so
there is no need to redefine it. Simply delete the first line.
plugins/ommysql/Makefile.am should be fixed accordingly.

It also seems that the configure.ac bits are missing.

Cheers,
Michael

[1] http://www.gnu.org/software/automake/manual/html_node/Uniform.html#Uniform
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Mon Dec  3 08:10:44 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 3 Dec 2007 08:10:44 +0100
Subject: [rsyslog] [patch] output module for PostgreSQL
In-Reply-To: <aa76cdac0712021410l2a218cftb5a6883c38bacd32@mail.gmail.com>
References: <20071201223644.036dccd6@marathon.haufe><20071201230126.3537f120@marathon.haufe><20071201231011.5a27a361@marathon.haufe>
	<aa76cdac0712021410l2a218cftb5a6883c38bacd32@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA30856B@grfint2.intern.adiscon.com>

Hi,

thanks to all for the great contributions! Let me just quickly outline
my plan, so that nobody is disappointed ;) I'll publish a new 1.19.12
release today. However, that will just hold some cleanup and fixes. When
I have done that, I'll begin to merge the patches into the
already-modified 1.20.0. I am not yet sure how much issues I will see
because there are a number of code changes in that release. I am also
not sure yet in which order I apply the patches, because one might be
better than the other. I'll announce when I have merged and will
probably post a test tarball here on the list before officially
releasing. My target for release is mid to end of week.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Sunday, December 02, 2007 11:10 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] [patch] output module for PostgreSQL
> 
> 2007/12/1, Jakob Haufe <sur5r at sur5r.net>:
> > Jakob Haufe <sur5r at sur5r.net> wrote on Sat, 1 Dec 2007 23:01:26
+0100
> > in 20071201230126.3537f120 at marathon.haufe:
> >
> > > Jakob Haufe <sur5r at sur5r.net> wrote on Sat, 1 Dec 2007 22:36:44
> +0100
> > > in 20071201223644.036dccd6 at marathon.haufe:
> > >
> > > > The attached patch does the following:
> > > >
> > > > * add plugins/ompgsql/*
> > >
> > > Erm, yes. Maybe i should have noticed, that cvs doesn't recurse
> into
> > > new subdirectories... so here are the files for the
plugins/ompgsql
> > > directory.
> > Ok, mailman doesn't like any mimetypes but text/plain... next try
> *grml*
> 
> Hi,
> 
> just a very minor nitpick in Makefile.am:
> pkglibdir is by default defined to be $(libdir)/@PACKAGE@/ [1], so
> there is no need to redefine it. Simply delete the first line.
> plugins/ommysql/Makefile.am should be fixed accordingly.
> 
> It also seems that the configure.ac bits are missing.
> 
> Cheers,
> Michael
> 
> [1]
>
http://www.gnu.org/software/automake/manual/html_node/Uniform.html#Unif
> orm
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Mon Dec  3 10:01:24 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 3 Dec 2007 10:01:24 +0100
Subject: [rsyslog] rsyslog 1.19.12 released
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308570@grfint2.intern.adiscon.com>

As promised this morning, I have just released 1.19.12. It is a cleanup
release, no need to install it if you do not experience any issues.
Users of 1.19.11 using mysql should update, as the new -pthread switch
was not applied to ommysql in 1.19.11.

Download:

   http://www.rsyslog.com/Downloads-req-getit-lid-64.phtml

As always, feedback is appreciated.

Best regards,
Rainer Gerhards

Rsyslog homepage: http://www.rsyslog.com
My blog: http://rgerhards.blogspot.com



From sur5r at sur5r.net  Mon Dec  3 10:37:48 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Mon, 3 Dec 2007 10:37:48 +0100
Subject: [rsyslog] [patch] output module for PostgreSQL
References: <20071201223644.036dccd6@marathon.haufe>
	<20071201230126.3537f120@marathon.haufe>
	<20071201231011.5a27a361@marathon.haufe>
	<aa76cdac0712021410l2a218cftb5a6883c38bacd32@mail.gmail.com>
Message-ID: <20071203103748.781bfba2@mp-atlantis3.informatik.uni-mannheim.de>

On Sun, 2 Dec 2007 23:10:07 +0100
"Michael Biebl" <mbiebl at gmail.com> wrote:

> just a very minor nitpick in Makefile.am:
> pkglibdir is by default defined to be $(libdir)/@PACKAGE@/ [1], so
> there is no need to redefine it. Simply delete the first line.
> plugins/ommysql/Makefile.am should be fixed accordingly.

Thanks for pointing this out. Unfortunately, I have no in-depth
knowledge of autotools, so I'm thankful for any suggestions.

> It also seems that the configure.ac bits are missing.

They are there, though hard to find. 

MSGID 20071201223644.036dccd6 at marathon.haufe
contains the patch modifying existing files, namely:
msg.[ch], syslogd.[ch], template.[ch], configure.ac, Makefile.am

MSGID 20071201231011.5a27a361 at marathon.haufe
contains the files for plugins/ompgsql/*

Regards,
Jakob



From sur5r at sur5r.net  Mon Dec  3 11:09:12 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Mon, 3 Dec 2007 11:09:12 +0100
Subject: [rsyslog] [patch] output module for PostgreSQL
References: <20071201223644.036dccd6@marathon.haufe>
	<20071201230126.3537f120@marathon.haufe>
	<20071201231011.5a27a361@marathon.haufe>
	<aa76cdac0712021410l2a218cftb5a6883c38bacd32@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA30856B@grfint2.intern.adiscon.com>
Message-ID: <20071203110912.50657ea6@mp-atlantis3.informatik.uni-mannheim.de>

On Mon, 3 Dec 2007 08:10:44 +0100
"Rainer Gerhards" <rgerhards at hq.adiscon.com> wrote:

> thanks to all for the great contributions! Let me just quickly outline
> my plan, so that nobody is disappointed ;) I'll publish a new 1.19.12
> release today. However, that will just hold some cleanup and fixes.
> When I have done that, I'll begin to merge the patches into the
> already-modified 1.20.0. I am not yet sure how much issues I will see
> because there are a number of code changes in that release. I am also
> not sure yet in which order I apply the patches, because one might be
> better than the other. I'll announce when I have merged and will
> probably post a test tarball here on the list before officially
> releasing. My target for release is mid to end of week.

That's perfectly ok for me. But I definitely would appreciate any
feedback on ompgsql before officially releasing it. I do not expect any
problems with it but you never know...

So if anybody is going to test it, please give feedback.

Regards,
sur5r



From rgerhards at hq.adiscon.com  Mon Dec  3 11:11:25 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 3 Dec 2007 11:11:25 +0100
Subject: [rsyslog] [patch] output module for PostgreSQL
In-Reply-To: <20071203110912.50657ea6@mp-atlantis3.informatik.uni-mannheim.de>
References: <20071201223644.036dccd6@marathon.haufe><20071201230126.3537f120@marathon.haufe><20071201231011.5a27a361@marathon.haufe><aa76cdac0712021410l2a218cftb5a6883c38bacd32@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA30856B@grfint2.intern.adiscon.com>
	<20071203110912.50657ea6@mp-atlantis3.informatik.uni-mannheim.de>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308574@grfint2.intern.adiscon.com>

Just a quick update. I am right now integrating it and will review code
during this and once I am through. As I suspected, I need to do at least
some manual merging, as the source tree already changed and in some
places considerably (talking about 1.20.0!). So it would be good if you
could-re-test it once I can release an early test tarball.

I keep you updated.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Jakob Haufe
> Sent: Monday, December 03, 2007 11:09 AM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] [patch] output module for PostgreSQL
> 
> On Mon, 3 Dec 2007 08:10:44 +0100
> "Rainer Gerhards" <rgerhards at hq.adiscon.com> wrote:
> 
> > thanks to all for the great contributions! Let me just quickly
> outline
> > my plan, so that nobody is disappointed ;) I'll publish a new
1.19.12
> > release today. However, that will just hold some cleanup and fixes.
> > When I have done that, I'll begin to merge the patches into the
> > already-modified 1.20.0. I am not yet sure how much issues I will
see
> > because there are a number of code changes in that release. I am
also
> > not sure yet in which order I apply the patches, because one might
be
> > better than the other. I'll announce when I have merged and will
> > probably post a test tarball here on the list before officially
> > releasing. My target for release is mid to end of week.
> 
> That's perfectly ok for me. But I definitely would appreciate any
> feedback on ompgsql before officially releasing it. I do not expect
any
> problems with it but you never know...
> 
> So if anybody is going to test it, please give feedback.
> 
> Regards,
> sur5r
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From mbiebl at gmail.com  Tue Dec  4 03:58:21 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Tue, 4 Dec 2007 03:58:21 +0100
Subject: [rsyslog] [PATCH] Fix latest "NetBSD fix"
Message-ID: <aa76cdac0712031858w52c6c63bq529b7496cb8a5cf3@mail.gmail.com>

Hi,

the latest NetBSD related commits (configure.ac/Makefile.am) break the
compilation on Linux. Variable assignments mustn't have whitespace,
otherwise have_libdl and as a result HAVE_LIBDL is not defined.

To fix the issue though, I'd simply recommend to remove the hard coded
-ldl from Makefile.am instead of the proposed solution.
The AC_CHECK_LIB macro [2] already appends -ldl to LIBS, depending on
the outcome of the check [2]. The global LIBS variable (similar to
CFLAGS e.g.) is used for linking all binaries.

Linked (see, I learned my lesson :-p) is the patch [1] to address this.

The only downside of AC_CHECK_LIB is, that it assigns -ldl (and -lrt)
to the global LIBS variable. This means that all binaries are linked
with -ldl -lrt, as mentioned above, even if they don't need them (i.e.
ommysql/ompsql plugin,  rklogd rfc3195d).

I can prepare a patch to address this issue correctly, if wanted.

Cheers,
Michael

[1] http://debs.michaelbiebl.de/rsyslog/fix_dl_compilation.patch
[2] http://www.gnu.org/software/automake/manual/autoconf/Libraries.html

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Tue Dec  4 11:15:57 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 04 Dec 2007 11:15:57 +0100
Subject: [rsyslog] [PATCH] Fix latest "NetBSD fix"
In-Reply-To: <aa76cdac0712031858w52c6c63bq529b7496cb8a5cf3@mail.gmail.com>
References: <aa76cdac0712031858w52c6c63bq529b7496cb8a5cf3@mail.gmail.com>
Message-ID: <1196763357.12493.6.camel@localhost.localdomain>

Hi Michael,

many thanks for the information. Actually, you were quicker than me ;) I
worked with Bartosz on the issue yesterday, but needed to leave before I
could finalize it.

I would appreciate if you could provide the "big patch" you wrote about.
That will ensure that autotools is used in the best possible way :)

Rainer

On Tue, 2007-12-04 at 03:58 +0100, Michael Biebl wrote:
> Hi,
> 
> the latest NetBSD related commits (configure.ac/Makefile.am) break the
> compilation on Linux. Variable assignments mustn't have whitespace,
> otherwise have_libdl and as a result HAVE_LIBDL is not defined.
> 
> To fix the issue though, I'd simply recommend to remove the hard coded
> -ldl from Makefile.am instead of the proposed solution.
> The AC_CHECK_LIB macro [2] already appends -ldl to LIBS, depending on
> the outcome of the check [2]. The global LIBS variable (similar to
> CFLAGS e.g.) is used for linking all binaries.
> 
> Linked (see, I learned my lesson :-p) is the patch [1] to address this.
> 
> The only downside of AC_CHECK_LIB is, that it assigns -ldl (and -lrt)
> to the global LIBS variable. This means that all binaries are linked
> with -ldl -lrt, as mentioned above, even if they don't need them (i.e.
> ommysql/ompsql plugin,  rklogd rfc3195d).
> 
> I can prepare a patch to address this issue correctly, if wanted.
> 
> Cheers,
> Michael
> 
> [1] http://debs.michaelbiebl.de/rsyslog/fix_dl_compilation.patch
> [2] http://www.gnu.org/software/automake/manual/autoconf/Libraries.html
> 


From rgerhards at hq.adiscon.com  Tue Dec  4 11:23:26 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 4 Dec 2007 11:23:26 +0100
Subject: [rsyslog] rsyslog 1.20.0 preview available
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308596@grfint2.intern.adiscon.com>

Hi all,

there is a preview of rsyslog 1.20.0 available at

http://download.rsyslog.com/rsyslog/rsyslog-1.20.0.tar.gz

It obviously does not contains Michael Biebl's upcoming autotools patch,
but should be fairly well. I would be most interested in feedback from
the field, especially on the new postgres module by sur5r (thanks
again!). I plan to release end of week if now show stoppers show up.

Please note that I have disabled GSS-API in a rough way: you can simply
not select it in configure. This is because GSS-API support does need
some more work or it could become an interop problem in the future. So
we decided to push it back until that is solved.

Feedback is highly appreciated. If I updated the tarball, I'll announce
that. Oh, yes, doc is not finished yet, so do not wonder...

Rainer


From sur5r at sur5r.net  Tue Dec  4 14:00:14 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Tue, 4 Dec 2007 14:00:14 +0100
Subject: [rsyslog] rsyslog 1.20.0 preview available
References: <577465F99B41C842AAFBE9ED71E70ABA308596@grfint2.intern.adiscon.com>
Message-ID: <20071204140014.7c2187b2@mp-atlantis3.informatik.uni-mannheim.de>

On Tue, 4 Dec 2007 11:23:26 +0100
"Rainer Gerhards" <rgerhards at hq.adiscon.com> wrote:

> Feedback is highly appreciated. If I updated the tarball, I'll
> announce that. Oh, yes, doc is not finished yet, so do not wonder...

Just a small thing I already mentioned on the forum but unfortunately
got overlooked:

ommysql, line 236: p+=sizeof(":ommysql:");

This adds increments the pointer one byte too far (trailing \0) and
thus eats the first byte of the hostname part.

It's a bit unfortunate that strlen() doesn't count the \0 while
sizeof() does...

Regards,
sur5r



From rgerhards at hq.adiscon.com  Tue Dec  4 14:10:06 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 4 Dec 2007 14:10:06 +0100
Subject: [rsyslog] rsyslog 1.20.0 preview available
In-Reply-To: <20071204140014.7c2187b2@mp-atlantis3.informatik.uni-mannheim.de>
References: <577465F99B41C842AAFBE9ED71E70ABA308596@grfint2.intern.adiscon.com>
	<20071204140014.7c2187b2@mp-atlantis3.informatik.uni-mannheim.de>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA30859F@grfint2.intern.adiscon.com>

Indeed, thanks for pointing it out. Its fixed now, in cvs and I also
re-released the tarball (just for completeness).

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Jakob Haufe
> Sent: Tuesday, December 04, 2007 2:00 PM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] rsyslog 1.20.0 preview available
> 
> On Tue, 4 Dec 2007 11:23:26 +0100
> "Rainer Gerhards" <rgerhards at hq.adiscon.com> wrote:
> 
> > Feedback is highly appreciated. If I updated the tarball, I'll
> > announce that. Oh, yes, doc is not finished yet, so do not wonder...
> 
> Just a small thing I already mentioned on the forum but unfortunately
> got overlooked:
> 
> ommysql, line 236: p+=sizeof(":ommysql:");
> 
> This adds increments the pointer one byte too far (trailing \0) and
> thus eats the first byte of the hostname part.
> 
> It's a bit unfortunate that strlen() doesn't count the \0 while
> sizeof() does...
> 
> Regards,
> sur5r
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From stephen.carville at gmail.com  Fri Dec  7 22:09:18 2007
From: stephen.carville at gmail.com (Stephen Carville)
Date: Fri, 7 Dec 2007 13:09:18 -0800
Subject: [rsyslog] Last Message repeated N times
Message-ID: <2428c0380712071309w31f4fc20qb933dfef47f445ac@mail.gmail.com>

I am using rsyslog 1.19.10 on a central server to collect messages and
store them in a mysql database.  One thing I've noticed is that the
"last message repeated NNN times" entries are alway entered with
FromHost="last", SysLogTag="message", and Message="repeated NNN
times".  Can this be fixed?

I still use the regular sysklogd package on the remote machines to
forward the messages.

-- 
Stephen Carville


From rgerhards at hq.adiscon.com  Tue Dec 11 11:06:00 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 11 Dec 2007 11:06:00 +0100
Subject: [rsyslog] Asking for help...
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA30862F@grfint2.intern.adiscon.com>

Hi folks,

I have a design question with upcoming rsyslog changes and could not
even find an appropriate place to ask. I have blogged about it, so the
information is available to everyone:

http://rgerhards.blogspot.com/2007/12/design-problem.html

If you happen to know a good place to ask (or even know a solution), I
would really appreciate to hear about that!

Thanks
Rainer


From rgerhards at hq.adiscon.com  Tue Dec 11 12:01:07 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 11 Dec 2007 12:01:07 +0100
Subject: [rsyslog] Last Message repeated N times
In-Reply-To: <2428c0380712071309w31f4fc20qb933dfef47f445ac@mail.gmail.com>
References: <2428c0380712071309w31f4fc20qb933dfef47f445ac@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308637@grfint2.intern.adiscon.com>

Hi Stephen,

I was totally swamped the past days, sorry for the missing response.
I'll check what causes this. As a side-note, I think that the message
reduction code does not work well with the database - as you can no
longer query the individual messages. Is this your intent?

Rainer 

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Stephen Carville
> Sent: Friday, December 07, 2007 10:09 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] Last Message repeated N times
> 
> I am using rsyslog 1.19.10 on a central server to collect messages and
> store them in a mysql database.  One thing I've noticed is that the
> "last message repeated NNN times" entries are alway entered with
> FromHost="last", SysLogTag="message", and Message="repeated NNN
> times".  Can this be fixed?
> 
> I still use the regular sysklogd package on the remote machines to
> forward the messages.
> 
> --
> Stephen Carville
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Tue Dec 11 12:26:32 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 11 Dec 2007 12:26:32 +0100
Subject: [rsyslog] Last Message repeated N times
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308637@grfint2.intern.adiscon.com>
References: <2428c0380712071309w31f4fc20qb933dfef47f445ac@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA308637@grfint2.intern.adiscon.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA30863D@grfint2.intern.adiscon.com>

I think I misunderstood your problem... The "last message repeated n
times" stems back from the reporting sysklogd's - right? Mhhh... that's
a problem. I need to do some hardcoded processing in the parser to cover
part of it. The problem is that sysklogd emits a totally non-standard
message in that case. Even worse, I think it does not even contain a
clue at the missing information. I could probably use the sender's IP
address as source in  that case, but the tag seems to be impossible to
obtain.

Can you post me a copy of the %rawmsg% property, so that I can have a
look at what exactly you are seing?

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Tuesday, December 11, 2007 12:01 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Last Message repeated N times
> 
> Hi Stephen,
> 
> I was totally swamped the past days, sorry for the missing response.
> I'll check what causes this. As a side-note, I think that the message
> reduction code does not work well with the database - as you can no
> longer query the individual messages. Is this your intent?
> 
> Rainer
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Stephen Carville
> > Sent: Friday, December 07, 2007 10:09 PM
> > To: rsyslog at lists.adiscon.com
> > Subject: [rsyslog] Last Message repeated N times
> >
> > I am using rsyslog 1.19.10 on a central server to collect messages
> and
> > store them in a mysql database.  One thing I've noticed is that the
> > "last message repeated NNN times" entries are alway entered with
> > FromHost="last", SysLogTag="message", and Message="repeated NNN
> > times".  Can this be fixed?
> >
> > I still use the regular sysklogd package on the remote machines to
> > forward the messages.
> >
> > --
> > Stephen Carville
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From sur5r at sur5r.net  Tue Dec 11 14:48:15 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Tue, 11 Dec 2007 14:48:15 +0100
Subject: [rsyslog] Asking for help...
References: <577465F99B41C842AAFBE9ED71E70ABA30862F@grfint2.intern.adiscon.com>
Message-ID: <20071211144815.6a2ef123@marathon.haufe>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

"Rainer Gerhards" <rgerhards at hq.adiscon.com> wrote on Tue, 11 Dec 2007
11:06:00 +0100:

> http://rgerhards.blogspot.com/2007/12/design-problem.html

Maybe I'm a bit short-sighted here, but isn't this exactly the situation
for non-blocking sockets? Or will that raise other problems?

Regards,
sur5r
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHXpUj1YAhDic+adYRAnBZAKCWNXoWIKHhnhm/Pe9CMi1MT5MOWQCeJknc
AvFjtLOkeHTNz/mdpz3Gy2k=
=YZgt
-----END PGP SIGNATURE-----

From rgerhards at hq.adiscon.com  Tue Dec 11 14:55:21 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 11 Dec 2007 14:55:21 +0100
Subject: [rsyslog] Asking for help...
In-Reply-To: <20071211144815.6a2ef123@marathon.haufe>
References: <577465F99B41C842AAFBE9ED71E70ABA30862F@grfint2.intern.adiscon.com>
	<20071211144815.6a2ef123@marathon.haufe>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308646@grfint2.intern.adiscon.com>

Thanks for the feedback - is it OK if we discuss this in the forum? I
think not everyone here on the list is interested in that discussion...

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Jakob Haufe
> Sent: Tuesday, December 11, 2007 2:48 PM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] Asking for help...
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> "Rainer Gerhards" <rgerhards at hq.adiscon.com> wrote on Tue, 11 Dec 2007
> 11:06:00 +0100:
> 
> > http://rgerhards.blogspot.com/2007/12/design-problem.html
> 
> Maybe I'm a bit short-sighted here, but isn't this exactly the
> situation
> for non-blocking sockets? Or will that raise other problems?
> 
> Regards,
> sur5r
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
> 
> iD8DBQFHXpUj1YAhDic+adYRAnBZAKCWNXoWIKHhnhm/Pe9CMi1MT5MOWQCeJknc
> AvFjtLOkeHTNz/mdpz3Gy2k=
> =YZgt
> -----END PGP SIGNATURE-----
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Wed Dec 12 17:06:48 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 12 Dec 2007 17:06:48 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>

Hi all,

I have just released rsyslog 1.20.1. It primarily contains bug fixes: a
memory leak in regard to PostgreSQL date formatting and a potential race
condition within repeated message processing have been fixed. Also,
there are new runtime options to harden ACL processing when DNS is not
ready while rsyslogd start up. There are also some minor changes and
clean-ups. Version 1.20.1 is recommended to anyone using previous
releases. Users of PostgreSQL are urged to update to this version.

Download:
   http://www.rsyslog.com/Downloads-req-getit-lid-66.phtml

Changelog:
   http://www.rsyslog.com/Article147.phtml

You may want to watch my blog for upcoming developments:
http://rgerhards.blogspot.com. There was recently a lot of activity and
more is expected.

As always, feedback is appreciated.
Rainer Gerhards


From stephen.carville at gmail.com  Wed Dec 12 18:38:15 2007
From: stephen.carville at gmail.com (Stephen Carville)
Date: Wed, 12 Dec 2007 09:38:15 -0800
Subject: [rsyslog] Last Message repeated N times
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308637@grfint2.intern.adiscon.com>
References: <2428c0380712071309w31f4fc20qb933dfef47f445ac@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA308637@grfint2.intern.adiscon.com>
Message-ID: <2428c0380712120938x72f55b9na4aea606374f0255@mail.gmail.com>

On Dec 11, 2007 3:01 AM, Rainer Gerhards <rgerhards at hq.adiscon.com> wrote:
> Hi Stephen,
>
> I was totally swamped the past days, sorry for the missing response.
> I'll check what causes this. As a side-note, I think that the message
> reduction code does not work well with the database - as you can no
> longer query the individual messages. Is this your intent?

My original intent was to have a centralized copy of all the changes
that cfengine makes.  That works extremely well and saves me a
buttload of time over having to log by hand in a shared Excel
spreadsheet every time I update a hosts file or tweak httpd.conf or
tomcat's server.xml.  That and logging the messages from auditd were
the main goals.  I added /var/log/messages and /var/log/secure mainly
because I could and because it gives the Linux side of the shop
greater credibility with the SOX (expletive deleted).  Now that
messages and secure are in the database I'll have to run a log
analyzer periodically looking for indications of "stuff" that's not
right but that comes after about six other projects.

This isn't a show stopper and I could probably just drop the "last
messages repeated.." entries without any harm.  Tho I;m not sure how
jsut yet.  Out of 2.4M rows about 385K are of this type.

> Rainer
>
>
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Stephen Carville
> > Sent: Friday, December 07, 2007 10:09 PM
> > To: rsyslog at lists.adiscon.com
> > Subject: [rsyslog] Last Message repeated N times
> >
> > I am using rsyslog 1.19.10 on a central server to collect messages and
> > store them in a mysql database.  One thing I've noticed is that the
> > "last message repeated NNN times" entries are alway entered with
> > FromHost="last", SysLogTag="message", and Message="repeated NNN
> > times".  Can this be fixed?
> >
> > I still use the regular sysklogd package on the remote machines to
> > forward the messages.
> >
> > --
> > Stephen Carville
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
>

-- 
Stephen Carville


From stephen.carville at gmail.com  Wed Dec 12 19:02:09 2007
From: stephen.carville at gmail.com (Stephen Carville)
Date: Wed, 12 Dec 2007 10:02:09 -0800
Subject: [rsyslog] Last Message repeated N times
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA30863D@grfint2.intern.adiscon.com>
References: <2428c0380712071309w31f4fc20qb933dfef47f445ac@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA308637@grfint2.intern.adiscon.com>
	<577465F99B41C842AAFBE9ED71E70ABA30863D@grfint2.intern.adiscon.com>
Message-ID: <2428c0380712121002l70f37260p869ab35b8acb7068@mail.gmail.com>

On Dec 11, 2007 3:26 AM, Rainer Gerhards <rgerhards at hq.adiscon.com> wrote:
> I think I misunderstood your problem... The "last message repeated n
> times" stems back from the reporting sysklogd's - right?

Would ugrading to rsyslogd on the client machines help?  I'm getting
ready for an upgrade cycle where everything except a couple of boxes
running old hardware not suppoert by the 2.6 kernel will be either
Oracle ULN 4.X or Redhat/CentOS 5.X  I didn't see a spec file in the
tar.gz but I can write one myself if there is no good one available.

> Mhhh... that's
> a problem. I need to do some hardcoded processing in the parser to cover
> part of it. The problem is that sysklogd emits a totally non-standard
> message in that case. Even worse, I think it does not even contain a
> clue at the missing information. I could probably use the sender's IP
> address as source in  that case, but the tag seems to be impossible to
> obtain.
>
> Can you post me a copy of the %rawmsg% property, so that I can have a
> look at what exactly you are seing?
>
> Rainer
>
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>
> > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> > Sent: Tuesday, December 11, 2007 12:01 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] Last Message repeated N times
> >
> > Hi Stephen,
> >
> > I was totally swamped the past days, sorry for the missing response.
> > I'll check what causes this. As a side-note, I think that the message
> > reduction code does not work well with the database - as you can no
> > longer query the individual messages. Is this your intent?
> >
> > Rainer
> >
> > > -----Original Message-----
> > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > > bounces at lists.adiscon.com] On Behalf Of Stephen Carville
> > > Sent: Friday, December 07, 2007 10:09 PM
> > > To: rsyslog at lists.adiscon.com
> > > Subject: [rsyslog] Last Message repeated N times
> > >
> > > I am using rsyslog 1.19.10 on a central server to collect messages
> > and
> > > store them in a mysql database.  One thing I've noticed is that the
> > > "last message repeated NNN times" entries are alway entered with
> > > FromHost="last", SysLogTag="message", and Message="repeated NNN
> > > times".  Can this be fixed?
> > >
> > > I still use the regular sysklogd package on the remote machines to
> > > forward the messages.
> > >
> > > --
> > > Stephen Carville
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
>



-- 
Stephen Carville


From mbiebl at gmail.com  Wed Dec 12 19:11:55 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 12 Dec 2007 19:11:55 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com>

2007/12/12, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Hi all,
>
> I have just released rsyslog 1.20.1. It primarily contains bug fixes: a

1.20.1 unfortunately fails to compile (unless you use --enable-debug):
gcc -g -O2 -W -Wall -Wformat-security -Wshadow -Wcast-align
-Wpointer-arith -Wmissing-format-attribute -g -o rsyslogd
rsyslogd-syslogd.o rsyslogd-pidfile.o rsyslogd-template.o
rsyslogd-outchannel.o rsyslogd-stringbuf.o rsyslogd-srUtils.o
rsyslogd-parse.o rsyslogd-sync.o rsyslogd-net.o rsyslogd-msg.o
rsyslogd-expr.o rsyslogd-omshell.o rsyslogd-omusrmsg.o
rsyslogd-omfwd.o rsyslogd-tcpsyslog.o rsyslogd-omfile.o
rsyslogd-omdiscard.o rsyslogd-modules.o rsyslogd-objomsr.o
rsyslogd-cfsysline.o rsyslogd-linkedlist.o rsyslogd-iminternal.o
rsyslogd-action.o rsyslogd-gss-misc.o -Wl,--export-dynamic  -lz
-lpthread -ldl -lrt
rsyslogd-syslogd.o: In function `domarkActions':
/home/michael/cvs/rsyslog/build/../syslogd.c:3431: undefined reference
to `lockObj'
/home/michael/cvs/rsyslog/build/../syslogd.c:3439: undefined reference
to `unlockObj'
rsyslogd-syslogd.o: In function `callAction':
/home/michael/cvs/rsyslog/build/../syslogd.c:2442: undefined reference
to `lockObj'
/home/michael/cvs/rsyslog/build/../syslogd.c:2508: undefined reference
to `unlockObj'
/home/michael/cvs/rsyslog/build/../syslogd.c:2508: undefined reference
to `unlockObj'
collect2: ld returned 1 exit status
make[2]: *** [rsyslogd] Fehler 1
make[2]: Leaving directory `/home/michael/cvs/rsyslog/build'
make[1]: *** [all-recursive] Fehler 1
make[1]: Leaving directory `/home/michael/cvs/rsyslog/build'
make: *** [all] Fehler 2


sync.h declares lockObj/unlockObj, but sync.c only has these symbols
if compiled without NDEBUG.

Cheers,
Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From infofarmer at FreeBSD.org  Wed Dec 12 19:16:36 2007
From: infofarmer at FreeBSD.org (Andrew Pantyukhin)
Date: Wed, 12 Dec 2007 21:16:36 +0300
Subject: [rsyslog] rsyslog 1.20.1 released
In-Reply-To: <aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>
	<aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com>
Message-ID: <20071212181635.GQ51627@amilo.cenkes.org>

On Wed, Dec 12, 2007 at 07:11:55PM +0100, Michael Biebl wrote:
> 2007/12/12, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > Hi all,
> >
> > I have just released rsyslog 1.20.1. It primarily contains bug fixes: a
> 
> sync.h declares lockObj/unlockObj, but sync.c only has these symbols
> if compiled without NDEBUG.

Try including config.h in sync.h


From mbiebl at gmail.com  Wed Dec 12 19:30:34 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 12 Dec 2007 19:30:34 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
In-Reply-To: <20071212181635.GQ51627@amilo.cenkes.org>
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>
	<aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com>
	<20071212181635.GQ51627@amilo.cenkes.org>
Message-ID: <aa76cdac0712121030y49b70fc0r6379af2b361a5f06@mail.gmail.com>

2007/12/12, Andrew Pantyukhin <infofarmer at freebsd.org>:
> On Wed, Dec 12, 2007 at 07:11:55PM +0100, Michael Biebl wrote:
> > 2007/12/12, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > > Hi all,
> > >
> > > I have just released rsyslog 1.20.1. It primarily contains bug fixes: a
> >
> > sync.h declares lockObj/unlockObj, but sync.c only has these symbols
> > if compiled without NDEBUG.
>
> Try including config.h in sync.h
>

No difference.

The other header files (using NDEBUG) don't include config.h either btw.
The rely on the c file, which includes them, to include config.h.

Cheers,
Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From mbiebl at gmail.com  Wed Dec 12 20:18:38 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 12 Dec 2007 20:18:38 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
In-Reply-To: <aa76cdac0712121030y49b70fc0r6379af2b361a5f06@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>
	<aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com>
	<20071212181635.GQ51627@amilo.cenkes.org>
	<aa76cdac0712121030y49b70fc0r6379af2b361a5f06@mail.gmail.com>
Message-ID: <aa76cdac0712121118kea18ee7md111ed3558bb7a87@mail.gmail.com>

2007/12/12, Michael Biebl <mbiebl at gmail.com>:
> 2007/12/12, Andrew Pantyukhin <infofarmer at freebsd.org>:
> > On Wed, Dec 12, 2007 at 07:11:55PM +0100, Michael Biebl wrote:
> > > 2007/12/12, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > > > Hi all,
> > > >
> > > > I have just released rsyslog 1.20.1. It primarily contains bug fixes: a
> > >
> > > sync.h declares lockObj/unlockObj, but sync.c only has these symbols
> > > if compiled without NDEBUG.
> >
> > Try including config.h in sync.h
> >
>

Found the culprit. Patch is attached.
Imho calling the define NDEBUG makes the code unnecessarily hard to
read (#ifndef NDEBUG ..). I had to stare at the code for a while until
I noticed that it was negated one time to often.

Cheers,
Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

From sur5r at sur5r.net  Wed Dec 12 20:35:38 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Wed, 12 Dec 2007 20:35:38 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>
	<aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com>
	<20071212181635.GQ51627@amilo.cenkes.org>
	<aa76cdac0712121030y49b70fc0r6379af2b361a5f06@mail.gmail.com>
	<aa76cdac0712121118kea18ee7md111ed3558bb7a87@mail.gmail.com>
Message-ID: <20071212203538.6dbb00d1@marathon.haufe>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

"Michael Biebl" <mbiebl at gmail.com> wrote on Wed, 12 Dec 2007 20:18:38
+0100:

> Found the culprit. Patch is attached.

Mailman ate it again. Try attaching it with mimetype text/plain, seems
to work then.

Regards,
sur5r
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHYDgQ1YAhDic+adYRAtxQAKCspmrE+HfH12YVBzJcIBQ/GqNKXACfdSl3
NkY9aWuB+ftzvx2t4MtKPHw=
=BODR
-----END PGP SIGNATURE-----

From rgerhards at hq.adiscon.com  Wed Dec 12 21:04:09 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 12 Dec 2007 21:04:09 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
In-Reply-To: <20071212203538.6dbb00d1@marathon.haufe>
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com><aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com><20071212181635.GQ51627@amilo.cenkes.org><aa76cdac0712121030y49b70fc0r6379af2b361a5f06@mail.gmail.com><aa76cdac0712121118kea18ee7md111ed3558bb7a87@mail.gmail.com>
	<20071212203538.6dbb00d1@marathon.haufe>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308665@grfint2.intern.adiscon.com>

Hold for a moment, the actual patch is a bit different . more soon.

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Jakob Haufe
> Sent: Wednesday, December 12, 2007 8:36 PM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] rsyslog 1.20.1 released
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> "Michael Biebl" <mbiebl at gmail.com> wrote on Wed, 12 Dec 2007 20:18:38
> +0100:
> 
> > Found the culprit. Patch is attached.
> 
> Mailman ate it again. Try attaching it with mimetype text/plain, seems
> to work then.
> 
> Regards,
> sur5r
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
> 
> iD8DBQFHYDgQ1YAhDic+adYRAtxQAKCspmrE+HfH12YVBzJcIBQ/GqNKXACfdSl3
> NkY9aWuB+ftzvx2t4MtKPHw=
> =BODR
> -----END PGP SIGNATURE-----
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Wed Dec 12 21:07:26 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 12 Dec 2007 21:07:26 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308665@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com><aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com><20071212181635.GQ51627@amilo.cenkes.org><aa76cdac0712121030y49b70fc0r6379af2b361a5f06@mail.gmail.com><aa76cdac0712121118kea18ee7md111ed3558bb7a87@mail.gmail.com><20071212203538.6dbb00d1@marathon.haufe>
	<577465F99B41C842AAFBE9ED71E70ABA308665@grfint2.intern.adiscon.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308666@grfint2.intern.adiscon.com>

Re-released, info follows soon...

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Wednesday, December 12, 2007 9:04 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 1.20.1 released
> 
> Hold for a moment, the actual patch is a bit different . more soon.
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Jakob Haufe
> > Sent: Wednesday, December 12, 2007 8:36 PM
> > To: rsyslog at lists.adiscon.com
> > Subject: Re: [rsyslog] rsyslog 1.20.1 released
> >
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > "Michael Biebl" <mbiebl at gmail.com> wrote on Wed, 12 Dec 2007
20:18:38
> > +0100:
> >
> > > Found the culprit. Patch is attached.
> >
> > Mailman ate it again. Try attaching it with mimetype text/plain,
> seems
> > to work then.
> >
> > Regards,
> > sur5r
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v1.4.6 (GNU/Linux)
> >
> > iD8DBQFHYDgQ1YAhDic+adYRAtxQAKCspmrE+HfH12YVBzJcIBQ/GqNKXACfdSl3
> > NkY9aWuB+ftzvx2t4MtKPHw=
> > =BODR
> > -----END PGP SIGNATURE-----
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From mbiebl at gmail.com  Wed Dec 12 21:31:23 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 12 Dec 2007 21:31:23 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308665@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>
	<aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com>
	<20071212181635.GQ51627@amilo.cenkes.org>
	<aa76cdac0712121030y49b70fc0r6379af2b361a5f06@mail.gmail.com>
	<aa76cdac0712121118kea18ee7md111ed3558bb7a87@mail.gmail.com>
	<20071212203538.6dbb00d1@marathon.haufe>
	<577465F99B41C842AAFBE9ED71E70ABA308665@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712121231i62631494p4ea39e41136adac3@mail.gmail.com>

2007/12/12, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Hold for a moment, the actual patch is a bit different . more soon.

Yeah, got it the wrong way around. Didn't read the comment in sync.h
carefully enough.
Thanks for you sharp eye!

As I said, imho it's easier to read and grasp if the define is called DEBUG.

Cheers,
Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Wed Dec 12 21:40:25 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 12 Dec 2007 21:40:25 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
In-Reply-To: <aa76cdac0712121130x4c96b883t4ee91e3362fb83d6@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>
	<aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com>
	<20071212181635.GQ51627@amilo.cenkes.org>
	<aa76cdac0712121030y49b70fc0r6379af2b361a5f06@mail.gmail.com>
	<aa76cdac0712121118kea18ee7md111ed3558bb7a87@mail.gmail.com>
	<aa76cdac0712121130x4c96b883t4ee91e3362fb83d6@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308667@grfint2.intern.adiscon.com>

Hi all,

Man, when did I really screw up the last time? And I thought I had run
all test...

Anyhow, thanks all for helping to solve this issue.

Michael got it right, it is the NDEBUG. But is not the .c file but the
header file that was wrong. The idea is that we have functions only in
debugging mode, because functions make it easier to debug. But they are
slower than macro expansions. So in release mode, the functions are
replace with the mutex calls directly (which, I think, a macros, too).
If I recall it correctly, I screwed up after I created that
optimization.

> > Found the culprit. Patch is attached.
> > Imho calling the define NDEBUG makes the code unnecessarily hard to
> > read (#ifndef NDEBUG ..). I had to stare at the code for a 
> while until
> > I noticed that it was negated one time to often.

The NDEBUG is an assert() inheritance. Assert() uses it to indicate a
release build. I've build on the same tool to avoid introducing another
one. But obviously, it has its drawbacks, too. I think I'll solve that
by adding a comment.

I hope this clarifies. Drop me a note if I got it wrong again or you
have a question ;)

The tarball that is now in the repository should solve the issue. 

Rainer


From mbiebl at gmail.com  Wed Dec 12 21:47:32 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 12 Dec 2007 21:47:32 +0100
Subject: [rsyslog] Problems with IncludeConfig
Message-ID: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>

Hi everyone,

the current IncludeConfig implementation has some issues, especially
for distributers:
All files in the directory as specified by $IncludeConfig are included.
This means, temp files like *~ *.bak etc that are left behind by
editors, or stuff like *.rpmsave or *.dpkg-old/*.dpkg-new.
This is obviously unfortunate.

How shall we solve this?
1.) I patched the Debian package to only include files ending with
*.conf. If we document that properly this is imho an acceptable
solution.
2.) Implement an internal blacklist of endings that are ignored (see above)
3.) Instead of hardcoding the file ending to *.conf (as in 1.), use
the same syntax as e.g. apache, which allows to specify shell globs:
example:
Include /etc/apache2/conf.d/*.conf

What do you think?

Michael


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From r.bhatia at ipax.at  Thu Dec 13 09:55:49 2007
From: r.bhatia at ipax.at (Raoul Bhatia [IPAX])
Date: Thu, 13 Dec 2007 09:55:49 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
Message-ID: <4760F395.4030502@ipax.at>

Michael Biebl wrote:
> the current IncludeConfig implementation has some issues, especially
> for distributers:
> All files in the directory as specified by $IncludeConfig are included.
> This means, temp files like *~ *.bak etc that are left behind by
> editors, or stuff like *.rpmsave or *.dpkg-old/*.dpkg-new.
> This is obviously unfortunate.
> 
> How shall we solve this?
> 1.) I patched the Debian package to only include files ending with
> *.conf. If we document that properly this is imho an acceptable
> solution.
> 2.) Implement an internal blacklist of endings that are ignored (see above)
> 3.) Instead of hardcoding the file ending to *.conf (as in 1.), use
> the same syntax as e.g. apache, which allows to specify shell globs:
> example:
> Include /etc/apache2/conf.d/*.conf

i would opt for the third option - thou i'm not sure how much work
this creates ;)

cheers,
raoul
-- 
____________________________________________________________________
DI (FH) Raoul Bhatia M.Sc.          email.          r.bhatia at ipax.at
Technischer Leiter

IPAX - Aloy Bhatia Hava OEG         web.          http://www.ipax.at
Barawitzkagasse 10/2/2/11           email.            office at ipax.at
1190 Wien                           tel.               +43 1 3670030
FN 277995t HG Wien                  fax.            +43 1 3670030 15
____________________________________________________________________


From rgerhards at hq.adiscon.com  Thu Dec 13 13:04:33 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 13 Dec 2007 13:04:33 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <4760F395.4030502@ipax.at>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
	<4760F395.4030502@ipax.at>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>

Lol... the initial message seems not to have passed my spam filter... ;)

I, too, tend to option three. The problem, though is
backwards-compatibility... Any suggestions and concerns are appreciated.

I'll look at the code to see how much work that creates. I think the
complexity is quite acceptable. I am right now in a threading redesign,
so I'd like to push this away for a little while... Anyhow, please keep
comments floating - I'll listen and drop in any thoughts I have.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Raoul Bhatia [IPAX]
> Sent: Thursday, December 13, 2007 9:56 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Problems with IncludeConfig
> 
> Michael Biebl wrote:
> > the current IncludeConfig implementation has some issues, especially
> > for distributers:
> > All files in the directory as specified by $IncludeConfig are
> included.
> > This means, temp files like *~ *.bak etc that are left behind by
> > editors, or stuff like *.rpmsave or *.dpkg-old/*.dpkg-new.
> > This is obviously unfortunate.
> >
> > How shall we solve this?
> > 1.) I patched the Debian package to only include files ending with
> > *.conf. If we document that properly this is imho an acceptable
> > solution.
> > 2.) Implement an internal blacklist of endings that are ignored (see
> above)
> > 3.) Instead of hardcoding the file ending to *.conf (as in 1.), use
> > the same syntax as e.g. apache, which allows to specify shell globs:
> > example:
> > Include /etc/apache2/conf.d/*.conf
> 
> i would opt for the third option - thou i'm not sure how much work
> this creates ;)
> 
> cheers,
> raoul
> --
> ____________________________________________________________________
> DI (FH) Raoul Bhatia M.Sc.          email.          r.bhatia at ipax.at
> Technischer Leiter
> 
> IPAX - Aloy Bhatia Hava OEG         web.          http://www.ipax.at
> Barawitzkagasse 10/2/2/11           email.            office at ipax.at
> 1190 Wien                           tel.               +43 1 3670030
> FN 277995t HG Wien                  fax.            +43 1 3670030 15
> ____________________________________________________________________
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Fri Dec 14 09:29:19 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 14 Dec 2007 09:29:19 +0100
Subject: [rsyslog] IMPORTANT: Version Changes
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308674@grfint2.intern.adiscon.com>

Hi all,

some information first here on the mailing list before I announce it. As
you know, I am looking at the way threading is supposed to work in
future releases and, most importantly, looking at the inputs (like mark
message generation).

Around summer, I wrote that I will probably need to release new major
versions when we go into multithreading redesign. It looks like we have
reached this stage. I tried to keep a single code base that still
supports both single- and multi-threaded operations. I have looked into
this the past days and I need to say that it creates a lot of complexity
and hard to understand code.

For this reason, I think it is finally time to branch the code based and
release some new versions.

Soon, I will create a branch for the current 1.20.1 code base. That will
only receive bug fixes, but no new development (except, I guess, GSSAPI
which I about to be contributed by Red Hat). When we are confident the
last changes worked well and introduced no new bugs, there will be a
version 2.0.0 stable release based on that code base.

CVS head, however, will then be rsyslog version 3. It will receive the
new input module interface. It requires pthreads, because there is no
way input modules and many more of the new desired features can be
implemented without them. Consequently, I will remove all
single-threading code from it, resulting in an easier to understand code
base. Please note that I expect this code to change dramatically when it
is being modified to be more modular (much like it was when I introduced
modular outputs in summer). Please note that I will apply any non-bugfix
patches to this code base, only.

I have somewhat bad feeling of going ahead with implementing a more
sophisticated and more parallel multi-threading while we still have an
issue with the segfault. However, I think by now we did everything
imaginable to capture that rare bug. I have come to the conclusion that
the best chance to find it is go ahead and implement the more
sophisticated design. That will lead to a review, and rewrite, of much
of the code in question, uncovering this we didn't think about before.
The recently discovered race condition is an excellent sample.

One thing about the license: rsyslog 2 will stay with "GPL v2 and above"
license, but rsyslog V3 will be licensed under "GPL v3 and above". I
already wrote about that change. It is my firm believe that GPL v3
brings benefit to our freedom to use digital goods. I am a strong oppose
of digital restrictions management (DRM) and software patens and I do
not like the idea that rsyslog benefits anyone who encourages these
things. I hope for your understanding.

I will set stage now for these changes and will do a web announcement
soon. Please don't be surprised that rsyslog v3 will be available before
v2, you now know the reason.

Rainer


From mbiebl at gmail.com  Fri Dec 14 14:36:03 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Fri, 14 Dec 2007 14:36:03 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
	<4760F395.4030502@ipax.at>
	<577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>

2007/12/13, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Lol... the initial message seems not to have passed my spam filter... ;)
>
> I, too, tend to option three. The problem, though is

Yeah, it's the most flexible one.

> backwards-compatibility... Any suggestions and concerns are appreciated.

This feature has been pretty new, and as such probably isn't widely used.
I would simply emphasize this change in the release notes and the Changelog.
The worst that can happen is, that a file is not included anymore.

>
> I'll look at the code to see how much work that creates. I think the
> complexity is quite acceptable. I am right now in a threading redesign,
> so I'd like to push this away for a little while... Anyhow, please keep
> comments floating - I'll listen and drop in any thoughts I have.

Should be fairly straightforward to implement using
http://www.gnu.org/software/libc/manual/html_node/Globbing.html#Globbing

Cheers,
Michael


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From mbiebl at gmail.com  Fri Dec 14 17:24:02 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Fri, 14 Dec 2007 17:24:02 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
	<4760F395.4030502@ipax.at>
	<577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>
	<aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>
Message-ID: <aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com>

2007/12/14, Michael Biebl <mbiebl at gmail.com>:
> 2007/12/13, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > Lol... the initial message seems not to have passed my spam filter... ;)
> >
> > I, too, tend to option three. The problem, though is
>
> Yeah, it's the most flexible one.
>
> > backwards-compatibility... Any suggestions and concerns are appreciated.
>
> This feature has been pretty new, and as such probably isn't widely used.
> I would simply emphasize this change in the release notes and the Changelog.
> The worst that can happen is, that a file is not included anymore.
>
> >
> > I'll look at the code to see how much work that creates. I think the
> > complexity is quite acceptable. I am right now in a threading redesign,
> > so I'd like to push this away for a little while... Anyhow, please keep
> > comments floating - I'll listen and drop in any thoughts I have.
>
> Should be fairly straightforward to implement using
> http://www.gnu.org/software/libc/manual/html_node/Globbing.html#Globbing
>

Small example using glob:

#include <glob.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char **argv)
{
        glob_t files;
        int ret = 0;
        int i = 0;
        char *path;

        if (argc > 1) {
                path = argv[1];
        } else {
                path = strdup("/etc/rsyslog.d/*.conf");
        }

        ret = glob(path, GLOB_ERR, NULL, &files);
        printf("All files matching %s:\n", path);
        for (i = 0; i < files.gl_pathc; i++) {
                printf("\t%s\n", files.gl_pathv[i]);
        }

        globfree(&files);
        printf("Finished\n");

        return EXIT_SUCCESS;
}

man 3 glob has more information.

If wanted, I can prepare a patch for rsyslog.

Cheers,
Michael
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Fri Dec 14 17:29:38 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 14 Dec 2007 17:29:38 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com><4760F395.4030502@ipax.at><577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com><aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>
	<aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com>

Michael,

sorry for the sluggish reply. I am right in the middle of the initial
threading code (have a look at CVS ;)) and that blocks most of
everything else ;)

It would be great if you could supply a patch, I'd apply it as soon as I
got something together with the new code. I try to do the threading in
an as atomic code change as possible, but it will proabbyl busy me for a
while ;)

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Friday, December 14, 2007 5:24 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Problems with IncludeConfig
> 
> 2007/12/14, Michael Biebl <mbiebl at gmail.com>:
> > 2007/12/13, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > > Lol... the initial message seems not to have passed my spam
> filter... ;)
> > >
> > > I, too, tend to option three. The problem, though is
> >
> > Yeah, it's the most flexible one.
> >
> > > backwards-compatibility... Any suggestions and concerns are
> appreciated.
> >
> > This feature has been pretty new, and as such probably isn't widely
> used.
> > I would simply emphasize this change in the release notes and the
> Changelog.
> > The worst that can happen is, that a file is not included anymore.
> >
> > >
> > > I'll look at the code to see how much work that creates. I think
> the
> > > complexity is quite acceptable. I am right now in a threading
> redesign,
> > > so I'd like to push this away for a little while... Anyhow, please
> keep
> > > comments floating - I'll listen and drop in any thoughts I have.
> >
> > Should be fairly straightforward to implement using
> >
>
http://www.gnu.org/software/libc/manual/html_node/Globbing.html#Globbin
> g
> >
> 
> Small example using glob:
> 
> #include <glob.h>
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
> 
> int main(int argc, char **argv)
> {
>         glob_t files;
>         int ret = 0;
>         int i = 0;
>         char *path;
> 
>         if (argc > 1) {
>                 path = argv[1];
>         } else {
>                 path = strdup("/etc/rsyslog.d/*.conf");
>         }
> 
>         ret = glob(path, GLOB_ERR, NULL, &files);
>         printf("All files matching %s:\n", path);
>         for (i = 0; i < files.gl_pathc; i++) {
>                 printf("\t%s\n", files.gl_pathv[i]);
>         }
> 
>         globfree(&files);
>         printf("Finished\n");
> 
>         return EXIT_SUCCESS;
> }
> 
> man 3 glob has more information.
> 
> If wanted, I can prepare a patch for rsyslog.
> 
> Cheers,
> Michael
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From mbiebl at gmail.com  Sun Dec 16 23:38:14 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Sun, 16 Dec 2007 23:38:14 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
	<4760F395.4030502@ipax.at>
	<577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>
	<aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>
	<aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712161438n69382660u9d2a082914d717c9@mail.gmail.com>

2007/12/14, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Michael,
>
> sorry for the sluggish reply. I am right in the middle of the initial
> threading code (have a look at CVS ;)) and that blocks most of
> everything else ;)
>
> It would be great if you could supply a patch, I'd apply it as soon as I
> got something together with the new code. I try to do the threading in
> an as atomic code change as possible, but it will proabbyl busy me for a
> while ;)

Unfortunately I can't set the mimetype via gmail, so the patch is at

http://debs.michaelbiebl.de/rsyslog/rsyslog-glob.patch

I didn't check the return values for glob. I wanted I can implement
that and printout debugging messsages in that case.
The patch also doesn't contain any changes to the documentation (yet).
I wanted to get some feedback first.

Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From mbiebl at gmail.com  Mon Dec 17 05:09:59 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Mon, 17 Dec 2007 05:09:59 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <aa76cdac0712161438n69382660u9d2a082914d717c9@mail.gmail.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
	<4760F395.4030502@ipax.at>
	<577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>
	<aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>
	<aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com>
	<aa76cdac0712161438n69382660u9d2a082914d717c9@mail.gmail.com>
Message-ID: <aa76cdac0712162009q9fe3ecaq407df79296e4baff@mail.gmail.com>

2007/12/16, Michael Biebl <mbiebl at gmail.com>:
> 2007/12/14, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > Michael,
> >
> > sorry for the sluggish reply. I am right in the middle of the initial
> > threading code (have a look at CVS ;)) and that blocks most of
> > everything else ;)
> >
> > It would be great if you could supply a patch, I'd apply it as soon as I
> > got something together with the new code. I try to do the threading in
> > an as atomic code change as possible, but it will proabbyl busy me for a
> > while ;)
>
> Unfortunately I can't set the mimetype via gmail, so the patch is at
>
> http://debs.michaelbiebl.de/rsyslog/rsyslog-glob.patch
>

I have an updated patch
 http://debs.michaelbiebl.de/rsyslog/rsyslog-glob-2.patch
which is backward compatible.
It allows the following

include config files:
 $IncludeConfig /etc/rsyslog.d/foo.conf
include config directories:
 $IncludeConfig /etc/rsyslog.d/ (with or without trailing slash)
include via shell glob:
 $IncludeConfig /etc/rsyslog.d/*.conf

One remaining issue is, how to handle symlinks. I made the patch
consistent to the current behaviour of doIncludeDirectories to ignore
symlinks.

I know that the apache Include directive allows symlinks, and
supporting it could be handy.
So my question is, what do others think about it. Please let me know.

Cheers,
Michael

P.S: A personal wish: as the list is subscriber only, would it be
possible to allow attachements (other than text/plain, gmail is
obviously too stupid)?


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From mbiebl at gmail.com  Mon Dec 17 05:27:28 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Mon, 17 Dec 2007 05:27:28 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <aa76cdac0712162009q9fe3ecaq407df79296e4baff@mail.gmail.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
	<4760F395.4030502@ipax.at>
	<577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>
	<aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>
	<aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com>
	<aa76cdac0712161438n69382660u9d2a082914d717c9@mail.gmail.com>
	<aa76cdac0712162009q9fe3ecaq407df79296e4baff@mail.gmail.com>
Message-ID: <aa76cdac0712162027r85875d2rc7223f9c6a61a43@mail.gmail.com>

2007/12/17, Michael Biebl <mbiebl at gmail.com>:
> 2007/12/16, Michael Biebl <mbiebl at gmail.com>:
> > 2007/12/14, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> I have an updated patch
>  http://debs.michaelbiebl.de/rsyslog/rsyslog-glob-2.patch

Spotted a minor typo, so here goes v3
http://debs.michaelbiebl.de/rsyslog/rsyslog-glob-3.patch

Michael

Btw. gmail uses
Content-Type: multipart/mixed
and then
Content-Type: text/plain for the mail
and
Content-Type: text/x-diff for the patch
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Mon Dec 17 08:04:46 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 17 Dec 2007 08:04:46 +0100
Subject: [rsyslog] Tackeling the segfault...
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308683@grfint2.intern.adiscon.com>

Hi all,

thanks to mildew, I could create another fix that may solve the issue. I
will do a writeup of what happens, but that'll take a short while. In
the mean time, I would appreciate if those of you that experience the
segfault could try the new version. I have released a pre-1.20.2 tarball
at

http://download.rsyslog.com/rsyslog/rsyslog-1.20.2.tar.gz

As usual, this is not yet 1.20.2, so whenever it is re-released be sure
to update your installation. I would appreciate feedback on it very
much.

Many thanks,
Rainer


From mbiebl at gmail.com  Mon Dec 17 08:20:59 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Mon, 17 Dec 2007 08:20:59 +0100
Subject: [rsyslog] Tackeling the segfault...
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308683@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308683@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712162320l3b089b2eue6a3c4699ad0b277@mail.gmail.com>

2007/12/17, Rainer Gerhards <rgerhards at hq.adiscon.com>:
>
> http://download.rsyslog.com/rsyslog/rsyslog-1.20.2.tar.gz
>
> As usual, this is not yet 1.20.2, so whenever it is re-released be sure
> to update your installation. I would appreciate feedback on it very
> much.

Hi Rainer,

it would be handy if you would post md5sums or sha1sums of the tarball.
This would make it easier to identify/distinguish the prereleases and
the final tarball release.

Cheers,
Michael

P.S: current CVS head doesn't compile with USE_NETZIP.

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Mon Dec 17 08:26:38 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 17 Dec 2007 08:26:38 +0100
Subject: [rsyslog] Tackeling the segfault...
In-Reply-To: <aa76cdac0712162320l3b089b2eue6a3c4699ad0b277@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308683@grfint2.intern.adiscon.com>
	<aa76cdac0712162320l3b089b2eue6a3c4699ad0b277@mail.gmail.com>
Message-ID: <1197876398.21389.3.camel@localhost.localdomain>


On Mon, 2007-12-17 at 08:20 +0100, Michael Biebl wrote:
> 2007/12/17, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> >
> > http://download.rsyslog.com/rsyslog/rsyslog-1.20.2.tar.gz
> >
> > As usual, this is not yet 1.20.2, so whenever it is re-released be sure
> > to update your installation. I would appreciate feedback on it very
> > much.
> 
> Hi Rainer,
> 
> it would be handy if you would post md5sums or sha1sums of the tarball.
> This would make it easier to identify/distinguish the prereleases and
> the final tarball release.

good suggestion. For this release the md5sum is
45e6f14094c26f2dd3a3ff7a4d025381 

> 
> Cheers,
> Michael
> 
> P.S: current CVS head doesn't compile with USE_NETZIP.
Will look into that. I have to admit that CVS head may have some issues
the next few days. I thought about holding off commits until everything
is perfect, that would result in one gigantic diff once it is done -
which would make it hard to follow what I am doing. So I am now doing
commits whenever something seems to work reasonably well, but definitely
not prime time yet. The 3.0 branch is not ready now for being used in
practice. But, NETZIP should compile and I'll look into it. Thanks :)


From rgerhards at hq.adiscon.com  Mon Dec 17 08:27:34 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 17 Dec 2007 08:27:34 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <aa76cdac0712162027r85875d2rc7223f9c6a61a43@mail.gmail.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com><4760F395.4030502@ipax.at><577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com><aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com><aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com><aa76cdac0712161438n69382660u9d2a082914d717c9@mail.gmail.com><aa76cdac0712162009q9fe3ecaq407df79296e4baff@mail.gmail.com>
	<aa76cdac0712162027r85875d2rc7223f9c6a61a43@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308685@grfint2.intern.adiscon.com>

Hi Michael,

I like this patch, especially it being backwards-compatible. And I also
tend to agree that symlinks would be nice to have.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Monday, December 17, 2007 5:27 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Problems with IncludeConfig
> 
> 2007/12/17, Michael Biebl <mbiebl at gmail.com>:
> > 2007/12/16, Michael Biebl <mbiebl at gmail.com>:
> > > 2007/12/14, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > I have an updated patch
> >  http://debs.michaelbiebl.de/rsyslog/rsyslog-glob-2.patch
> 
> Spotted a minor typo, so here goes v3
> http://debs.michaelbiebl.de/rsyslog/rsyslog-glob-3.patch
> 
> Michael
> 
> Btw. gmail uses
> Content-Type: multipart/mixed
> and then
> Content-Type: text/plain for the mail
> and
> Content-Type: text/x-diff for the patch
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Mon Dec 17 08:48:01 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 17 Dec 2007 08:48:01 +0100
Subject: [rsyslog] Tackeling the segfault...
In-Reply-To: <aa76cdac0712162320l3b089b2eue6a3c4699ad0b277@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308683@grfint2.intern.adiscon.com>
	<aa76cdac0712162320l3b089b2eue6a3c4699ad0b277@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308686@grfint2.intern.adiscon.com>

> P.S: current CVS head doesn't compile with USE_NETZIP.

Mmmhhh... I cannot reproduce this on my machine (using Fedora 8). Can
you supply me the error messages - I guess best via private mail because
this will get into too much detail for most on this list (Or we move it
to the dev corner in the forum, so that those interested can follow)...

Thanks,
Rainer


From r.bhatia at ipax.at  Mon Dec 17 10:47:50 2007
From: r.bhatia at ipax.at (Raoul Bhatia [IPAX])
Date: Mon, 17 Dec 2007 10:47:50 +0100
Subject: [rsyslog] Tackeling the segfault...
In-Reply-To: <aa76cdac0712162320l3b089b2eue6a3c4699ad0b277@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308683@grfint2.intern.adiscon.com>
	<aa76cdac0712162320l3b089b2eue6a3c4699ad0b277@mail.gmail.com>
Message-ID: <476645C6.2080909@ipax.at>

Michael Biebl wrote:
> 2007/12/17, Rainer Gerhards <rgerhards at hq.adiscon.com>:
>> http://download.rsyslog.com/rsyslog/rsyslog-1.20.2.tar.gz
>>
>> As usual, this is not yet 1.20.2, so whenever it is re-released be sure
>> to update your installation. I would appreciate feedback on it very
>> much.
> 
> Hi Rainer,
> 
> it would be handy if you would post md5sums or sha1sums of the tarball.
> This would make it easier to identify/distinguish the prereleases and
> the final tarball release.

or simply use the well known -pre or -rc tags?

cheers,
raoul
-- 
____________________________________________________________________
DI (FH) Raoul Bhatia M.Sc.          email.          r.bhatia at ipax.at
Technischer Leiter

IPAX - Aloy Bhatia Hava OEG         web.          http://www.ipax.at
Barawitzkagasse 10/2/2/11           email.            office at ipax.at
1190 Wien                           tel.               +43 1 3670030
FN 277995t HG Wien                  fax.            +43 1 3670030 15
____________________________________________________________________


From rgerhards at hq.adiscon.com  Mon Dec 17 11:04:33 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 17 Dec 2007 11:04:33 +0100
Subject: [rsyslog] backward compatibility...
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>

Hi all,

I would appreciate your feedback on backward compatibility. I have
started to create a loadable input module interface in rsyslog v3. My
first input module is the mark message handler (because it has almost no
code, so it was a good example of what needs to be done...). I have now
basically have finished an initial version. The interface needs to be
changed in the longer term, but I'll postpone that until I've done a bit
more work on the config interface. Now I am facing a problem with
backward compatibility:

In sysklogd/rsyslog pre-v3, the -m option controls how often mark
messages are written (with -m0 disabling this mechanism). In v3, the
$MarkMessagePeriod config file directive does the same. However, a
$modload immark is necessary to activate that functionality because
otherwise the module is not loaded (which is good if it is not needed).
When trying to remain backward-compatible, I have an issue when the -m
option is absent. If so, that means a mark message period of 20 minutes.
However, it is hard to detect what may be intended in v3 if this option
is not present. Does it mean the former default (running as a drop-in
replacement) or does it actually mean that no mark messages should
appear if no such directive is in rsyslog.conf?

Do you think it would be acceptable to not emit messages in this case,
slightly breaking backward compatibility? Or should I come up with a
solution. The later would probably require a specific option setting to
disable the -m option and its default (somehow telling "I am not running
in v2 compatibility mode").

I am not sure what is best...

Rainer 


From r.bhatia at ipax.at  Mon Dec 17 11:17:03 2007
From: r.bhatia at ipax.at (Raoul Bhatia [IPAX])
Date: Mon, 17 Dec 2007 11:17:03 +0100
Subject: [rsyslog] backward compatibility...
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
Message-ID: <47664C9F.5030301@ipax.at>

Rainer Gerhards wrote:
> Hi all,
> 
> I would appreciate your feedback on backward compatibility. I have
> started to create a loadable input module interface in rsyslog v3. My
> first input module is the mark message handler (because it has almost no
> code, so it was a good example of what needs to be done...). I have now
> basically have finished an initial version. The interface needs to be
> changed in the longer term, but I'll postpone that until I've done a bit
> more work on the config interface. Now I am facing a problem with
> backward compatibility:
> 
> In sysklogd/rsyslog pre-v3, the -m option controls how often mark
> messages are written (with -m0 disabling this mechanism). In v3, the
> $MarkMessagePeriod config file directive does the same. However, a
> $modload immark is necessary to activate that functionality because
> otherwise the module is not loaded (which is good if it is not needed).
> When trying to remain backward-compatible, I have an issue when the -m
> option is absent. If so, that means a mark message period of 20 minutes.
> However, it is hard to detect what may be intended in v3 if this option
> is not present. Does it mean the former default (running as a drop-in
> replacement) or does it actually mean that no mark messages should
> appear if no such directive is in rsyslog.conf?
> 
> Do you think it would be acceptable to not emit messages in this case,
> slightly breaking backward compatibility? Or should I come up with a
> solution. The later would probably require a specific option setting to
> disable the -m option and its default (somehow telling "I am not running
> in v2 compatibility mode").
> 
> I am not sure what is best...

i would use the opportunity and not hesitate to use a
make-everything-right-even-if-we-break-something approach.

upgrading seems to not be trivial, as one needs to go through the config
files anyway. so lets do things right and maybe issue some error/warning
message on obsolete options (like -m).

cheers,
raoul
ps: i hope i'm getting you right ;)
-- 
____________________________________________________________________
DI (FH) Raoul Bhatia M.Sc.          email.          r.bhatia at ipax.at
Technischer Leiter

IPAX - Aloy Bhatia Hava OEG         web.          http://www.ipax.at
Barawitzkagasse 10/2/2/11           email.            office at ipax.at
1190 Wien                           tel.               +43 1 3670030
FN 277995t HG Wien                  fax.            +43 1 3670030 15
____________________________________________________________________


From rgerhards at hq.adiscon.com  Mon Dec 17 11:35:02 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 17 Dec 2007 11:35:02 +0100
Subject: [rsyslog] backward compatibility...
In-Reply-To: <47664C9F.5030301@ipax.at>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
	<47664C9F.5030301@ipax.at>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308692@grfint2.intern.adiscon.com>

> i would use the opportunity and not hesitate to use a
> make-everything-right-even-if-we-break-something approach.
> 
> upgrading seems to not be trivial, as one needs to go through the
> config
> files anyway. so lets do things right and maybe issue some
> error/warning
> message on obsolete options (like -m).
> 
> cheers,
> raoul
> ps: i hope i'm getting you right ;)

Thanks - yes, this was the kind of feedback I am looking for. Sorry for
the quite complex description, I could not come up with a better one...

Rainer


From rgerhards at hq.adiscon.com  Tue Dec 18 17:21:26 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 18 Dec 2007 17:21:26 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <aa76cdac0712162009q9fe3ecaq407df79296e4baff@mail.gmail.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
	<4760F395.4030502@ipax.at>
	<577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>
	<aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>
	<aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com>
	<aa76cdac0712161438n69382660u9d2a082914d717c9@mail.gmail.com>
	<aa76cdac0712162009q9fe3ecaq407df79296e4baff@mail.gmail.com>
Message-ID: <1197994886.14826.2.camel@localhost.localdomain>

This patch now also made it to the repository. Applied to both v2
(previously 1.20.2) and v3.

Rainer

On Mon, 2007-12-17 at 05:09 +0100, Michael Biebl wrote:
> 2007/12/16, Michael Biebl <mbiebl at gmail.com>:
> > 2007/12/14, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > > Michael,
> > >
> > > sorry for the sluggish reply. I am right in the middle of the initial
> > > threading code (have a look at CVS ;)) and that blocks most of
> > > everything else ;)
> > >
> > > It would be great if you could supply a patch, I'd apply it as soon as I
> > > got something together with the new code. I try to do the threading in
> > > an as atomic code change as possible, but it will proabbyl busy me for a
> > > while ;)
> >
> > Unfortunately I can't set the mimetype via gmail, so the patch is at
> >
> > http://debs.michaelbiebl.de/rsyslog/rsyslog-glob.patch
> >
> 
> I have an updated patch
>  http://debs.michaelbiebl.de/rsyslog/rsyslog-glob-2.patch
> which is backward compatible.
> It allows the following
> 
> include config files:
>  $IncludeConfig /etc/rsyslog.d/foo.conf
> include config directories:
>  $IncludeConfig /etc/rsyslog.d/ (with or without trailing slash)
> include via shell glob:
>  $IncludeConfig /etc/rsyslog.d/*.conf
> 
> One remaining issue is, how to handle symlinks. I made the patch
> consistent to the current behaviour of doIncludeDirectories to ignore
> symlinks.
> 
> I know that the apache Include directive allows symlinks, and
> supporting it could be handy.
> So my question is, what do others think about it. Please let me know.
> 
> Cheers,
> Michael
> 
> P.S: A personal wish: as the list is subscriber only, would it be
> possible to allow attachements (other than text/plain, gmail is
> obviously too stupid)?
> 
> 


From sur5r at sur5r.net  Tue Dec 18 22:16:54 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Tue, 18 Dec 2007 22:16:54 +0100
Subject: [rsyslog] backward compatibility...
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
Message-ID: <20071218221654.27ec59a2@marathon.haufe>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

"Rainer Gerhards" <rgerhards at hq.adiscon.com> wrote on Mon, 17 Dec 2007
11:04:33 +0100:

> Do you think it would be acceptable to not emit messages in this case,
> slightly breaking backward compatibility? Or should I come up with a
> solution. The later would probably require a specific option setting
> to disable the -m option and its default (somehow telling "I am not
> running in v2 compatibility mode").

This makes me think about sysklogd compatibility. How about the
following: If there's no rsyslog.conf but a syslog.conf then rsyslogd
could go to v2/sysklogd-compatibility mode. If not, it should run in
native mode.
This should improve compatibility in "drop-in replacement for stock
sysklogd" setups.

Regards,
sur5r
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHaDjK1YAhDic+adYRArd6AJ9fXRZU1TnW4L/eKRSfcE3x/G/PMACfbDEF
GgU6vviKYQQofAeAJ9vEnDw=
=6u5b
-----END PGP SIGNATURE-----

From mic at npgx.com.au  Wed Dec 19 04:45:00 2007
From: mic at npgx.com.au (Michael Mansour)
Date: Wed, 19 Dec 2007 13:45:00 +1000
Subject: [rsyslog] backward compatibility...
In-Reply-To: <20071218221654.27ec59a2@marathon.haufe>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
	<20071218221654.27ec59a2@marathon.haufe>
Message-ID: <20071219033907.M7098@npgx.com.au>

Hi,

> "Rainer Gerhards" <rgerhards at hq.adiscon.com> wrote on Mon, 17 Dec 2007
> 11:04:33 +0100:
> 
> > Do you think it would be acceptable to not emit messages in this case,
> > slightly breaking backward compatibility? Or should I come up with a
> > solution. The later would probably require a specific option setting
> > to disable the -m option and its default (somehow telling "I am not
> > running in v2 compatibility mode").
> 
> This makes me think about sysklogd compatibility. How about the
> following: If there's no rsyslog.conf but a syslog.conf then rsyslogd
> could go to v2/sysklogd-compatibility mode. If not, it should run in
> native mode.
> This should improve compatibility in "drop-in replacement for stock
> sysklogd" setups.

That's true, but I've also found developers are happy to cater for rsyslog
through updates to their software. For example, I use drbdlinks (software for
clusters) and when I introduced rsyslog into my environment I found drbdlinks
incompatible with it. 

I put a workaround in place and at the same time emailed the drbdlinks
developer, and he introduced rsyslog support into his code, provided an update
to his software and mentioned me in the suggestion.

Jamie Cameron of Webmin, Virtualmin and Virtualmin Pro is/has also
introduced/introducing support for rsyslog into his products, since some
Webmin functionality looks for the /etc/syslog.conf file to allow some
functionality in the apps and will/now looks for /etc/rsyslog.conf also.

I saw will/now since I have his confirmation it's on his Todo list from a few
months back, but I haven't tested any updates from him since we spoke about it
back then.

The moral of this story is, for any software you use, ask the developers to
add rsyslog support if things they develop only supports syslog.

Regards,

Michael.

> Regards,
> sur5r
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
> 
> iD8DBQFHaDjK1YAhDic+adYRArd6AJ9fXRZU1TnW4L/eKRSfcE3x/G/PMACfbDEF
> GgU6vviKYQQofAeAJ9vEnDw=
> =6u5b
> -----END PGP SIGNATURE-----
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
------- End of Original Message -------



From mbiebl at gmail.com  Wed Dec 19 05:51:57 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 19 Dec 2007 05:51:57 +0100
Subject: [rsyslog] backward compatibility...
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712182051m2ef76c2sb401e0bd45904382@mail.gmail.com>

2007/12/17, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Hi all,
>
> I would appreciate your feedback on backward compatibility. I have
> started to create a loadable input module interface in rsyslog v3. My
> first input module is the mark message handler (because it has almost no
> code, so it was a good example of what needs to be done...). I have now
> basically have finished an initial version. The interface needs to be
> changed in the longer term, but I'll postpone that until I've done a bit
> more work on the config interface. Now I am facing a problem with
> backward compatibility:
>
> In sysklogd/rsyslog pre-v3, the -m option controls how often mark
> messages are written (with -m0 disabling this mechanism). In v3, the

One thing I was wondering:
If you intend to shift all (even core) functionality into loadable
modules, how do do you handle things like --help or available command
line options like -m
Do you want to hardcode it  or will you provide an interface, where
rsyslog will query the module about its help message and available
options.

I'm also still a bit uncertain, if moving everything resp. core
functionality to modules is a good idea (for problems you already
mentioned).
Imho having all core functionality in a single binary is simply much
more robust and fool proof. For things like the SQL db output plugin,
the module interface is great, because it avoids to pull in large
library and package dependencies and allows to install them on a as
need basis. For other functionality I still need to recognize the
benefits.

Rainer, could you roughly sketch, how you envision to break rsyslog
into loadable modules in v3. Which kind of functionality would be
loadable as module, which functionality do you plan to keep in the
rsyslogd binary. A listing of all (planned) modules + the provided
functionality and requirements would really help.

Another thing: Say you move the regexp support into a separate module.
If a regexp is then used in rsyslog.conf, will you bail out with an
error, simply print a warning (which could go unnoticed and the poor
administrator doesn't know why his regexp doesn't know) or load
modules on demand.
For the latter you'd need some kind of interface to query the *.so
files for their supported functionality. I.e. the modules would export
a list of config directives it supports and rsyslog could upon startup
query each available module and create a map.
So, e.g. the ommysql module would export its support for the :ommysql:
config directive.
Whenever rsyslog finds such a config directive it could/would load the
modules on demand.
Same could be done for the command line parameters.
The imklog module would export, that it supports the -m command line
parameter. Whenever that commandline parameter is used, rsyslog would
know which module to load.

There are only rough ideas and there is certainly still much to
consider. But what do you think about the basic idea?

Cheers,
Michael

Cheers,
Michael




-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From mbiebl at gmail.com  Wed Dec 19 09:27:13 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 19 Dec 2007 09:27:13 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <1197994886.14826.2.camel@localhost.localdomain>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
	<4760F395.4030502@ipax.at>
	<577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>
	<aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>
	<aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com>
	<aa76cdac0712161438n69382660u9d2a082914d717c9@mail.gmail.com>
	<aa76cdac0712162009q9fe3ecaq407df79296e4baff@mail.gmail.com>
	<1197994886.14826.2.camel@localhost.localdomain>
Message-ID: <aa76cdac0712190027k55824b7cn582acd6e7a1a6feb@mail.gmail.com>

2007/12/18, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> This patch now also made it to the repository. Applied to both v2
> (previously 1.20.2) and v3.

Cool, thanks.
Remains to update the documentation doc/rsconf1_includeconfig.html.
I'm usually not good a writing docs, but if you want I can provide a patch ;-)

Cheers,
Michael
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Wed Dec 19 09:33:01 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 19 Dec 2007 09:33:01 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <aa76cdac0712190027k55824b7cn582acd6e7a1a6feb@mail.gmail.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com><4760F395.4030502@ipax.at><577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com><aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com><aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com><aa76cdac0712161438n69382660u9d2a082914d717c9@mail.gmail.com><aa76cdac0712162009q9fe3ecaq407df79296e4baff@mail.gmail.com><1197994886.14826.2.camel@localhost.localdomain>
	<aa76cdac0712190027k55824b7cn582acd6e7a1a6feb@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086A8@grfint2.intern.adiscon.com>

Oops... this time I forgot the doc (I always try to have at least a
minimal description there). I just added two lines, maybe you have a
look. I am a bit under pressure because I just released 1.21.1, but now
re-created the tarball with the somewhat enhanced doc. If you are up for
a better doc, just sent me a patch ;) It'll appear in the next release.

Release notification is due in a few minutes...

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Wednesday, December 19, 2007 9:27 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Problems with IncludeConfig
> 
> 2007/12/18, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > This patch now also made it to the repository. Applied to both v2
> > (previously 1.20.2) and v3.
> 
> Cool, thanks.
> Remains to update the documentation doc/rsconf1_includeconfig.html.
> I'm usually not good a writing docs, but if you want I can provide a
> patch ;-)
> 
> Cheers,
> Michael
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Wed Dec 19 09:42:35 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 19 Dec 2007 09:42:35 +0100
Subject: [rsyslog] rsyslog 1.21.0 released
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086A9@grfint2.intern.adiscon.com>

Hi all,

as I mentioned a few minutes ago, I have just released version 1.21.0.
First question may be "why 1.21.0 and not 2.0.0"? The answer is that
varmojfekoj provided a patch that allowed me to finally enable GSS-API
support in rsyslog. While I really like this, I did not find it
appropriate to include such a major new feature in the first release of
a 2.0.0 *stable*. So I went back to the v1 releases. If I do not hear of
any serious problems, I'll re-release that version as 2.0.0.

What also brings us to the new features: GSS-API is obviously now there.
There is very limited documentation that goes with it. I'll focus on
that in the v3 releases, with docs being propagated down to v2 as
possible. If anyone would like to contribute doc, I'd obviously be very
happy ;) I currently do not intend to spent too much time with it,
because I do not like to distract myself from v3 until I managed to get
at least the most important things up and running.

V1.21.0 also contains Michael Biebl's $IncludeConfig enhancement, many
cleanups as well as the fix that hopefully addresses the segfault. So
far, I have neither heard a success or failure report. For success, it
obviously is too early, rsyslogd often needed a few days to experience
the fault. I still hope for the best...

A quick note on v3: things are going pretty well. I'll post an answer to
Michael Biebl's note on what will become modular a bit later, it needs
some time. GSS-API was the last major new feature I'll accept for v2.
Everything else will go in v3. I expect to release v2 by the end of the
week and the initial v3 maybe a week later. I will make a preview
tarball of v3 available soon so that those interested can have a look at
it (and, of course, CVS head already has it).

Download links and changelog can be found on the usual places on
www.rsyslog.com.

As always, feedback is appreciated.

Rainer Gerhards 


From mbiebl at gmail.com  Wed Dec 19 09:50:44 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 19 Dec 2007 09:50:44 +0100
Subject: [rsyslog] rsyslog 1.21.0 released
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086A9@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA3086A9@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712190050u5ec134fbp6675ab637602ae89@mail.gmail.com>

2007/12/19, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Download links and changelog can be found on the usual places on
> www.rsyslog.com.

Download link seems to be missing.

Cheers,
Michael

P.S: what about posting md5sums for release tarballs in the annoucment
email and on www.rsyslog.com?

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From mbiebl at gmail.com  Wed Dec 19 09:52:04 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 19 Dec 2007 09:52:04 +0100
Subject: [rsyslog] rsyslog 1.21.0 released
In-Reply-To: <aa76cdac0712190050u5ec134fbp6675ab637602ae89@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA3086A9@grfint2.intern.adiscon.com>
	<aa76cdac0712190050u5ec134fbp6675ab637602ae89@mail.gmail.com>
Message-ID: <aa76cdac0712190052g1ec7035bhf68270abd32bf45b@mail.gmail.com>

2007/12/19, Michael Biebl <mbiebl at gmail.com>:
> 2007/12/19, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > Download links and changelog can be found on the usual places on
> > www.rsyslog.com.
>
> Download link seems to be missing.

Ah, seems to be there now.

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Wed Dec 19 10:13:18 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 19 Dec 2007 10:13:18 +0100
Subject: [rsyslog] rsyslog 1.21.0 released
In-Reply-To: <aa76cdac0712190052g1ec7035bhf68270abd32bf45b@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA3086A9@grfint2.intern.adiscon.com><aa76cdac0712190050u5ec134fbp6675ab637602ae89@mail.gmail.com>
	<aa76cdac0712190052g1ec7035bhf68270abd32bf45b@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086AB@grfint2.intern.adiscon.com>

I guess it was a cache issue, I always start by creating the download,
then changelog entry, etc... The md5sum is part of the download
description. I thought it is OK to post it there, only. It also somewhat
saved me when I applied the last-minute doc patch and re-released. With
the md5sum in the mail, it would have created some confusion. What do
you say?

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Wednesday, December 19, 2007 9:52 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 1.21.0 released
> 
> 2007/12/19, Michael Biebl <mbiebl at gmail.com>:
> > 2007/12/19, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > > Download links and changelog can be found on the usual places on
> > > www.rsyslog.com.
> >
> > Download link seems to be missing.
> 
> Ah, seems to be there now.
> 
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From mbiebl at gmail.com  Wed Dec 19 10:21:29 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 19 Dec 2007 10:21:29 +0100
Subject: [rsyslog] rsyslog 1.21.0 released
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086AB@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA3086A9@grfint2.intern.adiscon.com>
	<aa76cdac0712190050u5ec134fbp6675ab637602ae89@mail.gmail.com>
	<aa76cdac0712190052g1ec7035bhf68270abd32bf45b@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086AB@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712190121icbf2cb7v70c672a68bba9329@mail.gmail.com>

2007/12/19, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> I guess it was a cache issue, I always start by creating the download,
> then changelog entry, etc... The md5sum is part of the download
> description. I thought it is OK to post it there, only. It also somewhat
> saved me when I applied the last-minute doc patch and re-released. With
> the md5sum in the mail, it would have created some confusion. What do
> you say?

Agreed. Posting the md5sum on the web page should be fine.

Cheers,
Michael
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Wed Dec 19 12:37:09 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 19 Dec 2007 12:37:09 +0100
Subject: [rsyslog] backward compatibility...
In-Reply-To: <aa76cdac0712182051m2ef76c2sb401e0bd45904382@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
	<aa76cdac0712182051m2ef76c2sb401e0bd45904382@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086B1@grfint2.intern.adiscon.com>

Michael and all,

I took me a while to craft a response to your excellent question. I have
done this as a blog post so that it is easier to reference it in the
future.

I suggest that everyone interested in the v3 design has a look at it,
because it describes the way I am heading. If someone doesn't like that,
it is now time to speak up - in a few weeks it will probably be
impossible to change routes.

http://rgerhards.blogspot.com/2007/12/modules-core-functionality-and-rsy
slog.html

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Wednesday, December 19, 2007 5:52 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] backward compatibility...
> 
> 2007/12/17, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > Hi all,
> >
> > I would appreciate your feedback on backward compatibility. I have
> > started to create a loadable input module interface in rsyslog v3.
My
> > first input module is the mark message handler (because it has
almost
> no
> > code, so it was a good example of what needs to be done...). I have
> now
> > basically have finished an initial version. The interface needs to
be
> > changed in the longer term, but I'll postpone that until I've done a
> bit
> > more work on the config interface. Now I am facing a problem with
> > backward compatibility:
> >
> > In sysklogd/rsyslog pre-v3, the -m option controls how often mark
> > messages are written (with -m0 disabling this mechanism). In v3, the
> 
> One thing I was wondering:
> If you intend to shift all (even core) functionality into loadable
> modules, how do do you handle things like --help or available command
> line options like -m
> Do you want to hardcode it  or will you provide an interface, where
> rsyslog will query the module about its help message and available
> options.
> 
> I'm also still a bit uncertain, if moving everything resp. core
> functionality to modules is a good idea (for problems you already
> mentioned).
> Imho having all core functionality in a single binary is simply much
> more robust and fool proof. For things like the SQL db output plugin,
> the module interface is great, because it avoids to pull in large
> library and package dependencies and allows to install them on a as
> need basis. For other functionality I still need to recognize the
> benefits.
> 
> Rainer, could you roughly sketch, how you envision to break rsyslog
> into loadable modules in v3. Which kind of functionality would be
> loadable as module, which functionality do you plan to keep in the
> rsyslogd binary. A listing of all (planned) modules + the provided
> functionality and requirements would really help.
> 
> Another thing: Say you move the regexp support into a separate module.
> If a regexp is then used in rsyslog.conf, will you bail out with an
> error, simply print a warning (which could go unnoticed and the poor
> administrator doesn't know why his regexp doesn't know) or load
> modules on demand.
> For the latter you'd need some kind of interface to query the *.so
> files for their supported functionality. I.e. the modules would export
> a list of config directives it supports and rsyslog could upon startup
> query each available module and create a map.
> So, e.g. the ommysql module would export its support for the :ommysql:
> config directive.
> Whenever rsyslog finds such a config directive it could/would load the
> modules on demand.
> Same could be done for the command line parameters.
> The imklog module would export, that it supports the -m command line
> parameter. Whenever that commandline parameter is used, rsyslog would
> know which module to load.
> 
> There are only rough ideas and there is certainly still much to
> consider. But what do you think about the basic idea?
> 
> Cheers,
> Michael
> 
> Cheers,
> Michael
> 
> 
> 
> 
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From mbiebl at gmail.com  Thu Dec 20 10:08:56 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Thu, 20 Dec 2007 10:08:56 +0100
Subject: [rsyslog] backward compatibility...
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086B1@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
	<aa76cdac0712182051m2ef76c2sb401e0bd45904382@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086B1@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712200108q7e9eb058g2e179c2fe00f65d0@mail.gmail.com>

2007/12/19, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Michael and all,
>
> I took me a while to craft a response to your excellent question. I have
> done this as a blog post so that it is easier to reference it in the
> future.
>
> I suggest that everyone interested in the v3 design has a look at it,
> because it describes the way I am heading. If someone doesn't like that,
> it is now time to speak up - in a few weeks it will probably be
> impossible to change routes.

Based on your recent blog post here are some thoughts of mine. Please
keep in mind, that being (Debian) package maintainer, so I speak from
a distributors pov.

1.) external dependencies
Having loadable modules for stuff like MySQL/PostgreSQL output support
is great (for a package maintainer)
This means I can package a basic rsyslog package with minimal
dependencies (glibc) and people can chose to install
rsyslog-(mysql,pgsql) based on their needs.
Here in this case, loadable modules have a real benefit.
The only remaining external dependencies currently in rsyslog are libz
(NETZIP support) and gssapi-krb (KERBEROS support). If those
functionality would be put into a loadable module, I'd support that.
2.) memory usage
I don't think this is a real issue, even for embedded systems. See
point 1. If we manage to put external dependencies into loadable
modules, that would be sufficient imho.
Modularising everything won't have that much of an memory benefit
imho. Given that a standard rsyslogd is about 1M RSS, this doesn't
imply the need for more modularization.
3.) Security
You mentioned, that you try to improve security through modules.
Usually, having loadable module support is considered a security risk.
One messed up $IncludeConfig directive (or manipulated through a
malicious attacker), and you load potentially hazardous modules.
Loadable modules support introduces a much bigger attack vector.

I'm not a SELinux guy. But I'd be interested if loadable modules could
cause trouble in putting rsyslog in it's own security domain. Maybe
the fedora guys can comment on this.

4.) (code) modularization
Writing modular code doesn't strictly imply that the modules have to
be loadable *.so files.
You can still write modular code, with a strict API etc. and organize
it e.g. via sub directories.
5.) Performance penalty through loadable modules.
I could be wrong on this point, but given that you have pointers on
functions, when you use modules (dlsym), there is an additional
indirection on each function call. This could have a performance
impact for core functionality. This is just speculation and should be
tested/evaluated beforehand. After all, rsyslog tries to meet
high-performance needs, too.
6.) Inconvenience
This is just a gut feeling, but having to load all sorts of moduls
first, before rsyslog does anything, could prove cumbersome. As
administrator you'd have to know, which modules to load, to get
rsyslog to do what you want. This means additional effort (reading
docs) and inconvenience.
7.) Robustness
Having a single binary can prove to be a live safer. E.g. you could
simply copy the rsyslogd binary from another machine if there is
something wrong with your local copy.
As you were talking about embedded systems, having the ability to
compile a static binary including all functionality is a definite
plus. There might be platforms out there which don't support dlopen().

I'm still missing the complete picture, too. It's still a bit too vague for me.

You were talking about input, output and filter modules. Rainer, could
cou try to complete this list somehow, maybe draw/sketch a flow
diagram, marking the modules.

input: local, kernel and network, ...
filter: regexp, ...
output: mysql, pgsql, file, forward...
authorization?


I hope, this all doesn't sound too negative. But before going all
modular, all these issues should be considered imho.

Hopefully these comments will help.

Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Thu Dec 20 10:24:49 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 20 Dec 2007 10:24:49 +0100
Subject: [rsyslog] backward compatibility...
In-Reply-To: <aa76cdac0712200108q7e9eb058g2e179c2fe00f65d0@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com><aa76cdac0712182051m2ef76c2sb401e0bd45904382@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA3086B1@grfint2.intern.adiscon.com>
	<aa76cdac0712200108q7e9eb058g2e179c2fe00f65d0@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086B9@grfint2.intern.adiscon.com>

Michael,

thanks for the good points. I'll provide a more in-depth reply later,
but noticed that I missed one important point in the blog post.

Loadable modules are also (at least partly) about platform independence.
Let me elaborate this with an example:

I have yet to fully port rsyslog to solaris. The reason I stopped is
that Solaris uses a mechanism different from Linux to handle local
logging (e.g. via logger). I didn't like to mess up even more with the
code that pulls different log sources.

With the loadable input plugins, there is a very clean solution. If you
run on Linux, just load imLocaLogLinux and if you happen to run under
Solaris load imLocalLogSolaris instead. No conditional compilation, no
complex code. Plus, anyone interested to provide an input implementation
for whatever weird system is free to do so.

So far to what I forgot to mention...

A quick reply: I think we agree that modular code is good to have. Our
main difference seems to be if it should be dynamically or statically
linked together. I am right now modularizing the inputs, have done two
so far and going to the local log socket next. That will also bring some
practical experience, which it already did. So I'll do a round of
modularization and then look at the result to fine-tune it. If it will
be loadable or not will not really be a major concern at that stage. I
just wanted to provide reasoning why I develop concurrently to this very
good discussion.

Again, a more in-depth reply a bit later.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Thursday, December 20, 2007 10:09 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] backward compatibility...
> 
> 2007/12/19, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > Michael and all,
> >
> > I took me a while to craft a response to your excellent question. I
> have
> > done this as a blog post so that it is easier to reference it in the
> > future.
> >
> > I suggest that everyone interested in the v3 design has a look at
it,
> > because it describes the way I am heading. If someone doesn't like
> that,
> > it is now time to speak up - in a few weeks it will probably be
> > impossible to change routes.
> 
> Based on your recent blog post here are some thoughts of mine. Please
> keep in mind, that being (Debian) package maintainer, so I speak from
> a distributors pov.
> 
> 1.) external dependencies
> Having loadable modules for stuff like MySQL/PostgreSQL output support
> is great (for a package maintainer)
> This means I can package a basic rsyslog package with minimal
> dependencies (glibc) and people can chose to install
> rsyslog-(mysql,pgsql) based on their needs.
> Here in this case, loadable modules have a real benefit.
> The only remaining external dependencies currently in rsyslog are libz
> (NETZIP support) and gssapi-krb (KERBEROS support). If those
> functionality would be put into a loadable module, I'd support that.
> 2.) memory usage
> I don't think this is a real issue, even for embedded systems. See
> point 1. If we manage to put external dependencies into loadable
> modules, that would be sufficient imho.
> Modularising everything won't have that much of an memory benefit
> imho. Given that a standard rsyslogd is about 1M RSS, this doesn't
> imply the need for more modularization.
> 3.) Security
> You mentioned, that you try to improve security through modules.
> Usually, having loadable module support is considered a security risk.
> One messed up $IncludeConfig directive (or manipulated through a
> malicious attacker), and you load potentially hazardous modules.
> Loadable modules support introduces a much bigger attack vector.
> 
> I'm not a SELinux guy. But I'd be interested if loadable modules could
> cause trouble in putting rsyslog in it's own security domain. Maybe
> the fedora guys can comment on this.
> 
> 4.) (code) modularization
> Writing modular code doesn't strictly imply that the modules have to
> be loadable *.so files.
> You can still write modular code, with a strict API etc. and organize
> it e.g. via sub directories.
> 5.) Performance penalty through loadable modules.
> I could be wrong on this point, but given that you have pointers on
> functions, when you use modules (dlsym), there is an additional
> indirection on each function call. This could have a performance
> impact for core functionality. This is just speculation and should be
> tested/evaluated beforehand. After all, rsyslog tries to meet
> high-performance needs, too.
> 6.) Inconvenience
> This is just a gut feeling, but having to load all sorts of moduls
> first, before rsyslog does anything, could prove cumbersome. As
> administrator you'd have to know, which modules to load, to get
> rsyslog to do what you want. This means additional effort (reading
> docs) and inconvenience.
> 7.) Robustness
> Having a single binary can prove to be a live safer. E.g. you could
> simply copy the rsyslogd binary from another machine if there is
> something wrong with your local copy.
> As you were talking about embedded systems, having the ability to
> compile a static binary including all functionality is a definite
> plus. There might be platforms out there which don't support dlopen().
> 
> I'm still missing the complete picture, too. It's still a bit too
vague
> for me.
> 
> You were talking about input, output and filter modules. Rainer, could
> cou try to complete this list somehow, maybe draw/sketch a flow
> diagram, marking the modules.
> 
> input: local, kernel and network, ...
> filter: regexp, ...
> output: mysql, pgsql, file, forward...
> authorization?
> 
> 
> I hope, this all doesn't sound too negative. But before going all
> modular, all these issues should be considered imho.
> 
> Hopefully these comments will help.
> 
> Michael
> 
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From pvrabec at redhat.com  Thu Dec 20 10:47:56 2007
From: pvrabec at redhat.com (Peter Vrabec)
Date: Thu, 20 Dec 2007 10:47:56 +0100
Subject: [rsyslog] backward compatibility...
In-Reply-To: <aa76cdac0712200108q7e9eb058g2e179c2fe00f65d0@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086B1@grfint2.intern.adiscon.com>
	<aa76cdac0712200108q7e9eb058g2e179c2fe00f65d0@mail.gmail.com>
Message-ID: <200712201047.56857.pvrabec@redhat.com>

just a quick response,

On Thursday 20 December 2007 10:08:56 am Michael Biebl wrote:
> 2007/12/19, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > Michael and all,
> >
> > I took me a while to craft a response to your excellent question. I have
> > done this as a blog post so that it is easier to reference it in the
> > future.
> >
> > I suggest that everyone interested in the v3 design has a look at it,
> > because it describes the way I am heading. If someone doesn't like that,
> > it is now time to speak up - in a few weeks it will probably be
> > impossible to change routes.
>
> Based on your recent blog post here are some thoughts of mine. Please
> keep in mind, that being (Debian) package maintainer, so I speak from
> a distributors pov.
>

> 3.) Security
> You mentioned, that you try to improve security through modules.
> Usually, having loadable module support is considered a security risk.
> One messed up $IncludeConfig directive (or manipulated through a
> malicious attacker), and you load potentially hazardous modules.
> Loadable modules support introduces a much bigger attack vector.
>
> I'm not a SELinux guy. But I'd be interested if loadable modules could
> cause trouble in putting rsyslog in it's own security domain. Maybe
> the fedora guys can comment on this.
I'm not SELinux guy either, but I don't see any problem here. Modules and 
binary will be in same domain.



From rgerhards at hq.adiscon.com  Thu Dec 20 15:56:33 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 20 Dec 2007 15:56:33 +0100
Subject: [rsyslog] Another compatibility question...
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086BF@grfint2.intern.adiscon.com>

Hi all,

I'd appreciate your feedback on this question.

Historically, syslog messages were not only received on port 514, but
were also send from that port. This is an artifact from the fact that
early syslogd used a single port both for sending and receiving. So far,
I have maintained that and done so with some pain. With v3, I would like
to get rid of this legacy (because it introduces inter-module
dependencies and also makes multiple listeners really complicated).

So I wonder if anyone over here finds this "feature" (syslog messages
are sent from port 514 by default) important. What pops up my mind are
firewall rules. Is there anything else broken if we change that?

Feedback is highly appreciated,
Rainer


From bakers at web-ster.com  Thu Dec 20 18:04:33 2007
From: bakers at web-ster.com (Scott Baker)
Date: Thu, 20 Dec 2007 09:04:33 -0800
Subject: [rsyslog] Syslog logging taking too much disk write?
Message-ID: <476AA0A1.5080300@web-ster.com>

I have a semi busy syslog server which logs about 40 or 50 lines per
second. I noticed using dstat that it's taking a TON of disk write.
In the megaBYTES per second range. This seems extremely high to me
so I wrote a script to test it.

http://www.perturb.org/code/log_lines.pl

First download dstat, gkrellm, or iostat and look at what your
normal disk write is per second. On my underused desktop it hovers
around zero or maybe 25KB/s per second.

If I run my script and tell it to log 10 lines to syslog per second

perl log_lines.pl 10

the write usage spikes up to ~160KB/s. We're only writing about 740
bytes of data per second so why is the disk write so high?

If I tell my script to instead log to a local file in /tmp at 10
lines per second

perl log_lines.pl 10 -t

The write usage spikes to ~120KB/s about once every 5 seconds (as
the buffer fills up). I'm curious why syslog writing is so much more
intense? It seems to me on a really busy syslog server that you
could seriously smoke your HDs with all the writing it's doing. Am I
missing something?

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Thu Dec 20 18:39:40 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 20 Dec 2007 18:39:40 +0100
Subject: [rsyslog] Syslog logging taking too much disk write?
In-Reply-To: <476AA0A1.5080300@web-ster.com>
References: <476AA0A1.5080300@web-ster.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086C1@grfint2.intern.adiscon.com>

Could you post your rsyslog.conf please. Without seeing it, it sounds
like you sync the disc with every write. Unfortunately, this is the
legacy default. You can turn it off with placing a dash before the
selector line:

e.g. *.* /var/log/whatever

syncs after each write, while

*.* -/var/log/whatever

uses buffering - which should be fairly safe with decent hardware and a
UPS...

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Scott Baker
> Sent: Thursday, December 20, 2007 6:05 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] Syslog logging taking too much disk write?
> 
> I have a semi busy syslog server which logs about 40 or 50 lines per
> second. I noticed using dstat that it's taking a TON of disk write.
> In the megaBYTES per second range. This seems extremely high to me
> so I wrote a script to test it.
> 
> http://www.perturb.org/code/log_lines.pl
> 
> First download dstat, gkrellm, or iostat and look at what your
> normal disk write is per second. On my underused desktop it hovers
> around zero or maybe 25KB/s per second.
> 
> If I run my script and tell it to log 10 lines to syslog per second
> 
> perl log_lines.pl 10
> 
> the write usage spikes up to ~160KB/s. We're only writing about 740
> bytes of data per second so why is the disk write so high?
> 
> If I tell my script to instead log to a local file in /tmp at 10
> lines per second
> 
> perl log_lines.pl 10 -t
> 
> The write usage spikes to ~120KB/s about once every 5 seconds (as
> the buffer fills up). I'm curious why syslog writing is so much more
> intense? It seems to me on a really busy syslog server that you
> could seriously smoke your HDs with all the writing it's doing. Am I
> missing something?
> 
> --
> Scott Baker - Canby Telcom
> RHCE - System Administrator - 503.266.8253
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From bakers at web-ster.com  Thu Dec 20 19:09:06 2007
From: bakers at web-ster.com (Scott Baker)
Date: Thu, 20 Dec 2007 10:09:06 -0800
Subject: [rsyslog] Syslog logging taking too much disk write?
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086C1@grfint2.intern.adiscon.com>
References: <476AA0A1.5080300@web-ster.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086C1@grfint2.intern.adiscon.com>
Message-ID: <476AAFC2.207@web-ster.com>

Rainer Gerhards wrote:
> Could you post your rsyslog.conf please. Without seeing it, it sounds
> like you sync the disc with every write. Unfortunately, this is the
> legacy default. You can turn it off with placing a dash before the
> selector line:
> 
> e.g. *.* /var/log/whatever
> 
> syncs after each write, while
> 
> *.* -/var/log/whatever
> 
> uses buffering - which should be fairly safe with decent hardware and a
> UPS...

I was using the stock Fedora 8 config which does indeed sync every
write to disk. I put a - in front of all the options and my disk
usage has gone way down!

What sort of buffer does rsyslog use? How often does that buffer
fill up and write to disk? I only ask because sometimes we tail
logfiles in real time to watch things like DHCP etc as they happen.

If the log is 30 seconds behind that may cause a problem.

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From bakers at web-ster.com  Thu Dec 20 19:34:06 2007
From: bakers at web-ster.com (Scott Baker)
Date: Thu, 20 Dec 2007 10:34:06 -0800
Subject: [rsyslog] Port 514
Message-ID: <476AB59E.9050706@web-ster.com>

> Hi all,
> 
> I'd appreciate your feedback on this question.
> 
> Historically, syslog messages were not only received on port 514, but
> were also send from that port. This is an artifact from the fact that
> early syslogd used a single port both for sending and receiving. So far,
> I have maintained that and done so with some pain. With v3, I would like
> to get rid of this legacy (because it introduces inter-module
> dependencies and also makes multiple listeners really complicated).
> 
> So I wonder if anyone over here finds this "feature" (syslog messages
> are sent from port 514 by default) important. What pops up my mind are
> firewall rules. Is there anything else broken if we change that?

We use syslog pretty extensively here at the ISP I work for and I
don't see how this would affect anything we do. Firewall rules are
almost always bast on destination port, not sending port. I don't
think changing this would cause any problems.

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Thu Dec 20 21:29:53 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 20 Dec 2007 21:29:53 +0100
Subject: [rsyslog] Syslog logging taking too much disk write?
In-Reply-To: <476AAFC2.207@web-ster.com>
References: <476AA0A1.5080300@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C1@grfint2.intern.adiscon.com>
	<476AAFC2.207@web-ster.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086C6@grfint2.intern.adiscon.com>

Hi Scott, 

[snip]

> What sort of buffer does rsyslog use? How often does that buffer
> fill up and write to disk? I only ask because sometimes we tail
> logfiles in real time to watch things like DHCP etc as they happen.
> 
> If the log is 30 seconds behind that may cause a problem.

I suggest you first have a look at:

http://rgerhards.blogspot.com/2007/08/why-is-rsyslog-multi-threaded-and-
is-it.html

The bottom line with the current release is that there is no specific
"output file buffer" - rsyslog writes to the OS and it buffers. Works
pretty well for rapidly incoming traffic. What we do, however, is have a
buffer between messages received and processed. This is primarily meant
for taking care of either a) message bursts or b) temporarily slow
outputs (e.g., initial database connection). If you don't like that
buffer, you can limit it via rsyslog.conf (the default is 10,000
messages max).

With rsyslog v3, there will be a more complex threading scheme and also
some more buffers. I can also envision it will support buffered writes,
enhancing the IO throughput for files (e.g. write in 64K chunks or every
n (milli)seconds, whatever comes first). I am currently in the early
stages of v3, but it will be a very exciting release. And I of course
appreciate feedback from demanding environments (hint, hint ;)).

But v3 will never hold messages longer than a) absolutely necessary or
b) configured by operator. There is a good post on v3 right on the main
page of my blog.

I hope that clarifies, if not, please let me know.

Rainer


From bakers at web-ster.com  Thu Dec 20 22:29:39 2007
From: bakers at web-ster.com (Scott Baker)
Date: Thu, 20 Dec 2007 13:29:39 -0800
Subject: [rsyslog] Separating logs by host?
Message-ID: <476ADEC3.5040506@web-ster.com>

Is it possible with rsyslog to separate logs based on IP address and
not logging facility?

I have about 10 pieces of equipment that I'd like to get syslog
fromt, but I've run out of log facilities (local1 through local7).
It'd be nice if each of those pieces of equipment could have it's
own log. I can't find anything in the documentation that covers this.

Am I missing it?

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Thu Dec 20 22:33:46 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 20 Dec 2007 22:33:46 +0100
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <476ADEC3.5040506@web-ster.com>
References: <476ADEC3.5040506@web-ster.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>

Looks like doc can need some more guides ;)

Does this help: http://www.rsyslog.com/Article60.phtml

Rainer 

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Scott Baker
> Sent: Thursday, December 20, 2007 10:30 PM
> To: rsyslog-users
> Subject: [rsyslog] Separating logs by host?
> 
> Is it possible with rsyslog to separate logs based on IP address and
> not logging facility?
> 
> I have about 10 pieces of equipment that I'd like to get syslog
> fromt, but I've run out of log facilities (local1 through local7).
> It'd be nice if each of those pieces of equipment could have it's
> own log. I can't find anything in the documentation that covers this.
> 
> Am I missing it?
> 
> -- 
> Scott Baker - Canby Telcom
> RHCE - System Administrator - 503.266.8253
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> 


From rgerhards at hq.adiscon.com  Thu Dec 20 22:45:06 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 20 Dec 2007 22:45:06 +0100
Subject: [rsyslog] backward compatibility...
In-Reply-To: <aa76cdac0712200108q7e9eb058g2e179c2fe00f65d0@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com><aa76cdac0712182051m2ef76c2sb401e0bd45904382@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA3086B1@grfint2.intern.adiscon.com>
	<aa76cdac0712200108q7e9eb058g2e179c2fe00f65d0@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086C9@grfint2.intern.adiscon.com>

Hi Michael,

as promised, a more complete response. See inline below... 

> > I took me a while to craft a response to your excellent 
> question. I have
> > done this as a blog post so that it is easier to reference it in the
> > future.
> >
> > I suggest that everyone interested in the v3 design has a 
> look at it,
> > because it describes the way I am heading. If someone 
> doesn't like that,
> > it is now time to speak up - in a few weeks it will probably be
> > impossible to change routes.
> 
> Based on your recent blog post here are some thoughts of mine. Please
> keep in mind, that being (Debian) package maintainer, so I speak from
> a distributors pov.
> 
> 1.) external dependencies
> Having loadable modules for stuff like MySQL/PostgreSQL output support
> is great (for a package maintainer)
> This means I can package a basic rsyslog package with minimal
> dependencies (glibc) and people can chose to install
> rsyslog-(mysql,pgsql) based on their needs.
> Here in this case, loadable modules have a real benefit.
> The only remaining external dependencies currently in rsyslog are libz
> (NETZIP support) and gssapi-krb (KERBEROS support). If those
> functionality would be put into a loadable module, I'd support that.
> 2.) memory usage
> I don't think this is a real issue, even for embedded systems. See
> point 1. If we manage to put external dependencies into loadable
> modules, that would be sufficient imho.
> Modularising everything won't have that much of an memory benefit
> imho. Given that a standard rsyslogd is about 1M RSS, this doesn't
> imply the need for more modularization.

I tend to agree here with you. However, I think that a clear line needs
to be drawn between "modularization" and "loadable modules". Probably I
did cross this line myself a couple of times without even noticing.

The modularization effort *is* needed, because the current monolythic
code base is hard to maintain and has "evolved" - in contrast to be
designed. While this is not bad, you notice in many instants that the
underlaying design is no longer appropriate to do a number of things. In
order to add more really useful features, the code must be modularized.
This will reduce complexity and hopefully make it less subject to
programming error (which, of course, do still happen). [That doesn't
mean I am about to rewrite everything, more on that below.]

So the bottom line is we need modularization for new features. It also
helps greatly with portability, as I outlined in my other mail.

However, we do NOT necessarily need loadble modules - but so far, I find
them desirable. Either way, it seems to be very important to
differentiate between these two concepts.

> 3.) Security
> You mentioned, that you try to improve security through modules.
> Usually, having loadable module support is considered a security risk.

Sure - but it is the the ability to have loadable modules. This risk
does not scale with the number of modules. The problem is that the
interface can be attacked. It's only safe if you do not offer that
functionality at all. At least this is my point of view.

> One messed up $IncludeConfig directive (or manipulated through a
> malicious attacker), and you load potentially hazardous modules.
> Loadable modules support introduces a much bigger attack vector.

Agreed - but not by providing more than a view, as outlined above. The
module interface tries to mitigate against some risk. For example,
output modules never see the internal structure but only the resulting
strings. Of course, so far everything is in-process so a malicious
module may access whatever it wants. I have plans, howerver, to
optionally execute modules out of process. That would be a good option
for not well known third party modules. Obviously at the expense of
quite a performance hit...

> 
> I'm not a SELinux guy. But I'd be interested if loadable modules could
> cause trouble in putting rsyslog in it's own security domain. Maybe
> the fedora guys can comment on this.

I am not qualified to talk about this, but I think along the lines that
Peter posted...

> 
> 4.) (code) modularization
> Writing modular code doesn't strictly imply that the modules have to
> be loadable *.so files.
> You can still write modular code, with a strict API etc. and organize
> it e.g. via sub directories.

Fully agree.

> 5.) Performance penalty through loadable modules.
> I could be wrong on this point, but given that you have pointers on
> functions, when you use modules (dlsym), there is an additional
> indirection on each function call. This could have a performance
> impact for core functionality. This is just speculation and should be
> tested/evaluated beforehand. After all, rsyslog tries to meet
> high-performance needs, too.

No, that doesn't happen (by design). There is a very slight overhead
during module load, when interfaces are queried and symbols are fixed
up. Some indirection, of course, results from a generic approach, but
that is inevidable if the design is modular. With today's hardware, an
indirect function call is not as much of a problem. Again, that has
nothing to do with loable modules but with non-monolythic code.

> 6.) Inconvenience
> This is just a gut feeling, but having to load all sorts of moduls
> first, before rsyslog does anything, could prove cumbersome. As
> administrator you'd have to know, which modules to load, to get
> rsyslog to do what you want. This means additional effort (reading
> docs) and inconvenience.

That's why I said the "core modules" could be present in the default.
But more later below...

> 7.) Robustness
> Having a single binary can prove to be a live safer. E.g. you could
> simply copy the rsyslogd binary from another machine if there is
> something wrong with your local copy.

Agreed, but copying a directory with a number of executables shouldn't
be that much harder ;)

> As you were talking about embedded systems, having the ability to
> compile a static binary including all functionality is a definite
> plus. There might be platforms out there which don't support dlopen().
That's an excellent point!

> 
> I'm still missing the complete picture, too. It's still a bit 
> too vague for me.

That's probably a problem with what I can convey. As I wrote above, I am
not doing a rewrite, with a new design and a new code base. My goal is
to *migrate* rsyslog (which originally based on sysklogd code) over to a
new, better architecture. I try to make as few compromises as possible,
but every now and then there are some things that can't be ignored. My
backward-compatibility questions on this list are part of the process.
So while I have an idea of where I am heading and what I want to achive,
I do not have a crystal clear view of how that will actually happen. I
am going iteratively, tackeling one feature after another and feeding
experience gained back into the process. I am currently at the input
modules and there are definitely some lessons to learn ;)

> 
> You were talking about input, output and filter modules. Rainer, could
> cou try to complete this list somehow, maybe draw/sketch a flow
> diagram, marking the modules.
> 
> input: local, kernel and network, ...
> filter: regexp, ...
> output: mysql, pgsql, file, forward...
> authorization?

Let me give it a try:

Input: immark, imrfc3195, imklog, imfile, imudp, imtcp, imgssapi?,
imtls, imuxsock, ...

Filter: many (many!)... regexp, substr, tolower, toupper, dnslookup?,
left, right, mysqldate, pgsqldate, whateverdate, md5sum, ...

Output: ommysql, ompgsql, omusrmsg, omfwd (split into omudp, omtcp,
omgssapi, ...), ...

Authorization will be provided by the inputs... But there will be some
helpers. How this will happen is not yet clear to me - that's part of
the iterative process. Initially it will be with the modules and then
possibly be migrated to somewhere else.

I'll try to get you a drawing, but that's always a pain for me ;)

> 
> I hope, this all doesn't sound too negative. 

No, much appreciated. It is extremely useful to get that kind of
feedback.

> But before going all
> modular, all these issues should be considered imho.
> 
> Hopefully these comments will help.

After going through all of this, I begin to think that libtool probably
has the answer. I don't know exactly how to, but I think it can take
loadbale modules and I can tell it to make them into static modules. If
that's the case, I can create loable modules, but the build process will
change that into static linking. If I am wrong with that, I can mangle a
bit with the main entry point names in the modules and achive the same
result with just changes to the build process (I don't know yet how to
modify autotools to do that, but I hope there will be some helping hands
;)).

How does this sound?

Rainer
PS: please keep the comments coming, especially if you don't agree! 


From bakers at web-ster.com  Thu Dec 20 22:46:33 2007
From: bakers at web-ster.com (Scott Baker)
Date: Thu, 20 Dec 2007 13:46:33 -0800
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>
References: <476ADEC3.5040506@web-ster.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>
Message-ID: <476AE2B9.30002@web-ster.com>

Rainer Gerhards wrote:
> Looks like doc can need some more guides ;)
> 
> Does this help: http://www.rsyslog.com/Article60.phtml

Excellent... doing this causes all messages to be logged in that
format. Even local messages from the box (cron, mail, authpriv, etc)
to go into that format.

Can I apply that logging statement to a group of IPs? I have a bunch
of DSL equipment that logs. It would be nice if I could break all my
equipment out like:

Routers: /var/log/routers-%$NOW%.log
DSL: /var/log/dsl-%$NOW%.log
Dial-up: /var/log/dial--%$NOW%.log

Each/Any of those groups could have one or many sources. We have
about 13 unique DSL devices that should all probably go in the same
log for simplicity. It would be REALLY cool to be able to say
something like:

192.168.1.1/24:*.*                    /var/log/dsl-%$NOW%.log
192.168.3.1/24,10.1.1.1/24:*.*        /var/log/routers-%$NOW%.log

Am I just dreaming or is that possible?

Also why is the next version of rsyslog (after reading your blog)
going to be version 3 and not 2.0?

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Thu Dec 20 22:50:11 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 20 Dec 2007 22:50:11 +0100
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <476AE2B9.30002@web-ster.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>
	<476AE2B9.30002@web-ster.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>

I need to think about the real question a bit ;) Not an immediate
solution ... but not hopeless either ;)

On v2/v3 - v2 will be the next stable release:
http://rgerhards.blogspot.com/2007/12/begun-working-on-rsyslog-v3.html 

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Scott Baker
> Sent: Thursday, December 20, 2007 10:47 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Separating logs by host?
> 
> Rainer Gerhards wrote:
> > Looks like doc can need some more guides ;)
> > 
> > Does this help: http://www.rsyslog.com/Article60.phtml
> 
> Excellent... doing this causes all messages to be logged in that
> format. Even local messages from the box (cron, mail, authpriv, etc)
> to go into that format.
> 
> Can I apply that logging statement to a group of IPs? I have a bunch
> of DSL equipment that logs. It would be nice if I could break all my
> equipment out like:
> 
> Routers: /var/log/routers-%$NOW%.log
> DSL: /var/log/dsl-%$NOW%.log
> Dial-up: /var/log/dial--%$NOW%.log
> 
> Each/Any of those groups could have one or many sources. We have
> about 13 unique DSL devices that should all probably go in the same
> log for simplicity. It would be REALLY cool to be able to say
> something like:
> 
> 192.168.1.1/24:*.*                    /var/log/dsl-%$NOW%.log
> 192.168.3.1/24,10.1.1.1/24:*.*        /var/log/routers-%$NOW%.log
> 
> Am I just dreaming or is that possible?
> 
> Also why is the next version of rsyslog (after reading your blog)
> going to be version 3 and not 2.0?
> 
> -- 
> Scott Baker - Canby Telcom
> RHCE - System Administrator - 503.266.8253
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> 


From bakers at web-ster.com  Fri Dec 21 01:23:07 2007
From: bakers at web-ster.com (Scott Baker)
Date: Thu, 20 Dec 2007 16:23:07 -0800
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>	<476AE2B9.30002@web-ster.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>
Message-ID: <476B076B.7090003@web-ster.com>

Rainer Gerhards wrote:
> I need to think about the real question a bit ;) Not an immediate
> solution ... but not hopeless either ;)
> 
> On v2/v3 - v2 will be the next stable release:
> http://rgerhards.blogspot.com/2007/12/begun-working-on-rsyslog-v3.html 

Another good example is that we have several linux mail servers. I
like to aggregate all of their logging to one rsyslog server. They
all log to mail.* on their local machines, and then forward also to
my rsyslog server.

The problem is that shows up in my mail.* on the syslog server, and
thus gets mixed in with the local mail.* from the syslog server
itself. It would be nice to be able to separate those, by IP would
be the easiest.

mail.*				/var/log/maillog
1.2.3.4/32:mail.*		/var/log/mail-server.log

Or something like that...

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From r.bhatia at ipax.at  Fri Dec 21 09:03:04 2007
From: r.bhatia at ipax.at (Raoul Bhatia [IPAX])
Date: Fri, 21 Dec 2007 09:03:04 +0100
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <476B076B.7090003@web-ster.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>	<476AE2B9.30002@web-ster.com>	<577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>
	<476B076B.7090003@web-ster.com>
Message-ID: <476B7338.8050102@ipax.at>

Scott Baker wrote:
> Rainer Gerhards wrote:
>> I need to think about the real question a bit ;) Not an immediate
>> solution ... but not hopeless either ;)
>>
>> On v2/v3 - v2 will be the next stable release:
>> http://rgerhards.blogspot.com/2007/12/begun-working-on-rsyslog-v3.html 
> 
> Another good example is that we have several linux mail servers. I
> like to aggregate all of their logging to one rsyslog server. They
> all log to mail.* on their local machines, and then forward also to
> my rsyslog server.
> 
> The problem is that shows up in my mail.* on the syslog server, and
> thus gets mixed in with the local mail.* from the syslog server
> itself. It would be nice to be able to separate those, by IP would
> be the easiest.
> 
> mail.*				/var/log/maillog
> 1.2.3.4/32:mail.*		/var/log/mail-server.log
> 
> Or something like that...

might logging to an (my|pg)sql server solve all your issues?

cheers,
raoul
-- 
____________________________________________________________________
DI (FH) Raoul Bhatia M.Sc.          email.          r.bhatia at ipax.at
Technischer Leiter

IPAX - Aloy Bhatia Hava OEG         web.          http://www.ipax.at
Barawitzkagasse 10/2/2/11           email.            office at ipax.at
1190 Wien                           tel.               +43 1 3670030
FN 277995t HG Wien                  fax.            +43 1 3670030 15
____________________________________________________________________


From rgerhards at hq.adiscon.com  Fri Dec 21 11:07:36 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 21 Dec 2007 11:07:36 +0100
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <476B076B.7090003@web-ster.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>	<476AE2B9.30002@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>
	<476B076B.7090003@web-ster.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086D0@grfint2.intern.adiscon.com>

Mmhhh... once we have expression support, that will be an easy thing to
do. Currently, I think there are a number of clumpsy work-arounds. We
are around two month away from expressions, at least if all goes well.

I think you can achive something with BDS-style filter blocks
(!progname/-host/+host), something along these lines:

-host
mail.*	/var/log/maillog
+hostname
mail.*	/var/log/mail-server.log

Of course, this is hostname specific (and that again is not the IP but
what is in the message...)

To get to the IP, I think you can get at least some way with class A,B,C
addresses, but NOT with any other masks. Here is the idea:

$template dsl, /var/log/dsl-%$NOW%.log
$template routers, /var/log/routers-%$NOW%.log
:FROMHOST, startswith, "192.168.1."   -?dsl
:FROMHOST, startswith, "192.168.3."   -?routers
:FROMHOST, startswith, "10.1.1."      -?routers

Common pitfall: be sure to include the trailing dot in the condition to
match. If the rule were

:FROMHOST, startswith, "192.168.1"   -?dsl

It would match 192.168.1.1.2, 192.168.1.1.2 but also 192.168.1.11.2!

With the current engine, there are unfortunately no logical operations
available. So you cannot check for mail facility or whatever else. A
somewhat crude work-around would be to include the facility in the file
name, e.g. by specifying it as follows:

$template dsl, /var/log/dsl-%syslogfacility-text%-%$NOW%.log

That, of course, will result in a file written for each facility, even
those that you are not interested in. A unfortunately do not have a
solution for this now.

With v3 expressions, I envision something along these lines:

If maskmatch(FROMHOST, "192.168.1.0", 24) and syslog-facility-text ==
"mail" then 
    writefile "/var/log/dslmail-%$NOW%"

But that is not possible yet. And this is only an idea, not the actual
config file format we will have at that time. In my blog, there are a
number of posts about it, but the bottom line is that it is not yet
designed:

http://rgerhards.blogspot.com/2007/08/on-rsyslog-config-file-format.html

Doc about current capabilities is here:
 
http://www.rsyslog.com/module-Static_Docs-view-f-rsyslog_conf.html.phtml

search for "Filter Conditions" on that page.

The properties you need are documented here:
 
http://www.rsyslog.com/module-Static_Docs-view-f-property_replacer.html.
phtml

Keep in mind they ARE CASE-SENSITIVE! (and don't ask me why I had that
idea... ;))

As always, feedback is appreciated. I would be most interested to learn
at which final config you arrived.

Thanks,
Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Scott Baker
> Sent: Friday, December 21, 2007 1:23 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Separating logs by host?
> 
> Rainer Gerhards wrote:
> > I need to think about the real question a bit ;) Not an immediate
> > solution ... but not hopeless either ;)
> >
> > On v2/v3 - v2 will be the next stable release:
> > http://rgerhards.blogspot.com/2007/12/begun-working-on-rsyslog-
> v3.html
> 
> Another good example is that we have several linux mail servers. I
> like to aggregate all of their logging to one rsyslog server. They
> all log to mail.* on their local machines, and then forward also to
> my rsyslog server.
> 
> The problem is that shows up in my mail.* on the syslog server, and
> thus gets mixed in with the local mail.* from the syslog server
> itself. It would be nice to be able to separate those, by IP would
> be the easiest.
> 
> mail.*				/var/log/maillog
> 1.2.3.4/32:mail.*		/var/log/mail-server.log
> 
> Or something like that...
> 
> --
> Scott Baker - Canby Telcom
> RHCE - System Administrator - 503.266.8253
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Fri Dec 21 11:39:32 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 21 Dec 2007 11:39:32 +0100
Subject: [rsyslog] Very Early Preview of rsyslog 3.0.0
Message-ID: <1198233572.2492.16.camel@localhost.localdomain>

Hi all,

I have just reached a stage where everything in rsyslog 3.0.0 is a bit
stable. I thought it would be good to release a preview before I begin
to pull some other code and place it in its own module. There are
already a number of input plugins, so I think looking at that code
provides a very good idea of where we are heading.

Please note that there is a doc/v3compatibility.html document. It tells
about know compatibility issues with pre-v3 versions and other work that
is currently in progress. At least, as far as I wrote it down ;)

The tarball is available at

http://download.rsyslog.com/rsyslog/rsyslog-3.0.0-pre.tar.gz
md5sum: e9c807decf8acbb1b2697ec907d6af08

Please note that this tarball is a snapshot. I do not plan to maintain
it, development simply continues. I suggest NOT to use that version in
production. If you run it and find a bug, please report it. Also, any
other feedback will be used during future development.

Rainer


From bakers at web-ster.com  Fri Dec 21 17:48:44 2007
From: bakers at web-ster.com (Scott Baker)
Date: Fri, 21 Dec 2007 08:48:44 -0800
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086D0@grfint2.intern.adiscon.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>	<476AE2B9.30002@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>	<476B076B.7090003@web-ster.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086D0@grfint2.intern.adiscon.com>
Message-ID: <476BEE6C.50109@web-ster.com>

Rainer Gerhards wrote:
> Mmhhh... once we have expression support, that will be an easy thing to
> do. Currently, I think there are a number of clumpsy work-arounds. We
> are around two month away from expressions, at least if all goes well.
> 
> I think you can achive something with BDS-style filter blocks
> (!progname/-host/+host), something along these lines:
> 
> -host
> mail.*	/var/log/maillog
> +hostname
> mail.*	/var/log/mail-server.log
> 
> Of course, this is hostname specific (and that again is not the IP but
> what is in the message...)
> 
> To get to the IP, I think you can get at least some way with class A,B,C
> addresses, but NOT with any other masks. Here is the idea:
> 
> $template dsl, /var/log/dsl-%$NOW%.log
> $template routers, /var/log/routers-%$NOW%.log
> :FROMHOST, startswith, "192.168.1."   -?dsl
> :FROMHOST, startswith, "192.168.3."   -?routers
> :FROMHOST, startswith, "10.1.1."      -?routers
> 
> Common pitfall: be sure to include the trailing dot in the condition to
> match. If the rule were
> 
> :FROMHOST, startswith, "192.168.1"   -?dsl
> 
> It would match 192.168.1.1.2, 192.168.1.1.2 but also 192.168.1.11.2!
> 
> With the current engine, there are unfortunately no logical operations
> available. So you cannot check for mail facility or whatever else. A
> somewhat crude work-around would be to include the facility in the file
> name, e.g. by specifying it as follows:
> 
> $template dsl, /var/log/dsl-%syslogfacility-text%-%$NOW%.log
> 
> That, of course, will result in a file written for each facility, even
> those that you are not interested in. A unfortunately do not have a
> solution for this now.
> 
> With v3 expressions, I envision something along these lines:
> 
> If maskmatch(FROMHOST, "192.168.1.0", 24) and syslog-facility-text ==
> "mail" then 
>     writefile "/var/log/dslmail-%$NOW%"
> 
> But that is not possible yet. And this is only an idea, not the actual
> config file format we will have at that time. In my blog, there are a
> number of posts about it, but the bottom line is that it is not yet
> designed:
> 
> http://rgerhards.blogspot.com/2007/08/on-rsyslog-config-file-format.html
> 
> Doc about current capabilities is here:
>  
> http://www.rsyslog.com/module-Static_Docs-view-f-rsyslog_conf.html.phtml
> 
> search for "Filter Conditions" on that page.
> 
> The properties you need are documented here:
>  
> http://www.rsyslog.com/module-Static_Docs-view-f-property_replacer.html.
> phtml
> 
> Keep in mind they ARE CASE-SENSITIVE! (and don't ask me why I had that
> idea... ;))
> 
> As always, feedback is appreciated. I would be most interested to learn
> at which final config you arrived.

Awesome! This is TOTALLY workable for what we're doing. I'm trying
to set it up and getting a weird message:

------------------------------------------------------------------

Dec 21 08:43:17 green rsyslogd:error: extra characters in config
line ignored: '/var/log/cisco-dsl-%$NOW%.log'
Dec 21 08:43:17 green rsyslogd: Could not find template 'dsl' -
action disabled
Dec 21 08:43:17 green rsyslogd:the last error occured in
/etc/rsyslog.conf, line 37

------------------------------------------------------------------

The config I used it almost exactly what you provided...

$template dsl, /var/log/cisco-dsl-%$NOW%.log
:FROMHOST, startswith, "10.3."           -?dsl

Am I missing something basic?

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Fri Dec 21 17:53:32 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 21 Dec 2007 17:53:32 +0100
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <476BEE6C.50109@web-ster.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>	<476AE2B9.30002@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>	<476B076B.7090003@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086D0@grfint2.intern.adiscon.com>
	<476BEE6C.50109@web-ster.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086D8@grfint2.intern.adiscon.com>

[sniiiip]
> The config I used it almost exactly what you provided...
> 
> $template dsl, /var/log/cisco-dsl-%$NOW%.log
> :FROMHOST, startswith, "10.3."           -?dsl
> 
> Am I missing something basic?

I forgot the quotes around the template string ;) That should work (not
tested, though):

$template dsl, "/var/log/cisco-dsl-%$NOW%.log"
:FROMHOST, startswith, "10.3."           -?dsl

Rainer


From bakers at web-ster.com  Fri Dec 21 18:55:36 2007
From: bakers at web-ster.com (Scott Baker)
Date: Fri, 21 Dec 2007 09:55:36 -0800
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086D8@grfint2.intern.adiscon.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>	<476AE2B9.30002@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>	<476B076B.7090003@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086D0@grfint2.intern.adiscon.com>	<476BEE6C.50109@web-ster.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086D8@grfint2.intern.adiscon.com>
Message-ID: <476BFE18.4000008@web-ster.com>

Rainer Gerhards wrote:
> I forgot the quotes around the template string ;) That should work (not
> tested, though):
> 
> $template dsl, "/var/log/cisco-dsl-%$NOW%.log"
> :FROMHOST, startswith, "10.3."           -?dsl

That was it... now rsyslog starts without errors.

Howeever it's not logging to the file I have set to though, instead
it's going to /var/log/messages. I've verified with tcpdump that
it's sending from the correct IP address. I've tried both startswith
and isequal and neither seem to work.

I don't know how to troubleshoot any further? Is there some debug
mode or something that will give me more information about why it's
not logging the way I think it should?

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253



From rgerhards at hq.adiscon.com  Fri Dec 21 18:58:49 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 21 Dec 2007 18:58:49 +0100
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <476BFE18.4000008@web-ster.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>	<476AE2B9.30002@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>	<476B076B.7090003@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086D0@grfint2.intern.adiscon.com>	<476BEE6C.50109@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086D8@grfint2.intern.adiscon.com>
	<476BFE18.4000008@web-ster.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086D9@grfint2.intern.adiscon.com>

Debug mode is the keyword here ;)

Add -d -n to the command line and run it interactively with stdout
redirected to a file. Maybe I've made a mistake with the directives,
maybe we have another issue. Debug output can get quite large, please do
not send it to the list (I think mailman will bounce it anyhow). You can
email me privately (rgerhards at adiscon.com) if you like.

Just be warned: its 7p right now here and I can not promise to fix a
larger problem before xmas ;) 

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Scott Baker
> Sent: Friday, December 21, 2007 6:56 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Separating logs by host?
> 
> Rainer Gerhards wrote:
> > I forgot the quotes around the template string ;) That should work
> (not
> > tested, though):
> >
> > $template dsl, "/var/log/cisco-dsl-%$NOW%.log"
> > :FROMHOST, startswith, "10.3."           -?dsl
> 
> That was it... now rsyslog starts without errors.
> 
> Howeever it's not logging to the file I have set to though, instead
> it's going to /var/log/messages. I've verified with tcpdump that
> it's sending from the correct IP address. I've tried both startswith
> and isequal and neither seem to work.
> 
> I don't know how to troubleshoot any further? Is there some debug
> mode or something that will give me more information about why it's
> not logging the way I think it should?
> 
> --
> Scott Baker - Canby Telcom
> RHCE - System Administrator - 503.266.8253
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From bakers at web-ster.com  Fri Dec 21 19:13:33 2007
From: bakers at web-ster.com (Scott Baker)
Date: Fri, 21 Dec 2007 10:13:33 -0800
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086D9@grfint2.intern.adiscon.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>	<476AE2B9.30002@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>	<476B076B.7090003@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086D0@grfint2.intern.adiscon.com>	<476BEE6C.50109@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086D8@grfint2.intern.adiscon.com>	<476BFE18.4000008@web-ster.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086D9@grfint2.intern.adiscon.com>
Message-ID: <476C024D.30200@web-ster.com>

Rainer Gerhards wrote:
> Debug mode is the keyword here ;)
> 
> Add -d -n to the command line and run it interactively with stdout
> redirected to a file. Maybe I've made a mistake with the directives,
> maybe we have another issue. Debug output can get quite large, please do
> not send it to the list (I think mailman will bounce it anyhow). You can
> email me privately (rgerhards at adiscon.com) if you like.
> 
> Just be warned: its 7p right now here and I can not promise to fix a
> larger problem before xmas ;) 

You rock! Rsyslog Debug Mode = Good stuff! I was able to get it
working just fine with the BSD style selectors. The FROMHOST part is
still giving me grief though:

Filter: check for property 'FROMHOST' (value 'extirpate') isequal
'1.2.3.4': FALSE

So my question is... for the BSD style selectors and FROM it appears
that rsyslog is using just the hostname and not the FULL reverse dns
name: extirpate vs extirpate.web-ster.com. Is that by design? I
don't think it will be a problem but I could see where you would
want to separate.

mail.domain1.com
vs
mail.domain2.com

Anyway, FROMHOST is using the name and not the IP address to match
against. Is there another property I should use? I don't see
anything in the docs... In reality I could match about the hostname
with a regexp since all my DSL is named the same...

Again IP would be ideal and easier...

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Fri Dec 21 19:20:07 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 21 Dec 2007 19:20:07 +0100
Subject: [rsyslog] Separating logs by host?
Message-ID: <002e01c843fe$24521c22$060013ac@intern.adiscon.com>

Just out of my head and from the pda while being out of office: try the -x option. Will check mail later again :)

----- Urspr?ngliche Nachricht -----
Von: "Scott Baker" <bakers at web-ster.com>
An: "rsyslog-users" <rsyslog at lists.adiscon.com>
Gesendet: 21.12.07 19:13
Betreff: Re: [rsyslog] Separating logs by host?

Rainer Gerhards wrote:
> Debug mode is the keyword here ;)
> 
> Add -d -n to the command line and run it interactively with stdout
> redirected to a file. Maybe I've made a mistake with the directives,
> maybe we have another issue. Debug output can get quite large, please do
> not send it to the list (I think mailman will bounce it anyhow). You can
> email me privately (rgerhards at adiscon.com) if you like.
> 
> Just be warned: its 7p right now here and I can not promise to fix a
> larger problem before xmas ;) 

You rock! Rsyslog Debug Mode = Good stuff! I was able to get it
working just fine with the BSD style selectors. The FROMHOST part is
still giving me grief though:

Filter: check for property 'FROMHOST' (value 'extirpate') isequal
'1.2.3.4': FALSE

So my question is... for the BSD style selectors and FROM it appears
that rsyslog is using just the hostname and not the FULL reverse dns
name: extirpate vs extirpate.web-ster.com. Is that by design? I
don't think it will be a problem but I could see where you would
want to separate.

mail.domain1.com
vs
mail.domain2.com

Anyway, FROMHOST is using the name and not the IP address to match
against. Is there another property I should use? I don't see
anything in the docs... In reality I could match about the hostname
with a regexp since all my DSL is named the same...

Again IP would be ideal and easier...

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog


From bakers at web-ster.com  Sat Dec 22 01:13:01 2007
From: bakers at web-ster.com (Scott Baker)
Date: Fri, 21 Dec 2007 16:13:01 -0800
Subject: [rsyslog] Rsyslog is awesome
Message-ID: <476C568D.9050807@web-ster.com>

Rainer:

I spent the better part of today setting up my centralized rsyslog
today. It's logging for about 20 different hosts now and working well.

I do have some questions about hostname matching DNS versus hosts
files. I think I may have found a crash bug too. I had a host that
was rsyslog was logging from without reverse DNS. As soon as I would
add the reverse DNS record rsyslog would crash. As if that source
name changing caused rsyslog to get confused.

I'm very happy to go over this in more detail with you when you have
some more time. With Christmas coming up I'm sure we'll touch base
after. Overall I'm very impressed with rsyslog. Good work!

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From mbiebl at gmail.com  Sun Dec 23 15:46:26 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Sun, 23 Dec 2007 15:46:26 +0100
Subject: [rsyslog] Very Early Preview of rsyslog 3.0.0
In-Reply-To: <1198233572.2492.16.camel@localhost.localdomain>
References: <1198233572.2492.16.camel@localhost.localdomain>
Message-ID: <aa76cdac0712230646r273b2a1aj13fec13695a3013f@mail.gmail.com>

2007/12/21, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Hi all,
>
> I have just reached a stage where everything in rsyslog 3.0.0 is a bit
> stable. I thought it would be good to release a preview before I begin
> to pull some other code and place it in its own module. There are
> already a number of input plugins, so I think looking at that code
> provides a very good idea of where we are heading.
>
> Please note that there is a doc/v3compatibility.html document. It tells
> about know compatibility issues with pre-v3 versions and other work that
> is currently in progress. At least, as far as I wrote it down ;)

I tested imuxsock and imklog (yay for getting rid of rklogd) and it
seems to work fine so far.
Nice work.

Michael
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Sun Dec 23 18:20:00 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Sun, 23 Dec 2007 18:20:00 +0100
Subject: [rsyslog] rsyslog 1.21.1 released
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086DE@grfint2.intern.adiscon.com>

Dear All,

I have just released rsyslog 1.21.1. It is a maintenance release to
1.21.0. Most importantly, it fixes a memory leak introduced in 1.21.0.
There are also some other bugfixes present. Users of 1.21.0 should
upgrade to this release.

If you run 1.21.0 and experience(d) the segfault previously, please let
me know your results with 1.21.0 before upgrading. This would be most
useful for me.

Download and Changelog can be found on the usual places of
www.rsyslog.com.

I would also like to wish a great holiday season to everyone! Thanks for
your interest in rsyslog and all the good feedback provided.

All the best,
Rainer


From bakers at web-ster.com  Mon Dec 24 17:33:11 2007
From: bakers at web-ster.com (Scott Baker)
Date: Mon, 24 Dec 2007 08:33:11 -0800
Subject: [rsyslog] rsyslog 1.21.1 released
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086DE@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA3086DE@grfint2.intern.adiscon.com>
Message-ID: <476FDF47.1060607@web-ster.com>

Rainer Gerhards wrote:
> Dear All,
> 
> I have just released rsyslog 1.21.1. It is a maintenance release to
> 1.21.0. Most importantly, it fixes a memory leak introduced in 1.21.0.
> There are also some other bugfixes present. Users of 1.21.0 should
> upgrade to this release.
> 
> If you run 1.21.0 and experience(d) the segfault previously, please let
> me know your results with 1.21.0 before upgrading. This would be most
> useful for me.
> 
> Download and Changelog can be found on the usual places of
> www.rsyslog.com.
> 
> I would also like to wish a great holiday season to everyone! Thanks for
> your interest in rsyslog and all the good feedback provided.

Rainer:

I just got to work and I checked my rsyslog server today and it's
running happily. After upgrading from 1.19.0 to 1.21.0 I didn't have
those segfaults that were happening about every 3 hours. In fact
rsyslog logged just over 1.3g of logs in 3 days.

I'll update to 1.21.1 right now and we'll see how she does.

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From bakers at web-ster.com  Mon Dec 24 17:57:02 2007
From: bakers at web-ster.com (Scott Baker)
Date: Mon, 24 Dec 2007 08:57:02 -0800
Subject: [rsyslog] BSD Style Selectors
Message-ID: <476FE4DE.1020201@web-ster.com>

It's been my experience that if I have a config like:

+server1
# The authpriv file has restricted access.
authpriv.*                          -/var/log/secure

# Log all the mail messages in one place.
mail.*                       -/var/log/maillog

# Log cron stuff
cron.*                        -/var/log/cron

-----------------------------------------------------

That +server1 setting applies to everything below it until it's
changed by another BSD style selector? In other words if server2
sent cron.* syslog in this config it would not log that in
/var/log/cron ??

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Mon Dec 24 18:35:52 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 24 Dec 2007 18:35:52 +0100
Subject: [rsyslog] BSD Style Selectors
In-Reply-To: <476FE4DE.1020201@web-ster.com>
References: <476FE4DE.1020201@web-ster.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086E1@grfint2.intern.adiscon.com>

Yes, that's right - and: merry xmas :)

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Scott Baker
> Sent: Monday, December 24, 2007 5:57 PM
> To: rsyslog-users
> Subject: [rsyslog] BSD Style Selectors
> 
> It's been my experience that if I have a config like:
> 
> +server1
> # The authpriv file has restricted access.
> authpriv.*                          -/var/log/secure
> 
> # Log all the mail messages in one place.
> mail.*                       -/var/log/maillog
> 
> # Log cron stuff
> cron.*                        -/var/log/cron
> 
> -----------------------------------------------------
> 
> That +server1 setting applies to everything below it until it's
> changed by another BSD style selector? In other words if server2
> sent cron.* syslog in this config it would not log that in
> /var/log/cron ??
> 
> --
> Scott Baker - Canby Telcom
> RHCE - System Administrator - 503.266.8253
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From bakers at web-ster.com  Mon Dec 24 19:10:35 2007
From: bakers at web-ster.com (Scott Baker)
Date: Mon, 24 Dec 2007 10:10:35 -0800
Subject: [rsyslog] Hostname matching with DNS
Message-ID: <476FF61B.3020006@web-ster.com>

I have a couple host on private IPs 10.x.x.x and thus they have no
DNS entries. So rather that log the IP in syslog I setup host
entries for them.

If I do something like

:FROMHOST, isequal, "foobar"                -?dialup

it doesn't match the /etc/hosts entry I have for foobar. If I setup
a catchall entry that goes to a test log I see the line

Dec 24 10:06:23 foobar [This is the message]

So it's logging the hostname like I would expect it to (rsyslog is
aware of the host entry) but I can't match against it? Unfortunately
my server is SUPER busy now and I can't put the server in debug mode
to check what's coming across. Is there another way I could check this?

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Mon Dec 24 19:59:18 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 24 Dec 2007 19:59:18 +0100
Subject: [rsyslog] Hostname matching with DNS
Message-ID: <002f01c8465f$209fa2ec$060013ac@intern.adiscon.com>

Really quick: check the HOSTNAME (or so ;)) property. 

----- Urspr?ngliche Nachricht -----
Von: "Scott Baker" <bakers at web-ster.com>
An: "rsyslog-users" <rsyslog at lists.adiscon.com>
Gesendet: 24.12.07 19:10
Betreff: [rsyslog] Hostname matching with DNS

I have a couple host on private IPs 10.x.x.x and thus they have no
DNS entries. So rather that log the IP in syslog I setup host
entries for them.

If I do something like

:FROMHOST, isequal, "foobar"                -?dialup

it doesn't match the /etc/hosts entry I have for foobar. If I setup
a catchall entry that goes to a test log I see the line

Dec 24 10:06:23 foobar [This is the message]

So it's logging the hostname like I would expect it to (rsyslog is
aware of the host entry) but I can't match against it? Unfortunately
my server is SUPER busy now and I can't put the server in debug mode
to check what's coming across. Is there another way I could check this?

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Mon Dec 24 20:59:02 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 24 Dec 2007 20:59:02 +0100
Subject: [rsyslog] Hostname matching with DNS
In-Reply-To: <002f01c8465f$209fa2ec$060013ac@intern.adiscon.com>
References: <002f01c8465f$209fa2ec$060013ac@intern.adiscon.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086E2@grfint2.intern.adiscon.com>

Scott,

So now a bit more in-depth: the HOSTNAME is taken form the syslog message, while FROMHOST is the last hope. There is only a difference in relay scenarios - or, like here, based on DNS resolution. This is why you see different values. The point is to match against the same one that is used in the catchall rule.

However, I think the most appropriate thing to do is add a FROMHOST-IP property, which always has the IP address of the sender, no matter if the -x option is given or not.

Would that help?

Rainer 

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of 
> Rainer Gerhards
> Sent: Monday, December 24, 2007 7:59 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Hostname matching with DNS
> 
> Really quick: check the HOSTNAME (or so ;)) property. 
> 
> ----- Urspr?ngliche Nachricht -----
> Von: "Scott Baker" <bakers at web-ster.com>
> An: "rsyslog-users" <rsyslog at lists.adiscon.com>
> Gesendet: 24.12.07 19:10
> Betreff: [rsyslog] Hostname matching with DNS
> 
> I have a couple host on private IPs 10.x.x.x and thus they have no
> DNS entries. So rather that log the IP in syslog I setup host
> entries for them.
> 
> If I do something like
> 
> :FROMHOST, isequal, "foobar"                -?dialup
> 
> it doesn't match the /etc/hosts entry I have for foobar. If I setup
> a catchall entry that goes to a test log I see the line
> 
> Dec 24 10:06:23 foobar [This is the message]
> 
> So it's logging the hostname like I would expect it to (rsyslog is
> aware of the host entry) but I can't match against it? Unfortunately
> my server is SUPER busy now and I can't put the server in debug mode
> to check what's coming across. Is there another way I could 
> check this?
> 
> -- 
> Scott Baker - Canby Telcom
> RHCE - System Administrator - 503.266.8253
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> 


From rgerhards at hq.adiscon.com  Mon Dec 24 21:01:21 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 24 Dec 2007 21:01:21 +0100
Subject: [rsyslog] Hostname matching with DNS
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086E2@grfint2.intern.adiscon.com>
References: <002f01c8465f$209fa2ec$060013ac@intern.adiscon.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086E2@grfint2.intern.adiscon.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086E3@grfint2.intern.adiscon.com>

> while FROMHOST is the last hope 


Lol... No, it is not my last *hope*, even though it may be for the time
being ;) That should have been "last hop". Nice typo ;)

Rainer


From bakers at web-ster.com  Mon Dec 24 21:06:48 2007
From: bakers at web-ster.com (Scott Baker)
Date: Mon, 24 Dec 2007 12:06:48 -0800
Subject: [rsyslog] Hostname matching with DNS
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086E2@grfint2.intern.adiscon.com>
References: <002f01c8465f$209fa2ec$060013ac@intern.adiscon.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086E2@grfint2.intern.adiscon.com>
Message-ID: <47701158.9080506@web-ster.com>

Rainer Gerhards wrote:
> Scott,
> 
> So now a bit more in-depth: the HOSTNAME is taken form the syslog message, while FROMHOST is the last hope. There is only a difference in relay scenarios - or, like here, based on DNS resolution. This is why you see different values. The point is to match against the same one that is used in the catchall rule.
> 
> However, I think the most appropriate thing to do is add a FROMHOST-IP property, which always has the IP address of the sender, no matter if the -x option is given or not.
> 
> Would that help?

Ya I think a FROMHOST-IP property would be great. It'd make setting
up my rules easier. Tell me more about hosts resolution vs DNS
resolution?

I have some FROMHOST rules that work just fine, but as soon as I add
a hosts entry it stops logging. So somewhere it's using the host
entry but I can't figure out how/where or what precedence that has
over DNS.

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Mon Dec 24 21:23:58 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 24 Dec 2007 21:23:58 +0100
Subject: [rsyslog] Hostname matching with DNS
Message-ID: <003001c8466a$f111b332$060013ac@intern.adiscon.com>

Rsyslog doesn't do anything specific with the hosts file - it just uses the system resolver library and everything else is handled in there...

I'll have a look at hostname-ip as soon as i can.

rainer

----- Urspr?ngliche Nachricht -----
Von: "Scott Baker" <bakers at web-ster.com>
An: "rsyslog-users" <rsyslog at lists.adiscon.com>
Gesendet: 24.12.07 21:06
Betreff: Re: [rsyslog] Hostname matching with DNS

Rainer Gerhards wrote:
> Scott,
> 
> So now a bit more in-depth: the HOSTNAME is taken form the syslog message, while FROMHOST is the last hope. There is only a difference in relay scenarios - or, like here, based on DNS resolution. This is why you see different values. The point is to match against the same one that is used in the catchall rule.
> 
> However, I think the most appropriate thing to do is add a FROMHOST-IP property, which always has the IP address of the sender, no matter if the -x option is given or not.
> 
> Would that help?

Ya I think a FROMHOST-IP property would be great. It'd make setting
up my rules easier. Tell me more about hosts resolution vs DNS
resolution?

I have some FROMHOST rules that work just fine, but as soon as I add
a hosts entry it stops logging. So somewhere it's using the host
entry but I can't figure out how/where or what precedence that has
over DNS.

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog


From bakers at web-ster.com  Mon Dec 24 22:53:26 2007
From: bakers at web-ster.com (Scott Baker)
Date: Mon, 24 Dec 2007 13:53:26 -0800
Subject: [rsyslog] Log Permissions?
Message-ID: <47702A56.7030805@web-ster.com>

Is there a way to set the owner/group or read permissions of log
files created by rsyslog? I can't find much in the man pages.

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Mon Dec 24 23:06:52 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 24 Dec 2007 23:06:52 +0100
Subject: [rsyslog] Log Permissions?
Message-ID: <003101c84679$50a4fc86$060013ac@intern.adiscon.com>

Look at www.rsyslog.com/doc Not the man pages. It is too much for man (but the above url is in it ;)). Check the config directives, there are a number of settings for that.

rainer

----- Urspr?ngliche Nachricht -----
Von: "Scott Baker" <bakers at web-ster.com>
An: "rsyslog-users" <rsyslog at lists.adiscon.com>
Gesendet: 24.12.07 22:53
Betreff: [rsyslog] Log Permissions?

Is there a way to set the owner/group or read permissions of log
files created by rsyslog? I can't find much in the man pages.

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog


From janfrode at tanso.net  Tue Dec 25 00:34:02 2007
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Tue, 25 Dec 2007 00:34:02 +0100
Subject: [rsyslog] rsyslog 1.21.1 released
References: <577465F99B41C842AAFBE9ED71E70ABA3086DE@grfint2.intern.adiscon.com>
Message-ID: <slrnfn0gfa.j64.janfrode@lc4eb8045376502.ibm.com>

On 2007-12-23, Rainer Gerhards <rgerhards at hq.adiscon.com> wrote:
>
> If you run 1.21.0 and experience(d) the segfault previously, please let
> me know your results with 1.21.0 before upgrading. This would be most
> useful for me.

I upgraded to 1.21.0 december 19., and it's been up since (4h19m cputime, 
134h process uptime). No segfaults yet. It hasn't been eating any noticeable
amount of memory either, so I'll probably hold off on the upgrade a bit.



  -jf



From rgerhards at hq.adiscon.com  Tue Dec 25 10:21:07 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 25 Dec 2007 10:21:07 +0100
Subject: [rsyslog] rsyslog 1.21.1 released
Message-ID: <003201c846d7$84c7c4a0$060013ac@intern.adiscon.com>

Ah, thats good news! And, yes, if you don't see the mem leak it is probably better to hild off the update, so that we get some more information on the stability. The memleak occurs only when the internal message queue is full, which normally shouldn't happen. So there is a good chance you never see it...

Rainer

----- Urspr?ngliche Nachricht -----
Von: "Jan-Frode Myklebust" <janfrode at tanso.net>
An: "rsyslog at lists.adiscon.com" <rsyslog at lists.adiscon.com>
Gesendet: 25.12.07 00:34
Betreff: Re: [rsyslog] rsyslog 1.21.1 released

On 2007-12-23, Rainer Gerhards <rgerhards at hq.adiscon.com> wrote:
>
> If you run 1.21.0 and experience(d) the segfault previously, please let
> me know your results with 1.21.0 before upgrading. This would be most
> useful for me.

I upgraded to 1.21.0 december 19., and it's been up since (4h19m cputime, 
134h process uptime). No segfaults yet. It hasn't been eating any noticeable
amount of memory either, so I'll probably hold off on the upgrade a bit.



  -jf

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog


From janfrode at tanso.net  Wed Dec 26 17:42:53 2007
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Wed, 26 Dec 2007 17:42:53 +0100
Subject: [rsyslog] Log Permissions?
References: <47702A56.7030805@web-ster.com>
Message-ID: <slrnfn514d.7qs.janfrode@lc4eb8045376502.ibm.com>

On 2007-12-24, Scott Baker <bakers at web-ster.com> wrote:
> Is there a way to set the owner/group or read permissions of log
> files created by rsyslog? I can't find much in the man pages.
>

I use these in my /etc/rsyslog.conf to give group "logread" read
permissions of all files:

$DirGroup logread
$DirCreateMode 0750
$FileGroup logread
$umask 0027



  -jf



From rgerhards at hq.adiscon.com  Thu Dec 27 14:19:18 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 27 Dec 2007 14:19:18 +0100
Subject: [rsyslog] New rsyslog 3.0.0 preview - with new features
Message-ID: <1198761558.10564.3.camel@localhost.localdomain>

Hi all,

I have just uploaded a new preview of 3.0.0. It now has all input
modules (though they still have a lot of internal dependencies).
Finally, I also managed to get in some new features: the UDP listener
can now be bound to specific local interfaces and there can be multiple
UDP listeners active at the same time (on different interface and/or
different ports). The same feature for TCP (including GSS) is planned,
but I may need to do some more stage work before.

The preview is available at

http://download.rsyslog.com/rsyslog/rsyslog-3.0.0-pre.tar.gz
md5sum: 979763fe645ad6ee6fddf11737af7ebd 

This is a snapshot. Development continues. Feedback is appreciated.

Rainer


From rgerhards at hq.adiscon.com  Fri Dec 28 09:26:49 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 28 Dec 2007 09:26:49 +0100
Subject: [rsyslog] omfwd/gss-api in rsyslog v1/v2
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086F0@grfint2.intern.adiscon.com>

Hi all,

as you know, I am working on the modular structure of rsyslog v3. I am
currently revisiting gss-api support. I notice that with the current
omfwd, it will be extremely hard to separate gss-api support into its
own module. Doing so will break backward compatibility to the
configuration file.

GSS-API has been out only for a few days, and mostly over the holiday
period. So it is much less of a concern if we introduce now some changes
that will case rsyslog.conf format modifications. Much less trouble than
when we release v2, a release expected to be in wide use for at least
half a year, if not much longer. V2 released with the current syntax
would require me to do some tricks in v3 to keep compatibility. Quite
complex.

So I decided to create a omgssapi for v3 and extract the gss-api code
from omfwd. It looks like this can be done without too much code
duplication. There will be some duplicate code, but it will shrink as v3
continues to be developed. Once I have a good working version, which I
expect very soon, I will backport that to the v1/2 source tree. I'll
then do a new v1 release with a slightly incompatible gss-api config
file syntax. After this is out for a few days, I hope I can than finally
push out that version as v2.

I hope this is a good decision. I think it will save us major future
trouble at the expense of a relatively slight disturbance in the late v1
timeline. I guess most user's won't even notice there is a change.

As always, Feedback is appreciated.

Rainer 


From bakers at web-ster.com  Fri Dec 28 17:15:07 2007
From: bakers at web-ster.com (Scott Baker)
Date: Fri, 28 Dec 2007 08:15:07 -0800
Subject: [rsyslog] omfwd/gss-api in rsyslog v1/v2
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086F0@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA3086F0@grfint2.intern.adiscon.com>
Message-ID: <4775210B.804@web-ster.com>

Rainer Gerhards wrote:
> Hi all,
> 
> as you know, I am working on the modular structure of rsyslog v3. I am
> currently revisiting gss-api support. I notice that with the current
> omfwd, it will be extremely hard to separate gss-api support into its
> own module. Doing so will break backward compatibility to the
> configuration file.
> 
> GSS-API has been out only for a few days, and mostly over the holiday
> period. So it is much less of a concern if we introduce now some changes
> that will case rsyslog.conf format modifications. Much less trouble than
> when we release v2, a release expected to be in wide use for at least
> half a year, if not much longer. V2 released with the current syntax
> would require me to do some tricks in v3 to keep compatibility. Quite
> complex.
> 
> So I decided to create a omgssapi for v3 and extract the gss-api code
> from omfwd. It looks like this can be done without too much code
> duplication. There will be some duplicate code, but it will shrink as v3
> continues to be developed. Once I have a good working version, which I
> expect very soon, I will backport that to the v1/2 source tree. I'll
> then do a new v1 release with a slightly incompatible gss-api config
> file syntax. After this is out for a few days, I hope I can than finally
> push out that version as v2.
> 
> I hope this is a good decision. I think it will save us major future
> trouble at the expense of a relatively slight disturbance in the late v1
> timeline. I guess most user's won't even notice there is a change.
> 
> As always, Feedback is appreciated.

I think this is the best solution moving forward. I don't think
configuration changes moving to a new major version are a huge
headache, especially since GSS-API is so new.

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Fri Dec 28 22:39:59 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 28 Dec 2007 22:39:59 +0100
Subject: [rsyslog] Rsyslog 1.21.2 released
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086F7@grfint2.intern.adiscon.com>

Hi all,

just a quick note:

I have released rsyslog 1.21.2. It is a maintenance release. The most
important new feature is that GSS-API support for sending TCP messages
has been moved to a separate output plugin. This is an important
forward-compatibility option for rsyslog v3. This resulted in some
configuration file changes. This release also fixes a number of minor
bugs. This is a recommended update for all users.

Please also keep my message from earlier today in mind.

Rainer


From mbiebl at gmail.com  Sat Dec  1 00:14:35 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Sat, 1 Dec 2007 00:14:35 +0100
Subject: [rsyslog] gss-api support for rsyslog
In-Reply-To: <473B2961.9010705@redhat.com>
References: <473B2961.9010705@redhat.com>
Message-ID: <aa76cdac0711301514x711721a7i2767e8a99e175f79@mail.gmail.com>

2007/11/14, theinric at redhat.com <theinric at redhat.com>:
> Hi,
>
> here is a patch that enables rsyslog to use gss-api for authentication
> and encryption.

Hi,

ist seems not all of the patch has made it into CVS. The part in
Makefile.am is missing, so the compilation fails. Please apply the
(inlined) patch ;-)

Index: Makefile.am
===================================================================
RCS file: /cvsroot/rsyslog/rsyslog/Makefile.am,v
retrieving revision 1.36
diff -u -3 -p -r1.36 Makefile.am
--- Makefile.am 30 Nov 2007 13:43:12 -0000      1.36
+++ Makefile.am 30 Nov 2007 23:10:27 -0000
@@ -63,7 +63,7 @@ rsyslogd_SOURCES = \
        gss-misc.h

 rsyslogd_CPPFLAGS = -D_PATH_MODDIR=\"$(pkglibdir)/\" $(pthreads_cflags)
-rsyslogd_LDADD = $(zlib_libs) $(pthreads_libs) -ldl
+rsyslogd_LDADD = $(zlib_libs) $(pthreads_libs) $(gss_libs) -ldl
 rsyslogd_LDFLAGS = -export-dynamic

 man_MANS = rfc3195d.8 rklogd.8 rsyslogd.8 rsyslog.conf.5



Cheers,
Michael
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From sur5r at sur5r.net  Sat Dec  1 22:36:44 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Sat, 1 Dec 2007 22:36:44 +0100
Subject: [rsyslog] [patch] output module for PostgreSQL
Message-ID: <20071201223644.036dccd6@marathon.haufe>

Hi all!

As already mentioned in the forum: I've created an output module for
PostgreSQL.

The attached patch does the following:

* add plugins/ompgsql/*
* extend configure.ac and Makefile.am
* msg.h
** struct msg_t: add pszTIMESTAMP_PgSQL, add pszRcvdAt_PgSQL, fix typo
* msg.c
** getTimeGenerated(): add handling of tplFmtPgSQLDate
** getTimeReported(): add handling of tplFmtPgSQLDate
* syslogd.h:
** add definition of formatTimestampToPgSQL()
* syslogd.c:
** add StdPgSQLFmt
** mainThread(): include StdPgSQLFmt
* template.h:
** add tplFmtPgSQLDate to tplFormatTypes
* template.c:
** doOptions(): date-pgsql -> tplFmtPgSQLDate
** tplPrintList(): add tplFmtPgSQLDate to switch()

Please note that the patch has not yet been tested thoroughly, so until
that has been done, it is to be considered alpha quality.

If anyone wants to try it: any feedback is appreciated.

Regards,
sur5r

P.S.: I hope the patch survives mailman... ;)
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: rsyslog_ompgsql.diff
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20071201/3543bb53/attachment-0001.txt>

From sur5r at sur5r.net  Sat Dec  1 23:01:26 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Sat, 1 Dec 2007 23:01:26 +0100
Subject: [rsyslog] [patch] output module for PostgreSQL
References: <20071201223644.036dccd6@marathon.haufe>
Message-ID: <20071201230126.3537f120@marathon.haufe>

Jakob Haufe <sur5r at sur5r.net> wrote on Sat, 1 Dec 2007 22:36:44 +0100
in 20071201223644.036dccd6 at marathon.haufe:

> The attached patch does the following:
> 
> * add plugins/ompgsql/*

Erm, yes. Maybe i should have noticed, that cvs doesn't recurse into
new subdirectories... so here are the files for the plugins/ompgsql
directory.

From sur5r at sur5r.net  Sat Dec  1 23:10:11 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Sat, 1 Dec 2007 23:10:11 +0100
Subject: [rsyslog] [patch] output module for PostgreSQL
References: <20071201223644.036dccd6@marathon.haufe>
	<20071201230126.3537f120@marathon.haufe>
Message-ID: <20071201231011.5a27a361@marathon.haufe>

Jakob Haufe <sur5r at sur5r.net> wrote on Sat, 1 Dec 2007 23:01:26 +0100
in 20071201230126.3537f120 at marathon.haufe:

> Jakob Haufe <sur5r at sur5r.net> wrote on Sat, 1 Dec 2007 22:36:44 +0100
> in 20071201223644.036dccd6 at marathon.haufe:
> 
> > The attached patch does the following:
> > 
> > * add plugins/ompgsql/*
> 
> Erm, yes. Maybe i should have noticed, that cvs doesn't recurse into
> new subdirectories... so here are the files for the plugins/ompgsql
> directory.
Ok, mailman doesn't like any mimetypes but text/plain... next try *grml*
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: Makefile.am
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20071201/0f6fc89b/attachment-0002.asc>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: createDB.sql
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20071201/0f6fc89b/attachment-0002.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: ompgsql.c
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20071201/0f6fc89b/attachment-0003.asc>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: ompgsql.h
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20071201/0f6fc89b/attachment-0003.txt>

From mbiebl at gmail.com  Sun Dec  2 23:10:07 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Sun, 2 Dec 2007 23:10:07 +0100
Subject: [rsyslog] [patch] output module for PostgreSQL
In-Reply-To: <20071201231011.5a27a361@marathon.haufe>
References: <20071201223644.036dccd6@marathon.haufe>
	<20071201230126.3537f120@marathon.haufe>
	<20071201231011.5a27a361@marathon.haufe>
Message-ID: <aa76cdac0712021410l2a218cftb5a6883c38bacd32@mail.gmail.com>

2007/12/1, Jakob Haufe <sur5r at sur5r.net>:
> Jakob Haufe <sur5r at sur5r.net> wrote on Sat, 1 Dec 2007 23:01:26 +0100
> in 20071201230126.3537f120 at marathon.haufe:
>
> > Jakob Haufe <sur5r at sur5r.net> wrote on Sat, 1 Dec 2007 22:36:44 +0100
> > in 20071201223644.036dccd6 at marathon.haufe:
> >
> > > The attached patch does the following:
> > >
> > > * add plugins/ompgsql/*
> >
> > Erm, yes. Maybe i should have noticed, that cvs doesn't recurse into
> > new subdirectories... so here are the files for the plugins/ompgsql
> > directory.
> Ok, mailman doesn't like any mimetypes but text/plain... next try *grml*

Hi,

just a very minor nitpick in Makefile.am:
pkglibdir is by default defined to be $(libdir)/@PACKAGE@/ [1], so
there is no need to redefine it. Simply delete the first line.
plugins/ommysql/Makefile.am should be fixed accordingly.

It also seems that the configure.ac bits are missing.

Cheers,
Michael

[1] http://www.gnu.org/software/automake/manual/html_node/Uniform.html#Uniform
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Mon Dec  3 08:10:44 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 3 Dec 2007 08:10:44 +0100
Subject: [rsyslog] [patch] output module for PostgreSQL
In-Reply-To: <aa76cdac0712021410l2a218cftb5a6883c38bacd32@mail.gmail.com>
References: <20071201223644.036dccd6@marathon.haufe><20071201230126.3537f120@marathon.haufe><20071201231011.5a27a361@marathon.haufe>
	<aa76cdac0712021410l2a218cftb5a6883c38bacd32@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA30856B@grfint2.intern.adiscon.com>

Hi,

thanks to all for the great contributions! Let me just quickly outline
my plan, so that nobody is disappointed ;) I'll publish a new 1.19.12
release today. However, that will just hold some cleanup and fixes. When
I have done that, I'll begin to merge the patches into the
already-modified 1.20.0. I am not yet sure how much issues I will see
because there are a number of code changes in that release. I am also
not sure yet in which order I apply the patches, because one might be
better than the other. I'll announce when I have merged and will
probably post a test tarball here on the list before officially
releasing. My target for release is mid to end of week.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Sunday, December 02, 2007 11:10 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] [patch] output module for PostgreSQL
> 
> 2007/12/1, Jakob Haufe <sur5r at sur5r.net>:
> > Jakob Haufe <sur5r at sur5r.net> wrote on Sat, 1 Dec 2007 23:01:26
+0100
> > in 20071201230126.3537f120 at marathon.haufe:
> >
> > > Jakob Haufe <sur5r at sur5r.net> wrote on Sat, 1 Dec 2007 22:36:44
> +0100
> > > in 20071201223644.036dccd6 at marathon.haufe:
> > >
> > > > The attached patch does the following:
> > > >
> > > > * add plugins/ompgsql/*
> > >
> > > Erm, yes. Maybe i should have noticed, that cvs doesn't recurse
> into
> > > new subdirectories... so here are the files for the
plugins/ompgsql
> > > directory.
> > Ok, mailman doesn't like any mimetypes but text/plain... next try
> *grml*
> 
> Hi,
> 
> just a very minor nitpick in Makefile.am:
> pkglibdir is by default defined to be $(libdir)/@PACKAGE@/ [1], so
> there is no need to redefine it. Simply delete the first line.
> plugins/ommysql/Makefile.am should be fixed accordingly.
> 
> It also seems that the configure.ac bits are missing.
> 
> Cheers,
> Michael
> 
> [1]
>
http://www.gnu.org/software/automake/manual/html_node/Uniform.html#Unif
> orm
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Mon Dec  3 10:01:24 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 3 Dec 2007 10:01:24 +0100
Subject: [rsyslog] rsyslog 1.19.12 released
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308570@grfint2.intern.adiscon.com>

As promised this morning, I have just released 1.19.12. It is a cleanup
release, no need to install it if you do not experience any issues.
Users of 1.19.11 using mysql should update, as the new -pthread switch
was not applied to ommysql in 1.19.11.

Download:

   http://www.rsyslog.com/Downloads-req-getit-lid-64.phtml

As always, feedback is appreciated.

Best regards,
Rainer Gerhards

Rsyslog homepage: http://www.rsyslog.com
My blog: http://rgerhards.blogspot.com



From sur5r at sur5r.net  Mon Dec  3 10:37:48 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Mon, 3 Dec 2007 10:37:48 +0100
Subject: [rsyslog] [patch] output module for PostgreSQL
References: <20071201223644.036dccd6@marathon.haufe>
	<20071201230126.3537f120@marathon.haufe>
	<20071201231011.5a27a361@marathon.haufe>
	<aa76cdac0712021410l2a218cftb5a6883c38bacd32@mail.gmail.com>
Message-ID: <20071203103748.781bfba2@mp-atlantis3.informatik.uni-mannheim.de>

On Sun, 2 Dec 2007 23:10:07 +0100
"Michael Biebl" <mbiebl at gmail.com> wrote:

> just a very minor nitpick in Makefile.am:
> pkglibdir is by default defined to be $(libdir)/@PACKAGE@/ [1], so
> there is no need to redefine it. Simply delete the first line.
> plugins/ommysql/Makefile.am should be fixed accordingly.

Thanks for pointing this out. Unfortunately, I have no in-depth
knowledge of autotools, so I'm thankful for any suggestions.

> It also seems that the configure.ac bits are missing.

They are there, though hard to find. 

MSGID 20071201223644.036dccd6 at marathon.haufe
contains the patch modifying existing files, namely:
msg.[ch], syslogd.[ch], template.[ch], configure.ac, Makefile.am

MSGID 20071201231011.5a27a361 at marathon.haufe
contains the files for plugins/ompgsql/*

Regards,
Jakob



From sur5r at sur5r.net  Mon Dec  3 11:09:12 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Mon, 3 Dec 2007 11:09:12 +0100
Subject: [rsyslog] [patch] output module for PostgreSQL
References: <20071201223644.036dccd6@marathon.haufe>
	<20071201230126.3537f120@marathon.haufe>
	<20071201231011.5a27a361@marathon.haufe>
	<aa76cdac0712021410l2a218cftb5a6883c38bacd32@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA30856B@grfint2.intern.adiscon.com>
Message-ID: <20071203110912.50657ea6@mp-atlantis3.informatik.uni-mannheim.de>

On Mon, 3 Dec 2007 08:10:44 +0100
"Rainer Gerhards" <rgerhards at hq.adiscon.com> wrote:

> thanks to all for the great contributions! Let me just quickly outline
> my plan, so that nobody is disappointed ;) I'll publish a new 1.19.12
> release today. However, that will just hold some cleanup and fixes.
> When I have done that, I'll begin to merge the patches into the
> already-modified 1.20.0. I am not yet sure how much issues I will see
> because there are a number of code changes in that release. I am also
> not sure yet in which order I apply the patches, because one might be
> better than the other. I'll announce when I have merged and will
> probably post a test tarball here on the list before officially
> releasing. My target for release is mid to end of week.

That's perfectly ok for me. But I definitely would appreciate any
feedback on ompgsql before officially releasing it. I do not expect any
problems with it but you never know...

So if anybody is going to test it, please give feedback.

Regards,
sur5r



From rgerhards at hq.adiscon.com  Mon Dec  3 11:11:25 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 3 Dec 2007 11:11:25 +0100
Subject: [rsyslog] [patch] output module for PostgreSQL
In-Reply-To: <20071203110912.50657ea6@mp-atlantis3.informatik.uni-mannheim.de>
References: <20071201223644.036dccd6@marathon.haufe><20071201230126.3537f120@marathon.haufe><20071201231011.5a27a361@marathon.haufe><aa76cdac0712021410l2a218cftb5a6883c38bacd32@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA30856B@grfint2.intern.adiscon.com>
	<20071203110912.50657ea6@mp-atlantis3.informatik.uni-mannheim.de>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308574@grfint2.intern.adiscon.com>

Just a quick update. I am right now integrating it and will review code
during this and once I am through. As I suspected, I need to do at least
some manual merging, as the source tree already changed and in some
places considerably (talking about 1.20.0!). So it would be good if you
could-re-test it once I can release an early test tarball.

I keep you updated.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Jakob Haufe
> Sent: Monday, December 03, 2007 11:09 AM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] [patch] output module for PostgreSQL
> 
> On Mon, 3 Dec 2007 08:10:44 +0100
> "Rainer Gerhards" <rgerhards at hq.adiscon.com> wrote:
> 
> > thanks to all for the great contributions! Let me just quickly
> outline
> > my plan, so that nobody is disappointed ;) I'll publish a new
1.19.12
> > release today. However, that will just hold some cleanup and fixes.
> > When I have done that, I'll begin to merge the patches into the
> > already-modified 1.20.0. I am not yet sure how much issues I will
see
> > because there are a number of code changes in that release. I am
also
> > not sure yet in which order I apply the patches, because one might
be
> > better than the other. I'll announce when I have merged and will
> > probably post a test tarball here on the list before officially
> > releasing. My target for release is mid to end of week.
> 
> That's perfectly ok for me. But I definitely would appreciate any
> feedback on ompgsql before officially releasing it. I do not expect
any
> problems with it but you never know...
> 
> So if anybody is going to test it, please give feedback.
> 
> Regards,
> sur5r
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From mbiebl at gmail.com  Tue Dec  4 03:58:21 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Tue, 4 Dec 2007 03:58:21 +0100
Subject: [rsyslog] [PATCH] Fix latest "NetBSD fix"
Message-ID: <aa76cdac0712031858w52c6c63bq529b7496cb8a5cf3@mail.gmail.com>

Hi,

the latest NetBSD related commits (configure.ac/Makefile.am) break the
compilation on Linux. Variable assignments mustn't have whitespace,
otherwise have_libdl and as a result HAVE_LIBDL is not defined.

To fix the issue though, I'd simply recommend to remove the hard coded
-ldl from Makefile.am instead of the proposed solution.
The AC_CHECK_LIB macro [2] already appends -ldl to LIBS, depending on
the outcome of the check [2]. The global LIBS variable (similar to
CFLAGS e.g.) is used for linking all binaries.

Linked (see, I learned my lesson :-p) is the patch [1] to address this.

The only downside of AC_CHECK_LIB is, that it assigns -ldl (and -lrt)
to the global LIBS variable. This means that all binaries are linked
with -ldl -lrt, as mentioned above, even if they don't need them (i.e.
ommysql/ompsql plugin,  rklogd rfc3195d).

I can prepare a patch to address this issue correctly, if wanted.

Cheers,
Michael

[1] http://debs.michaelbiebl.de/rsyslog/fix_dl_compilation.patch
[2] http://www.gnu.org/software/automake/manual/autoconf/Libraries.html

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Tue Dec  4 11:15:57 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 04 Dec 2007 11:15:57 +0100
Subject: [rsyslog] [PATCH] Fix latest "NetBSD fix"
In-Reply-To: <aa76cdac0712031858w52c6c63bq529b7496cb8a5cf3@mail.gmail.com>
References: <aa76cdac0712031858w52c6c63bq529b7496cb8a5cf3@mail.gmail.com>
Message-ID: <1196763357.12493.6.camel@localhost.localdomain>

Hi Michael,

many thanks for the information. Actually, you were quicker than me ;) I
worked with Bartosz on the issue yesterday, but needed to leave before I
could finalize it.

I would appreciate if you could provide the "big patch" you wrote about.
That will ensure that autotools is used in the best possible way :)

Rainer

On Tue, 2007-12-04 at 03:58 +0100, Michael Biebl wrote:
> Hi,
> 
> the latest NetBSD related commits (configure.ac/Makefile.am) break the
> compilation on Linux. Variable assignments mustn't have whitespace,
> otherwise have_libdl and as a result HAVE_LIBDL is not defined.
> 
> To fix the issue though, I'd simply recommend to remove the hard coded
> -ldl from Makefile.am instead of the proposed solution.
> The AC_CHECK_LIB macro [2] already appends -ldl to LIBS, depending on
> the outcome of the check [2]. The global LIBS variable (similar to
> CFLAGS e.g.) is used for linking all binaries.
> 
> Linked (see, I learned my lesson :-p) is the patch [1] to address this.
> 
> The only downside of AC_CHECK_LIB is, that it assigns -ldl (and -lrt)
> to the global LIBS variable. This means that all binaries are linked
> with -ldl -lrt, as mentioned above, even if they don't need them (i.e.
> ommysql/ompsql plugin,  rklogd rfc3195d).
> 
> I can prepare a patch to address this issue correctly, if wanted.
> 
> Cheers,
> Michael
> 
> [1] http://debs.michaelbiebl.de/rsyslog/fix_dl_compilation.patch
> [2] http://www.gnu.org/software/automake/manual/autoconf/Libraries.html
> 


From rgerhards at hq.adiscon.com  Tue Dec  4 11:23:26 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 4 Dec 2007 11:23:26 +0100
Subject: [rsyslog] rsyslog 1.20.0 preview available
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308596@grfint2.intern.adiscon.com>

Hi all,

there is a preview of rsyslog 1.20.0 available at

http://download.rsyslog.com/rsyslog/rsyslog-1.20.0.tar.gz

It obviously does not contains Michael Biebl's upcoming autotools patch,
but should be fairly well. I would be most interested in feedback from
the field, especially on the new postgres module by sur5r (thanks
again!). I plan to release end of week if now show stoppers show up.

Please note that I have disabled GSS-API in a rough way: you can simply
not select it in configure. This is because GSS-API support does need
some more work or it could become an interop problem in the future. So
we decided to push it back until that is solved.

Feedback is highly appreciated. If I updated the tarball, I'll announce
that. Oh, yes, doc is not finished yet, so do not wonder...

Rainer


From sur5r at sur5r.net  Tue Dec  4 14:00:14 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Tue, 4 Dec 2007 14:00:14 +0100
Subject: [rsyslog] rsyslog 1.20.0 preview available
References: <577465F99B41C842AAFBE9ED71E70ABA308596@grfint2.intern.adiscon.com>
Message-ID: <20071204140014.7c2187b2@mp-atlantis3.informatik.uni-mannheim.de>

On Tue, 4 Dec 2007 11:23:26 +0100
"Rainer Gerhards" <rgerhards at hq.adiscon.com> wrote:

> Feedback is highly appreciated. If I updated the tarball, I'll
> announce that. Oh, yes, doc is not finished yet, so do not wonder...

Just a small thing I already mentioned on the forum but unfortunately
got overlooked:

ommysql, line 236: p+=sizeof(":ommysql:");

This adds increments the pointer one byte too far (trailing \0) and
thus eats the first byte of the hostname part.

It's a bit unfortunate that strlen() doesn't count the \0 while
sizeof() does...

Regards,
sur5r



From rgerhards at hq.adiscon.com  Tue Dec  4 14:10:06 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 4 Dec 2007 14:10:06 +0100
Subject: [rsyslog] rsyslog 1.20.0 preview available
In-Reply-To: <20071204140014.7c2187b2@mp-atlantis3.informatik.uni-mannheim.de>
References: <577465F99B41C842AAFBE9ED71E70ABA308596@grfint2.intern.adiscon.com>
	<20071204140014.7c2187b2@mp-atlantis3.informatik.uni-mannheim.de>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA30859F@grfint2.intern.adiscon.com>

Indeed, thanks for pointing it out. Its fixed now, in cvs and I also
re-released the tarball (just for completeness).

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Jakob Haufe
> Sent: Tuesday, December 04, 2007 2:00 PM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] rsyslog 1.20.0 preview available
> 
> On Tue, 4 Dec 2007 11:23:26 +0100
> "Rainer Gerhards" <rgerhards at hq.adiscon.com> wrote:
> 
> > Feedback is highly appreciated. If I updated the tarball, I'll
> > announce that. Oh, yes, doc is not finished yet, so do not wonder...
> 
> Just a small thing I already mentioned on the forum but unfortunately
> got overlooked:
> 
> ommysql, line 236: p+=sizeof(":ommysql:");
> 
> This adds increments the pointer one byte too far (trailing \0) and
> thus eats the first byte of the hostname part.
> 
> It's a bit unfortunate that strlen() doesn't count the \0 while
> sizeof() does...
> 
> Regards,
> sur5r
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From stephen.carville at gmail.com  Fri Dec  7 22:09:18 2007
From: stephen.carville at gmail.com (Stephen Carville)
Date: Fri, 7 Dec 2007 13:09:18 -0800
Subject: [rsyslog] Last Message repeated N times
Message-ID: <2428c0380712071309w31f4fc20qb933dfef47f445ac@mail.gmail.com>

I am using rsyslog 1.19.10 on a central server to collect messages and
store them in a mysql database.  One thing I've noticed is that the
"last message repeated NNN times" entries are alway entered with
FromHost="last", SysLogTag="message", and Message="repeated NNN
times".  Can this be fixed?

I still use the regular sysklogd package on the remote machines to
forward the messages.

-- 
Stephen Carville


From rgerhards at hq.adiscon.com  Tue Dec 11 11:06:00 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 11 Dec 2007 11:06:00 +0100
Subject: [rsyslog] Asking for help...
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA30862F@grfint2.intern.adiscon.com>

Hi folks,

I have a design question with upcoming rsyslog changes and could not
even find an appropriate place to ask. I have blogged about it, so the
information is available to everyone:

http://rgerhards.blogspot.com/2007/12/design-problem.html

If you happen to know a good place to ask (or even know a solution), I
would really appreciate to hear about that!

Thanks
Rainer


From rgerhards at hq.adiscon.com  Tue Dec 11 12:01:07 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 11 Dec 2007 12:01:07 +0100
Subject: [rsyslog] Last Message repeated N times
In-Reply-To: <2428c0380712071309w31f4fc20qb933dfef47f445ac@mail.gmail.com>
References: <2428c0380712071309w31f4fc20qb933dfef47f445ac@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308637@grfint2.intern.adiscon.com>

Hi Stephen,

I was totally swamped the past days, sorry for the missing response.
I'll check what causes this. As a side-note, I think that the message
reduction code does not work well with the database - as you can no
longer query the individual messages. Is this your intent?

Rainer 

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Stephen Carville
> Sent: Friday, December 07, 2007 10:09 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] Last Message repeated N times
> 
> I am using rsyslog 1.19.10 on a central server to collect messages and
> store them in a mysql database.  One thing I've noticed is that the
> "last message repeated NNN times" entries are alway entered with
> FromHost="last", SysLogTag="message", and Message="repeated NNN
> times".  Can this be fixed?
> 
> I still use the regular sysklogd package on the remote machines to
> forward the messages.
> 
> --
> Stephen Carville
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Tue Dec 11 12:26:32 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 11 Dec 2007 12:26:32 +0100
Subject: [rsyslog] Last Message repeated N times
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308637@grfint2.intern.adiscon.com>
References: <2428c0380712071309w31f4fc20qb933dfef47f445ac@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA308637@grfint2.intern.adiscon.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA30863D@grfint2.intern.adiscon.com>

I think I misunderstood your problem... The "last message repeated n
times" stems back from the reporting sysklogd's - right? Mhhh... that's
a problem. I need to do some hardcoded processing in the parser to cover
part of it. The problem is that sysklogd emits a totally non-standard
message in that case. Even worse, I think it does not even contain a
clue at the missing information. I could probably use the sender's IP
address as source in  that case, but the tag seems to be impossible to
obtain.

Can you post me a copy of the %rawmsg% property, so that I can have a
look at what exactly you are seing?

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Tuesday, December 11, 2007 12:01 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Last Message repeated N times
> 
> Hi Stephen,
> 
> I was totally swamped the past days, sorry for the missing response.
> I'll check what causes this. As a side-note, I think that the message
> reduction code does not work well with the database - as you can no
> longer query the individual messages. Is this your intent?
> 
> Rainer
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Stephen Carville
> > Sent: Friday, December 07, 2007 10:09 PM
> > To: rsyslog at lists.adiscon.com
> > Subject: [rsyslog] Last Message repeated N times
> >
> > I am using rsyslog 1.19.10 on a central server to collect messages
> and
> > store them in a mysql database.  One thing I've noticed is that the
> > "last message repeated NNN times" entries are alway entered with
> > FromHost="last", SysLogTag="message", and Message="repeated NNN
> > times".  Can this be fixed?
> >
> > I still use the regular sysklogd package on the remote machines to
> > forward the messages.
> >
> > --
> > Stephen Carville
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From sur5r at sur5r.net  Tue Dec 11 14:48:15 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Tue, 11 Dec 2007 14:48:15 +0100
Subject: [rsyslog] Asking for help...
References: <577465F99B41C842AAFBE9ED71E70ABA30862F@grfint2.intern.adiscon.com>
Message-ID: <20071211144815.6a2ef123@marathon.haufe>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

"Rainer Gerhards" <rgerhards at hq.adiscon.com> wrote on Tue, 11 Dec 2007
11:06:00 +0100:

> http://rgerhards.blogspot.com/2007/12/design-problem.html

Maybe I'm a bit short-sighted here, but isn't this exactly the situation
for non-blocking sockets? Or will that raise other problems?

Regards,
sur5r
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHXpUj1YAhDic+adYRAnBZAKCWNXoWIKHhnhm/Pe9CMi1MT5MOWQCeJknc
AvFjtLOkeHTNz/mdpz3Gy2k=
=YZgt
-----END PGP SIGNATURE-----

From rgerhards at hq.adiscon.com  Tue Dec 11 14:55:21 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 11 Dec 2007 14:55:21 +0100
Subject: [rsyslog] Asking for help...
In-Reply-To: <20071211144815.6a2ef123@marathon.haufe>
References: <577465F99B41C842AAFBE9ED71E70ABA30862F@grfint2.intern.adiscon.com>
	<20071211144815.6a2ef123@marathon.haufe>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308646@grfint2.intern.adiscon.com>

Thanks for the feedback - is it OK if we discuss this in the forum? I
think not everyone here on the list is interested in that discussion...

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Jakob Haufe
> Sent: Tuesday, December 11, 2007 2:48 PM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] Asking for help...
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> "Rainer Gerhards" <rgerhards at hq.adiscon.com> wrote on Tue, 11 Dec 2007
> 11:06:00 +0100:
> 
> > http://rgerhards.blogspot.com/2007/12/design-problem.html
> 
> Maybe I'm a bit short-sighted here, but isn't this exactly the
> situation
> for non-blocking sockets? Or will that raise other problems?
> 
> Regards,
> sur5r
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
> 
> iD8DBQFHXpUj1YAhDic+adYRAnBZAKCWNXoWIKHhnhm/Pe9CMi1MT5MOWQCeJknc
> AvFjtLOkeHTNz/mdpz3Gy2k=
> =YZgt
> -----END PGP SIGNATURE-----
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Wed Dec 12 17:06:48 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 12 Dec 2007 17:06:48 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>

Hi all,

I have just released rsyslog 1.20.1. It primarily contains bug fixes: a
memory leak in regard to PostgreSQL date formatting and a potential race
condition within repeated message processing have been fixed. Also,
there are new runtime options to harden ACL processing when DNS is not
ready while rsyslogd start up. There are also some minor changes and
clean-ups. Version 1.20.1 is recommended to anyone using previous
releases. Users of PostgreSQL are urged to update to this version.

Download:
   http://www.rsyslog.com/Downloads-req-getit-lid-66.phtml

Changelog:
   http://www.rsyslog.com/Article147.phtml

You may want to watch my blog for upcoming developments:
http://rgerhards.blogspot.com. There was recently a lot of activity and
more is expected.

As always, feedback is appreciated.
Rainer Gerhards


From stephen.carville at gmail.com  Wed Dec 12 18:38:15 2007
From: stephen.carville at gmail.com (Stephen Carville)
Date: Wed, 12 Dec 2007 09:38:15 -0800
Subject: [rsyslog] Last Message repeated N times
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308637@grfint2.intern.adiscon.com>
References: <2428c0380712071309w31f4fc20qb933dfef47f445ac@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA308637@grfint2.intern.adiscon.com>
Message-ID: <2428c0380712120938x72f55b9na4aea606374f0255@mail.gmail.com>

On Dec 11, 2007 3:01 AM, Rainer Gerhards <rgerhards at hq.adiscon.com> wrote:
> Hi Stephen,
>
> I was totally swamped the past days, sorry for the missing response.
> I'll check what causes this. As a side-note, I think that the message
> reduction code does not work well with the database - as you can no
> longer query the individual messages. Is this your intent?

My original intent was to have a centralized copy of all the changes
that cfengine makes.  That works extremely well and saves me a
buttload of time over having to log by hand in a shared Excel
spreadsheet every time I update a hosts file or tweak httpd.conf or
tomcat's server.xml.  That and logging the messages from auditd were
the main goals.  I added /var/log/messages and /var/log/secure mainly
because I could and because it gives the Linux side of the shop
greater credibility with the SOX (expletive deleted).  Now that
messages and secure are in the database I'll have to run a log
analyzer periodically looking for indications of "stuff" that's not
right but that comes after about six other projects.

This isn't a show stopper and I could probably just drop the "last
messages repeated.." entries without any harm.  Tho I;m not sure how
jsut yet.  Out of 2.4M rows about 385K are of this type.

> Rainer
>
>
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Stephen Carville
> > Sent: Friday, December 07, 2007 10:09 PM
> > To: rsyslog at lists.adiscon.com
> > Subject: [rsyslog] Last Message repeated N times
> >
> > I am using rsyslog 1.19.10 on a central server to collect messages and
> > store them in a mysql database.  One thing I've noticed is that the
> > "last message repeated NNN times" entries are alway entered with
> > FromHost="last", SysLogTag="message", and Message="repeated NNN
> > times".  Can this be fixed?
> >
> > I still use the regular sysklogd package on the remote machines to
> > forward the messages.
> >
> > --
> > Stephen Carville
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
>

-- 
Stephen Carville


From stephen.carville at gmail.com  Wed Dec 12 19:02:09 2007
From: stephen.carville at gmail.com (Stephen Carville)
Date: Wed, 12 Dec 2007 10:02:09 -0800
Subject: [rsyslog] Last Message repeated N times
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA30863D@grfint2.intern.adiscon.com>
References: <2428c0380712071309w31f4fc20qb933dfef47f445ac@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA308637@grfint2.intern.adiscon.com>
	<577465F99B41C842AAFBE9ED71E70ABA30863D@grfint2.intern.adiscon.com>
Message-ID: <2428c0380712121002l70f37260p869ab35b8acb7068@mail.gmail.com>

On Dec 11, 2007 3:26 AM, Rainer Gerhards <rgerhards at hq.adiscon.com> wrote:
> I think I misunderstood your problem... The "last message repeated n
> times" stems back from the reporting sysklogd's - right?

Would ugrading to rsyslogd on the client machines help?  I'm getting
ready for an upgrade cycle where everything except a couple of boxes
running old hardware not suppoert by the 2.6 kernel will be either
Oracle ULN 4.X or Redhat/CentOS 5.X  I didn't see a spec file in the
tar.gz but I can write one myself if there is no good one available.

> Mhhh... that's
> a problem. I need to do some hardcoded processing in the parser to cover
> part of it. The problem is that sysklogd emits a totally non-standard
> message in that case. Even worse, I think it does not even contain a
> clue at the missing information. I could probably use the sender's IP
> address as source in  that case, but the tag seems to be impossible to
> obtain.
>
> Can you post me a copy of the %rawmsg% property, so that I can have a
> look at what exactly you are seing?
>
> Rainer
>
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>
> > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> > Sent: Tuesday, December 11, 2007 12:01 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] Last Message repeated N times
> >
> > Hi Stephen,
> >
> > I was totally swamped the past days, sorry for the missing response.
> > I'll check what causes this. As a side-note, I think that the message
> > reduction code does not work well with the database - as you can no
> > longer query the individual messages. Is this your intent?
> >
> > Rainer
> >
> > > -----Original Message-----
> > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > > bounces at lists.adiscon.com] On Behalf Of Stephen Carville
> > > Sent: Friday, December 07, 2007 10:09 PM
> > > To: rsyslog at lists.adiscon.com
> > > Subject: [rsyslog] Last Message repeated N times
> > >
> > > I am using rsyslog 1.19.10 on a central server to collect messages
> > and
> > > store them in a mysql database.  One thing I've noticed is that the
> > > "last message repeated NNN times" entries are alway entered with
> > > FromHost="last", SysLogTag="message", and Message="repeated NNN
> > > times".  Can this be fixed?
> > >
> > > I still use the regular sysklogd package on the remote machines to
> > > forward the messages.
> > >
> > > --
> > > Stephen Carville
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
>



-- 
Stephen Carville


From mbiebl at gmail.com  Wed Dec 12 19:11:55 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 12 Dec 2007 19:11:55 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com>

2007/12/12, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Hi all,
>
> I have just released rsyslog 1.20.1. It primarily contains bug fixes: a

1.20.1 unfortunately fails to compile (unless you use --enable-debug):
gcc -g -O2 -W -Wall -Wformat-security -Wshadow -Wcast-align
-Wpointer-arith -Wmissing-format-attribute -g -o rsyslogd
rsyslogd-syslogd.o rsyslogd-pidfile.o rsyslogd-template.o
rsyslogd-outchannel.o rsyslogd-stringbuf.o rsyslogd-srUtils.o
rsyslogd-parse.o rsyslogd-sync.o rsyslogd-net.o rsyslogd-msg.o
rsyslogd-expr.o rsyslogd-omshell.o rsyslogd-omusrmsg.o
rsyslogd-omfwd.o rsyslogd-tcpsyslog.o rsyslogd-omfile.o
rsyslogd-omdiscard.o rsyslogd-modules.o rsyslogd-objomsr.o
rsyslogd-cfsysline.o rsyslogd-linkedlist.o rsyslogd-iminternal.o
rsyslogd-action.o rsyslogd-gss-misc.o -Wl,--export-dynamic  -lz
-lpthread -ldl -lrt
rsyslogd-syslogd.o: In function `domarkActions':
/home/michael/cvs/rsyslog/build/../syslogd.c:3431: undefined reference
to `lockObj'
/home/michael/cvs/rsyslog/build/../syslogd.c:3439: undefined reference
to `unlockObj'
rsyslogd-syslogd.o: In function `callAction':
/home/michael/cvs/rsyslog/build/../syslogd.c:2442: undefined reference
to `lockObj'
/home/michael/cvs/rsyslog/build/../syslogd.c:2508: undefined reference
to `unlockObj'
/home/michael/cvs/rsyslog/build/../syslogd.c:2508: undefined reference
to `unlockObj'
collect2: ld returned 1 exit status
make[2]: *** [rsyslogd] Fehler 1
make[2]: Leaving directory `/home/michael/cvs/rsyslog/build'
make[1]: *** [all-recursive] Fehler 1
make[1]: Leaving directory `/home/michael/cvs/rsyslog/build'
make: *** [all] Fehler 2


sync.h declares lockObj/unlockObj, but sync.c only has these symbols
if compiled without NDEBUG.

Cheers,
Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From infofarmer at FreeBSD.org  Wed Dec 12 19:16:36 2007
From: infofarmer at FreeBSD.org (Andrew Pantyukhin)
Date: Wed, 12 Dec 2007 21:16:36 +0300
Subject: [rsyslog] rsyslog 1.20.1 released
In-Reply-To: <aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>
	<aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com>
Message-ID: <20071212181635.GQ51627@amilo.cenkes.org>

On Wed, Dec 12, 2007 at 07:11:55PM +0100, Michael Biebl wrote:
> 2007/12/12, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > Hi all,
> >
> > I have just released rsyslog 1.20.1. It primarily contains bug fixes: a
> 
> sync.h declares lockObj/unlockObj, but sync.c only has these symbols
> if compiled without NDEBUG.

Try including config.h in sync.h


From mbiebl at gmail.com  Wed Dec 12 19:30:34 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 12 Dec 2007 19:30:34 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
In-Reply-To: <20071212181635.GQ51627@amilo.cenkes.org>
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>
	<aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com>
	<20071212181635.GQ51627@amilo.cenkes.org>
Message-ID: <aa76cdac0712121030y49b70fc0r6379af2b361a5f06@mail.gmail.com>

2007/12/12, Andrew Pantyukhin <infofarmer at freebsd.org>:
> On Wed, Dec 12, 2007 at 07:11:55PM +0100, Michael Biebl wrote:
> > 2007/12/12, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > > Hi all,
> > >
> > > I have just released rsyslog 1.20.1. It primarily contains bug fixes: a
> >
> > sync.h declares lockObj/unlockObj, but sync.c only has these symbols
> > if compiled without NDEBUG.
>
> Try including config.h in sync.h
>

No difference.

The other header files (using NDEBUG) don't include config.h either btw.
The rely on the c file, which includes them, to include config.h.

Cheers,
Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From mbiebl at gmail.com  Wed Dec 12 20:18:38 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 12 Dec 2007 20:18:38 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
In-Reply-To: <aa76cdac0712121030y49b70fc0r6379af2b361a5f06@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>
	<aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com>
	<20071212181635.GQ51627@amilo.cenkes.org>
	<aa76cdac0712121030y49b70fc0r6379af2b361a5f06@mail.gmail.com>
Message-ID: <aa76cdac0712121118kea18ee7md111ed3558bb7a87@mail.gmail.com>

2007/12/12, Michael Biebl <mbiebl at gmail.com>:
> 2007/12/12, Andrew Pantyukhin <infofarmer at freebsd.org>:
> > On Wed, Dec 12, 2007 at 07:11:55PM +0100, Michael Biebl wrote:
> > > 2007/12/12, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > > > Hi all,
> > > >
> > > > I have just released rsyslog 1.20.1. It primarily contains bug fixes: a
> > >
> > > sync.h declares lockObj/unlockObj, but sync.c only has these symbols
> > > if compiled without NDEBUG.
> >
> > Try including config.h in sync.h
> >
>

Found the culprit. Patch is attached.
Imho calling the define NDEBUG makes the code unnecessarily hard to
read (#ifndef NDEBUG ..). I had to stare at the code for a while until
I noticed that it was negated one time to often.

Cheers,
Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

From sur5r at sur5r.net  Wed Dec 12 20:35:38 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Wed, 12 Dec 2007 20:35:38 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>
	<aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com>
	<20071212181635.GQ51627@amilo.cenkes.org>
	<aa76cdac0712121030y49b70fc0r6379af2b361a5f06@mail.gmail.com>
	<aa76cdac0712121118kea18ee7md111ed3558bb7a87@mail.gmail.com>
Message-ID: <20071212203538.6dbb00d1@marathon.haufe>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

"Michael Biebl" <mbiebl at gmail.com> wrote on Wed, 12 Dec 2007 20:18:38
+0100:

> Found the culprit. Patch is attached.

Mailman ate it again. Try attaching it with mimetype text/plain, seems
to work then.

Regards,
sur5r
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHYDgQ1YAhDic+adYRAtxQAKCspmrE+HfH12YVBzJcIBQ/GqNKXACfdSl3
NkY9aWuB+ftzvx2t4MtKPHw=
=BODR
-----END PGP SIGNATURE-----

From rgerhards at hq.adiscon.com  Wed Dec 12 21:04:09 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 12 Dec 2007 21:04:09 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
In-Reply-To: <20071212203538.6dbb00d1@marathon.haufe>
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com><aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com><20071212181635.GQ51627@amilo.cenkes.org><aa76cdac0712121030y49b70fc0r6379af2b361a5f06@mail.gmail.com><aa76cdac0712121118kea18ee7md111ed3558bb7a87@mail.gmail.com>
	<20071212203538.6dbb00d1@marathon.haufe>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308665@grfint2.intern.adiscon.com>

Hold for a moment, the actual patch is a bit different . more soon.

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Jakob Haufe
> Sent: Wednesday, December 12, 2007 8:36 PM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] rsyslog 1.20.1 released
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> "Michael Biebl" <mbiebl at gmail.com> wrote on Wed, 12 Dec 2007 20:18:38
> +0100:
> 
> > Found the culprit. Patch is attached.
> 
> Mailman ate it again. Try attaching it with mimetype text/plain, seems
> to work then.
> 
> Regards,
> sur5r
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
> 
> iD8DBQFHYDgQ1YAhDic+adYRAtxQAKCspmrE+HfH12YVBzJcIBQ/GqNKXACfdSl3
> NkY9aWuB+ftzvx2t4MtKPHw=
> =BODR
> -----END PGP SIGNATURE-----
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Wed Dec 12 21:07:26 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 12 Dec 2007 21:07:26 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308665@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com><aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com><20071212181635.GQ51627@amilo.cenkes.org><aa76cdac0712121030y49b70fc0r6379af2b361a5f06@mail.gmail.com><aa76cdac0712121118kea18ee7md111ed3558bb7a87@mail.gmail.com><20071212203538.6dbb00d1@marathon.haufe>
	<577465F99B41C842AAFBE9ED71E70ABA308665@grfint2.intern.adiscon.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308666@grfint2.intern.adiscon.com>

Re-released, info follows soon...

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Wednesday, December 12, 2007 9:04 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 1.20.1 released
> 
> Hold for a moment, the actual patch is a bit different . more soon.
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Jakob Haufe
> > Sent: Wednesday, December 12, 2007 8:36 PM
> > To: rsyslog at lists.adiscon.com
> > Subject: Re: [rsyslog] rsyslog 1.20.1 released
> >
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > "Michael Biebl" <mbiebl at gmail.com> wrote on Wed, 12 Dec 2007
20:18:38
> > +0100:
> >
> > > Found the culprit. Patch is attached.
> >
> > Mailman ate it again. Try attaching it with mimetype text/plain,
> seems
> > to work then.
> >
> > Regards,
> > sur5r
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v1.4.6 (GNU/Linux)
> >
> > iD8DBQFHYDgQ1YAhDic+adYRAtxQAKCspmrE+HfH12YVBzJcIBQ/GqNKXACfdSl3
> > NkY9aWuB+ftzvx2t4MtKPHw=
> > =BODR
> > -----END PGP SIGNATURE-----
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From mbiebl at gmail.com  Wed Dec 12 21:31:23 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 12 Dec 2007 21:31:23 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308665@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>
	<aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com>
	<20071212181635.GQ51627@amilo.cenkes.org>
	<aa76cdac0712121030y49b70fc0r6379af2b361a5f06@mail.gmail.com>
	<aa76cdac0712121118kea18ee7md111ed3558bb7a87@mail.gmail.com>
	<20071212203538.6dbb00d1@marathon.haufe>
	<577465F99B41C842AAFBE9ED71E70ABA308665@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712121231i62631494p4ea39e41136adac3@mail.gmail.com>

2007/12/12, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Hold for a moment, the actual patch is a bit different . more soon.

Yeah, got it the wrong way around. Didn't read the comment in sync.h
carefully enough.
Thanks for you sharp eye!

As I said, imho it's easier to read and grasp if the define is called DEBUG.

Cheers,
Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Wed Dec 12 21:40:25 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 12 Dec 2007 21:40:25 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
In-Reply-To: <aa76cdac0712121130x4c96b883t4ee91e3362fb83d6@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>
	<aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com>
	<20071212181635.GQ51627@amilo.cenkes.org>
	<aa76cdac0712121030y49b70fc0r6379af2b361a5f06@mail.gmail.com>
	<aa76cdac0712121118kea18ee7md111ed3558bb7a87@mail.gmail.com>
	<aa76cdac0712121130x4c96b883t4ee91e3362fb83d6@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308667@grfint2.intern.adiscon.com>

Hi all,

Man, when did I really screw up the last time? And I thought I had run
all test...

Anyhow, thanks all for helping to solve this issue.

Michael got it right, it is the NDEBUG. But is not the .c file but the
header file that was wrong. The idea is that we have functions only in
debugging mode, because functions make it easier to debug. But they are
slower than macro expansions. So in release mode, the functions are
replace with the mutex calls directly (which, I think, a macros, too).
If I recall it correctly, I screwed up after I created that
optimization.

> > Found the culprit. Patch is attached.
> > Imho calling the define NDEBUG makes the code unnecessarily hard to
> > read (#ifndef NDEBUG ..). I had to stare at the code for a 
> while until
> > I noticed that it was negated one time to often.

The NDEBUG is an assert() inheritance. Assert() uses it to indicate a
release build. I've build on the same tool to avoid introducing another
one. But obviously, it has its drawbacks, too. I think I'll solve that
by adding a comment.

I hope this clarifies. Drop me a note if I got it wrong again or you
have a question ;)

The tarball that is now in the repository should solve the issue. 

Rainer


From mbiebl at gmail.com  Wed Dec 12 21:47:32 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 12 Dec 2007 21:47:32 +0100
Subject: [rsyslog] Problems with IncludeConfig
Message-ID: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>

Hi everyone,

the current IncludeConfig implementation has some issues, especially
for distributers:
All files in the directory as specified by $IncludeConfig are included.
This means, temp files like *~ *.bak etc that are left behind by
editors, or stuff like *.rpmsave or *.dpkg-old/*.dpkg-new.
This is obviously unfortunate.

How shall we solve this?
1.) I patched the Debian package to only include files ending with
*.conf. If we document that properly this is imho an acceptable
solution.
2.) Implement an internal blacklist of endings that are ignored (see above)
3.) Instead of hardcoding the file ending to *.conf (as in 1.), use
the same syntax as e.g. apache, which allows to specify shell globs:
example:
Include /etc/apache2/conf.d/*.conf

What do you think?

Michael


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From r.bhatia at ipax.at  Thu Dec 13 09:55:49 2007
From: r.bhatia at ipax.at (Raoul Bhatia [IPAX])
Date: Thu, 13 Dec 2007 09:55:49 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
Message-ID: <4760F395.4030502@ipax.at>

Michael Biebl wrote:
> the current IncludeConfig implementation has some issues, especially
> for distributers:
> All files in the directory as specified by $IncludeConfig are included.
> This means, temp files like *~ *.bak etc that are left behind by
> editors, or stuff like *.rpmsave or *.dpkg-old/*.dpkg-new.
> This is obviously unfortunate.
> 
> How shall we solve this?
> 1.) I patched the Debian package to only include files ending with
> *.conf. If we document that properly this is imho an acceptable
> solution.
> 2.) Implement an internal blacklist of endings that are ignored (see above)
> 3.) Instead of hardcoding the file ending to *.conf (as in 1.), use
> the same syntax as e.g. apache, which allows to specify shell globs:
> example:
> Include /etc/apache2/conf.d/*.conf

i would opt for the third option - thou i'm not sure how much work
this creates ;)

cheers,
raoul
-- 
____________________________________________________________________
DI (FH) Raoul Bhatia M.Sc.          email.          r.bhatia at ipax.at
Technischer Leiter

IPAX - Aloy Bhatia Hava OEG         web.          http://www.ipax.at
Barawitzkagasse 10/2/2/11           email.            office at ipax.at
1190 Wien                           tel.               +43 1 3670030
FN 277995t HG Wien                  fax.            +43 1 3670030 15
____________________________________________________________________


From rgerhards at hq.adiscon.com  Thu Dec 13 13:04:33 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 13 Dec 2007 13:04:33 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <4760F395.4030502@ipax.at>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
	<4760F395.4030502@ipax.at>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>

Lol... the initial message seems not to have passed my spam filter... ;)

I, too, tend to option three. The problem, though is
backwards-compatibility... Any suggestions and concerns are appreciated.

I'll look at the code to see how much work that creates. I think the
complexity is quite acceptable. I am right now in a threading redesign,
so I'd like to push this away for a little while... Anyhow, please keep
comments floating - I'll listen and drop in any thoughts I have.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Raoul Bhatia [IPAX]
> Sent: Thursday, December 13, 2007 9:56 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Problems with IncludeConfig
> 
> Michael Biebl wrote:
> > the current IncludeConfig implementation has some issues, especially
> > for distributers:
> > All files in the directory as specified by $IncludeConfig are
> included.
> > This means, temp files like *~ *.bak etc that are left behind by
> > editors, or stuff like *.rpmsave or *.dpkg-old/*.dpkg-new.
> > This is obviously unfortunate.
> >
> > How shall we solve this?
> > 1.) I patched the Debian package to only include files ending with
> > *.conf. If we document that properly this is imho an acceptable
> > solution.
> > 2.) Implement an internal blacklist of endings that are ignored (see
> above)
> > 3.) Instead of hardcoding the file ending to *.conf (as in 1.), use
> > the same syntax as e.g. apache, which allows to specify shell globs:
> > example:
> > Include /etc/apache2/conf.d/*.conf
> 
> i would opt for the third option - thou i'm not sure how much work
> this creates ;)
> 
> cheers,
> raoul
> --
> ____________________________________________________________________
> DI (FH) Raoul Bhatia M.Sc.          email.          r.bhatia at ipax.at
> Technischer Leiter
> 
> IPAX - Aloy Bhatia Hava OEG         web.          http://www.ipax.at
> Barawitzkagasse 10/2/2/11           email.            office at ipax.at
> 1190 Wien                           tel.               +43 1 3670030
> FN 277995t HG Wien                  fax.            +43 1 3670030 15
> ____________________________________________________________________
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Fri Dec 14 09:29:19 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 14 Dec 2007 09:29:19 +0100
Subject: [rsyslog] IMPORTANT: Version Changes
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308674@grfint2.intern.adiscon.com>

Hi all,

some information first here on the mailing list before I announce it. As
you know, I am looking at the way threading is supposed to work in
future releases and, most importantly, looking at the inputs (like mark
message generation).

Around summer, I wrote that I will probably need to release new major
versions when we go into multithreading redesign. It looks like we have
reached this stage. I tried to keep a single code base that still
supports both single- and multi-threaded operations. I have looked into
this the past days and I need to say that it creates a lot of complexity
and hard to understand code.

For this reason, I think it is finally time to branch the code based and
release some new versions.

Soon, I will create a branch for the current 1.20.1 code base. That will
only receive bug fixes, but no new development (except, I guess, GSSAPI
which I about to be contributed by Red Hat). When we are confident the
last changes worked well and introduced no new bugs, there will be a
version 2.0.0 stable release based on that code base.

CVS head, however, will then be rsyslog version 3. It will receive the
new input module interface. It requires pthreads, because there is no
way input modules and many more of the new desired features can be
implemented without them. Consequently, I will remove all
single-threading code from it, resulting in an easier to understand code
base. Please note that I expect this code to change dramatically when it
is being modified to be more modular (much like it was when I introduced
modular outputs in summer). Please note that I will apply any non-bugfix
patches to this code base, only.

I have somewhat bad feeling of going ahead with implementing a more
sophisticated and more parallel multi-threading while we still have an
issue with the segfault. However, I think by now we did everything
imaginable to capture that rare bug. I have come to the conclusion that
the best chance to find it is go ahead and implement the more
sophisticated design. That will lead to a review, and rewrite, of much
of the code in question, uncovering this we didn't think about before.
The recently discovered race condition is an excellent sample.

One thing about the license: rsyslog 2 will stay with "GPL v2 and above"
license, but rsyslog V3 will be licensed under "GPL v3 and above". I
already wrote about that change. It is my firm believe that GPL v3
brings benefit to our freedom to use digital goods. I am a strong oppose
of digital restrictions management (DRM) and software patens and I do
not like the idea that rsyslog benefits anyone who encourages these
things. I hope for your understanding.

I will set stage now for these changes and will do a web announcement
soon. Please don't be surprised that rsyslog v3 will be available before
v2, you now know the reason.

Rainer


From mbiebl at gmail.com  Fri Dec 14 14:36:03 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Fri, 14 Dec 2007 14:36:03 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
	<4760F395.4030502@ipax.at>
	<577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>

2007/12/13, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Lol... the initial message seems not to have passed my spam filter... ;)
>
> I, too, tend to option three. The problem, though is

Yeah, it's the most flexible one.

> backwards-compatibility... Any suggestions and concerns are appreciated.

This feature has been pretty new, and as such probably isn't widely used.
I would simply emphasize this change in the release notes and the Changelog.
The worst that can happen is, that a file is not included anymore.

>
> I'll look at the code to see how much work that creates. I think the
> complexity is quite acceptable. I am right now in a threading redesign,
> so I'd like to push this away for a little while... Anyhow, please keep
> comments floating - I'll listen and drop in any thoughts I have.

Should be fairly straightforward to implement using
http://www.gnu.org/software/libc/manual/html_node/Globbing.html#Globbing

Cheers,
Michael


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From mbiebl at gmail.com  Fri Dec 14 17:24:02 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Fri, 14 Dec 2007 17:24:02 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
	<4760F395.4030502@ipax.at>
	<577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>
	<aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>
Message-ID: <aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com>

2007/12/14, Michael Biebl <mbiebl at gmail.com>:
> 2007/12/13, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > Lol... the initial message seems not to have passed my spam filter... ;)
> >
> > I, too, tend to option three. The problem, though is
>
> Yeah, it's the most flexible one.
>
> > backwards-compatibility... Any suggestions and concerns are appreciated.
>
> This feature has been pretty new, and as such probably isn't widely used.
> I would simply emphasize this change in the release notes and the Changelog.
> The worst that can happen is, that a file is not included anymore.
>
> >
> > I'll look at the code to see how much work that creates. I think the
> > complexity is quite acceptable. I am right now in a threading redesign,
> > so I'd like to push this away for a little while... Anyhow, please keep
> > comments floating - I'll listen and drop in any thoughts I have.
>
> Should be fairly straightforward to implement using
> http://www.gnu.org/software/libc/manual/html_node/Globbing.html#Globbing
>

Small example using glob:

#include <glob.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char **argv)
{
        glob_t files;
        int ret = 0;
        int i = 0;
        char *path;

        if (argc > 1) {
                path = argv[1];
        } else {
                path = strdup("/etc/rsyslog.d/*.conf");
        }

        ret = glob(path, GLOB_ERR, NULL, &files);
        printf("All files matching %s:\n", path);
        for (i = 0; i < files.gl_pathc; i++) {
                printf("\t%s\n", files.gl_pathv[i]);
        }

        globfree(&files);
        printf("Finished\n");

        return EXIT_SUCCESS;
}

man 3 glob has more information.

If wanted, I can prepare a patch for rsyslog.

Cheers,
Michael
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Fri Dec 14 17:29:38 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 14 Dec 2007 17:29:38 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com><4760F395.4030502@ipax.at><577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com><aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>
	<aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com>

Michael,

sorry for the sluggish reply. I am right in the middle of the initial
threading code (have a look at CVS ;)) and that blocks most of
everything else ;)

It would be great if you could supply a patch, I'd apply it as soon as I
got something together with the new code. I try to do the threading in
an as atomic code change as possible, but it will proabbyl busy me for a
while ;)

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Friday, December 14, 2007 5:24 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Problems with IncludeConfig
> 
> 2007/12/14, Michael Biebl <mbiebl at gmail.com>:
> > 2007/12/13, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > > Lol... the initial message seems not to have passed my spam
> filter... ;)
> > >
> > > I, too, tend to option three. The problem, though is
> >
> > Yeah, it's the most flexible one.
> >
> > > backwards-compatibility... Any suggestions and concerns are
> appreciated.
> >
> > This feature has been pretty new, and as such probably isn't widely
> used.
> > I would simply emphasize this change in the release notes and the
> Changelog.
> > The worst that can happen is, that a file is not included anymore.
> >
> > >
> > > I'll look at the code to see how much work that creates. I think
> the
> > > complexity is quite acceptable. I am right now in a threading
> redesign,
> > > so I'd like to push this away for a little while... Anyhow, please
> keep
> > > comments floating - I'll listen and drop in any thoughts I have.
> >
> > Should be fairly straightforward to implement using
> >
>
http://www.gnu.org/software/libc/manual/html_node/Globbing.html#Globbin
> g
> >
> 
> Small example using glob:
> 
> #include <glob.h>
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
> 
> int main(int argc, char **argv)
> {
>         glob_t files;
>         int ret = 0;
>         int i = 0;
>         char *path;
> 
>         if (argc > 1) {
>                 path = argv[1];
>         } else {
>                 path = strdup("/etc/rsyslog.d/*.conf");
>         }
> 
>         ret = glob(path, GLOB_ERR, NULL, &files);
>         printf("All files matching %s:\n", path);
>         for (i = 0; i < files.gl_pathc; i++) {
>                 printf("\t%s\n", files.gl_pathv[i]);
>         }
> 
>         globfree(&files);
>         printf("Finished\n");
> 
>         return EXIT_SUCCESS;
> }
> 
> man 3 glob has more information.
> 
> If wanted, I can prepare a patch for rsyslog.
> 
> Cheers,
> Michael
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From mbiebl at gmail.com  Sun Dec 16 23:38:14 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Sun, 16 Dec 2007 23:38:14 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
	<4760F395.4030502@ipax.at>
	<577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>
	<aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>
	<aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712161438n69382660u9d2a082914d717c9@mail.gmail.com>

2007/12/14, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Michael,
>
> sorry for the sluggish reply. I am right in the middle of the initial
> threading code (have a look at CVS ;)) and that blocks most of
> everything else ;)
>
> It would be great if you could supply a patch, I'd apply it as soon as I
> got something together with the new code. I try to do the threading in
> an as atomic code change as possible, but it will proabbyl busy me for a
> while ;)

Unfortunately I can't set the mimetype via gmail, so the patch is at

http://debs.michaelbiebl.de/rsyslog/rsyslog-glob.patch

I didn't check the return values for glob. I wanted I can implement
that and printout debugging messsages in that case.
The patch also doesn't contain any changes to the documentation (yet).
I wanted to get some feedback first.

Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From mbiebl at gmail.com  Mon Dec 17 05:09:59 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Mon, 17 Dec 2007 05:09:59 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <aa76cdac0712161438n69382660u9d2a082914d717c9@mail.gmail.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
	<4760F395.4030502@ipax.at>
	<577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>
	<aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>
	<aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com>
	<aa76cdac0712161438n69382660u9d2a082914d717c9@mail.gmail.com>
Message-ID: <aa76cdac0712162009q9fe3ecaq407df79296e4baff@mail.gmail.com>

2007/12/16, Michael Biebl <mbiebl at gmail.com>:
> 2007/12/14, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > Michael,
> >
> > sorry for the sluggish reply. I am right in the middle of the initial
> > threading code (have a look at CVS ;)) and that blocks most of
> > everything else ;)
> >
> > It would be great if you could supply a patch, I'd apply it as soon as I
> > got something together with the new code. I try to do the threading in
> > an as atomic code change as possible, but it will proabbyl busy me for a
> > while ;)
>
> Unfortunately I can't set the mimetype via gmail, so the patch is at
>
> http://debs.michaelbiebl.de/rsyslog/rsyslog-glob.patch
>

I have an updated patch
 http://debs.michaelbiebl.de/rsyslog/rsyslog-glob-2.patch
which is backward compatible.
It allows the following

include config files:
 $IncludeConfig /etc/rsyslog.d/foo.conf
include config directories:
 $IncludeConfig /etc/rsyslog.d/ (with or without trailing slash)
include via shell glob:
 $IncludeConfig /etc/rsyslog.d/*.conf

One remaining issue is, how to handle symlinks. I made the patch
consistent to the current behaviour of doIncludeDirectories to ignore
symlinks.

I know that the apache Include directive allows symlinks, and
supporting it could be handy.
So my question is, what do others think about it. Please let me know.

Cheers,
Michael

P.S: A personal wish: as the list is subscriber only, would it be
possible to allow attachements (other than text/plain, gmail is
obviously too stupid)?


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From mbiebl at gmail.com  Mon Dec 17 05:27:28 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Mon, 17 Dec 2007 05:27:28 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <aa76cdac0712162009q9fe3ecaq407df79296e4baff@mail.gmail.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
	<4760F395.4030502@ipax.at>
	<577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>
	<aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>
	<aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com>
	<aa76cdac0712161438n69382660u9d2a082914d717c9@mail.gmail.com>
	<aa76cdac0712162009q9fe3ecaq407df79296e4baff@mail.gmail.com>
Message-ID: <aa76cdac0712162027r85875d2rc7223f9c6a61a43@mail.gmail.com>

2007/12/17, Michael Biebl <mbiebl at gmail.com>:
> 2007/12/16, Michael Biebl <mbiebl at gmail.com>:
> > 2007/12/14, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> I have an updated patch
>  http://debs.michaelbiebl.de/rsyslog/rsyslog-glob-2.patch

Spotted a minor typo, so here goes v3
http://debs.michaelbiebl.de/rsyslog/rsyslog-glob-3.patch

Michael

Btw. gmail uses
Content-Type: multipart/mixed
and then
Content-Type: text/plain for the mail
and
Content-Type: text/x-diff for the patch
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Mon Dec 17 08:04:46 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 17 Dec 2007 08:04:46 +0100
Subject: [rsyslog] Tackeling the segfault...
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308683@grfint2.intern.adiscon.com>

Hi all,

thanks to mildew, I could create another fix that may solve the issue. I
will do a writeup of what happens, but that'll take a short while. In
the mean time, I would appreciate if those of you that experience the
segfault could try the new version. I have released a pre-1.20.2 tarball
at

http://download.rsyslog.com/rsyslog/rsyslog-1.20.2.tar.gz

As usual, this is not yet 1.20.2, so whenever it is re-released be sure
to update your installation. I would appreciate feedback on it very
much.

Many thanks,
Rainer


From mbiebl at gmail.com  Mon Dec 17 08:20:59 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Mon, 17 Dec 2007 08:20:59 +0100
Subject: [rsyslog] Tackeling the segfault...
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308683@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308683@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712162320l3b089b2eue6a3c4699ad0b277@mail.gmail.com>

2007/12/17, Rainer Gerhards <rgerhards at hq.adiscon.com>:
>
> http://download.rsyslog.com/rsyslog/rsyslog-1.20.2.tar.gz
>
> As usual, this is not yet 1.20.2, so whenever it is re-released be sure
> to update your installation. I would appreciate feedback on it very
> much.

Hi Rainer,

it would be handy if you would post md5sums or sha1sums of the tarball.
This would make it easier to identify/distinguish the prereleases and
the final tarball release.

Cheers,
Michael

P.S: current CVS head doesn't compile with USE_NETZIP.

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Mon Dec 17 08:26:38 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 17 Dec 2007 08:26:38 +0100
Subject: [rsyslog] Tackeling the segfault...
In-Reply-To: <aa76cdac0712162320l3b089b2eue6a3c4699ad0b277@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308683@grfint2.intern.adiscon.com>
	<aa76cdac0712162320l3b089b2eue6a3c4699ad0b277@mail.gmail.com>
Message-ID: <1197876398.21389.3.camel@localhost.localdomain>


On Mon, 2007-12-17 at 08:20 +0100, Michael Biebl wrote:
> 2007/12/17, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> >
> > http://download.rsyslog.com/rsyslog/rsyslog-1.20.2.tar.gz
> >
> > As usual, this is not yet 1.20.2, so whenever it is re-released be sure
> > to update your installation. I would appreciate feedback on it very
> > much.
> 
> Hi Rainer,
> 
> it would be handy if you would post md5sums or sha1sums of the tarball.
> This would make it easier to identify/distinguish the prereleases and
> the final tarball release.

good suggestion. For this release the md5sum is
45e6f14094c26f2dd3a3ff7a4d025381 

> 
> Cheers,
> Michael
> 
> P.S: current CVS head doesn't compile with USE_NETZIP.
Will look into that. I have to admit that CVS head may have some issues
the next few days. I thought about holding off commits until everything
is perfect, that would result in one gigantic diff once it is done -
which would make it hard to follow what I am doing. So I am now doing
commits whenever something seems to work reasonably well, but definitely
not prime time yet. The 3.0 branch is not ready now for being used in
practice. But, NETZIP should compile and I'll look into it. Thanks :)


From rgerhards at hq.adiscon.com  Mon Dec 17 08:27:34 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 17 Dec 2007 08:27:34 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <aa76cdac0712162027r85875d2rc7223f9c6a61a43@mail.gmail.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com><4760F395.4030502@ipax.at><577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com><aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com><aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com><aa76cdac0712161438n69382660u9d2a082914d717c9@mail.gmail.com><aa76cdac0712162009q9fe3ecaq407df79296e4baff@mail.gmail.com>
	<aa76cdac0712162027r85875d2rc7223f9c6a61a43@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308685@grfint2.intern.adiscon.com>

Hi Michael,

I like this patch, especially it being backwards-compatible. And I also
tend to agree that symlinks would be nice to have.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Monday, December 17, 2007 5:27 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Problems with IncludeConfig
> 
> 2007/12/17, Michael Biebl <mbiebl at gmail.com>:
> > 2007/12/16, Michael Biebl <mbiebl at gmail.com>:
> > > 2007/12/14, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > I have an updated patch
> >  http://debs.michaelbiebl.de/rsyslog/rsyslog-glob-2.patch
> 
> Spotted a minor typo, so here goes v3
> http://debs.michaelbiebl.de/rsyslog/rsyslog-glob-3.patch
> 
> Michael
> 
> Btw. gmail uses
> Content-Type: multipart/mixed
> and then
> Content-Type: text/plain for the mail
> and
> Content-Type: text/x-diff for the patch
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Mon Dec 17 08:48:01 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 17 Dec 2007 08:48:01 +0100
Subject: [rsyslog] Tackeling the segfault...
In-Reply-To: <aa76cdac0712162320l3b089b2eue6a3c4699ad0b277@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308683@grfint2.intern.adiscon.com>
	<aa76cdac0712162320l3b089b2eue6a3c4699ad0b277@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308686@grfint2.intern.adiscon.com>

> P.S: current CVS head doesn't compile with USE_NETZIP.

Mmmhhh... I cannot reproduce this on my machine (using Fedora 8). Can
you supply me the error messages - I guess best via private mail because
this will get into too much detail for most on this list (Or we move it
to the dev corner in the forum, so that those interested can follow)...

Thanks,
Rainer


From r.bhatia at ipax.at  Mon Dec 17 10:47:50 2007
From: r.bhatia at ipax.at (Raoul Bhatia [IPAX])
Date: Mon, 17 Dec 2007 10:47:50 +0100
Subject: [rsyslog] Tackeling the segfault...
In-Reply-To: <aa76cdac0712162320l3b089b2eue6a3c4699ad0b277@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308683@grfint2.intern.adiscon.com>
	<aa76cdac0712162320l3b089b2eue6a3c4699ad0b277@mail.gmail.com>
Message-ID: <476645C6.2080909@ipax.at>

Michael Biebl wrote:
> 2007/12/17, Rainer Gerhards <rgerhards at hq.adiscon.com>:
>> http://download.rsyslog.com/rsyslog/rsyslog-1.20.2.tar.gz
>>
>> As usual, this is not yet 1.20.2, so whenever it is re-released be sure
>> to update your installation. I would appreciate feedback on it very
>> much.
> 
> Hi Rainer,
> 
> it would be handy if you would post md5sums or sha1sums of the tarball.
> This would make it easier to identify/distinguish the prereleases and
> the final tarball release.

or simply use the well known -pre or -rc tags?

cheers,
raoul
-- 
____________________________________________________________________
DI (FH) Raoul Bhatia M.Sc.          email.          r.bhatia at ipax.at
Technischer Leiter

IPAX - Aloy Bhatia Hava OEG         web.          http://www.ipax.at
Barawitzkagasse 10/2/2/11           email.            office at ipax.at
1190 Wien                           tel.               +43 1 3670030
FN 277995t HG Wien                  fax.            +43 1 3670030 15
____________________________________________________________________


From rgerhards at hq.adiscon.com  Mon Dec 17 11:04:33 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 17 Dec 2007 11:04:33 +0100
Subject: [rsyslog] backward compatibility...
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>

Hi all,

I would appreciate your feedback on backward compatibility. I have
started to create a loadable input module interface in rsyslog v3. My
first input module is the mark message handler (because it has almost no
code, so it was a good example of what needs to be done...). I have now
basically have finished an initial version. The interface needs to be
changed in the longer term, but I'll postpone that until I've done a bit
more work on the config interface. Now I am facing a problem with
backward compatibility:

In sysklogd/rsyslog pre-v3, the -m option controls how often mark
messages are written (with -m0 disabling this mechanism). In v3, the
$MarkMessagePeriod config file directive does the same. However, a
$modload immark is necessary to activate that functionality because
otherwise the module is not loaded (which is good if it is not needed).
When trying to remain backward-compatible, I have an issue when the -m
option is absent. If so, that means a mark message period of 20 minutes.
However, it is hard to detect what may be intended in v3 if this option
is not present. Does it mean the former default (running as a drop-in
replacement) or does it actually mean that no mark messages should
appear if no such directive is in rsyslog.conf?

Do you think it would be acceptable to not emit messages in this case,
slightly breaking backward compatibility? Or should I come up with a
solution. The later would probably require a specific option setting to
disable the -m option and its default (somehow telling "I am not running
in v2 compatibility mode").

I am not sure what is best...

Rainer 


From r.bhatia at ipax.at  Mon Dec 17 11:17:03 2007
From: r.bhatia at ipax.at (Raoul Bhatia [IPAX])
Date: Mon, 17 Dec 2007 11:17:03 +0100
Subject: [rsyslog] backward compatibility...
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
Message-ID: <47664C9F.5030301@ipax.at>

Rainer Gerhards wrote:
> Hi all,
> 
> I would appreciate your feedback on backward compatibility. I have
> started to create a loadable input module interface in rsyslog v3. My
> first input module is the mark message handler (because it has almost no
> code, so it was a good example of what needs to be done...). I have now
> basically have finished an initial version. The interface needs to be
> changed in the longer term, but I'll postpone that until I've done a bit
> more work on the config interface. Now I am facing a problem with
> backward compatibility:
> 
> In sysklogd/rsyslog pre-v3, the -m option controls how often mark
> messages are written (with -m0 disabling this mechanism). In v3, the
> $MarkMessagePeriod config file directive does the same. However, a
> $modload immark is necessary to activate that functionality because
> otherwise the module is not loaded (which is good if it is not needed).
> When trying to remain backward-compatible, I have an issue when the -m
> option is absent. If so, that means a mark message period of 20 minutes.
> However, it is hard to detect what may be intended in v3 if this option
> is not present. Does it mean the former default (running as a drop-in
> replacement) or does it actually mean that no mark messages should
> appear if no such directive is in rsyslog.conf?
> 
> Do you think it would be acceptable to not emit messages in this case,
> slightly breaking backward compatibility? Or should I come up with a
> solution. The later would probably require a specific option setting to
> disable the -m option and its default (somehow telling "I am not running
> in v2 compatibility mode").
> 
> I am not sure what is best...

i would use the opportunity and not hesitate to use a
make-everything-right-even-if-we-break-something approach.

upgrading seems to not be trivial, as one needs to go through the config
files anyway. so lets do things right and maybe issue some error/warning
message on obsolete options (like -m).

cheers,
raoul
ps: i hope i'm getting you right ;)
-- 
____________________________________________________________________
DI (FH) Raoul Bhatia M.Sc.          email.          r.bhatia at ipax.at
Technischer Leiter

IPAX - Aloy Bhatia Hava OEG         web.          http://www.ipax.at
Barawitzkagasse 10/2/2/11           email.            office at ipax.at
1190 Wien                           tel.               +43 1 3670030
FN 277995t HG Wien                  fax.            +43 1 3670030 15
____________________________________________________________________


From rgerhards at hq.adiscon.com  Mon Dec 17 11:35:02 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 17 Dec 2007 11:35:02 +0100
Subject: [rsyslog] backward compatibility...
In-Reply-To: <47664C9F.5030301@ipax.at>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
	<47664C9F.5030301@ipax.at>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308692@grfint2.intern.adiscon.com>

> i would use the opportunity and not hesitate to use a
> make-everything-right-even-if-we-break-something approach.
> 
> upgrading seems to not be trivial, as one needs to go through the
> config
> files anyway. so lets do things right and maybe issue some
> error/warning
> message on obsolete options (like -m).
> 
> cheers,
> raoul
> ps: i hope i'm getting you right ;)

Thanks - yes, this was the kind of feedback I am looking for. Sorry for
the quite complex description, I could not come up with a better one...

Rainer


From rgerhards at hq.adiscon.com  Tue Dec 18 17:21:26 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 18 Dec 2007 17:21:26 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <aa76cdac0712162009q9fe3ecaq407df79296e4baff@mail.gmail.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
	<4760F395.4030502@ipax.at>
	<577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>
	<aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>
	<aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com>
	<aa76cdac0712161438n69382660u9d2a082914d717c9@mail.gmail.com>
	<aa76cdac0712162009q9fe3ecaq407df79296e4baff@mail.gmail.com>
Message-ID: <1197994886.14826.2.camel@localhost.localdomain>

This patch now also made it to the repository. Applied to both v2
(previously 1.20.2) and v3.

Rainer

On Mon, 2007-12-17 at 05:09 +0100, Michael Biebl wrote:
> 2007/12/16, Michael Biebl <mbiebl at gmail.com>:
> > 2007/12/14, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > > Michael,
> > >
> > > sorry for the sluggish reply. I am right in the middle of the initial
> > > threading code (have a look at CVS ;)) and that blocks most of
> > > everything else ;)
> > >
> > > It would be great if you could supply a patch, I'd apply it as soon as I
> > > got something together with the new code. I try to do the threading in
> > > an as atomic code change as possible, but it will proabbyl busy me for a
> > > while ;)
> >
> > Unfortunately I can't set the mimetype via gmail, so the patch is at
> >
> > http://debs.michaelbiebl.de/rsyslog/rsyslog-glob.patch
> >
> 
> I have an updated patch
>  http://debs.michaelbiebl.de/rsyslog/rsyslog-glob-2.patch
> which is backward compatible.
> It allows the following
> 
> include config files:
>  $IncludeConfig /etc/rsyslog.d/foo.conf
> include config directories:
>  $IncludeConfig /etc/rsyslog.d/ (with or without trailing slash)
> include via shell glob:
>  $IncludeConfig /etc/rsyslog.d/*.conf
> 
> One remaining issue is, how to handle symlinks. I made the patch
> consistent to the current behaviour of doIncludeDirectories to ignore
> symlinks.
> 
> I know that the apache Include directive allows symlinks, and
> supporting it could be handy.
> So my question is, what do others think about it. Please let me know.
> 
> Cheers,
> Michael
> 
> P.S: A personal wish: as the list is subscriber only, would it be
> possible to allow attachements (other than text/plain, gmail is
> obviously too stupid)?
> 
> 


From sur5r at sur5r.net  Tue Dec 18 22:16:54 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Tue, 18 Dec 2007 22:16:54 +0100
Subject: [rsyslog] backward compatibility...
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
Message-ID: <20071218221654.27ec59a2@marathon.haufe>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

"Rainer Gerhards" <rgerhards at hq.adiscon.com> wrote on Mon, 17 Dec 2007
11:04:33 +0100:

> Do you think it would be acceptable to not emit messages in this case,
> slightly breaking backward compatibility? Or should I come up with a
> solution. The later would probably require a specific option setting
> to disable the -m option and its default (somehow telling "I am not
> running in v2 compatibility mode").

This makes me think about sysklogd compatibility. How about the
following: If there's no rsyslog.conf but a syslog.conf then rsyslogd
could go to v2/sysklogd-compatibility mode. If not, it should run in
native mode.
This should improve compatibility in "drop-in replacement for stock
sysklogd" setups.

Regards,
sur5r
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHaDjK1YAhDic+adYRArd6AJ9fXRZU1TnW4L/eKRSfcE3x/G/PMACfbDEF
GgU6vviKYQQofAeAJ9vEnDw=
=6u5b
-----END PGP SIGNATURE-----

From mic at npgx.com.au  Wed Dec 19 04:45:00 2007
From: mic at npgx.com.au (Michael Mansour)
Date: Wed, 19 Dec 2007 13:45:00 +1000
Subject: [rsyslog] backward compatibility...
In-Reply-To: <20071218221654.27ec59a2@marathon.haufe>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
	<20071218221654.27ec59a2@marathon.haufe>
Message-ID: <20071219033907.M7098@npgx.com.au>

Hi,

> "Rainer Gerhards" <rgerhards at hq.adiscon.com> wrote on Mon, 17 Dec 2007
> 11:04:33 +0100:
> 
> > Do you think it would be acceptable to not emit messages in this case,
> > slightly breaking backward compatibility? Or should I come up with a
> > solution. The later would probably require a specific option setting
> > to disable the -m option and its default (somehow telling "I am not
> > running in v2 compatibility mode").
> 
> This makes me think about sysklogd compatibility. How about the
> following: If there's no rsyslog.conf but a syslog.conf then rsyslogd
> could go to v2/sysklogd-compatibility mode. If not, it should run in
> native mode.
> This should improve compatibility in "drop-in replacement for stock
> sysklogd" setups.

That's true, but I've also found developers are happy to cater for rsyslog
through updates to their software. For example, I use drbdlinks (software for
clusters) and when I introduced rsyslog into my environment I found drbdlinks
incompatible with it. 

I put a workaround in place and at the same time emailed the drbdlinks
developer, and he introduced rsyslog support into his code, provided an update
to his software and mentioned me in the suggestion.

Jamie Cameron of Webmin, Virtualmin and Virtualmin Pro is/has also
introduced/introducing support for rsyslog into his products, since some
Webmin functionality looks for the /etc/syslog.conf file to allow some
functionality in the apps and will/now looks for /etc/rsyslog.conf also.

I saw will/now since I have his confirmation it's on his Todo list from a few
months back, but I haven't tested any updates from him since we spoke about it
back then.

The moral of this story is, for any software you use, ask the developers to
add rsyslog support if things they develop only supports syslog.

Regards,

Michael.

> Regards,
> sur5r
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
> 
> iD8DBQFHaDjK1YAhDic+adYRArd6AJ9fXRZU1TnW4L/eKRSfcE3x/G/PMACfbDEF
> GgU6vviKYQQofAeAJ9vEnDw=
> =6u5b
> -----END PGP SIGNATURE-----
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
------- End of Original Message -------



From mbiebl at gmail.com  Wed Dec 19 05:51:57 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 19 Dec 2007 05:51:57 +0100
Subject: [rsyslog] backward compatibility...
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712182051m2ef76c2sb401e0bd45904382@mail.gmail.com>

2007/12/17, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Hi all,
>
> I would appreciate your feedback on backward compatibility. I have
> started to create a loadable input module interface in rsyslog v3. My
> first input module is the mark message handler (because it has almost no
> code, so it was a good example of what needs to be done...). I have now
> basically have finished an initial version. The interface needs to be
> changed in the longer term, but I'll postpone that until I've done a bit
> more work on the config interface. Now I am facing a problem with
> backward compatibility:
>
> In sysklogd/rsyslog pre-v3, the -m option controls how often mark
> messages are written (with -m0 disabling this mechanism). In v3, the

One thing I was wondering:
If you intend to shift all (even core) functionality into loadable
modules, how do do you handle things like --help or available command
line options like -m
Do you want to hardcode it  or will you provide an interface, where
rsyslog will query the module about its help message and available
options.

I'm also still a bit uncertain, if moving everything resp. core
functionality to modules is a good idea (for problems you already
mentioned).
Imho having all core functionality in a single binary is simply much
more robust and fool proof. For things like the SQL db output plugin,
the module interface is great, because it avoids to pull in large
library and package dependencies and allows to install them on a as
need basis. For other functionality I still need to recognize the
benefits.

Rainer, could you roughly sketch, how you envision to break rsyslog
into loadable modules in v3. Which kind of functionality would be
loadable as module, which functionality do you plan to keep in the
rsyslogd binary. A listing of all (planned) modules + the provided
functionality and requirements would really help.

Another thing: Say you move the regexp support into a separate module.
If a regexp is then used in rsyslog.conf, will you bail out with an
error, simply print a warning (which could go unnoticed and the poor
administrator doesn't know why his regexp doesn't know) or load
modules on demand.
For the latter you'd need some kind of interface to query the *.so
files for their supported functionality. I.e. the modules would export
a list of config directives it supports and rsyslog could upon startup
query each available module and create a map.
So, e.g. the ommysql module would export its support for the :ommysql:
config directive.
Whenever rsyslog finds such a config directive it could/would load the
modules on demand.
Same could be done for the command line parameters.
The imklog module would export, that it supports the -m command line
parameter. Whenever that commandline parameter is used, rsyslog would
know which module to load.

There are only rough ideas and there is certainly still much to
consider. But what do you think about the basic idea?

Cheers,
Michael

Cheers,
Michael




-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From mbiebl at gmail.com  Wed Dec 19 09:27:13 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 19 Dec 2007 09:27:13 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <1197994886.14826.2.camel@localhost.localdomain>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
	<4760F395.4030502@ipax.at>
	<577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>
	<aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>
	<aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com>
	<aa76cdac0712161438n69382660u9d2a082914d717c9@mail.gmail.com>
	<aa76cdac0712162009q9fe3ecaq407df79296e4baff@mail.gmail.com>
	<1197994886.14826.2.camel@localhost.localdomain>
Message-ID: <aa76cdac0712190027k55824b7cn582acd6e7a1a6feb@mail.gmail.com>

2007/12/18, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> This patch now also made it to the repository. Applied to both v2
> (previously 1.20.2) and v3.

Cool, thanks.
Remains to update the documentation doc/rsconf1_includeconfig.html.
I'm usually not good a writing docs, but if you want I can provide a patch ;-)

Cheers,
Michael
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Wed Dec 19 09:33:01 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 19 Dec 2007 09:33:01 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <aa76cdac0712190027k55824b7cn582acd6e7a1a6feb@mail.gmail.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com><4760F395.4030502@ipax.at><577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com><aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com><aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com><aa76cdac0712161438n69382660u9d2a082914d717c9@mail.gmail.com><aa76cdac0712162009q9fe3ecaq407df79296e4baff@mail.gmail.com><1197994886.14826.2.camel@localhost.localdomain>
	<aa76cdac0712190027k55824b7cn582acd6e7a1a6feb@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086A8@grfint2.intern.adiscon.com>

Oops... this time I forgot the doc (I always try to have at least a
minimal description there). I just added two lines, maybe you have a
look. I am a bit under pressure because I just released 1.21.1, but now
re-created the tarball with the somewhat enhanced doc. If you are up for
a better doc, just sent me a patch ;) It'll appear in the next release.

Release notification is due in a few minutes...

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Wednesday, December 19, 2007 9:27 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Problems with IncludeConfig
> 
> 2007/12/18, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > This patch now also made it to the repository. Applied to both v2
> > (previously 1.20.2) and v3.
> 
> Cool, thanks.
> Remains to update the documentation doc/rsconf1_includeconfig.html.
> I'm usually not good a writing docs, but if you want I can provide a
> patch ;-)
> 
> Cheers,
> Michael
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Wed Dec 19 09:42:35 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 19 Dec 2007 09:42:35 +0100
Subject: [rsyslog] rsyslog 1.21.0 released
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086A9@grfint2.intern.adiscon.com>

Hi all,

as I mentioned a few minutes ago, I have just released version 1.21.0.
First question may be "why 1.21.0 and not 2.0.0"? The answer is that
varmojfekoj provided a patch that allowed me to finally enable GSS-API
support in rsyslog. While I really like this, I did not find it
appropriate to include such a major new feature in the first release of
a 2.0.0 *stable*. So I went back to the v1 releases. If I do not hear of
any serious problems, I'll re-release that version as 2.0.0.

What also brings us to the new features: GSS-API is obviously now there.
There is very limited documentation that goes with it. I'll focus on
that in the v3 releases, with docs being propagated down to v2 as
possible. If anyone would like to contribute doc, I'd obviously be very
happy ;) I currently do not intend to spent too much time with it,
because I do not like to distract myself from v3 until I managed to get
at least the most important things up and running.

V1.21.0 also contains Michael Biebl's $IncludeConfig enhancement, many
cleanups as well as the fix that hopefully addresses the segfault. So
far, I have neither heard a success or failure report. For success, it
obviously is too early, rsyslogd often needed a few days to experience
the fault. I still hope for the best...

A quick note on v3: things are going pretty well. I'll post an answer to
Michael Biebl's note on what will become modular a bit later, it needs
some time. GSS-API was the last major new feature I'll accept for v2.
Everything else will go in v3. I expect to release v2 by the end of the
week and the initial v3 maybe a week later. I will make a preview
tarball of v3 available soon so that those interested can have a look at
it (and, of course, CVS head already has it).

Download links and changelog can be found on the usual places on
www.rsyslog.com.

As always, feedback is appreciated.

Rainer Gerhards 


From mbiebl at gmail.com  Wed Dec 19 09:50:44 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 19 Dec 2007 09:50:44 +0100
Subject: [rsyslog] rsyslog 1.21.0 released
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086A9@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA3086A9@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712190050u5ec134fbp6675ab637602ae89@mail.gmail.com>

2007/12/19, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Download links and changelog can be found on the usual places on
> www.rsyslog.com.

Download link seems to be missing.

Cheers,
Michael

P.S: what about posting md5sums for release tarballs in the annoucment
email and on www.rsyslog.com?

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From mbiebl at gmail.com  Wed Dec 19 09:52:04 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 19 Dec 2007 09:52:04 +0100
Subject: [rsyslog] rsyslog 1.21.0 released
In-Reply-To: <aa76cdac0712190050u5ec134fbp6675ab637602ae89@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA3086A9@grfint2.intern.adiscon.com>
	<aa76cdac0712190050u5ec134fbp6675ab637602ae89@mail.gmail.com>
Message-ID: <aa76cdac0712190052g1ec7035bhf68270abd32bf45b@mail.gmail.com>

2007/12/19, Michael Biebl <mbiebl at gmail.com>:
> 2007/12/19, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > Download links and changelog can be found on the usual places on
> > www.rsyslog.com.
>
> Download link seems to be missing.

Ah, seems to be there now.

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Wed Dec 19 10:13:18 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 19 Dec 2007 10:13:18 +0100
Subject: [rsyslog] rsyslog 1.21.0 released
In-Reply-To: <aa76cdac0712190052g1ec7035bhf68270abd32bf45b@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA3086A9@grfint2.intern.adiscon.com><aa76cdac0712190050u5ec134fbp6675ab637602ae89@mail.gmail.com>
	<aa76cdac0712190052g1ec7035bhf68270abd32bf45b@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086AB@grfint2.intern.adiscon.com>

I guess it was a cache issue, I always start by creating the download,
then changelog entry, etc... The md5sum is part of the download
description. I thought it is OK to post it there, only. It also somewhat
saved me when I applied the last-minute doc patch and re-released. With
the md5sum in the mail, it would have created some confusion. What do
you say?

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Wednesday, December 19, 2007 9:52 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 1.21.0 released
> 
> 2007/12/19, Michael Biebl <mbiebl at gmail.com>:
> > 2007/12/19, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > > Download links and changelog can be found on the usual places on
> > > www.rsyslog.com.
> >
> > Download link seems to be missing.
> 
> Ah, seems to be there now.
> 
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From mbiebl at gmail.com  Wed Dec 19 10:21:29 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 19 Dec 2007 10:21:29 +0100
Subject: [rsyslog] rsyslog 1.21.0 released
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086AB@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA3086A9@grfint2.intern.adiscon.com>
	<aa76cdac0712190050u5ec134fbp6675ab637602ae89@mail.gmail.com>
	<aa76cdac0712190052g1ec7035bhf68270abd32bf45b@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086AB@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712190121icbf2cb7v70c672a68bba9329@mail.gmail.com>

2007/12/19, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> I guess it was a cache issue, I always start by creating the download,
> then changelog entry, etc... The md5sum is part of the download
> description. I thought it is OK to post it there, only. It also somewhat
> saved me when I applied the last-minute doc patch and re-released. With
> the md5sum in the mail, it would have created some confusion. What do
> you say?

Agreed. Posting the md5sum on the web page should be fine.

Cheers,
Michael
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Wed Dec 19 12:37:09 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 19 Dec 2007 12:37:09 +0100
Subject: [rsyslog] backward compatibility...
In-Reply-To: <aa76cdac0712182051m2ef76c2sb401e0bd45904382@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
	<aa76cdac0712182051m2ef76c2sb401e0bd45904382@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086B1@grfint2.intern.adiscon.com>

Michael and all,

I took me a while to craft a response to your excellent question. I have
done this as a blog post so that it is easier to reference it in the
future.

I suggest that everyone interested in the v3 design has a look at it,
because it describes the way I am heading. If someone doesn't like that,
it is now time to speak up - in a few weeks it will probably be
impossible to change routes.

http://rgerhards.blogspot.com/2007/12/modules-core-functionality-and-rsy
slog.html

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Wednesday, December 19, 2007 5:52 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] backward compatibility...
> 
> 2007/12/17, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > Hi all,
> >
> > I would appreciate your feedback on backward compatibility. I have
> > started to create a loadable input module interface in rsyslog v3.
My
> > first input module is the mark message handler (because it has
almost
> no
> > code, so it was a good example of what needs to be done...). I have
> now
> > basically have finished an initial version. The interface needs to
be
> > changed in the longer term, but I'll postpone that until I've done a
> bit
> > more work on the config interface. Now I am facing a problem with
> > backward compatibility:
> >
> > In sysklogd/rsyslog pre-v3, the -m option controls how often mark
> > messages are written (with -m0 disabling this mechanism). In v3, the
> 
> One thing I was wondering:
> If you intend to shift all (even core) functionality into loadable
> modules, how do do you handle things like --help or available command
> line options like -m
> Do you want to hardcode it  or will you provide an interface, where
> rsyslog will query the module about its help message and available
> options.
> 
> I'm also still a bit uncertain, if moving everything resp. core
> functionality to modules is a good idea (for problems you already
> mentioned).
> Imho having all core functionality in a single binary is simply much
> more robust and fool proof. For things like the SQL db output plugin,
> the module interface is great, because it avoids to pull in large
> library and package dependencies and allows to install them on a as
> need basis. For other functionality I still need to recognize the
> benefits.
> 
> Rainer, could you roughly sketch, how you envision to break rsyslog
> into loadable modules in v3. Which kind of functionality would be
> loadable as module, which functionality do you plan to keep in the
> rsyslogd binary. A listing of all (planned) modules + the provided
> functionality and requirements would really help.
> 
> Another thing: Say you move the regexp support into a separate module.
> If a regexp is then used in rsyslog.conf, will you bail out with an
> error, simply print a warning (which could go unnoticed and the poor
> administrator doesn't know why his regexp doesn't know) or load
> modules on demand.
> For the latter you'd need some kind of interface to query the *.so
> files for their supported functionality. I.e. the modules would export
> a list of config directives it supports and rsyslog could upon startup
> query each available module and create a map.
> So, e.g. the ommysql module would export its support for the :ommysql:
> config directive.
> Whenever rsyslog finds such a config directive it could/would load the
> modules on demand.
> Same could be done for the command line parameters.
> The imklog module would export, that it supports the -m command line
> parameter. Whenever that commandline parameter is used, rsyslog would
> know which module to load.
> 
> There are only rough ideas and there is certainly still much to
> consider. But what do you think about the basic idea?
> 
> Cheers,
> Michael
> 
> Cheers,
> Michael
> 
> 
> 
> 
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From mbiebl at gmail.com  Thu Dec 20 10:08:56 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Thu, 20 Dec 2007 10:08:56 +0100
Subject: [rsyslog] backward compatibility...
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086B1@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
	<aa76cdac0712182051m2ef76c2sb401e0bd45904382@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086B1@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712200108q7e9eb058g2e179c2fe00f65d0@mail.gmail.com>

2007/12/19, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Michael and all,
>
> I took me a while to craft a response to your excellent question. I have
> done this as a blog post so that it is easier to reference it in the
> future.
>
> I suggest that everyone interested in the v3 design has a look at it,
> because it describes the way I am heading. If someone doesn't like that,
> it is now time to speak up - in a few weeks it will probably be
> impossible to change routes.

Based on your recent blog post here are some thoughts of mine. Please
keep in mind, that being (Debian) package maintainer, so I speak from
a distributors pov.

1.) external dependencies
Having loadable modules for stuff like MySQL/PostgreSQL output support
is great (for a package maintainer)
This means I can package a basic rsyslog package with minimal
dependencies (glibc) and people can chose to install
rsyslog-(mysql,pgsql) based on their needs.
Here in this case, loadable modules have a real benefit.
The only remaining external dependencies currently in rsyslog are libz
(NETZIP support) and gssapi-krb (KERBEROS support). If those
functionality would be put into a loadable module, I'd support that.
2.) memory usage
I don't think this is a real issue, even for embedded systems. See
point 1. If we manage to put external dependencies into loadable
modules, that would be sufficient imho.
Modularising everything won't have that much of an memory benefit
imho. Given that a standard rsyslogd is about 1M RSS, this doesn't
imply the need for more modularization.
3.) Security
You mentioned, that you try to improve security through modules.
Usually, having loadable module support is considered a security risk.
One messed up $IncludeConfig directive (or manipulated through a
malicious attacker), and you load potentially hazardous modules.
Loadable modules support introduces a much bigger attack vector.

I'm not a SELinux guy. But I'd be interested if loadable modules could
cause trouble in putting rsyslog in it's own security domain. Maybe
the fedora guys can comment on this.

4.) (code) modularization
Writing modular code doesn't strictly imply that the modules have to
be loadable *.so files.
You can still write modular code, with a strict API etc. and organize
it e.g. via sub directories.
5.) Performance penalty through loadable modules.
I could be wrong on this point, but given that you have pointers on
functions, when you use modules (dlsym), there is an additional
indirection on each function call. This could have a performance
impact for core functionality. This is just speculation and should be
tested/evaluated beforehand. After all, rsyslog tries to meet
high-performance needs, too.
6.) Inconvenience
This is just a gut feeling, but having to load all sorts of moduls
first, before rsyslog does anything, could prove cumbersome. As
administrator you'd have to know, which modules to load, to get
rsyslog to do what you want. This means additional effort (reading
docs) and inconvenience.
7.) Robustness
Having a single binary can prove to be a live safer. E.g. you could
simply copy the rsyslogd binary from another machine if there is
something wrong with your local copy.
As you were talking about embedded systems, having the ability to
compile a static binary including all functionality is a definite
plus. There might be platforms out there which don't support dlopen().

I'm still missing the complete picture, too. It's still a bit too vague for me.

You were talking about input, output and filter modules. Rainer, could
cou try to complete this list somehow, maybe draw/sketch a flow
diagram, marking the modules.

input: local, kernel and network, ...
filter: regexp, ...
output: mysql, pgsql, file, forward...
authorization?


I hope, this all doesn't sound too negative. But before going all
modular, all these issues should be considered imho.

Hopefully these comments will help.

Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Thu Dec 20 10:24:49 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 20 Dec 2007 10:24:49 +0100
Subject: [rsyslog] backward compatibility...
In-Reply-To: <aa76cdac0712200108q7e9eb058g2e179c2fe00f65d0@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com><aa76cdac0712182051m2ef76c2sb401e0bd45904382@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA3086B1@grfint2.intern.adiscon.com>
	<aa76cdac0712200108q7e9eb058g2e179c2fe00f65d0@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086B9@grfint2.intern.adiscon.com>

Michael,

thanks for the good points. I'll provide a more in-depth reply later,
but noticed that I missed one important point in the blog post.

Loadable modules are also (at least partly) about platform independence.
Let me elaborate this with an example:

I have yet to fully port rsyslog to solaris. The reason I stopped is
that Solaris uses a mechanism different from Linux to handle local
logging (e.g. via logger). I didn't like to mess up even more with the
code that pulls different log sources.

With the loadable input plugins, there is a very clean solution. If you
run on Linux, just load imLocaLogLinux and if you happen to run under
Solaris load imLocalLogSolaris instead. No conditional compilation, no
complex code. Plus, anyone interested to provide an input implementation
for whatever weird system is free to do so.

So far to what I forgot to mention...

A quick reply: I think we agree that modular code is good to have. Our
main difference seems to be if it should be dynamically or statically
linked together. I am right now modularizing the inputs, have done two
so far and going to the local log socket next. That will also bring some
practical experience, which it already did. So I'll do a round of
modularization and then look at the result to fine-tune it. If it will
be loadable or not will not really be a major concern at that stage. I
just wanted to provide reasoning why I develop concurrently to this very
good discussion.

Again, a more in-depth reply a bit later.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Thursday, December 20, 2007 10:09 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] backward compatibility...
> 
> 2007/12/19, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > Michael and all,
> >
> > I took me a while to craft a response to your excellent question. I
> have
> > done this as a blog post so that it is easier to reference it in the
> > future.
> >
> > I suggest that everyone interested in the v3 design has a look at
it,
> > because it describes the way I am heading. If someone doesn't like
> that,
> > it is now time to speak up - in a few weeks it will probably be
> > impossible to change routes.
> 
> Based on your recent blog post here are some thoughts of mine. Please
> keep in mind, that being (Debian) package maintainer, so I speak from
> a distributors pov.
> 
> 1.) external dependencies
> Having loadable modules for stuff like MySQL/PostgreSQL output support
> is great (for a package maintainer)
> This means I can package a basic rsyslog package with minimal
> dependencies (glibc) and people can chose to install
> rsyslog-(mysql,pgsql) based on their needs.
> Here in this case, loadable modules have a real benefit.
> The only remaining external dependencies currently in rsyslog are libz
> (NETZIP support) and gssapi-krb (KERBEROS support). If those
> functionality would be put into a loadable module, I'd support that.
> 2.) memory usage
> I don't think this is a real issue, even for embedded systems. See
> point 1. If we manage to put external dependencies into loadable
> modules, that would be sufficient imho.
> Modularising everything won't have that much of an memory benefit
> imho. Given that a standard rsyslogd is about 1M RSS, this doesn't
> imply the need for more modularization.
> 3.) Security
> You mentioned, that you try to improve security through modules.
> Usually, having loadable module support is considered a security risk.
> One messed up $IncludeConfig directive (or manipulated through a
> malicious attacker), and you load potentially hazardous modules.
> Loadable modules support introduces a much bigger attack vector.
> 
> I'm not a SELinux guy. But I'd be interested if loadable modules could
> cause trouble in putting rsyslog in it's own security domain. Maybe
> the fedora guys can comment on this.
> 
> 4.) (code) modularization
> Writing modular code doesn't strictly imply that the modules have to
> be loadable *.so files.
> You can still write modular code, with a strict API etc. and organize
> it e.g. via sub directories.
> 5.) Performance penalty through loadable modules.
> I could be wrong on this point, but given that you have pointers on
> functions, when you use modules (dlsym), there is an additional
> indirection on each function call. This could have a performance
> impact for core functionality. This is just speculation and should be
> tested/evaluated beforehand. After all, rsyslog tries to meet
> high-performance needs, too.
> 6.) Inconvenience
> This is just a gut feeling, but having to load all sorts of moduls
> first, before rsyslog does anything, could prove cumbersome. As
> administrator you'd have to know, which modules to load, to get
> rsyslog to do what you want. This means additional effort (reading
> docs) and inconvenience.
> 7.) Robustness
> Having a single binary can prove to be a live safer. E.g. you could
> simply copy the rsyslogd binary from another machine if there is
> something wrong with your local copy.
> As you were talking about embedded systems, having the ability to
> compile a static binary including all functionality is a definite
> plus. There might be platforms out there which don't support dlopen().
> 
> I'm still missing the complete picture, too. It's still a bit too
vague
> for me.
> 
> You were talking about input, output and filter modules. Rainer, could
> cou try to complete this list somehow, maybe draw/sketch a flow
> diagram, marking the modules.
> 
> input: local, kernel and network, ...
> filter: regexp, ...
> output: mysql, pgsql, file, forward...
> authorization?
> 
> 
> I hope, this all doesn't sound too negative. But before going all
> modular, all these issues should be considered imho.
> 
> Hopefully these comments will help.
> 
> Michael
> 
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From pvrabec at redhat.com  Thu Dec 20 10:47:56 2007
From: pvrabec at redhat.com (Peter Vrabec)
Date: Thu, 20 Dec 2007 10:47:56 +0100
Subject: [rsyslog] backward compatibility...
In-Reply-To: <aa76cdac0712200108q7e9eb058g2e179c2fe00f65d0@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086B1@grfint2.intern.adiscon.com>
	<aa76cdac0712200108q7e9eb058g2e179c2fe00f65d0@mail.gmail.com>
Message-ID: <200712201047.56857.pvrabec@redhat.com>

just a quick response,

On Thursday 20 December 2007 10:08:56 am Michael Biebl wrote:
> 2007/12/19, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > Michael and all,
> >
> > I took me a while to craft a response to your excellent question. I have
> > done this as a blog post so that it is easier to reference it in the
> > future.
> >
> > I suggest that everyone interested in the v3 design has a look at it,
> > because it describes the way I am heading. If someone doesn't like that,
> > it is now time to speak up - in a few weeks it will probably be
> > impossible to change routes.
>
> Based on your recent blog post here are some thoughts of mine. Please
> keep in mind, that being (Debian) package maintainer, so I speak from
> a distributors pov.
>

> 3.) Security
> You mentioned, that you try to improve security through modules.
> Usually, having loadable module support is considered a security risk.
> One messed up $IncludeConfig directive (or manipulated through a
> malicious attacker), and you load potentially hazardous modules.
> Loadable modules support introduces a much bigger attack vector.
>
> I'm not a SELinux guy. But I'd be interested if loadable modules could
> cause trouble in putting rsyslog in it's own security domain. Maybe
> the fedora guys can comment on this.
I'm not SELinux guy either, but I don't see any problem here. Modules and 
binary will be in same domain.



From rgerhards at hq.adiscon.com  Thu Dec 20 15:56:33 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 20 Dec 2007 15:56:33 +0100
Subject: [rsyslog] Another compatibility question...
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086BF@grfint2.intern.adiscon.com>

Hi all,

I'd appreciate your feedback on this question.

Historically, syslog messages were not only received on port 514, but
were also send from that port. This is an artifact from the fact that
early syslogd used a single port both for sending and receiving. So far,
I have maintained that and done so with some pain. With v3, I would like
to get rid of this legacy (because it introduces inter-module
dependencies and also makes multiple listeners really complicated).

So I wonder if anyone over here finds this "feature" (syslog messages
are sent from port 514 by default) important. What pops up my mind are
firewall rules. Is there anything else broken if we change that?

Feedback is highly appreciated,
Rainer


From bakers at web-ster.com  Thu Dec 20 18:04:33 2007
From: bakers at web-ster.com (Scott Baker)
Date: Thu, 20 Dec 2007 09:04:33 -0800
Subject: [rsyslog] Syslog logging taking too much disk write?
Message-ID: <476AA0A1.5080300@web-ster.com>

I have a semi busy syslog server which logs about 40 or 50 lines per
second. I noticed using dstat that it's taking a TON of disk write.
In the megaBYTES per second range. This seems extremely high to me
so I wrote a script to test it.

http://www.perturb.org/code/log_lines.pl

First download dstat, gkrellm, or iostat and look at what your
normal disk write is per second. On my underused desktop it hovers
around zero or maybe 25KB/s per second.

If I run my script and tell it to log 10 lines to syslog per second

perl log_lines.pl 10

the write usage spikes up to ~160KB/s. We're only writing about 740
bytes of data per second so why is the disk write so high?

If I tell my script to instead log to a local file in /tmp at 10
lines per second

perl log_lines.pl 10 -t

The write usage spikes to ~120KB/s about once every 5 seconds (as
the buffer fills up). I'm curious why syslog writing is so much more
intense? It seems to me on a really busy syslog server that you
could seriously smoke your HDs with all the writing it's doing. Am I
missing something?

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Thu Dec 20 18:39:40 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 20 Dec 2007 18:39:40 +0100
Subject: [rsyslog] Syslog logging taking too much disk write?
In-Reply-To: <476AA0A1.5080300@web-ster.com>
References: <476AA0A1.5080300@web-ster.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086C1@grfint2.intern.adiscon.com>

Could you post your rsyslog.conf please. Without seeing it, it sounds
like you sync the disc with every write. Unfortunately, this is the
legacy default. You can turn it off with placing a dash before the
selector line:

e.g. *.* /var/log/whatever

syncs after each write, while

*.* -/var/log/whatever

uses buffering - which should be fairly safe with decent hardware and a
UPS...

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Scott Baker
> Sent: Thursday, December 20, 2007 6:05 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] Syslog logging taking too much disk write?
> 
> I have a semi busy syslog server which logs about 40 or 50 lines per
> second. I noticed using dstat that it's taking a TON of disk write.
> In the megaBYTES per second range. This seems extremely high to me
> so I wrote a script to test it.
> 
> http://www.perturb.org/code/log_lines.pl
> 
> First download dstat, gkrellm, or iostat and look at what your
> normal disk write is per second. On my underused desktop it hovers
> around zero or maybe 25KB/s per second.
> 
> If I run my script and tell it to log 10 lines to syslog per second
> 
> perl log_lines.pl 10
> 
> the write usage spikes up to ~160KB/s. We're only writing about 740
> bytes of data per second so why is the disk write so high?
> 
> If I tell my script to instead log to a local file in /tmp at 10
> lines per second
> 
> perl log_lines.pl 10 -t
> 
> The write usage spikes to ~120KB/s about once every 5 seconds (as
> the buffer fills up). I'm curious why syslog writing is so much more
> intense? It seems to me on a really busy syslog server that you
> could seriously smoke your HDs with all the writing it's doing. Am I
> missing something?
> 
> --
> Scott Baker - Canby Telcom
> RHCE - System Administrator - 503.266.8253
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From bakers at web-ster.com  Thu Dec 20 19:09:06 2007
From: bakers at web-ster.com (Scott Baker)
Date: Thu, 20 Dec 2007 10:09:06 -0800
Subject: [rsyslog] Syslog logging taking too much disk write?
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086C1@grfint2.intern.adiscon.com>
References: <476AA0A1.5080300@web-ster.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086C1@grfint2.intern.adiscon.com>
Message-ID: <476AAFC2.207@web-ster.com>

Rainer Gerhards wrote:
> Could you post your rsyslog.conf please. Without seeing it, it sounds
> like you sync the disc with every write. Unfortunately, this is the
> legacy default. You can turn it off with placing a dash before the
> selector line:
> 
> e.g. *.* /var/log/whatever
> 
> syncs after each write, while
> 
> *.* -/var/log/whatever
> 
> uses buffering - which should be fairly safe with decent hardware and a
> UPS...

I was using the stock Fedora 8 config which does indeed sync every
write to disk. I put a - in front of all the options and my disk
usage has gone way down!

What sort of buffer does rsyslog use? How often does that buffer
fill up and write to disk? I only ask because sometimes we tail
logfiles in real time to watch things like DHCP etc as they happen.

If the log is 30 seconds behind that may cause a problem.

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From bakers at web-ster.com  Thu Dec 20 19:34:06 2007
From: bakers at web-ster.com (Scott Baker)
Date: Thu, 20 Dec 2007 10:34:06 -0800
Subject: [rsyslog] Port 514
Message-ID: <476AB59E.9050706@web-ster.com>

> Hi all,
> 
> I'd appreciate your feedback on this question.
> 
> Historically, syslog messages were not only received on port 514, but
> were also send from that port. This is an artifact from the fact that
> early syslogd used a single port both for sending and receiving. So far,
> I have maintained that and done so with some pain. With v3, I would like
> to get rid of this legacy (because it introduces inter-module
> dependencies and also makes multiple listeners really complicated).
> 
> So I wonder if anyone over here finds this "feature" (syslog messages
> are sent from port 514 by default) important. What pops up my mind are
> firewall rules. Is there anything else broken if we change that?

We use syslog pretty extensively here at the ISP I work for and I
don't see how this would affect anything we do. Firewall rules are
almost always bast on destination port, not sending port. I don't
think changing this would cause any problems.

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Thu Dec 20 21:29:53 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 20 Dec 2007 21:29:53 +0100
Subject: [rsyslog] Syslog logging taking too much disk write?
In-Reply-To: <476AAFC2.207@web-ster.com>
References: <476AA0A1.5080300@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C1@grfint2.intern.adiscon.com>
	<476AAFC2.207@web-ster.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086C6@grfint2.intern.adiscon.com>

Hi Scott, 

[snip]

> What sort of buffer does rsyslog use? How often does that buffer
> fill up and write to disk? I only ask because sometimes we tail
> logfiles in real time to watch things like DHCP etc as they happen.
> 
> If the log is 30 seconds behind that may cause a problem.

I suggest you first have a look at:

http://rgerhards.blogspot.com/2007/08/why-is-rsyslog-multi-threaded-and-
is-it.html

The bottom line with the current release is that there is no specific
"output file buffer" - rsyslog writes to the OS and it buffers. Works
pretty well for rapidly incoming traffic. What we do, however, is have a
buffer between messages received and processed. This is primarily meant
for taking care of either a) message bursts or b) temporarily slow
outputs (e.g., initial database connection). If you don't like that
buffer, you can limit it via rsyslog.conf (the default is 10,000
messages max).

With rsyslog v3, there will be a more complex threading scheme and also
some more buffers. I can also envision it will support buffered writes,
enhancing the IO throughput for files (e.g. write in 64K chunks or every
n (milli)seconds, whatever comes first). I am currently in the early
stages of v3, but it will be a very exciting release. And I of course
appreciate feedback from demanding environments (hint, hint ;)).

But v3 will never hold messages longer than a) absolutely necessary or
b) configured by operator. There is a good post on v3 right on the main
page of my blog.

I hope that clarifies, if not, please let me know.

Rainer


From bakers at web-ster.com  Thu Dec 20 22:29:39 2007
From: bakers at web-ster.com (Scott Baker)
Date: Thu, 20 Dec 2007 13:29:39 -0800
Subject: [rsyslog] Separating logs by host?
Message-ID: <476ADEC3.5040506@web-ster.com>

Is it possible with rsyslog to separate logs based on IP address and
not logging facility?

I have about 10 pieces of equipment that I'd like to get syslog
fromt, but I've run out of log facilities (local1 through local7).
It'd be nice if each of those pieces of equipment could have it's
own log. I can't find anything in the documentation that covers this.

Am I missing it?

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Thu Dec 20 22:33:46 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 20 Dec 2007 22:33:46 +0100
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <476ADEC3.5040506@web-ster.com>
References: <476ADEC3.5040506@web-ster.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>

Looks like doc can need some more guides ;)

Does this help: http://www.rsyslog.com/Article60.phtml

Rainer 

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Scott Baker
> Sent: Thursday, December 20, 2007 10:30 PM
> To: rsyslog-users
> Subject: [rsyslog] Separating logs by host?
> 
> Is it possible with rsyslog to separate logs based on IP address and
> not logging facility?
> 
> I have about 10 pieces of equipment that I'd like to get syslog
> fromt, but I've run out of log facilities (local1 through local7).
> It'd be nice if each of those pieces of equipment could have it's
> own log. I can't find anything in the documentation that covers this.
> 
> Am I missing it?
> 
> -- 
> Scott Baker - Canby Telcom
> RHCE - System Administrator - 503.266.8253
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> 


From rgerhards at hq.adiscon.com  Thu Dec 20 22:45:06 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 20 Dec 2007 22:45:06 +0100
Subject: [rsyslog] backward compatibility...
In-Reply-To: <aa76cdac0712200108q7e9eb058g2e179c2fe00f65d0@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com><aa76cdac0712182051m2ef76c2sb401e0bd45904382@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA3086B1@grfint2.intern.adiscon.com>
	<aa76cdac0712200108q7e9eb058g2e179c2fe00f65d0@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086C9@grfint2.intern.adiscon.com>

Hi Michael,

as promised, a more complete response. See inline below... 

> > I took me a while to craft a response to your excellent 
> question. I have
> > done this as a blog post so that it is easier to reference it in the
> > future.
> >
> > I suggest that everyone interested in the v3 design has a 
> look at it,
> > because it describes the way I am heading. If someone 
> doesn't like that,
> > it is now time to speak up - in a few weeks it will probably be
> > impossible to change routes.
> 
> Based on your recent blog post here are some thoughts of mine. Please
> keep in mind, that being (Debian) package maintainer, so I speak from
> a distributors pov.
> 
> 1.) external dependencies
> Having loadable modules for stuff like MySQL/PostgreSQL output support
> is great (for a package maintainer)
> This means I can package a basic rsyslog package with minimal
> dependencies (glibc) and people can chose to install
> rsyslog-(mysql,pgsql) based on their needs.
> Here in this case, loadable modules have a real benefit.
> The only remaining external dependencies currently in rsyslog are libz
> (NETZIP support) and gssapi-krb (KERBEROS support). If those
> functionality would be put into a loadable module, I'd support that.
> 2.) memory usage
> I don't think this is a real issue, even for embedded systems. See
> point 1. If we manage to put external dependencies into loadable
> modules, that would be sufficient imho.
> Modularising everything won't have that much of an memory benefit
> imho. Given that a standard rsyslogd is about 1M RSS, this doesn't
> imply the need for more modularization.

I tend to agree here with you. However, I think that a clear line needs
to be drawn between "modularization" and "loadable modules". Probably I
did cross this line myself a couple of times without even noticing.

The modularization effort *is* needed, because the current monolythic
code base is hard to maintain and has "evolved" - in contrast to be
designed. While this is not bad, you notice in many instants that the
underlaying design is no longer appropriate to do a number of things. In
order to add more really useful features, the code must be modularized.
This will reduce complexity and hopefully make it less subject to
programming error (which, of course, do still happen). [That doesn't
mean I am about to rewrite everything, more on that below.]

So the bottom line is we need modularization for new features. It also
helps greatly with portability, as I outlined in my other mail.

However, we do NOT necessarily need loadble modules - but so far, I find
them desirable. Either way, it seems to be very important to
differentiate between these two concepts.

> 3.) Security
> You mentioned, that you try to improve security through modules.
> Usually, having loadable module support is considered a security risk.

Sure - but it is the the ability to have loadable modules. This risk
does not scale with the number of modules. The problem is that the
interface can be attacked. It's only safe if you do not offer that
functionality at all. At least this is my point of view.

> One messed up $IncludeConfig directive (or manipulated through a
> malicious attacker), and you load potentially hazardous modules.
> Loadable modules support introduces a much bigger attack vector.

Agreed - but not by providing more than a view, as outlined above. The
module interface tries to mitigate against some risk. For example,
output modules never see the internal structure but only the resulting
strings. Of course, so far everything is in-process so a malicious
module may access whatever it wants. I have plans, howerver, to
optionally execute modules out of process. That would be a good option
for not well known third party modules. Obviously at the expense of
quite a performance hit...

> 
> I'm not a SELinux guy. But I'd be interested if loadable modules could
> cause trouble in putting rsyslog in it's own security domain. Maybe
> the fedora guys can comment on this.

I am not qualified to talk about this, but I think along the lines that
Peter posted...

> 
> 4.) (code) modularization
> Writing modular code doesn't strictly imply that the modules have to
> be loadable *.so files.
> You can still write modular code, with a strict API etc. and organize
> it e.g. via sub directories.

Fully agree.

> 5.) Performance penalty through loadable modules.
> I could be wrong on this point, but given that you have pointers on
> functions, when you use modules (dlsym), there is an additional
> indirection on each function call. This could have a performance
> impact for core functionality. This is just speculation and should be
> tested/evaluated beforehand. After all, rsyslog tries to meet
> high-performance needs, too.

No, that doesn't happen (by design). There is a very slight overhead
during module load, when interfaces are queried and symbols are fixed
up. Some indirection, of course, results from a generic approach, but
that is inevidable if the design is modular. With today's hardware, an
indirect function call is not as much of a problem. Again, that has
nothing to do with loable modules but with non-monolythic code.

> 6.) Inconvenience
> This is just a gut feeling, but having to load all sorts of moduls
> first, before rsyslog does anything, could prove cumbersome. As
> administrator you'd have to know, which modules to load, to get
> rsyslog to do what you want. This means additional effort (reading
> docs) and inconvenience.

That's why I said the "core modules" could be present in the default.
But more later below...

> 7.) Robustness
> Having a single binary can prove to be a live safer. E.g. you could
> simply copy the rsyslogd binary from another machine if there is
> something wrong with your local copy.

Agreed, but copying a directory with a number of executables shouldn't
be that much harder ;)

> As you were talking about embedded systems, having the ability to
> compile a static binary including all functionality is a definite
> plus. There might be platforms out there which don't support dlopen().
That's an excellent point!

> 
> I'm still missing the complete picture, too. It's still a bit 
> too vague for me.

That's probably a problem with what I can convey. As I wrote above, I am
not doing a rewrite, with a new design and a new code base. My goal is
to *migrate* rsyslog (which originally based on sysklogd code) over to a
new, better architecture. I try to make as few compromises as possible,
but every now and then there are some things that can't be ignored. My
backward-compatibility questions on this list are part of the process.
So while I have an idea of where I am heading and what I want to achive,
I do not have a crystal clear view of how that will actually happen. I
am going iteratively, tackeling one feature after another and feeding
experience gained back into the process. I am currently at the input
modules and there are definitely some lessons to learn ;)

> 
> You were talking about input, output and filter modules. Rainer, could
> cou try to complete this list somehow, maybe draw/sketch a flow
> diagram, marking the modules.
> 
> input: local, kernel and network, ...
> filter: regexp, ...
> output: mysql, pgsql, file, forward...
> authorization?

Let me give it a try:

Input: immark, imrfc3195, imklog, imfile, imudp, imtcp, imgssapi?,
imtls, imuxsock, ...

Filter: many (many!)... regexp, substr, tolower, toupper, dnslookup?,
left, right, mysqldate, pgsqldate, whateverdate, md5sum, ...

Output: ommysql, ompgsql, omusrmsg, omfwd (split into omudp, omtcp,
omgssapi, ...), ...

Authorization will be provided by the inputs... But there will be some
helpers. How this will happen is not yet clear to me - that's part of
the iterative process. Initially it will be with the modules and then
possibly be migrated to somewhere else.

I'll try to get you a drawing, but that's always a pain for me ;)

> 
> I hope, this all doesn't sound too negative. 

No, much appreciated. It is extremely useful to get that kind of
feedback.

> But before going all
> modular, all these issues should be considered imho.
> 
> Hopefully these comments will help.

After going through all of this, I begin to think that libtool probably
has the answer. I don't know exactly how to, but I think it can take
loadbale modules and I can tell it to make them into static modules. If
that's the case, I can create loable modules, but the build process will
change that into static linking. If I am wrong with that, I can mangle a
bit with the main entry point names in the modules and achive the same
result with just changes to the build process (I don't know yet how to
modify autotools to do that, but I hope there will be some helping hands
;)).

How does this sound?

Rainer
PS: please keep the comments coming, especially if you don't agree! 


From bakers at web-ster.com  Thu Dec 20 22:46:33 2007
From: bakers at web-ster.com (Scott Baker)
Date: Thu, 20 Dec 2007 13:46:33 -0800
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>
References: <476ADEC3.5040506@web-ster.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>
Message-ID: <476AE2B9.30002@web-ster.com>

Rainer Gerhards wrote:
> Looks like doc can need some more guides ;)
> 
> Does this help: http://www.rsyslog.com/Article60.phtml

Excellent... doing this causes all messages to be logged in that
format. Even local messages from the box (cron, mail, authpriv, etc)
to go into that format.

Can I apply that logging statement to a group of IPs? I have a bunch
of DSL equipment that logs. It would be nice if I could break all my
equipment out like:

Routers: /var/log/routers-%$NOW%.log
DSL: /var/log/dsl-%$NOW%.log
Dial-up: /var/log/dial--%$NOW%.log

Each/Any of those groups could have one or many sources. We have
about 13 unique DSL devices that should all probably go in the same
log for simplicity. It would be REALLY cool to be able to say
something like:

192.168.1.1/24:*.*                    /var/log/dsl-%$NOW%.log
192.168.3.1/24,10.1.1.1/24:*.*        /var/log/routers-%$NOW%.log

Am I just dreaming or is that possible?

Also why is the next version of rsyslog (after reading your blog)
going to be version 3 and not 2.0?

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Thu Dec 20 22:50:11 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 20 Dec 2007 22:50:11 +0100
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <476AE2B9.30002@web-ster.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>
	<476AE2B9.30002@web-ster.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>

I need to think about the real question a bit ;) Not an immediate
solution ... but not hopeless either ;)

On v2/v3 - v2 will be the next stable release:
http://rgerhards.blogspot.com/2007/12/begun-working-on-rsyslog-v3.html 

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Scott Baker
> Sent: Thursday, December 20, 2007 10:47 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Separating logs by host?
> 
> Rainer Gerhards wrote:
> > Looks like doc can need some more guides ;)
> > 
> > Does this help: http://www.rsyslog.com/Article60.phtml
> 
> Excellent... doing this causes all messages to be logged in that
> format. Even local messages from the box (cron, mail, authpriv, etc)
> to go into that format.
> 
> Can I apply that logging statement to a group of IPs? I have a bunch
> of DSL equipment that logs. It would be nice if I could break all my
> equipment out like:
> 
> Routers: /var/log/routers-%$NOW%.log
> DSL: /var/log/dsl-%$NOW%.log
> Dial-up: /var/log/dial--%$NOW%.log
> 
> Each/Any of those groups could have one or many sources. We have
> about 13 unique DSL devices that should all probably go in the same
> log for simplicity. It would be REALLY cool to be able to say
> something like:
> 
> 192.168.1.1/24:*.*                    /var/log/dsl-%$NOW%.log
> 192.168.3.1/24,10.1.1.1/24:*.*        /var/log/routers-%$NOW%.log
> 
> Am I just dreaming or is that possible?
> 
> Also why is the next version of rsyslog (after reading your blog)
> going to be version 3 and not 2.0?
> 
> -- 
> Scott Baker - Canby Telcom
> RHCE - System Administrator - 503.266.8253
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> 


From bakers at web-ster.com  Fri Dec 21 01:23:07 2007
From: bakers at web-ster.com (Scott Baker)
Date: Thu, 20 Dec 2007 16:23:07 -0800
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>	<476AE2B9.30002@web-ster.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>
Message-ID: <476B076B.7090003@web-ster.com>

Rainer Gerhards wrote:
> I need to think about the real question a bit ;) Not an immediate
> solution ... but not hopeless either ;)
> 
> On v2/v3 - v2 will be the next stable release:
> http://rgerhards.blogspot.com/2007/12/begun-working-on-rsyslog-v3.html 

Another good example is that we have several linux mail servers. I
like to aggregate all of their logging to one rsyslog server. They
all log to mail.* on their local machines, and then forward also to
my rsyslog server.

The problem is that shows up in my mail.* on the syslog server, and
thus gets mixed in with the local mail.* from the syslog server
itself. It would be nice to be able to separate those, by IP would
be the easiest.

mail.*				/var/log/maillog
1.2.3.4/32:mail.*		/var/log/mail-server.log

Or something like that...

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From r.bhatia at ipax.at  Fri Dec 21 09:03:04 2007
From: r.bhatia at ipax.at (Raoul Bhatia [IPAX])
Date: Fri, 21 Dec 2007 09:03:04 +0100
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <476B076B.7090003@web-ster.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>	<476AE2B9.30002@web-ster.com>	<577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>
	<476B076B.7090003@web-ster.com>
Message-ID: <476B7338.8050102@ipax.at>

Scott Baker wrote:
> Rainer Gerhards wrote:
>> I need to think about the real question a bit ;) Not an immediate
>> solution ... but not hopeless either ;)
>>
>> On v2/v3 - v2 will be the next stable release:
>> http://rgerhards.blogspot.com/2007/12/begun-working-on-rsyslog-v3.html 
> 
> Another good example is that we have several linux mail servers. I
> like to aggregate all of their logging to one rsyslog server. They
> all log to mail.* on their local machines, and then forward also to
> my rsyslog server.
> 
> The problem is that shows up in my mail.* on the syslog server, and
> thus gets mixed in with the local mail.* from the syslog server
> itself. It would be nice to be able to separate those, by IP would
> be the easiest.
> 
> mail.*				/var/log/maillog
> 1.2.3.4/32:mail.*		/var/log/mail-server.log
> 
> Or something like that...

might logging to an (my|pg)sql server solve all your issues?

cheers,
raoul
-- 
____________________________________________________________________
DI (FH) Raoul Bhatia M.Sc.          email.          r.bhatia at ipax.at
Technischer Leiter

IPAX - Aloy Bhatia Hava OEG         web.          http://www.ipax.at
Barawitzkagasse 10/2/2/11           email.            office at ipax.at
1190 Wien                           tel.               +43 1 3670030
FN 277995t HG Wien                  fax.            +43 1 3670030 15
____________________________________________________________________


From rgerhards at hq.adiscon.com  Fri Dec 21 11:07:36 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 21 Dec 2007 11:07:36 +0100
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <476B076B.7090003@web-ster.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>	<476AE2B9.30002@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>
	<476B076B.7090003@web-ster.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086D0@grfint2.intern.adiscon.com>

Mmhhh... once we have expression support, that will be an easy thing to
do. Currently, I think there are a number of clumpsy work-arounds. We
are around two month away from expressions, at least if all goes well.

I think you can achive something with BDS-style filter blocks
(!progname/-host/+host), something along these lines:

-host
mail.*	/var/log/maillog
+hostname
mail.*	/var/log/mail-server.log

Of course, this is hostname specific (and that again is not the IP but
what is in the message...)

To get to the IP, I think you can get at least some way with class A,B,C
addresses, but NOT with any other masks. Here is the idea:

$template dsl, /var/log/dsl-%$NOW%.log
$template routers, /var/log/routers-%$NOW%.log
:FROMHOST, startswith, "192.168.1."   -?dsl
:FROMHOST, startswith, "192.168.3."   -?routers
:FROMHOST, startswith, "10.1.1."      -?routers

Common pitfall: be sure to include the trailing dot in the condition to
match. If the rule were

:FROMHOST, startswith, "192.168.1"   -?dsl

It would match 192.168.1.1.2, 192.168.1.1.2 but also 192.168.1.11.2!

With the current engine, there are unfortunately no logical operations
available. So you cannot check for mail facility or whatever else. A
somewhat crude work-around would be to include the facility in the file
name, e.g. by specifying it as follows:

$template dsl, /var/log/dsl-%syslogfacility-text%-%$NOW%.log

That, of course, will result in a file written for each facility, even
those that you are not interested in. A unfortunately do not have a
solution for this now.

With v3 expressions, I envision something along these lines:

If maskmatch(FROMHOST, "192.168.1.0", 24) and syslog-facility-text ==
"mail" then 
    writefile "/var/log/dslmail-%$NOW%"

But that is not possible yet. And this is only an idea, not the actual
config file format we will have at that time. In my blog, there are a
number of posts about it, but the bottom line is that it is not yet
designed:

http://rgerhards.blogspot.com/2007/08/on-rsyslog-config-file-format.html

Doc about current capabilities is here:
 
http://www.rsyslog.com/module-Static_Docs-view-f-rsyslog_conf.html.phtml

search for "Filter Conditions" on that page.

The properties you need are documented here:
 
http://www.rsyslog.com/module-Static_Docs-view-f-property_replacer.html.
phtml

Keep in mind they ARE CASE-SENSITIVE! (and don't ask me why I had that
idea... ;))

As always, feedback is appreciated. I would be most interested to learn
at which final config you arrived.

Thanks,
Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Scott Baker
> Sent: Friday, December 21, 2007 1:23 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Separating logs by host?
> 
> Rainer Gerhards wrote:
> > I need to think about the real question a bit ;) Not an immediate
> > solution ... but not hopeless either ;)
> >
> > On v2/v3 - v2 will be the next stable release:
> > http://rgerhards.blogspot.com/2007/12/begun-working-on-rsyslog-
> v3.html
> 
> Another good example is that we have several linux mail servers. I
> like to aggregate all of their logging to one rsyslog server. They
> all log to mail.* on their local machines, and then forward also to
> my rsyslog server.
> 
> The problem is that shows up in my mail.* on the syslog server, and
> thus gets mixed in with the local mail.* from the syslog server
> itself. It would be nice to be able to separate those, by IP would
> be the easiest.
> 
> mail.*				/var/log/maillog
> 1.2.3.4/32:mail.*		/var/log/mail-server.log
> 
> Or something like that...
> 
> --
> Scott Baker - Canby Telcom
> RHCE - System Administrator - 503.266.8253
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Fri Dec 21 11:39:32 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 21 Dec 2007 11:39:32 +0100
Subject: [rsyslog] Very Early Preview of rsyslog 3.0.0
Message-ID: <1198233572.2492.16.camel@localhost.localdomain>

Hi all,

I have just reached a stage where everything in rsyslog 3.0.0 is a bit
stable. I thought it would be good to release a preview before I begin
to pull some other code and place it in its own module. There are
already a number of input plugins, so I think looking at that code
provides a very good idea of where we are heading.

Please note that there is a doc/v3compatibility.html document. It tells
about know compatibility issues with pre-v3 versions and other work that
is currently in progress. At least, as far as I wrote it down ;)

The tarball is available at

http://download.rsyslog.com/rsyslog/rsyslog-3.0.0-pre.tar.gz
md5sum: e9c807decf8acbb1b2697ec907d6af08

Please note that this tarball is a snapshot. I do not plan to maintain
it, development simply continues. I suggest NOT to use that version in
production. If you run it and find a bug, please report it. Also, any
other feedback will be used during future development.

Rainer


From bakers at web-ster.com  Fri Dec 21 17:48:44 2007
From: bakers at web-ster.com (Scott Baker)
Date: Fri, 21 Dec 2007 08:48:44 -0800
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086D0@grfint2.intern.adiscon.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>	<476AE2B9.30002@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>	<476B076B.7090003@web-ster.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086D0@grfint2.intern.adiscon.com>
Message-ID: <476BEE6C.50109@web-ster.com>

Rainer Gerhards wrote:
> Mmhhh... once we have expression support, that will be an easy thing to
> do. Currently, I think there are a number of clumpsy work-arounds. We
> are around two month away from expressions, at least if all goes well.
> 
> I think you can achive something with BDS-style filter blocks
> (!progname/-host/+host), something along these lines:
> 
> -host
> mail.*	/var/log/maillog
> +hostname
> mail.*	/var/log/mail-server.log
> 
> Of course, this is hostname specific (and that again is not the IP but
> what is in the message...)
> 
> To get to the IP, I think you can get at least some way with class A,B,C
> addresses, but NOT with any other masks. Here is the idea:
> 
> $template dsl, /var/log/dsl-%$NOW%.log
> $template routers, /var/log/routers-%$NOW%.log
> :FROMHOST, startswith, "192.168.1."   -?dsl
> :FROMHOST, startswith, "192.168.3."   -?routers
> :FROMHOST, startswith, "10.1.1."      -?routers
> 
> Common pitfall: be sure to include the trailing dot in the condition to
> match. If the rule were
> 
> :FROMHOST, startswith, "192.168.1"   -?dsl
> 
> It would match 192.168.1.1.2, 192.168.1.1.2 but also 192.168.1.11.2!
> 
> With the current engine, there are unfortunately no logical operations
> available. So you cannot check for mail facility or whatever else. A
> somewhat crude work-around would be to include the facility in the file
> name, e.g. by specifying it as follows:
> 
> $template dsl, /var/log/dsl-%syslogfacility-text%-%$NOW%.log
> 
> That, of course, will result in a file written for each facility, even
> those that you are not interested in. A unfortunately do not have a
> solution for this now.
> 
> With v3 expressions, I envision something along these lines:
> 
> If maskmatch(FROMHOST, "192.168.1.0", 24) and syslog-facility-text ==
> "mail" then 
>     writefile "/var/log/dslmail-%$NOW%"
> 
> But that is not possible yet. And this is only an idea, not the actual
> config file format we will have at that time. In my blog, there are a
> number of posts about it, but the bottom line is that it is not yet
> designed:
> 
> http://rgerhards.blogspot.com/2007/08/on-rsyslog-config-file-format.html
> 
> Doc about current capabilities is here:
>  
> http://www.rsyslog.com/module-Static_Docs-view-f-rsyslog_conf.html.phtml
> 
> search for "Filter Conditions" on that page.
> 
> The properties you need are documented here:
>  
> http://www.rsyslog.com/module-Static_Docs-view-f-property_replacer.html.
> phtml
> 
> Keep in mind they ARE CASE-SENSITIVE! (and don't ask me why I had that
> idea... ;))
> 
> As always, feedback is appreciated. I would be most interested to learn
> at which final config you arrived.

Awesome! This is TOTALLY workable for what we're doing. I'm trying
to set it up and getting a weird message:

------------------------------------------------------------------

Dec 21 08:43:17 green rsyslogd:error: extra characters in config
line ignored: '/var/log/cisco-dsl-%$NOW%.log'
Dec 21 08:43:17 green rsyslogd: Could not find template 'dsl' -
action disabled
Dec 21 08:43:17 green rsyslogd:the last error occured in
/etc/rsyslog.conf, line 37

------------------------------------------------------------------

The config I used it almost exactly what you provided...

$template dsl, /var/log/cisco-dsl-%$NOW%.log
:FROMHOST, startswith, "10.3."           -?dsl

Am I missing something basic?

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Fri Dec 21 17:53:32 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 21 Dec 2007 17:53:32 +0100
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <476BEE6C.50109@web-ster.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>	<476AE2B9.30002@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>	<476B076B.7090003@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086D0@grfint2.intern.adiscon.com>
	<476BEE6C.50109@web-ster.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086D8@grfint2.intern.adiscon.com>

[sniiiip]
> The config I used it almost exactly what you provided...
> 
> $template dsl, /var/log/cisco-dsl-%$NOW%.log
> :FROMHOST, startswith, "10.3."           -?dsl
> 
> Am I missing something basic?

I forgot the quotes around the template string ;) That should work (not
tested, though):

$template dsl, "/var/log/cisco-dsl-%$NOW%.log"
:FROMHOST, startswith, "10.3."           -?dsl

Rainer


From bakers at web-ster.com  Fri Dec 21 18:55:36 2007
From: bakers at web-ster.com (Scott Baker)
Date: Fri, 21 Dec 2007 09:55:36 -0800
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086D8@grfint2.intern.adiscon.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>	<476AE2B9.30002@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>	<476B076B.7090003@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086D0@grfint2.intern.adiscon.com>	<476BEE6C.50109@web-ster.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086D8@grfint2.intern.adiscon.com>
Message-ID: <476BFE18.4000008@web-ster.com>

Rainer Gerhards wrote:
> I forgot the quotes around the template string ;) That should work (not
> tested, though):
> 
> $template dsl, "/var/log/cisco-dsl-%$NOW%.log"
> :FROMHOST, startswith, "10.3."           -?dsl

That was it... now rsyslog starts without errors.

Howeever it's not logging to the file I have set to though, instead
it's going to /var/log/messages. I've verified with tcpdump that
it's sending from the correct IP address. I've tried both startswith
and isequal and neither seem to work.

I don't know how to troubleshoot any further? Is there some debug
mode or something that will give me more information about why it's
not logging the way I think it should?

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253



From rgerhards at hq.adiscon.com  Fri Dec 21 18:58:49 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 21 Dec 2007 18:58:49 +0100
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <476BFE18.4000008@web-ster.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>	<476AE2B9.30002@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>	<476B076B.7090003@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086D0@grfint2.intern.adiscon.com>	<476BEE6C.50109@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086D8@grfint2.intern.adiscon.com>
	<476BFE18.4000008@web-ster.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086D9@grfint2.intern.adiscon.com>

Debug mode is the keyword here ;)

Add -d -n to the command line and run it interactively with stdout
redirected to a file. Maybe I've made a mistake with the directives,
maybe we have another issue. Debug output can get quite large, please do
not send it to the list (I think mailman will bounce it anyhow). You can
email me privately (rgerhards at adiscon.com) if you like.

Just be warned: its 7p right now here and I can not promise to fix a
larger problem before xmas ;) 

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Scott Baker
> Sent: Friday, December 21, 2007 6:56 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Separating logs by host?
> 
> Rainer Gerhards wrote:
> > I forgot the quotes around the template string ;) That should work
> (not
> > tested, though):
> >
> > $template dsl, "/var/log/cisco-dsl-%$NOW%.log"
> > :FROMHOST, startswith, "10.3."           -?dsl
> 
> That was it... now rsyslog starts without errors.
> 
> Howeever it's not logging to the file I have set to though, instead
> it's going to /var/log/messages. I've verified with tcpdump that
> it's sending from the correct IP address. I've tried both startswith
> and isequal and neither seem to work.
> 
> I don't know how to troubleshoot any further? Is there some debug
> mode or something that will give me more information about why it's
> not logging the way I think it should?
> 
> --
> Scott Baker - Canby Telcom
> RHCE - System Administrator - 503.266.8253
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From bakers at web-ster.com  Fri Dec 21 19:13:33 2007
From: bakers at web-ster.com (Scott Baker)
Date: Fri, 21 Dec 2007 10:13:33 -0800
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086D9@grfint2.intern.adiscon.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>	<476AE2B9.30002@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>	<476B076B.7090003@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086D0@grfint2.intern.adiscon.com>	<476BEE6C.50109@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086D8@grfint2.intern.adiscon.com>	<476BFE18.4000008@web-ster.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086D9@grfint2.intern.adiscon.com>
Message-ID: <476C024D.30200@web-ster.com>

Rainer Gerhards wrote:
> Debug mode is the keyword here ;)
> 
> Add -d -n to the command line and run it interactively with stdout
> redirected to a file. Maybe I've made a mistake with the directives,
> maybe we have another issue. Debug output can get quite large, please do
> not send it to the list (I think mailman will bounce it anyhow). You can
> email me privately (rgerhards at adiscon.com) if you like.
> 
> Just be warned: its 7p right now here and I can not promise to fix a
> larger problem before xmas ;) 

You rock! Rsyslog Debug Mode = Good stuff! I was able to get it
working just fine with the BSD style selectors. The FROMHOST part is
still giving me grief though:

Filter: check for property 'FROMHOST' (value 'extirpate') isequal
'1.2.3.4': FALSE

So my question is... for the BSD style selectors and FROM it appears
that rsyslog is using just the hostname and not the FULL reverse dns
name: extirpate vs extirpate.web-ster.com. Is that by design? I
don't think it will be a problem but I could see where you would
want to separate.

mail.domain1.com
vs
mail.domain2.com

Anyway, FROMHOST is using the name and not the IP address to match
against. Is there another property I should use? I don't see
anything in the docs... In reality I could match about the hostname
with a regexp since all my DSL is named the same...

Again IP would be ideal and easier...

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Fri Dec 21 19:20:07 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 21 Dec 2007 19:20:07 +0100
Subject: [rsyslog] Separating logs by host?
Message-ID: <002e01c843fe$24521c22$060013ac@intern.adiscon.com>

Just out of my head and from the pda while being out of office: try the -x option. Will check mail later again :)

----- Urspr?ngliche Nachricht -----
Von: "Scott Baker" <bakers at web-ster.com>
An: "rsyslog-users" <rsyslog at lists.adiscon.com>
Gesendet: 21.12.07 19:13
Betreff: Re: [rsyslog] Separating logs by host?

Rainer Gerhards wrote:
> Debug mode is the keyword here ;)
> 
> Add -d -n to the command line and run it interactively with stdout
> redirected to a file. Maybe I've made a mistake with the directives,
> maybe we have another issue. Debug output can get quite large, please do
> not send it to the list (I think mailman will bounce it anyhow). You can
> email me privately (rgerhards at adiscon.com) if you like.
> 
> Just be warned: its 7p right now here and I can not promise to fix a
> larger problem before xmas ;) 

You rock! Rsyslog Debug Mode = Good stuff! I was able to get it
working just fine with the BSD style selectors. The FROMHOST part is
still giving me grief though:

Filter: check for property 'FROMHOST' (value 'extirpate') isequal
'1.2.3.4': FALSE

So my question is... for the BSD style selectors and FROM it appears
that rsyslog is using just the hostname and not the FULL reverse dns
name: extirpate vs extirpate.web-ster.com. Is that by design? I
don't think it will be a problem but I could see where you would
want to separate.

mail.domain1.com
vs
mail.domain2.com

Anyway, FROMHOST is using the name and not the IP address to match
against. Is there another property I should use? I don't see
anything in the docs... In reality I could match about the hostname
with a regexp since all my DSL is named the same...

Again IP would be ideal and easier...

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog


From bakers at web-ster.com  Sat Dec 22 01:13:01 2007
From: bakers at web-ster.com (Scott Baker)
Date: Fri, 21 Dec 2007 16:13:01 -0800
Subject: [rsyslog] Rsyslog is awesome
Message-ID: <476C568D.9050807@web-ster.com>

Rainer:

I spent the better part of today setting up my centralized rsyslog
today. It's logging for about 20 different hosts now and working well.

I do have some questions about hostname matching DNS versus hosts
files. I think I may have found a crash bug too. I had a host that
was rsyslog was logging from without reverse DNS. As soon as I would
add the reverse DNS record rsyslog would crash. As if that source
name changing caused rsyslog to get confused.

I'm very happy to go over this in more detail with you when you have
some more time. With Christmas coming up I'm sure we'll touch base
after. Overall I'm very impressed with rsyslog. Good work!

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From mbiebl at gmail.com  Sun Dec 23 15:46:26 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Sun, 23 Dec 2007 15:46:26 +0100
Subject: [rsyslog] Very Early Preview of rsyslog 3.0.0
In-Reply-To: <1198233572.2492.16.camel@localhost.localdomain>
References: <1198233572.2492.16.camel@localhost.localdomain>
Message-ID: <aa76cdac0712230646r273b2a1aj13fec13695a3013f@mail.gmail.com>

2007/12/21, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Hi all,
>
> I have just reached a stage where everything in rsyslog 3.0.0 is a bit
> stable. I thought it would be good to release a preview before I begin
> to pull some other code and place it in its own module. There are
> already a number of input plugins, so I think looking at that code
> provides a very good idea of where we are heading.
>
> Please note that there is a doc/v3compatibility.html document. It tells
> about know compatibility issues with pre-v3 versions and other work that
> is currently in progress. At least, as far as I wrote it down ;)

I tested imuxsock and imklog (yay for getting rid of rklogd) and it
seems to work fine so far.
Nice work.

Michael
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Sun Dec 23 18:20:00 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Sun, 23 Dec 2007 18:20:00 +0100
Subject: [rsyslog] rsyslog 1.21.1 released
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086DE@grfint2.intern.adiscon.com>

Dear All,

I have just released rsyslog 1.21.1. It is a maintenance release to
1.21.0. Most importantly, it fixes a memory leak introduced in 1.21.0.
There are also some other bugfixes present. Users of 1.21.0 should
upgrade to this release.

If you run 1.21.0 and experience(d) the segfault previously, please let
me know your results with 1.21.0 before upgrading. This would be most
useful for me.

Download and Changelog can be found on the usual places of
www.rsyslog.com.

I would also like to wish a great holiday season to everyone! Thanks for
your interest in rsyslog and all the good feedback provided.

All the best,
Rainer


From bakers at web-ster.com  Mon Dec 24 17:33:11 2007
From: bakers at web-ster.com (Scott Baker)
Date: Mon, 24 Dec 2007 08:33:11 -0800
Subject: [rsyslog] rsyslog 1.21.1 released
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086DE@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA3086DE@grfint2.intern.adiscon.com>
Message-ID: <476FDF47.1060607@web-ster.com>

Rainer Gerhards wrote:
> Dear All,
> 
> I have just released rsyslog 1.21.1. It is a maintenance release to
> 1.21.0. Most importantly, it fixes a memory leak introduced in 1.21.0.
> There are also some other bugfixes present. Users of 1.21.0 should
> upgrade to this release.
> 
> If you run 1.21.0 and experience(d) the segfault previously, please let
> me know your results with 1.21.0 before upgrading. This would be most
> useful for me.
> 
> Download and Changelog can be found on the usual places of
> www.rsyslog.com.
> 
> I would also like to wish a great holiday season to everyone! Thanks for
> your interest in rsyslog and all the good feedback provided.

Rainer:

I just got to work and I checked my rsyslog server today and it's
running happily. After upgrading from 1.19.0 to 1.21.0 I didn't have
those segfaults that were happening about every 3 hours. In fact
rsyslog logged just over 1.3g of logs in 3 days.

I'll update to 1.21.1 right now and we'll see how she does.

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From bakers at web-ster.com  Mon Dec 24 17:57:02 2007
From: bakers at web-ster.com (Scott Baker)
Date: Mon, 24 Dec 2007 08:57:02 -0800
Subject: [rsyslog] BSD Style Selectors
Message-ID: <476FE4DE.1020201@web-ster.com>

It's been my experience that if I have a config like:

+server1
# The authpriv file has restricted access.
authpriv.*                          -/var/log/secure

# Log all the mail messages in one place.
mail.*                       -/var/log/maillog

# Log cron stuff
cron.*                        -/var/log/cron

-----------------------------------------------------

That +server1 setting applies to everything below it until it's
changed by another BSD style selector? In other words if server2
sent cron.* syslog in this config it would not log that in
/var/log/cron ??

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Mon Dec 24 18:35:52 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 24 Dec 2007 18:35:52 +0100
Subject: [rsyslog] BSD Style Selectors
In-Reply-To: <476FE4DE.1020201@web-ster.com>
References: <476FE4DE.1020201@web-ster.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086E1@grfint2.intern.adiscon.com>

Yes, that's right - and: merry xmas :)

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Scott Baker
> Sent: Monday, December 24, 2007 5:57 PM
> To: rsyslog-users
> Subject: [rsyslog] BSD Style Selectors
> 
> It's been my experience that if I have a config like:
> 
> +server1
> # The authpriv file has restricted access.
> authpriv.*                          -/var/log/secure
> 
> # Log all the mail messages in one place.
> mail.*                       -/var/log/maillog
> 
> # Log cron stuff
> cron.*                        -/var/log/cron
> 
> -----------------------------------------------------
> 
> That +server1 setting applies to everything below it until it's
> changed by another BSD style selector? In other words if server2
> sent cron.* syslog in this config it would not log that in
> /var/log/cron ??
> 
> --
> Scott Baker - Canby Telcom
> RHCE - System Administrator - 503.266.8253
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From bakers at web-ster.com  Mon Dec 24 19:10:35 2007
From: bakers at web-ster.com (Scott Baker)
Date: Mon, 24 Dec 2007 10:10:35 -0800
Subject: [rsyslog] Hostname matching with DNS
Message-ID: <476FF61B.3020006@web-ster.com>

I have a couple host on private IPs 10.x.x.x and thus they have no
DNS entries. So rather that log the IP in syslog I setup host
entries for them.

If I do something like

:FROMHOST, isequal, "foobar"                -?dialup

it doesn't match the /etc/hosts entry I have for foobar. If I setup
a catchall entry that goes to a test log I see the line

Dec 24 10:06:23 foobar [This is the message]

So it's logging the hostname like I would expect it to (rsyslog is
aware of the host entry) but I can't match against it? Unfortunately
my server is SUPER busy now and I can't put the server in debug mode
to check what's coming across. Is there another way I could check this?

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Mon Dec 24 19:59:18 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 24 Dec 2007 19:59:18 +0100
Subject: [rsyslog] Hostname matching with DNS
Message-ID: <002f01c8465f$209fa2ec$060013ac@intern.adiscon.com>

Really quick: check the HOSTNAME (or so ;)) property. 

----- Urspr?ngliche Nachricht -----
Von: "Scott Baker" <bakers at web-ster.com>
An: "rsyslog-users" <rsyslog at lists.adiscon.com>
Gesendet: 24.12.07 19:10
Betreff: [rsyslog] Hostname matching with DNS

I have a couple host on private IPs 10.x.x.x and thus they have no
DNS entries. So rather that log the IP in syslog I setup host
entries for them.

If I do something like

:FROMHOST, isequal, "foobar"                -?dialup

it doesn't match the /etc/hosts entry I have for foobar. If I setup
a catchall entry that goes to a test log I see the line

Dec 24 10:06:23 foobar [This is the message]

So it's logging the hostname like I would expect it to (rsyslog is
aware of the host entry) but I can't match against it? Unfortunately
my server is SUPER busy now and I can't put the server in debug mode
to check what's coming across. Is there another way I could check this?

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Mon Dec 24 20:59:02 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 24 Dec 2007 20:59:02 +0100
Subject: [rsyslog] Hostname matching with DNS
In-Reply-To: <002f01c8465f$209fa2ec$060013ac@intern.adiscon.com>
References: <002f01c8465f$209fa2ec$060013ac@intern.adiscon.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086E2@grfint2.intern.adiscon.com>

Scott,

So now a bit more in-depth: the HOSTNAME is taken form the syslog message, while FROMHOST is the last hope. There is only a difference in relay scenarios - or, like here, based on DNS resolution. This is why you see different values. The point is to match against the same one that is used in the catchall rule.

However, I think the most appropriate thing to do is add a FROMHOST-IP property, which always has the IP address of the sender, no matter if the -x option is given or not.

Would that help?

Rainer 

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of 
> Rainer Gerhards
> Sent: Monday, December 24, 2007 7:59 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Hostname matching with DNS
> 
> Really quick: check the HOSTNAME (or so ;)) property. 
> 
> ----- Urspr?ngliche Nachricht -----
> Von: "Scott Baker" <bakers at web-ster.com>
> An: "rsyslog-users" <rsyslog at lists.adiscon.com>
> Gesendet: 24.12.07 19:10
> Betreff: [rsyslog] Hostname matching with DNS
> 
> I have a couple host on private IPs 10.x.x.x and thus they have no
> DNS entries. So rather that log the IP in syslog I setup host
> entries for them.
> 
> If I do something like
> 
> :FROMHOST, isequal, "foobar"                -?dialup
> 
> it doesn't match the /etc/hosts entry I have for foobar. If I setup
> a catchall entry that goes to a test log I see the line
> 
> Dec 24 10:06:23 foobar [This is the message]
> 
> So it's logging the hostname like I would expect it to (rsyslog is
> aware of the host entry) but I can't match against it? Unfortunately
> my server is SUPER busy now and I can't put the server in debug mode
> to check what's coming across. Is there another way I could 
> check this?
> 
> -- 
> Scott Baker - Canby Telcom
> RHCE - System Administrator - 503.266.8253
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> 


From rgerhards at hq.adiscon.com  Mon Dec 24 21:01:21 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 24 Dec 2007 21:01:21 +0100
Subject: [rsyslog] Hostname matching with DNS
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086E2@grfint2.intern.adiscon.com>
References: <002f01c8465f$209fa2ec$060013ac@intern.adiscon.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086E2@grfint2.intern.adiscon.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086E3@grfint2.intern.adiscon.com>

> while FROMHOST is the last hope 


Lol... No, it is not my last *hope*, even though it may be for the time
being ;) That should have been "last hop". Nice typo ;)

Rainer


From bakers at web-ster.com  Mon Dec 24 21:06:48 2007
From: bakers at web-ster.com (Scott Baker)
Date: Mon, 24 Dec 2007 12:06:48 -0800
Subject: [rsyslog] Hostname matching with DNS
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086E2@grfint2.intern.adiscon.com>
References: <002f01c8465f$209fa2ec$060013ac@intern.adiscon.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086E2@grfint2.intern.adiscon.com>
Message-ID: <47701158.9080506@web-ster.com>

Rainer Gerhards wrote:
> Scott,
> 
> So now a bit more in-depth: the HOSTNAME is taken form the syslog message, while FROMHOST is the last hope. There is only a difference in relay scenarios - or, like here, based on DNS resolution. This is why you see different values. The point is to match against the same one that is used in the catchall rule.
> 
> However, I think the most appropriate thing to do is add a FROMHOST-IP property, which always has the IP address of the sender, no matter if the -x option is given or not.
> 
> Would that help?

Ya I think a FROMHOST-IP property would be great. It'd make setting
up my rules easier. Tell me more about hosts resolution vs DNS
resolution?

I have some FROMHOST rules that work just fine, but as soon as I add
a hosts entry it stops logging. So somewhere it's using the host
entry but I can't figure out how/where or what precedence that has
over DNS.

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Mon Dec 24 21:23:58 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 24 Dec 2007 21:23:58 +0100
Subject: [rsyslog] Hostname matching with DNS
Message-ID: <003001c8466a$f111b332$060013ac@intern.adiscon.com>

Rsyslog doesn't do anything specific with the hosts file - it just uses the system resolver library and everything else is handled in there...

I'll have a look at hostname-ip as soon as i can.

rainer

----- Urspr?ngliche Nachricht -----
Von: "Scott Baker" <bakers at web-ster.com>
An: "rsyslog-users" <rsyslog at lists.adiscon.com>
Gesendet: 24.12.07 21:06
Betreff: Re: [rsyslog] Hostname matching with DNS

Rainer Gerhards wrote:
> Scott,
> 
> So now a bit more in-depth: the HOSTNAME is taken form the syslog message, while FROMHOST is the last hope. There is only a difference in relay scenarios - or, like here, based on DNS resolution. This is why you see different values. The point is to match against the same one that is used in the catchall rule.
> 
> However, I think the most appropriate thing to do is add a FROMHOST-IP property, which always has the IP address of the sender, no matter if the -x option is given or not.
> 
> Would that help?

Ya I think a FROMHOST-IP property would be great. It'd make setting
up my rules easier. Tell me more about hosts resolution vs DNS
resolution?

I have some FROMHOST rules that work just fine, but as soon as I add
a hosts entry it stops logging. So somewhere it's using the host
entry but I can't figure out how/where or what precedence that has
over DNS.

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog


From bakers at web-ster.com  Mon Dec 24 22:53:26 2007
From: bakers at web-ster.com (Scott Baker)
Date: Mon, 24 Dec 2007 13:53:26 -0800
Subject: [rsyslog] Log Permissions?
Message-ID: <47702A56.7030805@web-ster.com>

Is there a way to set the owner/group or read permissions of log
files created by rsyslog? I can't find much in the man pages.

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Mon Dec 24 23:06:52 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 24 Dec 2007 23:06:52 +0100
Subject: [rsyslog] Log Permissions?
Message-ID: <003101c84679$50a4fc86$060013ac@intern.adiscon.com>

Look at www.rsyslog.com/doc Not the man pages. It is too much for man (but the above url is in it ;)). Check the config directives, there are a number of settings for that.

rainer

----- Urspr?ngliche Nachricht -----
Von: "Scott Baker" <bakers at web-ster.com>
An: "rsyslog-users" <rsyslog at lists.adiscon.com>
Gesendet: 24.12.07 22:53
Betreff: [rsyslog] Log Permissions?

Is there a way to set the owner/group or read permissions of log
files created by rsyslog? I can't find much in the man pages.

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog


From janfrode at tanso.net  Tue Dec 25 00:34:02 2007
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Tue, 25 Dec 2007 00:34:02 +0100
Subject: [rsyslog] rsyslog 1.21.1 released
References: <577465F99B41C842AAFBE9ED71E70ABA3086DE@grfint2.intern.adiscon.com>
Message-ID: <slrnfn0gfa.j64.janfrode@lc4eb8045376502.ibm.com>

On 2007-12-23, Rainer Gerhards <rgerhards at hq.adiscon.com> wrote:
>
> If you run 1.21.0 and experience(d) the segfault previously, please let
> me know your results with 1.21.0 before upgrading. This would be most
> useful for me.

I upgraded to 1.21.0 december 19., and it's been up since (4h19m cputime, 
134h process uptime). No segfaults yet. It hasn't been eating any noticeable
amount of memory either, so I'll probably hold off on the upgrade a bit.



  -jf



From rgerhards at hq.adiscon.com  Tue Dec 25 10:21:07 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 25 Dec 2007 10:21:07 +0100
Subject: [rsyslog] rsyslog 1.21.1 released
Message-ID: <003201c846d7$84c7c4a0$060013ac@intern.adiscon.com>

Ah, thats good news! And, yes, if you don't see the mem leak it is probably better to hild off the update, so that we get some more information on the stability. The memleak occurs only when the internal message queue is full, which normally shouldn't happen. So there is a good chance you never see it...

Rainer

----- Urspr?ngliche Nachricht -----
Von: "Jan-Frode Myklebust" <janfrode at tanso.net>
An: "rsyslog at lists.adiscon.com" <rsyslog at lists.adiscon.com>
Gesendet: 25.12.07 00:34
Betreff: Re: [rsyslog] rsyslog 1.21.1 released

On 2007-12-23, Rainer Gerhards <rgerhards at hq.adiscon.com> wrote:
>
> If you run 1.21.0 and experience(d) the segfault previously, please let
> me know your results with 1.21.0 before upgrading. This would be most
> useful for me.

I upgraded to 1.21.0 december 19., and it's been up since (4h19m cputime, 
134h process uptime). No segfaults yet. It hasn't been eating any noticeable
amount of memory either, so I'll probably hold off on the upgrade a bit.



  -jf

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog


From janfrode at tanso.net  Wed Dec 26 17:42:53 2007
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Wed, 26 Dec 2007 17:42:53 +0100
Subject: [rsyslog] Log Permissions?
References: <47702A56.7030805@web-ster.com>
Message-ID: <slrnfn514d.7qs.janfrode@lc4eb8045376502.ibm.com>

On 2007-12-24, Scott Baker <bakers at web-ster.com> wrote:
> Is there a way to set the owner/group or read permissions of log
> files created by rsyslog? I can't find much in the man pages.
>

I use these in my /etc/rsyslog.conf to give group "logread" read
permissions of all files:

$DirGroup logread
$DirCreateMode 0750
$FileGroup logread
$umask 0027



  -jf



From rgerhards at hq.adiscon.com  Thu Dec 27 14:19:18 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 27 Dec 2007 14:19:18 +0100
Subject: [rsyslog] New rsyslog 3.0.0 preview - with new features
Message-ID: <1198761558.10564.3.camel@localhost.localdomain>

Hi all,

I have just uploaded a new preview of 3.0.0. It now has all input
modules (though they still have a lot of internal dependencies).
Finally, I also managed to get in some new features: the UDP listener
can now be bound to specific local interfaces and there can be multiple
UDP listeners active at the same time (on different interface and/or
different ports). The same feature for TCP (including GSS) is planned,
but I may need to do some more stage work before.

The preview is available at

http://download.rsyslog.com/rsyslog/rsyslog-3.0.0-pre.tar.gz
md5sum: 979763fe645ad6ee6fddf11737af7ebd 

This is a snapshot. Development continues. Feedback is appreciated.

Rainer


From rgerhards at hq.adiscon.com  Fri Dec 28 09:26:49 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 28 Dec 2007 09:26:49 +0100
Subject: [rsyslog] omfwd/gss-api in rsyslog v1/v2
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086F0@grfint2.intern.adiscon.com>

Hi all,

as you know, I am working on the modular structure of rsyslog v3. I am
currently revisiting gss-api support. I notice that with the current
omfwd, it will be extremely hard to separate gss-api support into its
own module. Doing so will break backward compatibility to the
configuration file.

GSS-API has been out only for a few days, and mostly over the holiday
period. So it is much less of a concern if we introduce now some changes
that will case rsyslog.conf format modifications. Much less trouble than
when we release v2, a release expected to be in wide use for at least
half a year, if not much longer. V2 released with the current syntax
would require me to do some tricks in v3 to keep compatibility. Quite
complex.

So I decided to create a omgssapi for v3 and extract the gss-api code
from omfwd. It looks like this can be done without too much code
duplication. There will be some duplicate code, but it will shrink as v3
continues to be developed. Once I have a good working version, which I
expect very soon, I will backport that to the v1/2 source tree. I'll
then do a new v1 release with a slightly incompatible gss-api config
file syntax. After this is out for a few days, I hope I can than finally
push out that version as v2.

I hope this is a good decision. I think it will save us major future
trouble at the expense of a relatively slight disturbance in the late v1
timeline. I guess most user's won't even notice there is a change.

As always, Feedback is appreciated.

Rainer 


From bakers at web-ster.com  Fri Dec 28 17:15:07 2007
From: bakers at web-ster.com (Scott Baker)
Date: Fri, 28 Dec 2007 08:15:07 -0800
Subject: [rsyslog] omfwd/gss-api in rsyslog v1/v2
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086F0@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA3086F0@grfint2.intern.adiscon.com>
Message-ID: <4775210B.804@web-ster.com>

Rainer Gerhards wrote:
> Hi all,
> 
> as you know, I am working on the modular structure of rsyslog v3. I am
> currently revisiting gss-api support. I notice that with the current
> omfwd, it will be extremely hard to separate gss-api support into its
> own module. Doing so will break backward compatibility to the
> configuration file.
> 
> GSS-API has been out only for a few days, and mostly over the holiday
> period. So it is much less of a concern if we introduce now some changes
> that will case rsyslog.conf format modifications. Much less trouble than
> when we release v2, a release expected to be in wide use for at least
> half a year, if not much longer. V2 released with the current syntax
> would require me to do some tricks in v3 to keep compatibility. Quite
> complex.
> 
> So I decided to create a omgssapi for v3 and extract the gss-api code
> from omfwd. It looks like this can be done without too much code
> duplication. There will be some duplicate code, but it will shrink as v3
> continues to be developed. Once I have a good working version, which I
> expect very soon, I will backport that to the v1/2 source tree. I'll
> then do a new v1 release with a slightly incompatible gss-api config
> file syntax. After this is out for a few days, I hope I can than finally
> push out that version as v2.
> 
> I hope this is a good decision. I think it will save us major future
> trouble at the expense of a relatively slight disturbance in the late v1
> timeline. I guess most user's won't even notice there is a change.
> 
> As always, Feedback is appreciated.

I think this is the best solution moving forward. I don't think
configuration changes moving to a new major version are a huge
headache, especially since GSS-API is so new.

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Fri Dec 28 22:39:59 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 28 Dec 2007 22:39:59 +0100
Subject: [rsyslog] Rsyslog 1.21.2 released
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086F7@grfint2.intern.adiscon.com>

Hi all,

just a quick note:

I have released rsyslog 1.21.2. It is a maintenance release. The most
important new feature is that GSS-API support for sending TCP messages
has been moved to a separate output plugin. This is an important
forward-compatibility option for rsyslog v3. This resulted in some
configuration file changes. This release also fixes a number of minor
bugs. This is a recommended update for all users.

Please also keep my message from earlier today in mind.

Rainer


From mbiebl at gmail.com  Sat Dec  1 00:14:35 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Sat, 1 Dec 2007 00:14:35 +0100
Subject: [rsyslog] gss-api support for rsyslog
In-Reply-To: <473B2961.9010705@redhat.com>
References: <473B2961.9010705@redhat.com>
Message-ID: <aa76cdac0711301514x711721a7i2767e8a99e175f79@mail.gmail.com>

2007/11/14, theinric at redhat.com <theinric at redhat.com>:
> Hi,
>
> here is a patch that enables rsyslog to use gss-api for authentication
> and encryption.

Hi,

ist seems not all of the patch has made it into CVS. The part in
Makefile.am is missing, so the compilation fails. Please apply the
(inlined) patch ;-)

Index: Makefile.am
===================================================================
RCS file: /cvsroot/rsyslog/rsyslog/Makefile.am,v
retrieving revision 1.36
diff -u -3 -p -r1.36 Makefile.am
--- Makefile.am 30 Nov 2007 13:43:12 -0000      1.36
+++ Makefile.am 30 Nov 2007 23:10:27 -0000
@@ -63,7 +63,7 @@ rsyslogd_SOURCES = \
        gss-misc.h

 rsyslogd_CPPFLAGS = -D_PATH_MODDIR=\"$(pkglibdir)/\" $(pthreads_cflags)
-rsyslogd_LDADD = $(zlib_libs) $(pthreads_libs) -ldl
+rsyslogd_LDADD = $(zlib_libs) $(pthreads_libs) $(gss_libs) -ldl
 rsyslogd_LDFLAGS = -export-dynamic

 man_MANS = rfc3195d.8 rklogd.8 rsyslogd.8 rsyslog.conf.5



Cheers,
Michael
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From sur5r at sur5r.net  Sat Dec  1 22:36:44 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Sat, 1 Dec 2007 22:36:44 +0100
Subject: [rsyslog] [patch] output module for PostgreSQL
Message-ID: <20071201223644.036dccd6@marathon.haufe>

Hi all!

As already mentioned in the forum: I've created an output module for
PostgreSQL.

The attached patch does the following:

* add plugins/ompgsql/*
* extend configure.ac and Makefile.am
* msg.h
** struct msg_t: add pszTIMESTAMP_PgSQL, add pszRcvdAt_PgSQL, fix typo
* msg.c
** getTimeGenerated(): add handling of tplFmtPgSQLDate
** getTimeReported(): add handling of tplFmtPgSQLDate
* syslogd.h:
** add definition of formatTimestampToPgSQL()
* syslogd.c:
** add StdPgSQLFmt
** mainThread(): include StdPgSQLFmt
* template.h:
** add tplFmtPgSQLDate to tplFormatTypes
* template.c:
** doOptions(): date-pgsql -> tplFmtPgSQLDate
** tplPrintList(): add tplFmtPgSQLDate to switch()

Please note that the patch has not yet been tested thoroughly, so until
that has been done, it is to be considered alpha quality.

If anyone wants to try it: any feedback is appreciated.

Regards,
sur5r

P.S.: I hope the patch survives mailman... ;)
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: rsyslog_ompgsql.diff
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20071201/3543bb53/attachment-0002.txt>

From sur5r at sur5r.net  Sat Dec  1 23:01:26 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Sat, 1 Dec 2007 23:01:26 +0100
Subject: [rsyslog] [patch] output module for PostgreSQL
References: <20071201223644.036dccd6@marathon.haufe>
Message-ID: <20071201230126.3537f120@marathon.haufe>

Jakob Haufe <sur5r at sur5r.net> wrote on Sat, 1 Dec 2007 22:36:44 +0100
in 20071201223644.036dccd6 at marathon.haufe:

> The attached patch does the following:
> 
> * add plugins/ompgsql/*

Erm, yes. Maybe i should have noticed, that cvs doesn't recurse into
new subdirectories... so here are the files for the plugins/ompgsql
directory.

From sur5r at sur5r.net  Sat Dec  1 23:10:11 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Sat, 1 Dec 2007 23:10:11 +0100
Subject: [rsyslog] [patch] output module for PostgreSQL
References: <20071201223644.036dccd6@marathon.haufe>
	<20071201230126.3537f120@marathon.haufe>
Message-ID: <20071201231011.5a27a361@marathon.haufe>

Jakob Haufe <sur5r at sur5r.net> wrote on Sat, 1 Dec 2007 23:01:26 +0100
in 20071201230126.3537f120 at marathon.haufe:

> Jakob Haufe <sur5r at sur5r.net> wrote on Sat, 1 Dec 2007 22:36:44 +0100
> in 20071201223644.036dccd6 at marathon.haufe:
> 
> > The attached patch does the following:
> > 
> > * add plugins/ompgsql/*
> 
> Erm, yes. Maybe i should have noticed, that cvs doesn't recurse into
> new subdirectories... so here are the files for the plugins/ompgsql
> directory.
Ok, mailman doesn't like any mimetypes but text/plain... next try *grml*
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: Makefile.am
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20071201/0f6fc89b/attachment-0004.asc>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: createDB.sql
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20071201/0f6fc89b/attachment-0004.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: ompgsql.c
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20071201/0f6fc89b/attachment-0005.asc>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: ompgsql.h
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20071201/0f6fc89b/attachment-0005.txt>

From mbiebl at gmail.com  Sun Dec  2 23:10:07 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Sun, 2 Dec 2007 23:10:07 +0100
Subject: [rsyslog] [patch] output module for PostgreSQL
In-Reply-To: <20071201231011.5a27a361@marathon.haufe>
References: <20071201223644.036dccd6@marathon.haufe>
	<20071201230126.3537f120@marathon.haufe>
	<20071201231011.5a27a361@marathon.haufe>
Message-ID: <aa76cdac0712021410l2a218cftb5a6883c38bacd32@mail.gmail.com>

2007/12/1, Jakob Haufe <sur5r at sur5r.net>:
> Jakob Haufe <sur5r at sur5r.net> wrote on Sat, 1 Dec 2007 23:01:26 +0100
> in 20071201230126.3537f120 at marathon.haufe:
>
> > Jakob Haufe <sur5r at sur5r.net> wrote on Sat, 1 Dec 2007 22:36:44 +0100
> > in 20071201223644.036dccd6 at marathon.haufe:
> >
> > > The attached patch does the following:
> > >
> > > * add plugins/ompgsql/*
> >
> > Erm, yes. Maybe i should have noticed, that cvs doesn't recurse into
> > new subdirectories... so here are the files for the plugins/ompgsql
> > directory.
> Ok, mailman doesn't like any mimetypes but text/plain... next try *grml*

Hi,

just a very minor nitpick in Makefile.am:
pkglibdir is by default defined to be $(libdir)/@PACKAGE@/ [1], so
there is no need to redefine it. Simply delete the first line.
plugins/ommysql/Makefile.am should be fixed accordingly.

It also seems that the configure.ac bits are missing.

Cheers,
Michael

[1] http://www.gnu.org/software/automake/manual/html_node/Uniform.html#Uniform
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Mon Dec  3 08:10:44 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 3 Dec 2007 08:10:44 +0100
Subject: [rsyslog] [patch] output module for PostgreSQL
In-Reply-To: <aa76cdac0712021410l2a218cftb5a6883c38bacd32@mail.gmail.com>
References: <20071201223644.036dccd6@marathon.haufe><20071201230126.3537f120@marathon.haufe><20071201231011.5a27a361@marathon.haufe>
	<aa76cdac0712021410l2a218cftb5a6883c38bacd32@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA30856B@grfint2.intern.adiscon.com>

Hi,

thanks to all for the great contributions! Let me just quickly outline
my plan, so that nobody is disappointed ;) I'll publish a new 1.19.12
release today. However, that will just hold some cleanup and fixes. When
I have done that, I'll begin to merge the patches into the
already-modified 1.20.0. I am not yet sure how much issues I will see
because there are a number of code changes in that release. I am also
not sure yet in which order I apply the patches, because one might be
better than the other. I'll announce when I have merged and will
probably post a test tarball here on the list before officially
releasing. My target for release is mid to end of week.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Sunday, December 02, 2007 11:10 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] [patch] output module for PostgreSQL
> 
> 2007/12/1, Jakob Haufe <sur5r at sur5r.net>:
> > Jakob Haufe <sur5r at sur5r.net> wrote on Sat, 1 Dec 2007 23:01:26
+0100
> > in 20071201230126.3537f120 at marathon.haufe:
> >
> > > Jakob Haufe <sur5r at sur5r.net> wrote on Sat, 1 Dec 2007 22:36:44
> +0100
> > > in 20071201223644.036dccd6 at marathon.haufe:
> > >
> > > > The attached patch does the following:
> > > >
> > > > * add plugins/ompgsql/*
> > >
> > > Erm, yes. Maybe i should have noticed, that cvs doesn't recurse
> into
> > > new subdirectories... so here are the files for the
plugins/ompgsql
> > > directory.
> > Ok, mailman doesn't like any mimetypes but text/plain... next try
> *grml*
> 
> Hi,
> 
> just a very minor nitpick in Makefile.am:
> pkglibdir is by default defined to be $(libdir)/@PACKAGE@/ [1], so
> there is no need to redefine it. Simply delete the first line.
> plugins/ommysql/Makefile.am should be fixed accordingly.
> 
> It also seems that the configure.ac bits are missing.
> 
> Cheers,
> Michael
> 
> [1]
>
http://www.gnu.org/software/automake/manual/html_node/Uniform.html#Unif
> orm
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Mon Dec  3 10:01:24 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 3 Dec 2007 10:01:24 +0100
Subject: [rsyslog] rsyslog 1.19.12 released
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308570@grfint2.intern.adiscon.com>

As promised this morning, I have just released 1.19.12. It is a cleanup
release, no need to install it if you do not experience any issues.
Users of 1.19.11 using mysql should update, as the new -pthread switch
was not applied to ommysql in 1.19.11.

Download:

   http://www.rsyslog.com/Downloads-req-getit-lid-64.phtml

As always, feedback is appreciated.

Best regards,
Rainer Gerhards

Rsyslog homepage: http://www.rsyslog.com
My blog: http://rgerhards.blogspot.com



From sur5r at sur5r.net  Mon Dec  3 10:37:48 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Mon, 3 Dec 2007 10:37:48 +0100
Subject: [rsyslog] [patch] output module for PostgreSQL
References: <20071201223644.036dccd6@marathon.haufe>
	<20071201230126.3537f120@marathon.haufe>
	<20071201231011.5a27a361@marathon.haufe>
	<aa76cdac0712021410l2a218cftb5a6883c38bacd32@mail.gmail.com>
Message-ID: <20071203103748.781bfba2@mp-atlantis3.informatik.uni-mannheim.de>

On Sun, 2 Dec 2007 23:10:07 +0100
"Michael Biebl" <mbiebl at gmail.com> wrote:

> just a very minor nitpick in Makefile.am:
> pkglibdir is by default defined to be $(libdir)/@PACKAGE@/ [1], so
> there is no need to redefine it. Simply delete the first line.
> plugins/ommysql/Makefile.am should be fixed accordingly.

Thanks for pointing this out. Unfortunately, I have no in-depth
knowledge of autotools, so I'm thankful for any suggestions.

> It also seems that the configure.ac bits are missing.

They are there, though hard to find. 

MSGID 20071201223644.036dccd6 at marathon.haufe
contains the patch modifying existing files, namely:
msg.[ch], syslogd.[ch], template.[ch], configure.ac, Makefile.am

MSGID 20071201231011.5a27a361 at marathon.haufe
contains the files for plugins/ompgsql/*

Regards,
Jakob



From sur5r at sur5r.net  Mon Dec  3 11:09:12 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Mon, 3 Dec 2007 11:09:12 +0100
Subject: [rsyslog] [patch] output module for PostgreSQL
References: <20071201223644.036dccd6@marathon.haufe>
	<20071201230126.3537f120@marathon.haufe>
	<20071201231011.5a27a361@marathon.haufe>
	<aa76cdac0712021410l2a218cftb5a6883c38bacd32@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA30856B@grfint2.intern.adiscon.com>
Message-ID: <20071203110912.50657ea6@mp-atlantis3.informatik.uni-mannheim.de>

On Mon, 3 Dec 2007 08:10:44 +0100
"Rainer Gerhards" <rgerhards at hq.adiscon.com> wrote:

> thanks to all for the great contributions! Let me just quickly outline
> my plan, so that nobody is disappointed ;) I'll publish a new 1.19.12
> release today. However, that will just hold some cleanup and fixes.
> When I have done that, I'll begin to merge the patches into the
> already-modified 1.20.0. I am not yet sure how much issues I will see
> because there are a number of code changes in that release. I am also
> not sure yet in which order I apply the patches, because one might be
> better than the other. I'll announce when I have merged and will
> probably post a test tarball here on the list before officially
> releasing. My target for release is mid to end of week.

That's perfectly ok for me. But I definitely would appreciate any
feedback on ompgsql before officially releasing it. I do not expect any
problems with it but you never know...

So if anybody is going to test it, please give feedback.

Regards,
sur5r



From rgerhards at hq.adiscon.com  Mon Dec  3 11:11:25 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 3 Dec 2007 11:11:25 +0100
Subject: [rsyslog] [patch] output module for PostgreSQL
In-Reply-To: <20071203110912.50657ea6@mp-atlantis3.informatik.uni-mannheim.de>
References: <20071201223644.036dccd6@marathon.haufe><20071201230126.3537f120@marathon.haufe><20071201231011.5a27a361@marathon.haufe><aa76cdac0712021410l2a218cftb5a6883c38bacd32@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA30856B@grfint2.intern.adiscon.com>
	<20071203110912.50657ea6@mp-atlantis3.informatik.uni-mannheim.de>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308574@grfint2.intern.adiscon.com>

Just a quick update. I am right now integrating it and will review code
during this and once I am through. As I suspected, I need to do at least
some manual merging, as the source tree already changed and in some
places considerably (talking about 1.20.0!). So it would be good if you
could-re-test it once I can release an early test tarball.

I keep you updated.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Jakob Haufe
> Sent: Monday, December 03, 2007 11:09 AM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] [patch] output module for PostgreSQL
> 
> On Mon, 3 Dec 2007 08:10:44 +0100
> "Rainer Gerhards" <rgerhards at hq.adiscon.com> wrote:
> 
> > thanks to all for the great contributions! Let me just quickly
> outline
> > my plan, so that nobody is disappointed ;) I'll publish a new
1.19.12
> > release today. However, that will just hold some cleanup and fixes.
> > When I have done that, I'll begin to merge the patches into the
> > already-modified 1.20.0. I am not yet sure how much issues I will
see
> > because there are a number of code changes in that release. I am
also
> > not sure yet in which order I apply the patches, because one might
be
> > better than the other. I'll announce when I have merged and will
> > probably post a test tarball here on the list before officially
> > releasing. My target for release is mid to end of week.
> 
> That's perfectly ok for me. But I definitely would appreciate any
> feedback on ompgsql before officially releasing it. I do not expect
any
> problems with it but you never know...
> 
> So if anybody is going to test it, please give feedback.
> 
> Regards,
> sur5r
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From mbiebl at gmail.com  Tue Dec  4 03:58:21 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Tue, 4 Dec 2007 03:58:21 +0100
Subject: [rsyslog] [PATCH] Fix latest "NetBSD fix"
Message-ID: <aa76cdac0712031858w52c6c63bq529b7496cb8a5cf3@mail.gmail.com>

Hi,

the latest NetBSD related commits (configure.ac/Makefile.am) break the
compilation on Linux. Variable assignments mustn't have whitespace,
otherwise have_libdl and as a result HAVE_LIBDL is not defined.

To fix the issue though, I'd simply recommend to remove the hard coded
-ldl from Makefile.am instead of the proposed solution.
The AC_CHECK_LIB macro [2] already appends -ldl to LIBS, depending on
the outcome of the check [2]. The global LIBS variable (similar to
CFLAGS e.g.) is used for linking all binaries.

Linked (see, I learned my lesson :-p) is the patch [1] to address this.

The only downside of AC_CHECK_LIB is, that it assigns -ldl (and -lrt)
to the global LIBS variable. This means that all binaries are linked
with -ldl -lrt, as mentioned above, even if they don't need them (i.e.
ommysql/ompsql plugin,  rklogd rfc3195d).

I can prepare a patch to address this issue correctly, if wanted.

Cheers,
Michael

[1] http://debs.michaelbiebl.de/rsyslog/fix_dl_compilation.patch
[2] http://www.gnu.org/software/automake/manual/autoconf/Libraries.html

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Tue Dec  4 11:15:57 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 04 Dec 2007 11:15:57 +0100
Subject: [rsyslog] [PATCH] Fix latest "NetBSD fix"
In-Reply-To: <aa76cdac0712031858w52c6c63bq529b7496cb8a5cf3@mail.gmail.com>
References: <aa76cdac0712031858w52c6c63bq529b7496cb8a5cf3@mail.gmail.com>
Message-ID: <1196763357.12493.6.camel@localhost.localdomain>

Hi Michael,

many thanks for the information. Actually, you were quicker than me ;) I
worked with Bartosz on the issue yesterday, but needed to leave before I
could finalize it.

I would appreciate if you could provide the "big patch" you wrote about.
That will ensure that autotools is used in the best possible way :)

Rainer

On Tue, 2007-12-04 at 03:58 +0100, Michael Biebl wrote:
> Hi,
> 
> the latest NetBSD related commits (configure.ac/Makefile.am) break the
> compilation on Linux. Variable assignments mustn't have whitespace,
> otherwise have_libdl and as a result HAVE_LIBDL is not defined.
> 
> To fix the issue though, I'd simply recommend to remove the hard coded
> -ldl from Makefile.am instead of the proposed solution.
> The AC_CHECK_LIB macro [2] already appends -ldl to LIBS, depending on
> the outcome of the check [2]. The global LIBS variable (similar to
> CFLAGS e.g.) is used for linking all binaries.
> 
> Linked (see, I learned my lesson :-p) is the patch [1] to address this.
> 
> The only downside of AC_CHECK_LIB is, that it assigns -ldl (and -lrt)
> to the global LIBS variable. This means that all binaries are linked
> with -ldl -lrt, as mentioned above, even if they don't need them (i.e.
> ommysql/ompsql plugin,  rklogd rfc3195d).
> 
> I can prepare a patch to address this issue correctly, if wanted.
> 
> Cheers,
> Michael
> 
> [1] http://debs.michaelbiebl.de/rsyslog/fix_dl_compilation.patch
> [2] http://www.gnu.org/software/automake/manual/autoconf/Libraries.html
> 


From rgerhards at hq.adiscon.com  Tue Dec  4 11:23:26 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 4 Dec 2007 11:23:26 +0100
Subject: [rsyslog] rsyslog 1.20.0 preview available
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308596@grfint2.intern.adiscon.com>

Hi all,

there is a preview of rsyslog 1.20.0 available at

http://download.rsyslog.com/rsyslog/rsyslog-1.20.0.tar.gz

It obviously does not contains Michael Biebl's upcoming autotools patch,
but should be fairly well. I would be most interested in feedback from
the field, especially on the new postgres module by sur5r (thanks
again!). I plan to release end of week if now show stoppers show up.

Please note that I have disabled GSS-API in a rough way: you can simply
not select it in configure. This is because GSS-API support does need
some more work or it could become an interop problem in the future. So
we decided to push it back until that is solved.

Feedback is highly appreciated. If I updated the tarball, I'll announce
that. Oh, yes, doc is not finished yet, so do not wonder...

Rainer


From sur5r at sur5r.net  Tue Dec  4 14:00:14 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Tue, 4 Dec 2007 14:00:14 +0100
Subject: [rsyslog] rsyslog 1.20.0 preview available
References: <577465F99B41C842AAFBE9ED71E70ABA308596@grfint2.intern.adiscon.com>
Message-ID: <20071204140014.7c2187b2@mp-atlantis3.informatik.uni-mannheim.de>

On Tue, 4 Dec 2007 11:23:26 +0100
"Rainer Gerhards" <rgerhards at hq.adiscon.com> wrote:

> Feedback is highly appreciated. If I updated the tarball, I'll
> announce that. Oh, yes, doc is not finished yet, so do not wonder...

Just a small thing I already mentioned on the forum but unfortunately
got overlooked:

ommysql, line 236: p+=sizeof(":ommysql:");

This adds increments the pointer one byte too far (trailing \0) and
thus eats the first byte of the hostname part.

It's a bit unfortunate that strlen() doesn't count the \0 while
sizeof() does...

Regards,
sur5r



From rgerhards at hq.adiscon.com  Tue Dec  4 14:10:06 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 4 Dec 2007 14:10:06 +0100
Subject: [rsyslog] rsyslog 1.20.0 preview available
In-Reply-To: <20071204140014.7c2187b2@mp-atlantis3.informatik.uni-mannheim.de>
References: <577465F99B41C842AAFBE9ED71E70ABA308596@grfint2.intern.adiscon.com>
	<20071204140014.7c2187b2@mp-atlantis3.informatik.uni-mannheim.de>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA30859F@grfint2.intern.adiscon.com>

Indeed, thanks for pointing it out. Its fixed now, in cvs and I also
re-released the tarball (just for completeness).

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Jakob Haufe
> Sent: Tuesday, December 04, 2007 2:00 PM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] rsyslog 1.20.0 preview available
> 
> On Tue, 4 Dec 2007 11:23:26 +0100
> "Rainer Gerhards" <rgerhards at hq.adiscon.com> wrote:
> 
> > Feedback is highly appreciated. If I updated the tarball, I'll
> > announce that. Oh, yes, doc is not finished yet, so do not wonder...
> 
> Just a small thing I already mentioned on the forum but unfortunately
> got overlooked:
> 
> ommysql, line 236: p+=sizeof(":ommysql:");
> 
> This adds increments the pointer one byte too far (trailing \0) and
> thus eats the first byte of the hostname part.
> 
> It's a bit unfortunate that strlen() doesn't count the \0 while
> sizeof() does...
> 
> Regards,
> sur5r
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From stephen.carville at gmail.com  Fri Dec  7 22:09:18 2007
From: stephen.carville at gmail.com (Stephen Carville)
Date: Fri, 7 Dec 2007 13:09:18 -0800
Subject: [rsyslog] Last Message repeated N times
Message-ID: <2428c0380712071309w31f4fc20qb933dfef47f445ac@mail.gmail.com>

I am using rsyslog 1.19.10 on a central server to collect messages and
store them in a mysql database.  One thing I've noticed is that the
"last message repeated NNN times" entries are alway entered with
FromHost="last", SysLogTag="message", and Message="repeated NNN
times".  Can this be fixed?

I still use the regular sysklogd package on the remote machines to
forward the messages.

-- 
Stephen Carville


From rgerhards at hq.adiscon.com  Tue Dec 11 11:06:00 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 11 Dec 2007 11:06:00 +0100
Subject: [rsyslog] Asking for help...
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA30862F@grfint2.intern.adiscon.com>

Hi folks,

I have a design question with upcoming rsyslog changes and could not
even find an appropriate place to ask. I have blogged about it, so the
information is available to everyone:

http://rgerhards.blogspot.com/2007/12/design-problem.html

If you happen to know a good place to ask (or even know a solution), I
would really appreciate to hear about that!

Thanks
Rainer


From rgerhards at hq.adiscon.com  Tue Dec 11 12:01:07 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 11 Dec 2007 12:01:07 +0100
Subject: [rsyslog] Last Message repeated N times
In-Reply-To: <2428c0380712071309w31f4fc20qb933dfef47f445ac@mail.gmail.com>
References: <2428c0380712071309w31f4fc20qb933dfef47f445ac@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308637@grfint2.intern.adiscon.com>

Hi Stephen,

I was totally swamped the past days, sorry for the missing response.
I'll check what causes this. As a side-note, I think that the message
reduction code does not work well with the database - as you can no
longer query the individual messages. Is this your intent?

Rainer 

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Stephen Carville
> Sent: Friday, December 07, 2007 10:09 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] Last Message repeated N times
> 
> I am using rsyslog 1.19.10 on a central server to collect messages and
> store them in a mysql database.  One thing I've noticed is that the
> "last message repeated NNN times" entries are alway entered with
> FromHost="last", SysLogTag="message", and Message="repeated NNN
> times".  Can this be fixed?
> 
> I still use the regular sysklogd package on the remote machines to
> forward the messages.
> 
> --
> Stephen Carville
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Tue Dec 11 12:26:32 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 11 Dec 2007 12:26:32 +0100
Subject: [rsyslog] Last Message repeated N times
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308637@grfint2.intern.adiscon.com>
References: <2428c0380712071309w31f4fc20qb933dfef47f445ac@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA308637@grfint2.intern.adiscon.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA30863D@grfint2.intern.adiscon.com>

I think I misunderstood your problem... The "last message repeated n
times" stems back from the reporting sysklogd's - right? Mhhh... that's
a problem. I need to do some hardcoded processing in the parser to cover
part of it. The problem is that sysklogd emits a totally non-standard
message in that case. Even worse, I think it does not even contain a
clue at the missing information. I could probably use the sender's IP
address as source in  that case, but the tag seems to be impossible to
obtain.

Can you post me a copy of the %rawmsg% property, so that I can have a
look at what exactly you are seing?

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Tuesday, December 11, 2007 12:01 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Last Message repeated N times
> 
> Hi Stephen,
> 
> I was totally swamped the past days, sorry for the missing response.
> I'll check what causes this. As a side-note, I think that the message
> reduction code does not work well with the database - as you can no
> longer query the individual messages. Is this your intent?
> 
> Rainer
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Stephen Carville
> > Sent: Friday, December 07, 2007 10:09 PM
> > To: rsyslog at lists.adiscon.com
> > Subject: [rsyslog] Last Message repeated N times
> >
> > I am using rsyslog 1.19.10 on a central server to collect messages
> and
> > store them in a mysql database.  One thing I've noticed is that the
> > "last message repeated NNN times" entries are alway entered with
> > FromHost="last", SysLogTag="message", and Message="repeated NNN
> > times".  Can this be fixed?
> >
> > I still use the regular sysklogd package on the remote machines to
> > forward the messages.
> >
> > --
> > Stephen Carville
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From sur5r at sur5r.net  Tue Dec 11 14:48:15 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Tue, 11 Dec 2007 14:48:15 +0100
Subject: [rsyslog] Asking for help...
References: <577465F99B41C842AAFBE9ED71E70ABA30862F@grfint2.intern.adiscon.com>
Message-ID: <20071211144815.6a2ef123@marathon.haufe>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

"Rainer Gerhards" <rgerhards at hq.adiscon.com> wrote on Tue, 11 Dec 2007
11:06:00 +0100:

> http://rgerhards.blogspot.com/2007/12/design-problem.html

Maybe I'm a bit short-sighted here, but isn't this exactly the situation
for non-blocking sockets? Or will that raise other problems?

Regards,
sur5r
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHXpUj1YAhDic+adYRAnBZAKCWNXoWIKHhnhm/Pe9CMi1MT5MOWQCeJknc
AvFjtLOkeHTNz/mdpz3Gy2k=
=YZgt
-----END PGP SIGNATURE-----

From rgerhards at hq.adiscon.com  Tue Dec 11 14:55:21 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 11 Dec 2007 14:55:21 +0100
Subject: [rsyslog] Asking for help...
In-Reply-To: <20071211144815.6a2ef123@marathon.haufe>
References: <577465F99B41C842AAFBE9ED71E70ABA30862F@grfint2.intern.adiscon.com>
	<20071211144815.6a2ef123@marathon.haufe>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308646@grfint2.intern.adiscon.com>

Thanks for the feedback - is it OK if we discuss this in the forum? I
think not everyone here on the list is interested in that discussion...

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Jakob Haufe
> Sent: Tuesday, December 11, 2007 2:48 PM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] Asking for help...
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> "Rainer Gerhards" <rgerhards at hq.adiscon.com> wrote on Tue, 11 Dec 2007
> 11:06:00 +0100:
> 
> > http://rgerhards.blogspot.com/2007/12/design-problem.html
> 
> Maybe I'm a bit short-sighted here, but isn't this exactly the
> situation
> for non-blocking sockets? Or will that raise other problems?
> 
> Regards,
> sur5r
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
> 
> iD8DBQFHXpUj1YAhDic+adYRAnBZAKCWNXoWIKHhnhm/Pe9CMi1MT5MOWQCeJknc
> AvFjtLOkeHTNz/mdpz3Gy2k=
> =YZgt
> -----END PGP SIGNATURE-----
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Wed Dec 12 17:06:48 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 12 Dec 2007 17:06:48 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>

Hi all,

I have just released rsyslog 1.20.1. It primarily contains bug fixes: a
memory leak in regard to PostgreSQL date formatting and a potential race
condition within repeated message processing have been fixed. Also,
there are new runtime options to harden ACL processing when DNS is not
ready while rsyslogd start up. There are also some minor changes and
clean-ups. Version 1.20.1 is recommended to anyone using previous
releases. Users of PostgreSQL are urged to update to this version.

Download:
   http://www.rsyslog.com/Downloads-req-getit-lid-66.phtml

Changelog:
   http://www.rsyslog.com/Article147.phtml

You may want to watch my blog for upcoming developments:
http://rgerhards.blogspot.com. There was recently a lot of activity and
more is expected.

As always, feedback is appreciated.
Rainer Gerhards


From stephen.carville at gmail.com  Wed Dec 12 18:38:15 2007
From: stephen.carville at gmail.com (Stephen Carville)
Date: Wed, 12 Dec 2007 09:38:15 -0800
Subject: [rsyslog] Last Message repeated N times
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308637@grfint2.intern.adiscon.com>
References: <2428c0380712071309w31f4fc20qb933dfef47f445ac@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA308637@grfint2.intern.adiscon.com>
Message-ID: <2428c0380712120938x72f55b9na4aea606374f0255@mail.gmail.com>

On Dec 11, 2007 3:01 AM, Rainer Gerhards <rgerhards at hq.adiscon.com> wrote:
> Hi Stephen,
>
> I was totally swamped the past days, sorry for the missing response.
> I'll check what causes this. As a side-note, I think that the message
> reduction code does not work well with the database - as you can no
> longer query the individual messages. Is this your intent?

My original intent was to have a centralized copy of all the changes
that cfengine makes.  That works extremely well and saves me a
buttload of time over having to log by hand in a shared Excel
spreadsheet every time I update a hosts file or tweak httpd.conf or
tomcat's server.xml.  That and logging the messages from auditd were
the main goals.  I added /var/log/messages and /var/log/secure mainly
because I could and because it gives the Linux side of the shop
greater credibility with the SOX (expletive deleted).  Now that
messages and secure are in the database I'll have to run a log
analyzer periodically looking for indications of "stuff" that's not
right but that comes after about six other projects.

This isn't a show stopper and I could probably just drop the "last
messages repeated.." entries without any harm.  Tho I;m not sure how
jsut yet.  Out of 2.4M rows about 385K are of this type.

> Rainer
>
>
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Stephen Carville
> > Sent: Friday, December 07, 2007 10:09 PM
> > To: rsyslog at lists.adiscon.com
> > Subject: [rsyslog] Last Message repeated N times
> >
> > I am using rsyslog 1.19.10 on a central server to collect messages and
> > store them in a mysql database.  One thing I've noticed is that the
> > "last message repeated NNN times" entries are alway entered with
> > FromHost="last", SysLogTag="message", and Message="repeated NNN
> > times".  Can this be fixed?
> >
> > I still use the regular sysklogd package on the remote machines to
> > forward the messages.
> >
> > --
> > Stephen Carville
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
>

-- 
Stephen Carville


From stephen.carville at gmail.com  Wed Dec 12 19:02:09 2007
From: stephen.carville at gmail.com (Stephen Carville)
Date: Wed, 12 Dec 2007 10:02:09 -0800
Subject: [rsyslog] Last Message repeated N times
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA30863D@grfint2.intern.adiscon.com>
References: <2428c0380712071309w31f4fc20qb933dfef47f445ac@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA308637@grfint2.intern.adiscon.com>
	<577465F99B41C842AAFBE9ED71E70ABA30863D@grfint2.intern.adiscon.com>
Message-ID: <2428c0380712121002l70f37260p869ab35b8acb7068@mail.gmail.com>

On Dec 11, 2007 3:26 AM, Rainer Gerhards <rgerhards at hq.adiscon.com> wrote:
> I think I misunderstood your problem... The "last message repeated n
> times" stems back from the reporting sysklogd's - right?

Would ugrading to rsyslogd on the client machines help?  I'm getting
ready for an upgrade cycle where everything except a couple of boxes
running old hardware not suppoert by the 2.6 kernel will be either
Oracle ULN 4.X or Redhat/CentOS 5.X  I didn't see a spec file in the
tar.gz but I can write one myself if there is no good one available.

> Mhhh... that's
> a problem. I need to do some hardcoded processing in the parser to cover
> part of it. The problem is that sysklogd emits a totally non-standard
> message in that case. Even worse, I think it does not even contain a
> clue at the missing information. I could probably use the sender's IP
> address as source in  that case, but the tag seems to be impossible to
> obtain.
>
> Can you post me a copy of the %rawmsg% property, so that I can have a
> look at what exactly you are seing?
>
> Rainer
>
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>
> > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> > Sent: Tuesday, December 11, 2007 12:01 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] Last Message repeated N times
> >
> > Hi Stephen,
> >
> > I was totally swamped the past days, sorry for the missing response.
> > I'll check what causes this. As a side-note, I think that the message
> > reduction code does not work well with the database - as you can no
> > longer query the individual messages. Is this your intent?
> >
> > Rainer
> >
> > > -----Original Message-----
> > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > > bounces at lists.adiscon.com] On Behalf Of Stephen Carville
> > > Sent: Friday, December 07, 2007 10:09 PM
> > > To: rsyslog at lists.adiscon.com
> > > Subject: [rsyslog] Last Message repeated N times
> > >
> > > I am using rsyslog 1.19.10 on a central server to collect messages
> > and
> > > store them in a mysql database.  One thing I've noticed is that the
> > > "last message repeated NNN times" entries are alway entered with
> > > FromHost="last", SysLogTag="message", and Message="repeated NNN
> > > times".  Can this be fixed?
> > >
> > > I still use the regular sysklogd package on the remote machines to
> > > forward the messages.
> > >
> > > --
> > > Stephen Carville
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
>



-- 
Stephen Carville


From mbiebl at gmail.com  Wed Dec 12 19:11:55 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 12 Dec 2007 19:11:55 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com>

2007/12/12, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Hi all,
>
> I have just released rsyslog 1.20.1. It primarily contains bug fixes: a

1.20.1 unfortunately fails to compile (unless you use --enable-debug):
gcc -g -O2 -W -Wall -Wformat-security -Wshadow -Wcast-align
-Wpointer-arith -Wmissing-format-attribute -g -o rsyslogd
rsyslogd-syslogd.o rsyslogd-pidfile.o rsyslogd-template.o
rsyslogd-outchannel.o rsyslogd-stringbuf.o rsyslogd-srUtils.o
rsyslogd-parse.o rsyslogd-sync.o rsyslogd-net.o rsyslogd-msg.o
rsyslogd-expr.o rsyslogd-omshell.o rsyslogd-omusrmsg.o
rsyslogd-omfwd.o rsyslogd-tcpsyslog.o rsyslogd-omfile.o
rsyslogd-omdiscard.o rsyslogd-modules.o rsyslogd-objomsr.o
rsyslogd-cfsysline.o rsyslogd-linkedlist.o rsyslogd-iminternal.o
rsyslogd-action.o rsyslogd-gss-misc.o -Wl,--export-dynamic  -lz
-lpthread -ldl -lrt
rsyslogd-syslogd.o: In function `domarkActions':
/home/michael/cvs/rsyslog/build/../syslogd.c:3431: undefined reference
to `lockObj'
/home/michael/cvs/rsyslog/build/../syslogd.c:3439: undefined reference
to `unlockObj'
rsyslogd-syslogd.o: In function `callAction':
/home/michael/cvs/rsyslog/build/../syslogd.c:2442: undefined reference
to `lockObj'
/home/michael/cvs/rsyslog/build/../syslogd.c:2508: undefined reference
to `unlockObj'
/home/michael/cvs/rsyslog/build/../syslogd.c:2508: undefined reference
to `unlockObj'
collect2: ld returned 1 exit status
make[2]: *** [rsyslogd] Fehler 1
make[2]: Leaving directory `/home/michael/cvs/rsyslog/build'
make[1]: *** [all-recursive] Fehler 1
make[1]: Leaving directory `/home/michael/cvs/rsyslog/build'
make: *** [all] Fehler 2


sync.h declares lockObj/unlockObj, but sync.c only has these symbols
if compiled without NDEBUG.

Cheers,
Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From infofarmer at FreeBSD.org  Wed Dec 12 19:16:36 2007
From: infofarmer at FreeBSD.org (Andrew Pantyukhin)
Date: Wed, 12 Dec 2007 21:16:36 +0300
Subject: [rsyslog] rsyslog 1.20.1 released
In-Reply-To: <aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>
	<aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com>
Message-ID: <20071212181635.GQ51627@amilo.cenkes.org>

On Wed, Dec 12, 2007 at 07:11:55PM +0100, Michael Biebl wrote:
> 2007/12/12, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > Hi all,
> >
> > I have just released rsyslog 1.20.1. It primarily contains bug fixes: a
> 
> sync.h declares lockObj/unlockObj, but sync.c only has these symbols
> if compiled without NDEBUG.

Try including config.h in sync.h


From mbiebl at gmail.com  Wed Dec 12 19:30:34 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 12 Dec 2007 19:30:34 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
In-Reply-To: <20071212181635.GQ51627@amilo.cenkes.org>
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>
	<aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com>
	<20071212181635.GQ51627@amilo.cenkes.org>
Message-ID: <aa76cdac0712121030y49b70fc0r6379af2b361a5f06@mail.gmail.com>

2007/12/12, Andrew Pantyukhin <infofarmer at freebsd.org>:
> On Wed, Dec 12, 2007 at 07:11:55PM +0100, Michael Biebl wrote:
> > 2007/12/12, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > > Hi all,
> > >
> > > I have just released rsyslog 1.20.1. It primarily contains bug fixes: a
> >
> > sync.h declares lockObj/unlockObj, but sync.c only has these symbols
> > if compiled without NDEBUG.
>
> Try including config.h in sync.h
>

No difference.

The other header files (using NDEBUG) don't include config.h either btw.
The rely on the c file, which includes them, to include config.h.

Cheers,
Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From mbiebl at gmail.com  Wed Dec 12 20:18:38 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 12 Dec 2007 20:18:38 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
In-Reply-To: <aa76cdac0712121030y49b70fc0r6379af2b361a5f06@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>
	<aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com>
	<20071212181635.GQ51627@amilo.cenkes.org>
	<aa76cdac0712121030y49b70fc0r6379af2b361a5f06@mail.gmail.com>
Message-ID: <aa76cdac0712121118kea18ee7md111ed3558bb7a87@mail.gmail.com>

2007/12/12, Michael Biebl <mbiebl at gmail.com>:
> 2007/12/12, Andrew Pantyukhin <infofarmer at freebsd.org>:
> > On Wed, Dec 12, 2007 at 07:11:55PM +0100, Michael Biebl wrote:
> > > 2007/12/12, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > > > Hi all,
> > > >
> > > > I have just released rsyslog 1.20.1. It primarily contains bug fixes: a
> > >
> > > sync.h declares lockObj/unlockObj, but sync.c only has these symbols
> > > if compiled without NDEBUG.
> >
> > Try including config.h in sync.h
> >
>

Found the culprit. Patch is attached.
Imho calling the define NDEBUG makes the code unnecessarily hard to
read (#ifndef NDEBUG ..). I had to stare at the code for a while until
I noticed that it was negated one time to often.

Cheers,
Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

From sur5r at sur5r.net  Wed Dec 12 20:35:38 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Wed, 12 Dec 2007 20:35:38 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>
	<aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com>
	<20071212181635.GQ51627@amilo.cenkes.org>
	<aa76cdac0712121030y49b70fc0r6379af2b361a5f06@mail.gmail.com>
	<aa76cdac0712121118kea18ee7md111ed3558bb7a87@mail.gmail.com>
Message-ID: <20071212203538.6dbb00d1@marathon.haufe>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

"Michael Biebl" <mbiebl at gmail.com> wrote on Wed, 12 Dec 2007 20:18:38
+0100:

> Found the culprit. Patch is attached.

Mailman ate it again. Try attaching it with mimetype text/plain, seems
to work then.

Regards,
sur5r
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHYDgQ1YAhDic+adYRAtxQAKCspmrE+HfH12YVBzJcIBQ/GqNKXACfdSl3
NkY9aWuB+ftzvx2t4MtKPHw=
=BODR
-----END PGP SIGNATURE-----

From rgerhards at hq.adiscon.com  Wed Dec 12 21:04:09 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 12 Dec 2007 21:04:09 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
In-Reply-To: <20071212203538.6dbb00d1@marathon.haufe>
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com><aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com><20071212181635.GQ51627@amilo.cenkes.org><aa76cdac0712121030y49b70fc0r6379af2b361a5f06@mail.gmail.com><aa76cdac0712121118kea18ee7md111ed3558bb7a87@mail.gmail.com>
	<20071212203538.6dbb00d1@marathon.haufe>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308665@grfint2.intern.adiscon.com>

Hold for a moment, the actual patch is a bit different . more soon.

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Jakob Haufe
> Sent: Wednesday, December 12, 2007 8:36 PM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] rsyslog 1.20.1 released
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> "Michael Biebl" <mbiebl at gmail.com> wrote on Wed, 12 Dec 2007 20:18:38
> +0100:
> 
> > Found the culprit. Patch is attached.
> 
> Mailman ate it again. Try attaching it with mimetype text/plain, seems
> to work then.
> 
> Regards,
> sur5r
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
> 
> iD8DBQFHYDgQ1YAhDic+adYRAtxQAKCspmrE+HfH12YVBzJcIBQ/GqNKXACfdSl3
> NkY9aWuB+ftzvx2t4MtKPHw=
> =BODR
> -----END PGP SIGNATURE-----
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Wed Dec 12 21:07:26 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 12 Dec 2007 21:07:26 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308665@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com><aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com><20071212181635.GQ51627@amilo.cenkes.org><aa76cdac0712121030y49b70fc0r6379af2b361a5f06@mail.gmail.com><aa76cdac0712121118kea18ee7md111ed3558bb7a87@mail.gmail.com><20071212203538.6dbb00d1@marathon.haufe>
	<577465F99B41C842AAFBE9ED71E70ABA308665@grfint2.intern.adiscon.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308666@grfint2.intern.adiscon.com>

Re-released, info follows soon...

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Wednesday, December 12, 2007 9:04 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 1.20.1 released
> 
> Hold for a moment, the actual patch is a bit different . more soon.
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Jakob Haufe
> > Sent: Wednesday, December 12, 2007 8:36 PM
> > To: rsyslog at lists.adiscon.com
> > Subject: Re: [rsyslog] rsyslog 1.20.1 released
> >
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > "Michael Biebl" <mbiebl at gmail.com> wrote on Wed, 12 Dec 2007
20:18:38
> > +0100:
> >
> > > Found the culprit. Patch is attached.
> >
> > Mailman ate it again. Try attaching it with mimetype text/plain,
> seems
> > to work then.
> >
> > Regards,
> > sur5r
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v1.4.6 (GNU/Linux)
> >
> > iD8DBQFHYDgQ1YAhDic+adYRAtxQAKCspmrE+HfH12YVBzJcIBQ/GqNKXACfdSl3
> > NkY9aWuB+ftzvx2t4MtKPHw=
> > =BODR
> > -----END PGP SIGNATURE-----
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From mbiebl at gmail.com  Wed Dec 12 21:31:23 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 12 Dec 2007 21:31:23 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308665@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>
	<aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com>
	<20071212181635.GQ51627@amilo.cenkes.org>
	<aa76cdac0712121030y49b70fc0r6379af2b361a5f06@mail.gmail.com>
	<aa76cdac0712121118kea18ee7md111ed3558bb7a87@mail.gmail.com>
	<20071212203538.6dbb00d1@marathon.haufe>
	<577465F99B41C842AAFBE9ED71E70ABA308665@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712121231i62631494p4ea39e41136adac3@mail.gmail.com>

2007/12/12, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Hold for a moment, the actual patch is a bit different . more soon.

Yeah, got it the wrong way around. Didn't read the comment in sync.h
carefully enough.
Thanks for you sharp eye!

As I said, imho it's easier to read and grasp if the define is called DEBUG.

Cheers,
Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Wed Dec 12 21:40:25 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 12 Dec 2007 21:40:25 +0100
Subject: [rsyslog] rsyslog 1.20.1 released
In-Reply-To: <aa76cdac0712121130x4c96b883t4ee91e3362fb83d6@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308661@grfint2.intern.adiscon.com>
	<aa76cdac0712121011v566469bbx381513b6e4dc6943@mail.gmail.com>
	<20071212181635.GQ51627@amilo.cenkes.org>
	<aa76cdac0712121030y49b70fc0r6379af2b361a5f06@mail.gmail.com>
	<aa76cdac0712121118kea18ee7md111ed3558bb7a87@mail.gmail.com>
	<aa76cdac0712121130x4c96b883t4ee91e3362fb83d6@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308667@grfint2.intern.adiscon.com>

Hi all,

Man, when did I really screw up the last time? And I thought I had run
all test...

Anyhow, thanks all for helping to solve this issue.

Michael got it right, it is the NDEBUG. But is not the .c file but the
header file that was wrong. The idea is that we have functions only in
debugging mode, because functions make it easier to debug. But they are
slower than macro expansions. So in release mode, the functions are
replace with the mutex calls directly (which, I think, a macros, too).
If I recall it correctly, I screwed up after I created that
optimization.

> > Found the culprit. Patch is attached.
> > Imho calling the define NDEBUG makes the code unnecessarily hard to
> > read (#ifndef NDEBUG ..). I had to stare at the code for a 
> while until
> > I noticed that it was negated one time to often.

The NDEBUG is an assert() inheritance. Assert() uses it to indicate a
release build. I've build on the same tool to avoid introducing another
one. But obviously, it has its drawbacks, too. I think I'll solve that
by adding a comment.

I hope this clarifies. Drop me a note if I got it wrong again or you
have a question ;)

The tarball that is now in the repository should solve the issue. 

Rainer


From mbiebl at gmail.com  Wed Dec 12 21:47:32 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 12 Dec 2007 21:47:32 +0100
Subject: [rsyslog] Problems with IncludeConfig
Message-ID: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>

Hi everyone,

the current IncludeConfig implementation has some issues, especially
for distributers:
All files in the directory as specified by $IncludeConfig are included.
This means, temp files like *~ *.bak etc that are left behind by
editors, or stuff like *.rpmsave or *.dpkg-old/*.dpkg-new.
This is obviously unfortunate.

How shall we solve this?
1.) I patched the Debian package to only include files ending with
*.conf. If we document that properly this is imho an acceptable
solution.
2.) Implement an internal blacklist of endings that are ignored (see above)
3.) Instead of hardcoding the file ending to *.conf (as in 1.), use
the same syntax as e.g. apache, which allows to specify shell globs:
example:
Include /etc/apache2/conf.d/*.conf

What do you think?

Michael


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From r.bhatia at ipax.at  Thu Dec 13 09:55:49 2007
From: r.bhatia at ipax.at (Raoul Bhatia [IPAX])
Date: Thu, 13 Dec 2007 09:55:49 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
Message-ID: <4760F395.4030502@ipax.at>

Michael Biebl wrote:
> the current IncludeConfig implementation has some issues, especially
> for distributers:
> All files in the directory as specified by $IncludeConfig are included.
> This means, temp files like *~ *.bak etc that are left behind by
> editors, or stuff like *.rpmsave or *.dpkg-old/*.dpkg-new.
> This is obviously unfortunate.
> 
> How shall we solve this?
> 1.) I patched the Debian package to only include files ending with
> *.conf. If we document that properly this is imho an acceptable
> solution.
> 2.) Implement an internal blacklist of endings that are ignored (see above)
> 3.) Instead of hardcoding the file ending to *.conf (as in 1.), use
> the same syntax as e.g. apache, which allows to specify shell globs:
> example:
> Include /etc/apache2/conf.d/*.conf

i would opt for the third option - thou i'm not sure how much work
this creates ;)

cheers,
raoul
-- 
____________________________________________________________________
DI (FH) Raoul Bhatia M.Sc.          email.          r.bhatia at ipax.at
Technischer Leiter

IPAX - Aloy Bhatia Hava OEG         web.          http://www.ipax.at
Barawitzkagasse 10/2/2/11           email.            office at ipax.at
1190 Wien                           tel.               +43 1 3670030
FN 277995t HG Wien                  fax.            +43 1 3670030 15
____________________________________________________________________


From rgerhards at hq.adiscon.com  Thu Dec 13 13:04:33 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 13 Dec 2007 13:04:33 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <4760F395.4030502@ipax.at>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
	<4760F395.4030502@ipax.at>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>

Lol... the initial message seems not to have passed my spam filter... ;)

I, too, tend to option three. The problem, though is
backwards-compatibility... Any suggestions and concerns are appreciated.

I'll look at the code to see how much work that creates. I think the
complexity is quite acceptable. I am right now in a threading redesign,
so I'd like to push this away for a little while... Anyhow, please keep
comments floating - I'll listen and drop in any thoughts I have.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Raoul Bhatia [IPAX]
> Sent: Thursday, December 13, 2007 9:56 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Problems with IncludeConfig
> 
> Michael Biebl wrote:
> > the current IncludeConfig implementation has some issues, especially
> > for distributers:
> > All files in the directory as specified by $IncludeConfig are
> included.
> > This means, temp files like *~ *.bak etc that are left behind by
> > editors, or stuff like *.rpmsave or *.dpkg-old/*.dpkg-new.
> > This is obviously unfortunate.
> >
> > How shall we solve this?
> > 1.) I patched the Debian package to only include files ending with
> > *.conf. If we document that properly this is imho an acceptable
> > solution.
> > 2.) Implement an internal blacklist of endings that are ignored (see
> above)
> > 3.) Instead of hardcoding the file ending to *.conf (as in 1.), use
> > the same syntax as e.g. apache, which allows to specify shell globs:
> > example:
> > Include /etc/apache2/conf.d/*.conf
> 
> i would opt for the third option - thou i'm not sure how much work
> this creates ;)
> 
> cheers,
> raoul
> --
> ____________________________________________________________________
> DI (FH) Raoul Bhatia M.Sc.          email.          r.bhatia at ipax.at
> Technischer Leiter
> 
> IPAX - Aloy Bhatia Hava OEG         web.          http://www.ipax.at
> Barawitzkagasse 10/2/2/11           email.            office at ipax.at
> 1190 Wien                           tel.               +43 1 3670030
> FN 277995t HG Wien                  fax.            +43 1 3670030 15
> ____________________________________________________________________
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Fri Dec 14 09:29:19 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 14 Dec 2007 09:29:19 +0100
Subject: [rsyslog] IMPORTANT: Version Changes
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308674@grfint2.intern.adiscon.com>

Hi all,

some information first here on the mailing list before I announce it. As
you know, I am looking at the way threading is supposed to work in
future releases and, most importantly, looking at the inputs (like mark
message generation).

Around summer, I wrote that I will probably need to release new major
versions when we go into multithreading redesign. It looks like we have
reached this stage. I tried to keep a single code base that still
supports both single- and multi-threaded operations. I have looked into
this the past days and I need to say that it creates a lot of complexity
and hard to understand code.

For this reason, I think it is finally time to branch the code based and
release some new versions.

Soon, I will create a branch for the current 1.20.1 code base. That will
only receive bug fixes, but no new development (except, I guess, GSSAPI
which I about to be contributed by Red Hat). When we are confident the
last changes worked well and introduced no new bugs, there will be a
version 2.0.0 stable release based on that code base.

CVS head, however, will then be rsyslog version 3. It will receive the
new input module interface. It requires pthreads, because there is no
way input modules and many more of the new desired features can be
implemented without them. Consequently, I will remove all
single-threading code from it, resulting in an easier to understand code
base. Please note that I expect this code to change dramatically when it
is being modified to be more modular (much like it was when I introduced
modular outputs in summer). Please note that I will apply any non-bugfix
patches to this code base, only.

I have somewhat bad feeling of going ahead with implementing a more
sophisticated and more parallel multi-threading while we still have an
issue with the segfault. However, I think by now we did everything
imaginable to capture that rare bug. I have come to the conclusion that
the best chance to find it is go ahead and implement the more
sophisticated design. That will lead to a review, and rewrite, of much
of the code in question, uncovering this we didn't think about before.
The recently discovered race condition is an excellent sample.

One thing about the license: rsyslog 2 will stay with "GPL v2 and above"
license, but rsyslog V3 will be licensed under "GPL v3 and above". I
already wrote about that change. It is my firm believe that GPL v3
brings benefit to our freedom to use digital goods. I am a strong oppose
of digital restrictions management (DRM) and software patens and I do
not like the idea that rsyslog benefits anyone who encourages these
things. I hope for your understanding.

I will set stage now for these changes and will do a web announcement
soon. Please don't be surprised that rsyslog v3 will be available before
v2, you now know the reason.

Rainer


From mbiebl at gmail.com  Fri Dec 14 14:36:03 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Fri, 14 Dec 2007 14:36:03 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
	<4760F395.4030502@ipax.at>
	<577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>

2007/12/13, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Lol... the initial message seems not to have passed my spam filter... ;)
>
> I, too, tend to option three. The problem, though is

Yeah, it's the most flexible one.

> backwards-compatibility... Any suggestions and concerns are appreciated.

This feature has been pretty new, and as such probably isn't widely used.
I would simply emphasize this change in the release notes and the Changelog.
The worst that can happen is, that a file is not included anymore.

>
> I'll look at the code to see how much work that creates. I think the
> complexity is quite acceptable. I am right now in a threading redesign,
> so I'd like to push this away for a little while... Anyhow, please keep
> comments floating - I'll listen and drop in any thoughts I have.

Should be fairly straightforward to implement using
http://www.gnu.org/software/libc/manual/html_node/Globbing.html#Globbing

Cheers,
Michael


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From mbiebl at gmail.com  Fri Dec 14 17:24:02 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Fri, 14 Dec 2007 17:24:02 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
	<4760F395.4030502@ipax.at>
	<577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>
	<aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>
Message-ID: <aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com>

2007/12/14, Michael Biebl <mbiebl at gmail.com>:
> 2007/12/13, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > Lol... the initial message seems not to have passed my spam filter... ;)
> >
> > I, too, tend to option three. The problem, though is
>
> Yeah, it's the most flexible one.
>
> > backwards-compatibility... Any suggestions and concerns are appreciated.
>
> This feature has been pretty new, and as such probably isn't widely used.
> I would simply emphasize this change in the release notes and the Changelog.
> The worst that can happen is, that a file is not included anymore.
>
> >
> > I'll look at the code to see how much work that creates. I think the
> > complexity is quite acceptable. I am right now in a threading redesign,
> > so I'd like to push this away for a little while... Anyhow, please keep
> > comments floating - I'll listen and drop in any thoughts I have.
>
> Should be fairly straightforward to implement using
> http://www.gnu.org/software/libc/manual/html_node/Globbing.html#Globbing
>

Small example using glob:

#include <glob.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char **argv)
{
        glob_t files;
        int ret = 0;
        int i = 0;
        char *path;

        if (argc > 1) {
                path = argv[1];
        } else {
                path = strdup("/etc/rsyslog.d/*.conf");
        }

        ret = glob(path, GLOB_ERR, NULL, &files);
        printf("All files matching %s:\n", path);
        for (i = 0; i < files.gl_pathc; i++) {
                printf("\t%s\n", files.gl_pathv[i]);
        }

        globfree(&files);
        printf("Finished\n");

        return EXIT_SUCCESS;
}

man 3 glob has more information.

If wanted, I can prepare a patch for rsyslog.

Cheers,
Michael
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Fri Dec 14 17:29:38 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 14 Dec 2007 17:29:38 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com><4760F395.4030502@ipax.at><577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com><aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>
	<aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com>

Michael,

sorry for the sluggish reply. I am right in the middle of the initial
threading code (have a look at CVS ;)) and that blocks most of
everything else ;)

It would be great if you could supply a patch, I'd apply it as soon as I
got something together with the new code. I try to do the threading in
an as atomic code change as possible, but it will proabbyl busy me for a
while ;)

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Friday, December 14, 2007 5:24 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Problems with IncludeConfig
> 
> 2007/12/14, Michael Biebl <mbiebl at gmail.com>:
> > 2007/12/13, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > > Lol... the initial message seems not to have passed my spam
> filter... ;)
> > >
> > > I, too, tend to option three. The problem, though is
> >
> > Yeah, it's the most flexible one.
> >
> > > backwards-compatibility... Any suggestions and concerns are
> appreciated.
> >
> > This feature has been pretty new, and as such probably isn't widely
> used.
> > I would simply emphasize this change in the release notes and the
> Changelog.
> > The worst that can happen is, that a file is not included anymore.
> >
> > >
> > > I'll look at the code to see how much work that creates. I think
> the
> > > complexity is quite acceptable. I am right now in a threading
> redesign,
> > > so I'd like to push this away for a little while... Anyhow, please
> keep
> > > comments floating - I'll listen and drop in any thoughts I have.
> >
> > Should be fairly straightforward to implement using
> >
>
http://www.gnu.org/software/libc/manual/html_node/Globbing.html#Globbin
> g
> >
> 
> Small example using glob:
> 
> #include <glob.h>
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
> 
> int main(int argc, char **argv)
> {
>         glob_t files;
>         int ret = 0;
>         int i = 0;
>         char *path;
> 
>         if (argc > 1) {
>                 path = argv[1];
>         } else {
>                 path = strdup("/etc/rsyslog.d/*.conf");
>         }
> 
>         ret = glob(path, GLOB_ERR, NULL, &files);
>         printf("All files matching %s:\n", path);
>         for (i = 0; i < files.gl_pathc; i++) {
>                 printf("\t%s\n", files.gl_pathv[i]);
>         }
> 
>         globfree(&files);
>         printf("Finished\n");
> 
>         return EXIT_SUCCESS;
> }
> 
> man 3 glob has more information.
> 
> If wanted, I can prepare a patch for rsyslog.
> 
> Cheers,
> Michael
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From mbiebl at gmail.com  Sun Dec 16 23:38:14 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Sun, 16 Dec 2007 23:38:14 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
	<4760F395.4030502@ipax.at>
	<577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>
	<aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>
	<aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712161438n69382660u9d2a082914d717c9@mail.gmail.com>

2007/12/14, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Michael,
>
> sorry for the sluggish reply. I am right in the middle of the initial
> threading code (have a look at CVS ;)) and that blocks most of
> everything else ;)
>
> It would be great if you could supply a patch, I'd apply it as soon as I
> got something together with the new code. I try to do the threading in
> an as atomic code change as possible, but it will proabbyl busy me for a
> while ;)

Unfortunately I can't set the mimetype via gmail, so the patch is at

http://debs.michaelbiebl.de/rsyslog/rsyslog-glob.patch

I didn't check the return values for glob. I wanted I can implement
that and printout debugging messsages in that case.
The patch also doesn't contain any changes to the documentation (yet).
I wanted to get some feedback first.

Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From mbiebl at gmail.com  Mon Dec 17 05:09:59 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Mon, 17 Dec 2007 05:09:59 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <aa76cdac0712161438n69382660u9d2a082914d717c9@mail.gmail.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
	<4760F395.4030502@ipax.at>
	<577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>
	<aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>
	<aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com>
	<aa76cdac0712161438n69382660u9d2a082914d717c9@mail.gmail.com>
Message-ID: <aa76cdac0712162009q9fe3ecaq407df79296e4baff@mail.gmail.com>

2007/12/16, Michael Biebl <mbiebl at gmail.com>:
> 2007/12/14, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > Michael,
> >
> > sorry for the sluggish reply. I am right in the middle of the initial
> > threading code (have a look at CVS ;)) and that blocks most of
> > everything else ;)
> >
> > It would be great if you could supply a patch, I'd apply it as soon as I
> > got something together with the new code. I try to do the threading in
> > an as atomic code change as possible, but it will proabbyl busy me for a
> > while ;)
>
> Unfortunately I can't set the mimetype via gmail, so the patch is at
>
> http://debs.michaelbiebl.de/rsyslog/rsyslog-glob.patch
>

I have an updated patch
 http://debs.michaelbiebl.de/rsyslog/rsyslog-glob-2.patch
which is backward compatible.
It allows the following

include config files:
 $IncludeConfig /etc/rsyslog.d/foo.conf
include config directories:
 $IncludeConfig /etc/rsyslog.d/ (with or without trailing slash)
include via shell glob:
 $IncludeConfig /etc/rsyslog.d/*.conf

One remaining issue is, how to handle symlinks. I made the patch
consistent to the current behaviour of doIncludeDirectories to ignore
symlinks.

I know that the apache Include directive allows symlinks, and
supporting it could be handy.
So my question is, what do others think about it. Please let me know.

Cheers,
Michael

P.S: A personal wish: as the list is subscriber only, would it be
possible to allow attachements (other than text/plain, gmail is
obviously too stupid)?


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From mbiebl at gmail.com  Mon Dec 17 05:27:28 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Mon, 17 Dec 2007 05:27:28 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <aa76cdac0712162009q9fe3ecaq407df79296e4baff@mail.gmail.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
	<4760F395.4030502@ipax.at>
	<577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>
	<aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>
	<aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com>
	<aa76cdac0712161438n69382660u9d2a082914d717c9@mail.gmail.com>
	<aa76cdac0712162009q9fe3ecaq407df79296e4baff@mail.gmail.com>
Message-ID: <aa76cdac0712162027r85875d2rc7223f9c6a61a43@mail.gmail.com>

2007/12/17, Michael Biebl <mbiebl at gmail.com>:
> 2007/12/16, Michael Biebl <mbiebl at gmail.com>:
> > 2007/12/14, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> I have an updated patch
>  http://debs.michaelbiebl.de/rsyslog/rsyslog-glob-2.patch

Spotted a minor typo, so here goes v3
http://debs.michaelbiebl.de/rsyslog/rsyslog-glob-3.patch

Michael

Btw. gmail uses
Content-Type: multipart/mixed
and then
Content-Type: text/plain for the mail
and
Content-Type: text/x-diff for the patch
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Mon Dec 17 08:04:46 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 17 Dec 2007 08:04:46 +0100
Subject: [rsyslog] Tackeling the segfault...
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308683@grfint2.intern.adiscon.com>

Hi all,

thanks to mildew, I could create another fix that may solve the issue. I
will do a writeup of what happens, but that'll take a short while. In
the mean time, I would appreciate if those of you that experience the
segfault could try the new version. I have released a pre-1.20.2 tarball
at

http://download.rsyslog.com/rsyslog/rsyslog-1.20.2.tar.gz

As usual, this is not yet 1.20.2, so whenever it is re-released be sure
to update your installation. I would appreciate feedback on it very
much.

Many thanks,
Rainer


From mbiebl at gmail.com  Mon Dec 17 08:20:59 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Mon, 17 Dec 2007 08:20:59 +0100
Subject: [rsyslog] Tackeling the segfault...
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308683@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308683@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712162320l3b089b2eue6a3c4699ad0b277@mail.gmail.com>

2007/12/17, Rainer Gerhards <rgerhards at hq.adiscon.com>:
>
> http://download.rsyslog.com/rsyslog/rsyslog-1.20.2.tar.gz
>
> As usual, this is not yet 1.20.2, so whenever it is re-released be sure
> to update your installation. I would appreciate feedback on it very
> much.

Hi Rainer,

it would be handy if you would post md5sums or sha1sums of the tarball.
This would make it easier to identify/distinguish the prereleases and
the final tarball release.

Cheers,
Michael

P.S: current CVS head doesn't compile with USE_NETZIP.

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Mon Dec 17 08:26:38 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 17 Dec 2007 08:26:38 +0100
Subject: [rsyslog] Tackeling the segfault...
In-Reply-To: <aa76cdac0712162320l3b089b2eue6a3c4699ad0b277@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308683@grfint2.intern.adiscon.com>
	<aa76cdac0712162320l3b089b2eue6a3c4699ad0b277@mail.gmail.com>
Message-ID: <1197876398.21389.3.camel@localhost.localdomain>


On Mon, 2007-12-17 at 08:20 +0100, Michael Biebl wrote:
> 2007/12/17, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> >
> > http://download.rsyslog.com/rsyslog/rsyslog-1.20.2.tar.gz
> >
> > As usual, this is not yet 1.20.2, so whenever it is re-released be sure
> > to update your installation. I would appreciate feedback on it very
> > much.
> 
> Hi Rainer,
> 
> it would be handy if you would post md5sums or sha1sums of the tarball.
> This would make it easier to identify/distinguish the prereleases and
> the final tarball release.

good suggestion. For this release the md5sum is
45e6f14094c26f2dd3a3ff7a4d025381 

> 
> Cheers,
> Michael
> 
> P.S: current CVS head doesn't compile with USE_NETZIP.
Will look into that. I have to admit that CVS head may have some issues
the next few days. I thought about holding off commits until everything
is perfect, that would result in one gigantic diff once it is done -
which would make it hard to follow what I am doing. So I am now doing
commits whenever something seems to work reasonably well, but definitely
not prime time yet. The 3.0 branch is not ready now for being used in
practice. But, NETZIP should compile and I'll look into it. Thanks :)


From rgerhards at hq.adiscon.com  Mon Dec 17 08:27:34 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 17 Dec 2007 08:27:34 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <aa76cdac0712162027r85875d2rc7223f9c6a61a43@mail.gmail.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com><4760F395.4030502@ipax.at><577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com><aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com><aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com><aa76cdac0712161438n69382660u9d2a082914d717c9@mail.gmail.com><aa76cdac0712162009q9fe3ecaq407df79296e4baff@mail.gmail.com>
	<aa76cdac0712162027r85875d2rc7223f9c6a61a43@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308685@grfint2.intern.adiscon.com>

Hi Michael,

I like this patch, especially it being backwards-compatible. And I also
tend to agree that symlinks would be nice to have.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Monday, December 17, 2007 5:27 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Problems with IncludeConfig
> 
> 2007/12/17, Michael Biebl <mbiebl at gmail.com>:
> > 2007/12/16, Michael Biebl <mbiebl at gmail.com>:
> > > 2007/12/14, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > I have an updated patch
> >  http://debs.michaelbiebl.de/rsyslog/rsyslog-glob-2.patch
> 
> Spotted a minor typo, so here goes v3
> http://debs.michaelbiebl.de/rsyslog/rsyslog-glob-3.patch
> 
> Michael
> 
> Btw. gmail uses
> Content-Type: multipart/mixed
> and then
> Content-Type: text/plain for the mail
> and
> Content-Type: text/x-diff for the patch
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Mon Dec 17 08:48:01 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 17 Dec 2007 08:48:01 +0100
Subject: [rsyslog] Tackeling the segfault...
In-Reply-To: <aa76cdac0712162320l3b089b2eue6a3c4699ad0b277@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308683@grfint2.intern.adiscon.com>
	<aa76cdac0712162320l3b089b2eue6a3c4699ad0b277@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308686@grfint2.intern.adiscon.com>

> P.S: current CVS head doesn't compile with USE_NETZIP.

Mmmhhh... I cannot reproduce this on my machine (using Fedora 8). Can
you supply me the error messages - I guess best via private mail because
this will get into too much detail for most on this list (Or we move it
to the dev corner in the forum, so that those interested can follow)...

Thanks,
Rainer


From r.bhatia at ipax.at  Mon Dec 17 10:47:50 2007
From: r.bhatia at ipax.at (Raoul Bhatia [IPAX])
Date: Mon, 17 Dec 2007 10:47:50 +0100
Subject: [rsyslog] Tackeling the segfault...
In-Reply-To: <aa76cdac0712162320l3b089b2eue6a3c4699ad0b277@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308683@grfint2.intern.adiscon.com>
	<aa76cdac0712162320l3b089b2eue6a3c4699ad0b277@mail.gmail.com>
Message-ID: <476645C6.2080909@ipax.at>

Michael Biebl wrote:
> 2007/12/17, Rainer Gerhards <rgerhards at hq.adiscon.com>:
>> http://download.rsyslog.com/rsyslog/rsyslog-1.20.2.tar.gz
>>
>> As usual, this is not yet 1.20.2, so whenever it is re-released be sure
>> to update your installation. I would appreciate feedback on it very
>> much.
> 
> Hi Rainer,
> 
> it would be handy if you would post md5sums or sha1sums of the tarball.
> This would make it easier to identify/distinguish the prereleases and
> the final tarball release.

or simply use the well known -pre or -rc tags?

cheers,
raoul
-- 
____________________________________________________________________
DI (FH) Raoul Bhatia M.Sc.          email.          r.bhatia at ipax.at
Technischer Leiter

IPAX - Aloy Bhatia Hava OEG         web.          http://www.ipax.at
Barawitzkagasse 10/2/2/11           email.            office at ipax.at
1190 Wien                           tel.               +43 1 3670030
FN 277995t HG Wien                  fax.            +43 1 3670030 15
____________________________________________________________________


From rgerhards at hq.adiscon.com  Mon Dec 17 11:04:33 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 17 Dec 2007 11:04:33 +0100
Subject: [rsyslog] backward compatibility...
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>

Hi all,

I would appreciate your feedback on backward compatibility. I have
started to create a loadable input module interface in rsyslog v3. My
first input module is the mark message handler (because it has almost no
code, so it was a good example of what needs to be done...). I have now
basically have finished an initial version. The interface needs to be
changed in the longer term, but I'll postpone that until I've done a bit
more work on the config interface. Now I am facing a problem with
backward compatibility:

In sysklogd/rsyslog pre-v3, the -m option controls how often mark
messages are written (with -m0 disabling this mechanism). In v3, the
$MarkMessagePeriod config file directive does the same. However, a
$modload immark is necessary to activate that functionality because
otherwise the module is not loaded (which is good if it is not needed).
When trying to remain backward-compatible, I have an issue when the -m
option is absent. If so, that means a mark message period of 20 minutes.
However, it is hard to detect what may be intended in v3 if this option
is not present. Does it mean the former default (running as a drop-in
replacement) or does it actually mean that no mark messages should
appear if no such directive is in rsyslog.conf?

Do you think it would be acceptable to not emit messages in this case,
slightly breaking backward compatibility? Or should I come up with a
solution. The later would probably require a specific option setting to
disable the -m option and its default (somehow telling "I am not running
in v2 compatibility mode").

I am not sure what is best...

Rainer 


From r.bhatia at ipax.at  Mon Dec 17 11:17:03 2007
From: r.bhatia at ipax.at (Raoul Bhatia [IPAX])
Date: Mon, 17 Dec 2007 11:17:03 +0100
Subject: [rsyslog] backward compatibility...
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
Message-ID: <47664C9F.5030301@ipax.at>

Rainer Gerhards wrote:
> Hi all,
> 
> I would appreciate your feedback on backward compatibility. I have
> started to create a loadable input module interface in rsyslog v3. My
> first input module is the mark message handler (because it has almost no
> code, so it was a good example of what needs to be done...). I have now
> basically have finished an initial version. The interface needs to be
> changed in the longer term, but I'll postpone that until I've done a bit
> more work on the config interface. Now I am facing a problem with
> backward compatibility:
> 
> In sysklogd/rsyslog pre-v3, the -m option controls how often mark
> messages are written (with -m0 disabling this mechanism). In v3, the
> $MarkMessagePeriod config file directive does the same. However, a
> $modload immark is necessary to activate that functionality because
> otherwise the module is not loaded (which is good if it is not needed).
> When trying to remain backward-compatible, I have an issue when the -m
> option is absent. If so, that means a mark message period of 20 minutes.
> However, it is hard to detect what may be intended in v3 if this option
> is not present. Does it mean the former default (running as a drop-in
> replacement) or does it actually mean that no mark messages should
> appear if no such directive is in rsyslog.conf?
> 
> Do you think it would be acceptable to not emit messages in this case,
> slightly breaking backward compatibility? Or should I come up with a
> solution. The later would probably require a specific option setting to
> disable the -m option and its default (somehow telling "I am not running
> in v2 compatibility mode").
> 
> I am not sure what is best...

i would use the opportunity and not hesitate to use a
make-everything-right-even-if-we-break-something approach.

upgrading seems to not be trivial, as one needs to go through the config
files anyway. so lets do things right and maybe issue some error/warning
message on obsolete options (like -m).

cheers,
raoul
ps: i hope i'm getting you right ;)
-- 
____________________________________________________________________
DI (FH) Raoul Bhatia M.Sc.          email.          r.bhatia at ipax.at
Technischer Leiter

IPAX - Aloy Bhatia Hava OEG         web.          http://www.ipax.at
Barawitzkagasse 10/2/2/11           email.            office at ipax.at
1190 Wien                           tel.               +43 1 3670030
FN 277995t HG Wien                  fax.            +43 1 3670030 15
____________________________________________________________________


From rgerhards at hq.adiscon.com  Mon Dec 17 11:35:02 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 17 Dec 2007 11:35:02 +0100
Subject: [rsyslog] backward compatibility...
In-Reply-To: <47664C9F.5030301@ipax.at>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
	<47664C9F.5030301@ipax.at>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308692@grfint2.intern.adiscon.com>

> i would use the opportunity and not hesitate to use a
> make-everything-right-even-if-we-break-something approach.
> 
> upgrading seems to not be trivial, as one needs to go through the
> config
> files anyway. so lets do things right and maybe issue some
> error/warning
> message on obsolete options (like -m).
> 
> cheers,
> raoul
> ps: i hope i'm getting you right ;)

Thanks - yes, this was the kind of feedback I am looking for. Sorry for
the quite complex description, I could not come up with a better one...

Rainer


From rgerhards at hq.adiscon.com  Tue Dec 18 17:21:26 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 18 Dec 2007 17:21:26 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <aa76cdac0712162009q9fe3ecaq407df79296e4baff@mail.gmail.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
	<4760F395.4030502@ipax.at>
	<577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>
	<aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>
	<aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com>
	<aa76cdac0712161438n69382660u9d2a082914d717c9@mail.gmail.com>
	<aa76cdac0712162009q9fe3ecaq407df79296e4baff@mail.gmail.com>
Message-ID: <1197994886.14826.2.camel@localhost.localdomain>

This patch now also made it to the repository. Applied to both v2
(previously 1.20.2) and v3.

Rainer

On Mon, 2007-12-17 at 05:09 +0100, Michael Biebl wrote:
> 2007/12/16, Michael Biebl <mbiebl at gmail.com>:
> > 2007/12/14, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > > Michael,
> > >
> > > sorry for the sluggish reply. I am right in the middle of the initial
> > > threading code (have a look at CVS ;)) and that blocks most of
> > > everything else ;)
> > >
> > > It would be great if you could supply a patch, I'd apply it as soon as I
> > > got something together with the new code. I try to do the threading in
> > > an as atomic code change as possible, but it will proabbyl busy me for a
> > > while ;)
> >
> > Unfortunately I can't set the mimetype via gmail, so the patch is at
> >
> > http://debs.michaelbiebl.de/rsyslog/rsyslog-glob.patch
> >
> 
> I have an updated patch
>  http://debs.michaelbiebl.de/rsyslog/rsyslog-glob-2.patch
> which is backward compatible.
> It allows the following
> 
> include config files:
>  $IncludeConfig /etc/rsyslog.d/foo.conf
> include config directories:
>  $IncludeConfig /etc/rsyslog.d/ (with or without trailing slash)
> include via shell glob:
>  $IncludeConfig /etc/rsyslog.d/*.conf
> 
> One remaining issue is, how to handle symlinks. I made the patch
> consistent to the current behaviour of doIncludeDirectories to ignore
> symlinks.
> 
> I know that the apache Include directive allows symlinks, and
> supporting it could be handy.
> So my question is, what do others think about it. Please let me know.
> 
> Cheers,
> Michael
> 
> P.S: A personal wish: as the list is subscriber only, would it be
> possible to allow attachements (other than text/plain, gmail is
> obviously too stupid)?
> 
> 


From sur5r at sur5r.net  Tue Dec 18 22:16:54 2007
From: sur5r at sur5r.net (Jakob Haufe)
Date: Tue, 18 Dec 2007 22:16:54 +0100
Subject: [rsyslog] backward compatibility...
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
Message-ID: <20071218221654.27ec59a2@marathon.haufe>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

"Rainer Gerhards" <rgerhards at hq.adiscon.com> wrote on Mon, 17 Dec 2007
11:04:33 +0100:

> Do you think it would be acceptable to not emit messages in this case,
> slightly breaking backward compatibility? Or should I come up with a
> solution. The later would probably require a specific option setting
> to disable the -m option and its default (somehow telling "I am not
> running in v2 compatibility mode").

This makes me think about sysklogd compatibility. How about the
following: If there's no rsyslog.conf but a syslog.conf then rsyslogd
could go to v2/sysklogd-compatibility mode. If not, it should run in
native mode.
This should improve compatibility in "drop-in replacement for stock
sysklogd" setups.

Regards,
sur5r
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHaDjK1YAhDic+adYRArd6AJ9fXRZU1TnW4L/eKRSfcE3x/G/PMACfbDEF
GgU6vviKYQQofAeAJ9vEnDw=
=6u5b
-----END PGP SIGNATURE-----

From mic at npgx.com.au  Wed Dec 19 04:45:00 2007
From: mic at npgx.com.au (Michael Mansour)
Date: Wed, 19 Dec 2007 13:45:00 +1000
Subject: [rsyslog] backward compatibility...
In-Reply-To: <20071218221654.27ec59a2@marathon.haufe>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
	<20071218221654.27ec59a2@marathon.haufe>
Message-ID: <20071219033907.M7098@npgx.com.au>

Hi,

> "Rainer Gerhards" <rgerhards at hq.adiscon.com> wrote on Mon, 17 Dec 2007
> 11:04:33 +0100:
> 
> > Do you think it would be acceptable to not emit messages in this case,
> > slightly breaking backward compatibility? Or should I come up with a
> > solution. The later would probably require a specific option setting
> > to disable the -m option and its default (somehow telling "I am not
> > running in v2 compatibility mode").
> 
> This makes me think about sysklogd compatibility. How about the
> following: If there's no rsyslog.conf but a syslog.conf then rsyslogd
> could go to v2/sysklogd-compatibility mode. If not, it should run in
> native mode.
> This should improve compatibility in "drop-in replacement for stock
> sysklogd" setups.

That's true, but I've also found developers are happy to cater for rsyslog
through updates to their software. For example, I use drbdlinks (software for
clusters) and when I introduced rsyslog into my environment I found drbdlinks
incompatible with it. 

I put a workaround in place and at the same time emailed the drbdlinks
developer, and he introduced rsyslog support into his code, provided an update
to his software and mentioned me in the suggestion.

Jamie Cameron of Webmin, Virtualmin and Virtualmin Pro is/has also
introduced/introducing support for rsyslog into his products, since some
Webmin functionality looks for the /etc/syslog.conf file to allow some
functionality in the apps and will/now looks for /etc/rsyslog.conf also.

I saw will/now since I have his confirmation it's on his Todo list from a few
months back, but I haven't tested any updates from him since we spoke about it
back then.

The moral of this story is, for any software you use, ask the developers to
add rsyslog support if things they develop only supports syslog.

Regards,

Michael.

> Regards,
> sur5r
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
> 
> iD8DBQFHaDjK1YAhDic+adYRArd6AJ9fXRZU1TnW4L/eKRSfcE3x/G/PMACfbDEF
> GgU6vviKYQQofAeAJ9vEnDw=
> =6u5b
> -----END PGP SIGNATURE-----
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
------- End of Original Message -------



From mbiebl at gmail.com  Wed Dec 19 05:51:57 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 19 Dec 2007 05:51:57 +0100
Subject: [rsyslog] backward compatibility...
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712182051m2ef76c2sb401e0bd45904382@mail.gmail.com>

2007/12/17, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Hi all,
>
> I would appreciate your feedback on backward compatibility. I have
> started to create a loadable input module interface in rsyslog v3. My
> first input module is the mark message handler (because it has almost no
> code, so it was a good example of what needs to be done...). I have now
> basically have finished an initial version. The interface needs to be
> changed in the longer term, but I'll postpone that until I've done a bit
> more work on the config interface. Now I am facing a problem with
> backward compatibility:
>
> In sysklogd/rsyslog pre-v3, the -m option controls how often mark
> messages are written (with -m0 disabling this mechanism). In v3, the

One thing I was wondering:
If you intend to shift all (even core) functionality into loadable
modules, how do do you handle things like --help or available command
line options like -m
Do you want to hardcode it  or will you provide an interface, where
rsyslog will query the module about its help message and available
options.

I'm also still a bit uncertain, if moving everything resp. core
functionality to modules is a good idea (for problems you already
mentioned).
Imho having all core functionality in a single binary is simply much
more robust and fool proof. For things like the SQL db output plugin,
the module interface is great, because it avoids to pull in large
library and package dependencies and allows to install them on a as
need basis. For other functionality I still need to recognize the
benefits.

Rainer, could you roughly sketch, how you envision to break rsyslog
into loadable modules in v3. Which kind of functionality would be
loadable as module, which functionality do you plan to keep in the
rsyslogd binary. A listing of all (planned) modules + the provided
functionality and requirements would really help.

Another thing: Say you move the regexp support into a separate module.
If a regexp is then used in rsyslog.conf, will you bail out with an
error, simply print a warning (which could go unnoticed and the poor
administrator doesn't know why his regexp doesn't know) or load
modules on demand.
For the latter you'd need some kind of interface to query the *.so
files for their supported functionality. I.e. the modules would export
a list of config directives it supports and rsyslog could upon startup
query each available module and create a map.
So, e.g. the ommysql module would export its support for the :ommysql:
config directive.
Whenever rsyslog finds such a config directive it could/would load the
modules on demand.
Same could be done for the command line parameters.
The imklog module would export, that it supports the -m command line
parameter. Whenever that commandline parameter is used, rsyslog would
know which module to load.

There are only rough ideas and there is certainly still much to
consider. But what do you think about the basic idea?

Cheers,
Michael

Cheers,
Michael




-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From mbiebl at gmail.com  Wed Dec 19 09:27:13 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 19 Dec 2007 09:27:13 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <1197994886.14826.2.camel@localhost.localdomain>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com>
	<4760F395.4030502@ipax.at>
	<577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com>
	<aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com>
	<aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com>
	<aa76cdac0712161438n69382660u9d2a082914d717c9@mail.gmail.com>
	<aa76cdac0712162009q9fe3ecaq407df79296e4baff@mail.gmail.com>
	<1197994886.14826.2.camel@localhost.localdomain>
Message-ID: <aa76cdac0712190027k55824b7cn582acd6e7a1a6feb@mail.gmail.com>

2007/12/18, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> This patch now also made it to the repository. Applied to both v2
> (previously 1.20.2) and v3.

Cool, thanks.
Remains to update the documentation doc/rsconf1_includeconfig.html.
I'm usually not good a writing docs, but if you want I can provide a patch ;-)

Cheers,
Michael
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Wed Dec 19 09:33:01 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 19 Dec 2007 09:33:01 +0100
Subject: [rsyslog] Problems with IncludeConfig
In-Reply-To: <aa76cdac0712190027k55824b7cn582acd6e7a1a6feb@mail.gmail.com>
References: <aa76cdac0712121247u52502fcaxf9fd0a33940f996f@mail.gmail.com><4760F395.4030502@ipax.at><577465F99B41C842AAFBE9ED71E70ABA30866E@grfint2.intern.adiscon.com><aa76cdac0712140536x4777273pd96abd7d5bc944eb@mail.gmail.com><aa76cdac0712140824s22f43398qe93b26462ed527f6@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA30867C@grfint2.intern.adiscon.com><aa76cdac0712161438n69382660u9d2a082914d717c9@mail.gmail.com><aa76cdac0712162009q9fe3ecaq407df79296e4baff@mail.gmail.com><1197994886.14826.2.camel@localhost.localdomain>
	<aa76cdac0712190027k55824b7cn582acd6e7a1a6feb@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086A8@grfint2.intern.adiscon.com>

Oops... this time I forgot the doc (I always try to have at least a
minimal description there). I just added two lines, maybe you have a
look. I am a bit under pressure because I just released 1.21.1, but now
re-created the tarball with the somewhat enhanced doc. If you are up for
a better doc, just sent me a patch ;) It'll appear in the next release.

Release notification is due in a few minutes...

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Wednesday, December 19, 2007 9:27 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Problems with IncludeConfig
> 
> 2007/12/18, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > This patch now also made it to the repository. Applied to both v2
> > (previously 1.20.2) and v3.
> 
> Cool, thanks.
> Remains to update the documentation doc/rsconf1_includeconfig.html.
> I'm usually not good a writing docs, but if you want I can provide a
> patch ;-)
> 
> Cheers,
> Michael
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Wed Dec 19 09:42:35 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 19 Dec 2007 09:42:35 +0100
Subject: [rsyslog] rsyslog 1.21.0 released
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086A9@grfint2.intern.adiscon.com>

Hi all,

as I mentioned a few minutes ago, I have just released version 1.21.0.
First question may be "why 1.21.0 and not 2.0.0"? The answer is that
varmojfekoj provided a patch that allowed me to finally enable GSS-API
support in rsyslog. While I really like this, I did not find it
appropriate to include such a major new feature in the first release of
a 2.0.0 *stable*. So I went back to the v1 releases. If I do not hear of
any serious problems, I'll re-release that version as 2.0.0.

What also brings us to the new features: GSS-API is obviously now there.
There is very limited documentation that goes with it. I'll focus on
that in the v3 releases, with docs being propagated down to v2 as
possible. If anyone would like to contribute doc, I'd obviously be very
happy ;) I currently do not intend to spent too much time with it,
because I do not like to distract myself from v3 until I managed to get
at least the most important things up and running.

V1.21.0 also contains Michael Biebl's $IncludeConfig enhancement, many
cleanups as well as the fix that hopefully addresses the segfault. So
far, I have neither heard a success or failure report. For success, it
obviously is too early, rsyslogd often needed a few days to experience
the fault. I still hope for the best...

A quick note on v3: things are going pretty well. I'll post an answer to
Michael Biebl's note on what will become modular a bit later, it needs
some time. GSS-API was the last major new feature I'll accept for v2.
Everything else will go in v3. I expect to release v2 by the end of the
week and the initial v3 maybe a week later. I will make a preview
tarball of v3 available soon so that those interested can have a look at
it (and, of course, CVS head already has it).

Download links and changelog can be found on the usual places on
www.rsyslog.com.

As always, feedback is appreciated.

Rainer Gerhards 


From mbiebl at gmail.com  Wed Dec 19 09:50:44 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 19 Dec 2007 09:50:44 +0100
Subject: [rsyslog] rsyslog 1.21.0 released
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086A9@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA3086A9@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712190050u5ec134fbp6675ab637602ae89@mail.gmail.com>

2007/12/19, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Download links and changelog can be found on the usual places on
> www.rsyslog.com.

Download link seems to be missing.

Cheers,
Michael

P.S: what about posting md5sums for release tarballs in the annoucment
email and on www.rsyslog.com?

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From mbiebl at gmail.com  Wed Dec 19 09:52:04 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 19 Dec 2007 09:52:04 +0100
Subject: [rsyslog] rsyslog 1.21.0 released
In-Reply-To: <aa76cdac0712190050u5ec134fbp6675ab637602ae89@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA3086A9@grfint2.intern.adiscon.com>
	<aa76cdac0712190050u5ec134fbp6675ab637602ae89@mail.gmail.com>
Message-ID: <aa76cdac0712190052g1ec7035bhf68270abd32bf45b@mail.gmail.com>

2007/12/19, Michael Biebl <mbiebl at gmail.com>:
> 2007/12/19, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > Download links and changelog can be found on the usual places on
> > www.rsyslog.com.
>
> Download link seems to be missing.

Ah, seems to be there now.

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Wed Dec 19 10:13:18 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 19 Dec 2007 10:13:18 +0100
Subject: [rsyslog] rsyslog 1.21.0 released
In-Reply-To: <aa76cdac0712190052g1ec7035bhf68270abd32bf45b@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA3086A9@grfint2.intern.adiscon.com><aa76cdac0712190050u5ec134fbp6675ab637602ae89@mail.gmail.com>
	<aa76cdac0712190052g1ec7035bhf68270abd32bf45b@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086AB@grfint2.intern.adiscon.com>

I guess it was a cache issue, I always start by creating the download,
then changelog entry, etc... The md5sum is part of the download
description. I thought it is OK to post it there, only. It also somewhat
saved me when I applied the last-minute doc patch and re-released. With
the md5sum in the mail, it would have created some confusion. What do
you say?

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Wednesday, December 19, 2007 9:52 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 1.21.0 released
> 
> 2007/12/19, Michael Biebl <mbiebl at gmail.com>:
> > 2007/12/19, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > > Download links and changelog can be found on the usual places on
> > > www.rsyslog.com.
> >
> > Download link seems to be missing.
> 
> Ah, seems to be there now.
> 
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From mbiebl at gmail.com  Wed Dec 19 10:21:29 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Wed, 19 Dec 2007 10:21:29 +0100
Subject: [rsyslog] rsyslog 1.21.0 released
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086AB@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA3086A9@grfint2.intern.adiscon.com>
	<aa76cdac0712190050u5ec134fbp6675ab637602ae89@mail.gmail.com>
	<aa76cdac0712190052g1ec7035bhf68270abd32bf45b@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086AB@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712190121icbf2cb7v70c672a68bba9329@mail.gmail.com>

2007/12/19, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> I guess it was a cache issue, I always start by creating the download,
> then changelog entry, etc... The md5sum is part of the download
> description. I thought it is OK to post it there, only. It also somewhat
> saved me when I applied the last-minute doc patch and re-released. With
> the md5sum in the mail, it would have created some confusion. What do
> you say?

Agreed. Posting the md5sum on the web page should be fine.

Cheers,
Michael
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Wed Dec 19 12:37:09 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 19 Dec 2007 12:37:09 +0100
Subject: [rsyslog] backward compatibility...
In-Reply-To: <aa76cdac0712182051m2ef76c2sb401e0bd45904382@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
	<aa76cdac0712182051m2ef76c2sb401e0bd45904382@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086B1@grfint2.intern.adiscon.com>

Michael and all,

I took me a while to craft a response to your excellent question. I have
done this as a blog post so that it is easier to reference it in the
future.

I suggest that everyone interested in the v3 design has a look at it,
because it describes the way I am heading. If someone doesn't like that,
it is now time to speak up - in a few weeks it will probably be
impossible to change routes.

http://rgerhards.blogspot.com/2007/12/modules-core-functionality-and-rsy
slog.html

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Wednesday, December 19, 2007 5:52 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] backward compatibility...
> 
> 2007/12/17, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > Hi all,
> >
> > I would appreciate your feedback on backward compatibility. I have
> > started to create a loadable input module interface in rsyslog v3.
My
> > first input module is the mark message handler (because it has
almost
> no
> > code, so it was a good example of what needs to be done...). I have
> now
> > basically have finished an initial version. The interface needs to
be
> > changed in the longer term, but I'll postpone that until I've done a
> bit
> > more work on the config interface. Now I am facing a problem with
> > backward compatibility:
> >
> > In sysklogd/rsyslog pre-v3, the -m option controls how often mark
> > messages are written (with -m0 disabling this mechanism). In v3, the
> 
> One thing I was wondering:
> If you intend to shift all (even core) functionality into loadable
> modules, how do do you handle things like --help or available command
> line options like -m
> Do you want to hardcode it  or will you provide an interface, where
> rsyslog will query the module about its help message and available
> options.
> 
> I'm also still a bit uncertain, if moving everything resp. core
> functionality to modules is a good idea (for problems you already
> mentioned).
> Imho having all core functionality in a single binary is simply much
> more robust and fool proof. For things like the SQL db output plugin,
> the module interface is great, because it avoids to pull in large
> library and package dependencies and allows to install them on a as
> need basis. For other functionality I still need to recognize the
> benefits.
> 
> Rainer, could you roughly sketch, how you envision to break rsyslog
> into loadable modules in v3. Which kind of functionality would be
> loadable as module, which functionality do you plan to keep in the
> rsyslogd binary. A listing of all (planned) modules + the provided
> functionality and requirements would really help.
> 
> Another thing: Say you move the regexp support into a separate module.
> If a regexp is then used in rsyslog.conf, will you bail out with an
> error, simply print a warning (which could go unnoticed and the poor
> administrator doesn't know why his regexp doesn't know) or load
> modules on demand.
> For the latter you'd need some kind of interface to query the *.so
> files for their supported functionality. I.e. the modules would export
> a list of config directives it supports and rsyslog could upon startup
> query each available module and create a map.
> So, e.g. the ommysql module would export its support for the :ommysql:
> config directive.
> Whenever rsyslog finds such a config directive it could/would load the
> modules on demand.
> Same could be done for the command line parameters.
> The imklog module would export, that it supports the -m command line
> parameter. Whenever that commandline parameter is used, rsyslog would
> know which module to load.
> 
> There are only rough ideas and there is certainly still much to
> consider. But what do you think about the basic idea?
> 
> Cheers,
> Michael
> 
> Cheers,
> Michael
> 
> 
> 
> 
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From mbiebl at gmail.com  Thu Dec 20 10:08:56 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Thu, 20 Dec 2007 10:08:56 +0100
Subject: [rsyslog] backward compatibility...
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086B1@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
	<aa76cdac0712182051m2ef76c2sb401e0bd45904382@mail.gmail.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086B1@grfint2.intern.adiscon.com>
Message-ID: <aa76cdac0712200108q7e9eb058g2e179c2fe00f65d0@mail.gmail.com>

2007/12/19, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Michael and all,
>
> I took me a while to craft a response to your excellent question. I have
> done this as a blog post so that it is easier to reference it in the
> future.
>
> I suggest that everyone interested in the v3 design has a look at it,
> because it describes the way I am heading. If someone doesn't like that,
> it is now time to speak up - in a few weeks it will probably be
> impossible to change routes.

Based on your recent blog post here are some thoughts of mine. Please
keep in mind, that being (Debian) package maintainer, so I speak from
a distributors pov.

1.) external dependencies
Having loadable modules for stuff like MySQL/PostgreSQL output support
is great (for a package maintainer)
This means I can package a basic rsyslog package with minimal
dependencies (glibc) and people can chose to install
rsyslog-(mysql,pgsql) based on their needs.
Here in this case, loadable modules have a real benefit.
The only remaining external dependencies currently in rsyslog are libz
(NETZIP support) and gssapi-krb (KERBEROS support). If those
functionality would be put into a loadable module, I'd support that.
2.) memory usage
I don't think this is a real issue, even for embedded systems. See
point 1. If we manage to put external dependencies into loadable
modules, that would be sufficient imho.
Modularising everything won't have that much of an memory benefit
imho. Given that a standard rsyslogd is about 1M RSS, this doesn't
imply the need for more modularization.
3.) Security
You mentioned, that you try to improve security through modules.
Usually, having loadable module support is considered a security risk.
One messed up $IncludeConfig directive (or manipulated through a
malicious attacker), and you load potentially hazardous modules.
Loadable modules support introduces a much bigger attack vector.

I'm not a SELinux guy. But I'd be interested if loadable modules could
cause trouble in putting rsyslog in it's own security domain. Maybe
the fedora guys can comment on this.

4.) (code) modularization
Writing modular code doesn't strictly imply that the modules have to
be loadable *.so files.
You can still write modular code, with a strict API etc. and organize
it e.g. via sub directories.
5.) Performance penalty through loadable modules.
I could be wrong on this point, but given that you have pointers on
functions, when you use modules (dlsym), there is an additional
indirection on each function call. This could have a performance
impact for core functionality. This is just speculation and should be
tested/evaluated beforehand. After all, rsyslog tries to meet
high-performance needs, too.
6.) Inconvenience
This is just a gut feeling, but having to load all sorts of moduls
first, before rsyslog does anything, could prove cumbersome. As
administrator you'd have to know, which modules to load, to get
rsyslog to do what you want. This means additional effort (reading
docs) and inconvenience.
7.) Robustness
Having a single binary can prove to be a live safer. E.g. you could
simply copy the rsyslogd binary from another machine if there is
something wrong with your local copy.
As you were talking about embedded systems, having the ability to
compile a static binary including all functionality is a definite
plus. There might be platforms out there which don't support dlopen().

I'm still missing the complete picture, too. It's still a bit too vague for me.

You were talking about input, output and filter modules. Rainer, could
cou try to complete this list somehow, maybe draw/sketch a flow
diagram, marking the modules.

input: local, kernel and network, ...
filter: regexp, ...
output: mysql, pgsql, file, forward...
authorization?


I hope, this all doesn't sound too negative. But before going all
modular, all these issues should be considered imho.

Hopefully these comments will help.

Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Thu Dec 20 10:24:49 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 20 Dec 2007 10:24:49 +0100
Subject: [rsyslog] backward compatibility...
In-Reply-To: <aa76cdac0712200108q7e9eb058g2e179c2fe00f65d0@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com><aa76cdac0712182051m2ef76c2sb401e0bd45904382@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA3086B1@grfint2.intern.adiscon.com>
	<aa76cdac0712200108q7e9eb058g2e179c2fe00f65d0@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086B9@grfint2.intern.adiscon.com>

Michael,

thanks for the good points. I'll provide a more in-depth reply later,
but noticed that I missed one important point in the blog post.

Loadable modules are also (at least partly) about platform independence.
Let me elaborate this with an example:

I have yet to fully port rsyslog to solaris. The reason I stopped is
that Solaris uses a mechanism different from Linux to handle local
logging (e.g. via logger). I didn't like to mess up even more with the
code that pulls different log sources.

With the loadable input plugins, there is a very clean solution. If you
run on Linux, just load imLocaLogLinux and if you happen to run under
Solaris load imLocalLogSolaris instead. No conditional compilation, no
complex code. Plus, anyone interested to provide an input implementation
for whatever weird system is free to do so.

So far to what I forgot to mention...

A quick reply: I think we agree that modular code is good to have. Our
main difference seems to be if it should be dynamically or statically
linked together. I am right now modularizing the inputs, have done two
so far and going to the local log socket next. That will also bring some
practical experience, which it already did. So I'll do a round of
modularization and then look at the result to fine-tune it. If it will
be loadable or not will not really be a major concern at that stage. I
just wanted to provide reasoning why I develop concurrently to this very
good discussion.

Again, a more in-depth reply a bit later.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Thursday, December 20, 2007 10:09 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] backward compatibility...
> 
> 2007/12/19, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > Michael and all,
> >
> > I took me a while to craft a response to your excellent question. I
> have
> > done this as a blog post so that it is easier to reference it in the
> > future.
> >
> > I suggest that everyone interested in the v3 design has a look at
it,
> > because it describes the way I am heading. If someone doesn't like
> that,
> > it is now time to speak up - in a few weeks it will probably be
> > impossible to change routes.
> 
> Based on your recent blog post here are some thoughts of mine. Please
> keep in mind, that being (Debian) package maintainer, so I speak from
> a distributors pov.
> 
> 1.) external dependencies
> Having loadable modules for stuff like MySQL/PostgreSQL output support
> is great (for a package maintainer)
> This means I can package a basic rsyslog package with minimal
> dependencies (glibc) and people can chose to install
> rsyslog-(mysql,pgsql) based on their needs.
> Here in this case, loadable modules have a real benefit.
> The only remaining external dependencies currently in rsyslog are libz
> (NETZIP support) and gssapi-krb (KERBEROS support). If those
> functionality would be put into a loadable module, I'd support that.
> 2.) memory usage
> I don't think this is a real issue, even for embedded systems. See
> point 1. If we manage to put external dependencies into loadable
> modules, that would be sufficient imho.
> Modularising everything won't have that much of an memory benefit
> imho. Given that a standard rsyslogd is about 1M RSS, this doesn't
> imply the need for more modularization.
> 3.) Security
> You mentioned, that you try to improve security through modules.
> Usually, having loadable module support is considered a security risk.
> One messed up $IncludeConfig directive (or manipulated through a
> malicious attacker), and you load potentially hazardous modules.
> Loadable modules support introduces a much bigger attack vector.
> 
> I'm not a SELinux guy. But I'd be interested if loadable modules could
> cause trouble in putting rsyslog in it's own security domain. Maybe
> the fedora guys can comment on this.
> 
> 4.) (code) modularization
> Writing modular code doesn't strictly imply that the modules have to
> be loadable *.so files.
> You can still write modular code, with a strict API etc. and organize
> it e.g. via sub directories.
> 5.) Performance penalty through loadable modules.
> I could be wrong on this point, but given that you have pointers on
> functions, when you use modules (dlsym), there is an additional
> indirection on each function call. This could have a performance
> impact for core functionality. This is just speculation and should be
> tested/evaluated beforehand. After all, rsyslog tries to meet
> high-performance needs, too.
> 6.) Inconvenience
> This is just a gut feeling, but having to load all sorts of moduls
> first, before rsyslog does anything, could prove cumbersome. As
> administrator you'd have to know, which modules to load, to get
> rsyslog to do what you want. This means additional effort (reading
> docs) and inconvenience.
> 7.) Robustness
> Having a single binary can prove to be a live safer. E.g. you could
> simply copy the rsyslogd binary from another machine if there is
> something wrong with your local copy.
> As you were talking about embedded systems, having the ability to
> compile a static binary including all functionality is a definite
> plus. There might be platforms out there which don't support dlopen().
> 
> I'm still missing the complete picture, too. It's still a bit too
vague
> for me.
> 
> You were talking about input, output and filter modules. Rainer, could
> cou try to complete this list somehow, maybe draw/sketch a flow
> diagram, marking the modules.
> 
> input: local, kernel and network, ...
> filter: regexp, ...
> output: mysql, pgsql, file, forward...
> authorization?
> 
> 
> I hope, this all doesn't sound too negative. But before going all
> modular, all these issues should be considered imho.
> 
> Hopefully these comments will help.
> 
> Michael
> 
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From pvrabec at redhat.com  Thu Dec 20 10:47:56 2007
From: pvrabec at redhat.com (Peter Vrabec)
Date: Thu, 20 Dec 2007 10:47:56 +0100
Subject: [rsyslog] backward compatibility...
In-Reply-To: <aa76cdac0712200108q7e9eb058g2e179c2fe00f65d0@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086B1@grfint2.intern.adiscon.com>
	<aa76cdac0712200108q7e9eb058g2e179c2fe00f65d0@mail.gmail.com>
Message-ID: <200712201047.56857.pvrabec@redhat.com>

just a quick response,

On Thursday 20 December 2007 10:08:56 am Michael Biebl wrote:
> 2007/12/19, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > Michael and all,
> >
> > I took me a while to craft a response to your excellent question. I have
> > done this as a blog post so that it is easier to reference it in the
> > future.
> >
> > I suggest that everyone interested in the v3 design has a look at it,
> > because it describes the way I am heading. If someone doesn't like that,
> > it is now time to speak up - in a few weeks it will probably be
> > impossible to change routes.
>
> Based on your recent blog post here are some thoughts of mine. Please
> keep in mind, that being (Debian) package maintainer, so I speak from
> a distributors pov.
>

> 3.) Security
> You mentioned, that you try to improve security through modules.
> Usually, having loadable module support is considered a security risk.
> One messed up $IncludeConfig directive (or manipulated through a
> malicious attacker), and you load potentially hazardous modules.
> Loadable modules support introduces a much bigger attack vector.
>
> I'm not a SELinux guy. But I'd be interested if loadable modules could
> cause trouble in putting rsyslog in it's own security domain. Maybe
> the fedora guys can comment on this.
I'm not SELinux guy either, but I don't see any problem here. Modules and 
binary will be in same domain.



From rgerhards at hq.adiscon.com  Thu Dec 20 15:56:33 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 20 Dec 2007 15:56:33 +0100
Subject: [rsyslog] Another compatibility question...
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086BF@grfint2.intern.adiscon.com>

Hi all,

I'd appreciate your feedback on this question.

Historically, syslog messages were not only received on port 514, but
were also send from that port. This is an artifact from the fact that
early syslogd used a single port both for sending and receiving. So far,
I have maintained that and done so with some pain. With v3, I would like
to get rid of this legacy (because it introduces inter-module
dependencies and also makes multiple listeners really complicated).

So I wonder if anyone over here finds this "feature" (syslog messages
are sent from port 514 by default) important. What pops up my mind are
firewall rules. Is there anything else broken if we change that?

Feedback is highly appreciated,
Rainer


From bakers at web-ster.com  Thu Dec 20 18:04:33 2007
From: bakers at web-ster.com (Scott Baker)
Date: Thu, 20 Dec 2007 09:04:33 -0800
Subject: [rsyslog] Syslog logging taking too much disk write?
Message-ID: <476AA0A1.5080300@web-ster.com>

I have a semi busy syslog server which logs about 40 or 50 lines per
second. I noticed using dstat that it's taking a TON of disk write.
In the megaBYTES per second range. This seems extremely high to me
so I wrote a script to test it.

http://www.perturb.org/code/log_lines.pl

First download dstat, gkrellm, or iostat and look at what your
normal disk write is per second. On my underused desktop it hovers
around zero or maybe 25KB/s per second.

If I run my script and tell it to log 10 lines to syslog per second

perl log_lines.pl 10

the write usage spikes up to ~160KB/s. We're only writing about 740
bytes of data per second so why is the disk write so high?

If I tell my script to instead log to a local file in /tmp at 10
lines per second

perl log_lines.pl 10 -t

The write usage spikes to ~120KB/s about once every 5 seconds (as
the buffer fills up). I'm curious why syslog writing is so much more
intense? It seems to me on a really busy syslog server that you
could seriously smoke your HDs with all the writing it's doing. Am I
missing something?

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Thu Dec 20 18:39:40 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 20 Dec 2007 18:39:40 +0100
Subject: [rsyslog] Syslog logging taking too much disk write?
In-Reply-To: <476AA0A1.5080300@web-ster.com>
References: <476AA0A1.5080300@web-ster.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086C1@grfint2.intern.adiscon.com>

Could you post your rsyslog.conf please. Without seeing it, it sounds
like you sync the disc with every write. Unfortunately, this is the
legacy default. You can turn it off with placing a dash before the
selector line:

e.g. *.* /var/log/whatever

syncs after each write, while

*.* -/var/log/whatever

uses buffering - which should be fairly safe with decent hardware and a
UPS...

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Scott Baker
> Sent: Thursday, December 20, 2007 6:05 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] Syslog logging taking too much disk write?
> 
> I have a semi busy syslog server which logs about 40 or 50 lines per
> second. I noticed using dstat that it's taking a TON of disk write.
> In the megaBYTES per second range. This seems extremely high to me
> so I wrote a script to test it.
> 
> http://www.perturb.org/code/log_lines.pl
> 
> First download dstat, gkrellm, or iostat and look at what your
> normal disk write is per second. On my underused desktop it hovers
> around zero or maybe 25KB/s per second.
> 
> If I run my script and tell it to log 10 lines to syslog per second
> 
> perl log_lines.pl 10
> 
> the write usage spikes up to ~160KB/s. We're only writing about 740
> bytes of data per second so why is the disk write so high?
> 
> If I tell my script to instead log to a local file in /tmp at 10
> lines per second
> 
> perl log_lines.pl 10 -t
> 
> The write usage spikes to ~120KB/s about once every 5 seconds (as
> the buffer fills up). I'm curious why syslog writing is so much more
> intense? It seems to me on a really busy syslog server that you
> could seriously smoke your HDs with all the writing it's doing. Am I
> missing something?
> 
> --
> Scott Baker - Canby Telcom
> RHCE - System Administrator - 503.266.8253
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From bakers at web-ster.com  Thu Dec 20 19:09:06 2007
From: bakers at web-ster.com (Scott Baker)
Date: Thu, 20 Dec 2007 10:09:06 -0800
Subject: [rsyslog] Syslog logging taking too much disk write?
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086C1@grfint2.intern.adiscon.com>
References: <476AA0A1.5080300@web-ster.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086C1@grfint2.intern.adiscon.com>
Message-ID: <476AAFC2.207@web-ster.com>

Rainer Gerhards wrote:
> Could you post your rsyslog.conf please. Without seeing it, it sounds
> like you sync the disc with every write. Unfortunately, this is the
> legacy default. You can turn it off with placing a dash before the
> selector line:
> 
> e.g. *.* /var/log/whatever
> 
> syncs after each write, while
> 
> *.* -/var/log/whatever
> 
> uses buffering - which should be fairly safe with decent hardware and a
> UPS...

I was using the stock Fedora 8 config which does indeed sync every
write to disk. I put a - in front of all the options and my disk
usage has gone way down!

What sort of buffer does rsyslog use? How often does that buffer
fill up and write to disk? I only ask because sometimes we tail
logfiles in real time to watch things like DHCP etc as they happen.

If the log is 30 seconds behind that may cause a problem.

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From bakers at web-ster.com  Thu Dec 20 19:34:06 2007
From: bakers at web-ster.com (Scott Baker)
Date: Thu, 20 Dec 2007 10:34:06 -0800
Subject: [rsyslog] Port 514
Message-ID: <476AB59E.9050706@web-ster.com>

> Hi all,
> 
> I'd appreciate your feedback on this question.
> 
> Historically, syslog messages were not only received on port 514, but
> were also send from that port. This is an artifact from the fact that
> early syslogd used a single port both for sending and receiving. So far,
> I have maintained that and done so with some pain. With v3, I would like
> to get rid of this legacy (because it introduces inter-module
> dependencies and also makes multiple listeners really complicated).
> 
> So I wonder if anyone over here finds this "feature" (syslog messages
> are sent from port 514 by default) important. What pops up my mind are
> firewall rules. Is there anything else broken if we change that?

We use syslog pretty extensively here at the ISP I work for and I
don't see how this would affect anything we do. Firewall rules are
almost always bast on destination port, not sending port. I don't
think changing this would cause any problems.

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Thu Dec 20 21:29:53 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 20 Dec 2007 21:29:53 +0100
Subject: [rsyslog] Syslog logging taking too much disk write?
In-Reply-To: <476AAFC2.207@web-ster.com>
References: <476AA0A1.5080300@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C1@grfint2.intern.adiscon.com>
	<476AAFC2.207@web-ster.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086C6@grfint2.intern.adiscon.com>

Hi Scott, 

[snip]

> What sort of buffer does rsyslog use? How often does that buffer
> fill up and write to disk? I only ask because sometimes we tail
> logfiles in real time to watch things like DHCP etc as they happen.
> 
> If the log is 30 seconds behind that may cause a problem.

I suggest you first have a look at:

http://rgerhards.blogspot.com/2007/08/why-is-rsyslog-multi-threaded-and-
is-it.html

The bottom line with the current release is that there is no specific
"output file buffer" - rsyslog writes to the OS and it buffers. Works
pretty well for rapidly incoming traffic. What we do, however, is have a
buffer between messages received and processed. This is primarily meant
for taking care of either a) message bursts or b) temporarily slow
outputs (e.g., initial database connection). If you don't like that
buffer, you can limit it via rsyslog.conf (the default is 10,000
messages max).

With rsyslog v3, there will be a more complex threading scheme and also
some more buffers. I can also envision it will support buffered writes,
enhancing the IO throughput for files (e.g. write in 64K chunks or every
n (milli)seconds, whatever comes first). I am currently in the early
stages of v3, but it will be a very exciting release. And I of course
appreciate feedback from demanding environments (hint, hint ;)).

But v3 will never hold messages longer than a) absolutely necessary or
b) configured by operator. There is a good post on v3 right on the main
page of my blog.

I hope that clarifies, if not, please let me know.

Rainer


From bakers at web-ster.com  Thu Dec 20 22:29:39 2007
From: bakers at web-ster.com (Scott Baker)
Date: Thu, 20 Dec 2007 13:29:39 -0800
Subject: [rsyslog] Separating logs by host?
Message-ID: <476ADEC3.5040506@web-ster.com>

Is it possible with rsyslog to separate logs based on IP address and
not logging facility?

I have about 10 pieces of equipment that I'd like to get syslog
fromt, but I've run out of log facilities (local1 through local7).
It'd be nice if each of those pieces of equipment could have it's
own log. I can't find anything in the documentation that covers this.

Am I missing it?

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Thu Dec 20 22:33:46 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 20 Dec 2007 22:33:46 +0100
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <476ADEC3.5040506@web-ster.com>
References: <476ADEC3.5040506@web-ster.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>

Looks like doc can need some more guides ;)

Does this help: http://www.rsyslog.com/Article60.phtml

Rainer 

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Scott Baker
> Sent: Thursday, December 20, 2007 10:30 PM
> To: rsyslog-users
> Subject: [rsyslog] Separating logs by host?
> 
> Is it possible with rsyslog to separate logs based on IP address and
> not logging facility?
> 
> I have about 10 pieces of equipment that I'd like to get syslog
> fromt, but I've run out of log facilities (local1 through local7).
> It'd be nice if each of those pieces of equipment could have it's
> own log. I can't find anything in the documentation that covers this.
> 
> Am I missing it?
> 
> -- 
> Scott Baker - Canby Telcom
> RHCE - System Administrator - 503.266.8253
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> 


From rgerhards at hq.adiscon.com  Thu Dec 20 22:45:06 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 20 Dec 2007 22:45:06 +0100
Subject: [rsyslog] backward compatibility...
In-Reply-To: <aa76cdac0712200108q7e9eb058g2e179c2fe00f65d0@mail.gmail.com>
References: <577465F99B41C842AAFBE9ED71E70ABA308691@grfint2.intern.adiscon.com><aa76cdac0712182051m2ef76c2sb401e0bd45904382@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA3086B1@grfint2.intern.adiscon.com>
	<aa76cdac0712200108q7e9eb058g2e179c2fe00f65d0@mail.gmail.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086C9@grfint2.intern.adiscon.com>

Hi Michael,

as promised, a more complete response. See inline below... 

> > I took me a while to craft a response to your excellent 
> question. I have
> > done this as a blog post so that it is easier to reference it in the
> > future.
> >
> > I suggest that everyone interested in the v3 design has a 
> look at it,
> > because it describes the way I am heading. If someone 
> doesn't like that,
> > it is now time to speak up - in a few weeks it will probably be
> > impossible to change routes.
> 
> Based on your recent blog post here are some thoughts of mine. Please
> keep in mind, that being (Debian) package maintainer, so I speak from
> a distributors pov.
> 
> 1.) external dependencies
> Having loadable modules for stuff like MySQL/PostgreSQL output support
> is great (for a package maintainer)
> This means I can package a basic rsyslog package with minimal
> dependencies (glibc) and people can chose to install
> rsyslog-(mysql,pgsql) based on their needs.
> Here in this case, loadable modules have a real benefit.
> The only remaining external dependencies currently in rsyslog are libz
> (NETZIP support) and gssapi-krb (KERBEROS support). If those
> functionality would be put into a loadable module, I'd support that.
> 2.) memory usage
> I don't think this is a real issue, even for embedded systems. See
> point 1. If we manage to put external dependencies into loadable
> modules, that would be sufficient imho.
> Modularising everything won't have that much of an memory benefit
> imho. Given that a standard rsyslogd is about 1M RSS, this doesn't
> imply the need for more modularization.

I tend to agree here with you. However, I think that a clear line needs
to be drawn between "modularization" and "loadable modules". Probably I
did cross this line myself a couple of times without even noticing.

The modularization effort *is* needed, because the current monolythic
code base is hard to maintain and has "evolved" - in contrast to be
designed. While this is not bad, you notice in many instants that the
underlaying design is no longer appropriate to do a number of things. In
order to add more really useful features, the code must be modularized.
This will reduce complexity and hopefully make it less subject to
programming error (which, of course, do still happen). [That doesn't
mean I am about to rewrite everything, more on that below.]

So the bottom line is we need modularization for new features. It also
helps greatly with portability, as I outlined in my other mail.

However, we do NOT necessarily need loadble modules - but so far, I find
them desirable. Either way, it seems to be very important to
differentiate between these two concepts.

> 3.) Security
> You mentioned, that you try to improve security through modules.
> Usually, having loadable module support is considered a security risk.

Sure - but it is the the ability to have loadable modules. This risk
does not scale with the number of modules. The problem is that the
interface can be attacked. It's only safe if you do not offer that
functionality at all. At least this is my point of view.

> One messed up $IncludeConfig directive (or manipulated through a
> malicious attacker), and you load potentially hazardous modules.
> Loadable modules support introduces a much bigger attack vector.

Agreed - but not by providing more than a view, as outlined above. The
module interface tries to mitigate against some risk. For example,
output modules never see the internal structure but only the resulting
strings. Of course, so far everything is in-process so a malicious
module may access whatever it wants. I have plans, howerver, to
optionally execute modules out of process. That would be a good option
for not well known third party modules. Obviously at the expense of
quite a performance hit...

> 
> I'm not a SELinux guy. But I'd be interested if loadable modules could
> cause trouble in putting rsyslog in it's own security domain. Maybe
> the fedora guys can comment on this.

I am not qualified to talk about this, but I think along the lines that
Peter posted...

> 
> 4.) (code) modularization
> Writing modular code doesn't strictly imply that the modules have to
> be loadable *.so files.
> You can still write modular code, with a strict API etc. and organize
> it e.g. via sub directories.

Fully agree.

> 5.) Performance penalty through loadable modules.
> I could be wrong on this point, but given that you have pointers on
> functions, when you use modules (dlsym), there is an additional
> indirection on each function call. This could have a performance
> impact for core functionality. This is just speculation and should be
> tested/evaluated beforehand. After all, rsyslog tries to meet
> high-performance needs, too.

No, that doesn't happen (by design). There is a very slight overhead
during module load, when interfaces are queried and symbols are fixed
up. Some indirection, of course, results from a generic approach, but
that is inevidable if the design is modular. With today's hardware, an
indirect function call is not as much of a problem. Again, that has
nothing to do with loable modules but with non-monolythic code.

> 6.) Inconvenience
> This is just a gut feeling, but having to load all sorts of moduls
> first, before rsyslog does anything, could prove cumbersome. As
> administrator you'd have to know, which modules to load, to get
> rsyslog to do what you want. This means additional effort (reading
> docs) and inconvenience.

That's why I said the "core modules" could be present in the default.
But more later below...

> 7.) Robustness
> Having a single binary can prove to be a live safer. E.g. you could
> simply copy the rsyslogd binary from another machine if there is
> something wrong with your local copy.

Agreed, but copying a directory with a number of executables shouldn't
be that much harder ;)

> As you were talking about embedded systems, having the ability to
> compile a static binary including all functionality is a definite
> plus. There might be platforms out there which don't support dlopen().
That's an excellent point!

> 
> I'm still missing the complete picture, too. It's still a bit 
> too vague for me.

That's probably a problem with what I can convey. As I wrote above, I am
not doing a rewrite, with a new design and a new code base. My goal is
to *migrate* rsyslog (which originally based on sysklogd code) over to a
new, better architecture. I try to make as few compromises as possible,
but every now and then there are some things that can't be ignored. My
backward-compatibility questions on this list are part of the process.
So while I have an idea of where I am heading and what I want to achive,
I do not have a crystal clear view of how that will actually happen. I
am going iteratively, tackeling one feature after another and feeding
experience gained back into the process. I am currently at the input
modules and there are definitely some lessons to learn ;)

> 
> You were talking about input, output and filter modules. Rainer, could
> cou try to complete this list somehow, maybe draw/sketch a flow
> diagram, marking the modules.
> 
> input: local, kernel and network, ...
> filter: regexp, ...
> output: mysql, pgsql, file, forward...
> authorization?

Let me give it a try:

Input: immark, imrfc3195, imklog, imfile, imudp, imtcp, imgssapi?,
imtls, imuxsock, ...

Filter: many (many!)... regexp, substr, tolower, toupper, dnslookup?,
left, right, mysqldate, pgsqldate, whateverdate, md5sum, ...

Output: ommysql, ompgsql, omusrmsg, omfwd (split into omudp, omtcp,
omgssapi, ...), ...

Authorization will be provided by the inputs... But there will be some
helpers. How this will happen is not yet clear to me - that's part of
the iterative process. Initially it will be with the modules and then
possibly be migrated to somewhere else.

I'll try to get you a drawing, but that's always a pain for me ;)

> 
> I hope, this all doesn't sound too negative. 

No, much appreciated. It is extremely useful to get that kind of
feedback.

> But before going all
> modular, all these issues should be considered imho.
> 
> Hopefully these comments will help.

After going through all of this, I begin to think that libtool probably
has the answer. I don't know exactly how to, but I think it can take
loadbale modules and I can tell it to make them into static modules. If
that's the case, I can create loable modules, but the build process will
change that into static linking. If I am wrong with that, I can mangle a
bit with the main entry point names in the modules and achive the same
result with just changes to the build process (I don't know yet how to
modify autotools to do that, but I hope there will be some helping hands
;)).

How does this sound?

Rainer
PS: please keep the comments coming, especially if you don't agree! 


From bakers at web-ster.com  Thu Dec 20 22:46:33 2007
From: bakers at web-ster.com (Scott Baker)
Date: Thu, 20 Dec 2007 13:46:33 -0800
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>
References: <476ADEC3.5040506@web-ster.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>
Message-ID: <476AE2B9.30002@web-ster.com>

Rainer Gerhards wrote:
> Looks like doc can need some more guides ;)
> 
> Does this help: http://www.rsyslog.com/Article60.phtml

Excellent... doing this causes all messages to be logged in that
format. Even local messages from the box (cron, mail, authpriv, etc)
to go into that format.

Can I apply that logging statement to a group of IPs? I have a bunch
of DSL equipment that logs. It would be nice if I could break all my
equipment out like:

Routers: /var/log/routers-%$NOW%.log
DSL: /var/log/dsl-%$NOW%.log
Dial-up: /var/log/dial--%$NOW%.log

Each/Any of those groups could have one or many sources. We have
about 13 unique DSL devices that should all probably go in the same
log for simplicity. It would be REALLY cool to be able to say
something like:

192.168.1.1/24:*.*                    /var/log/dsl-%$NOW%.log
192.168.3.1/24,10.1.1.1/24:*.*        /var/log/routers-%$NOW%.log

Am I just dreaming or is that possible?

Also why is the next version of rsyslog (after reading your blog)
going to be version 3 and not 2.0?

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Thu Dec 20 22:50:11 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 20 Dec 2007 22:50:11 +0100
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <476AE2B9.30002@web-ster.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>
	<476AE2B9.30002@web-ster.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>

I need to think about the real question a bit ;) Not an immediate
solution ... but not hopeless either ;)

On v2/v3 - v2 will be the next stable release:
http://rgerhards.blogspot.com/2007/12/begun-working-on-rsyslog-v3.html 

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Scott Baker
> Sent: Thursday, December 20, 2007 10:47 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Separating logs by host?
> 
> Rainer Gerhards wrote:
> > Looks like doc can need some more guides ;)
> > 
> > Does this help: http://www.rsyslog.com/Article60.phtml
> 
> Excellent... doing this causes all messages to be logged in that
> format. Even local messages from the box (cron, mail, authpriv, etc)
> to go into that format.
> 
> Can I apply that logging statement to a group of IPs? I have a bunch
> of DSL equipment that logs. It would be nice if I could break all my
> equipment out like:
> 
> Routers: /var/log/routers-%$NOW%.log
> DSL: /var/log/dsl-%$NOW%.log
> Dial-up: /var/log/dial--%$NOW%.log
> 
> Each/Any of those groups could have one or many sources. We have
> about 13 unique DSL devices that should all probably go in the same
> log for simplicity. It would be REALLY cool to be able to say
> something like:
> 
> 192.168.1.1/24:*.*                    /var/log/dsl-%$NOW%.log
> 192.168.3.1/24,10.1.1.1/24:*.*        /var/log/routers-%$NOW%.log
> 
> Am I just dreaming or is that possible?
> 
> Also why is the next version of rsyslog (after reading your blog)
> going to be version 3 and not 2.0?
> 
> -- 
> Scott Baker - Canby Telcom
> RHCE - System Administrator - 503.266.8253
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> 


From bakers at web-ster.com  Fri Dec 21 01:23:07 2007
From: bakers at web-ster.com (Scott Baker)
Date: Thu, 20 Dec 2007 16:23:07 -0800
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>	<476AE2B9.30002@web-ster.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>
Message-ID: <476B076B.7090003@web-ster.com>

Rainer Gerhards wrote:
> I need to think about the real question a bit ;) Not an immediate
> solution ... but not hopeless either ;)
> 
> On v2/v3 - v2 will be the next stable release:
> http://rgerhards.blogspot.com/2007/12/begun-working-on-rsyslog-v3.html 

Another good example is that we have several linux mail servers. I
like to aggregate all of their logging to one rsyslog server. They
all log to mail.* on their local machines, and then forward also to
my rsyslog server.

The problem is that shows up in my mail.* on the syslog server, and
thus gets mixed in with the local mail.* from the syslog server
itself. It would be nice to be able to separate those, by IP would
be the easiest.

mail.*				/var/log/maillog
1.2.3.4/32:mail.*		/var/log/mail-server.log

Or something like that...

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From r.bhatia at ipax.at  Fri Dec 21 09:03:04 2007
From: r.bhatia at ipax.at (Raoul Bhatia [IPAX])
Date: Fri, 21 Dec 2007 09:03:04 +0100
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <476B076B.7090003@web-ster.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>	<476AE2B9.30002@web-ster.com>	<577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>
	<476B076B.7090003@web-ster.com>
Message-ID: <476B7338.8050102@ipax.at>

Scott Baker wrote:
> Rainer Gerhards wrote:
>> I need to think about the real question a bit ;) Not an immediate
>> solution ... but not hopeless either ;)
>>
>> On v2/v3 - v2 will be the next stable release:
>> http://rgerhards.blogspot.com/2007/12/begun-working-on-rsyslog-v3.html 
> 
> Another good example is that we have several linux mail servers. I
> like to aggregate all of their logging to one rsyslog server. They
> all log to mail.* on their local machines, and then forward also to
> my rsyslog server.
> 
> The problem is that shows up in my mail.* on the syslog server, and
> thus gets mixed in with the local mail.* from the syslog server
> itself. It would be nice to be able to separate those, by IP would
> be the easiest.
> 
> mail.*				/var/log/maillog
> 1.2.3.4/32:mail.*		/var/log/mail-server.log
> 
> Or something like that...

might logging to an (my|pg)sql server solve all your issues?

cheers,
raoul
-- 
____________________________________________________________________
DI (FH) Raoul Bhatia M.Sc.          email.          r.bhatia at ipax.at
Technischer Leiter

IPAX - Aloy Bhatia Hava OEG         web.          http://www.ipax.at
Barawitzkagasse 10/2/2/11           email.            office at ipax.at
1190 Wien                           tel.               +43 1 3670030
FN 277995t HG Wien                  fax.            +43 1 3670030 15
____________________________________________________________________


From rgerhards at hq.adiscon.com  Fri Dec 21 11:07:36 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 21 Dec 2007 11:07:36 +0100
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <476B076B.7090003@web-ster.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>	<476AE2B9.30002@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>
	<476B076B.7090003@web-ster.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086D0@grfint2.intern.adiscon.com>

Mmhhh... once we have expression support, that will be an easy thing to
do. Currently, I think there are a number of clumpsy work-arounds. We
are around two month away from expressions, at least if all goes well.

I think you can achive something with BDS-style filter blocks
(!progname/-host/+host), something along these lines:

-host
mail.*	/var/log/maillog
+hostname
mail.*	/var/log/mail-server.log

Of course, this is hostname specific (and that again is not the IP but
what is in the message...)

To get to the IP, I think you can get at least some way with class A,B,C
addresses, but NOT with any other masks. Here is the idea:

$template dsl, /var/log/dsl-%$NOW%.log
$template routers, /var/log/routers-%$NOW%.log
:FROMHOST, startswith, "192.168.1."   -?dsl
:FROMHOST, startswith, "192.168.3."   -?routers
:FROMHOST, startswith, "10.1.1."      -?routers

Common pitfall: be sure to include the trailing dot in the condition to
match. If the rule were

:FROMHOST, startswith, "192.168.1"   -?dsl

It would match 192.168.1.1.2, 192.168.1.1.2 but also 192.168.1.11.2!

With the current engine, there are unfortunately no logical operations
available. So you cannot check for mail facility or whatever else. A
somewhat crude work-around would be to include the facility in the file
name, e.g. by specifying it as follows:

$template dsl, /var/log/dsl-%syslogfacility-text%-%$NOW%.log

That, of course, will result in a file written for each facility, even
those that you are not interested in. A unfortunately do not have a
solution for this now.

With v3 expressions, I envision something along these lines:

If maskmatch(FROMHOST, "192.168.1.0", 24) and syslog-facility-text ==
"mail" then 
    writefile "/var/log/dslmail-%$NOW%"

But that is not possible yet. And this is only an idea, not the actual
config file format we will have at that time. In my blog, there are a
number of posts about it, but the bottom line is that it is not yet
designed:

http://rgerhards.blogspot.com/2007/08/on-rsyslog-config-file-format.html

Doc about current capabilities is here:
 
http://www.rsyslog.com/module-Static_Docs-view-f-rsyslog_conf.html.phtml

search for "Filter Conditions" on that page.

The properties you need are documented here:
 
http://www.rsyslog.com/module-Static_Docs-view-f-property_replacer.html.
phtml

Keep in mind they ARE CASE-SENSITIVE! (and don't ask me why I had that
idea... ;))

As always, feedback is appreciated. I would be most interested to learn
at which final config you arrived.

Thanks,
Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Scott Baker
> Sent: Friday, December 21, 2007 1:23 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Separating logs by host?
> 
> Rainer Gerhards wrote:
> > I need to think about the real question a bit ;) Not an immediate
> > solution ... but not hopeless either ;)
> >
> > On v2/v3 - v2 will be the next stable release:
> > http://rgerhards.blogspot.com/2007/12/begun-working-on-rsyslog-
> v3.html
> 
> Another good example is that we have several linux mail servers. I
> like to aggregate all of their logging to one rsyslog server. They
> all log to mail.* on their local machines, and then forward also to
> my rsyslog server.
> 
> The problem is that shows up in my mail.* on the syslog server, and
> thus gets mixed in with the local mail.* from the syslog server
> itself. It would be nice to be able to separate those, by IP would
> be the easiest.
> 
> mail.*				/var/log/maillog
> 1.2.3.4/32:mail.*		/var/log/mail-server.log
> 
> Or something like that...
> 
> --
> Scott Baker - Canby Telcom
> RHCE - System Administrator - 503.266.8253
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Fri Dec 21 11:39:32 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 21 Dec 2007 11:39:32 +0100
Subject: [rsyslog] Very Early Preview of rsyslog 3.0.0
Message-ID: <1198233572.2492.16.camel@localhost.localdomain>

Hi all,

I have just reached a stage where everything in rsyslog 3.0.0 is a bit
stable. I thought it would be good to release a preview before I begin
to pull some other code and place it in its own module. There are
already a number of input plugins, so I think looking at that code
provides a very good idea of where we are heading.

Please note that there is a doc/v3compatibility.html document. It tells
about know compatibility issues with pre-v3 versions and other work that
is currently in progress. At least, as far as I wrote it down ;)

The tarball is available at

http://download.rsyslog.com/rsyslog/rsyslog-3.0.0-pre.tar.gz
md5sum: e9c807decf8acbb1b2697ec907d6af08

Please note that this tarball is a snapshot. I do not plan to maintain
it, development simply continues. I suggest NOT to use that version in
production. If you run it and find a bug, please report it. Also, any
other feedback will be used during future development.

Rainer


From bakers at web-ster.com  Fri Dec 21 17:48:44 2007
From: bakers at web-ster.com (Scott Baker)
Date: Fri, 21 Dec 2007 08:48:44 -0800
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086D0@grfint2.intern.adiscon.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>	<476AE2B9.30002@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>	<476B076B.7090003@web-ster.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086D0@grfint2.intern.adiscon.com>
Message-ID: <476BEE6C.50109@web-ster.com>

Rainer Gerhards wrote:
> Mmhhh... once we have expression support, that will be an easy thing to
> do. Currently, I think there are a number of clumpsy work-arounds. We
> are around two month away from expressions, at least if all goes well.
> 
> I think you can achive something with BDS-style filter blocks
> (!progname/-host/+host), something along these lines:
> 
> -host
> mail.*	/var/log/maillog
> +hostname
> mail.*	/var/log/mail-server.log
> 
> Of course, this is hostname specific (and that again is not the IP but
> what is in the message...)
> 
> To get to the IP, I think you can get at least some way with class A,B,C
> addresses, but NOT with any other masks. Here is the idea:
> 
> $template dsl, /var/log/dsl-%$NOW%.log
> $template routers, /var/log/routers-%$NOW%.log
> :FROMHOST, startswith, "192.168.1."   -?dsl
> :FROMHOST, startswith, "192.168.3."   -?routers
> :FROMHOST, startswith, "10.1.1."      -?routers
> 
> Common pitfall: be sure to include the trailing dot in the condition to
> match. If the rule were
> 
> :FROMHOST, startswith, "192.168.1"   -?dsl
> 
> It would match 192.168.1.1.2, 192.168.1.1.2 but also 192.168.1.11.2!
> 
> With the current engine, there are unfortunately no logical operations
> available. So you cannot check for mail facility or whatever else. A
> somewhat crude work-around would be to include the facility in the file
> name, e.g. by specifying it as follows:
> 
> $template dsl, /var/log/dsl-%syslogfacility-text%-%$NOW%.log
> 
> That, of course, will result in a file written for each facility, even
> those that you are not interested in. A unfortunately do not have a
> solution for this now.
> 
> With v3 expressions, I envision something along these lines:
> 
> If maskmatch(FROMHOST, "192.168.1.0", 24) and syslog-facility-text ==
> "mail" then 
>     writefile "/var/log/dslmail-%$NOW%"
> 
> But that is not possible yet. And this is only an idea, not the actual
> config file format we will have at that time. In my blog, there are a
> number of posts about it, but the bottom line is that it is not yet
> designed:
> 
> http://rgerhards.blogspot.com/2007/08/on-rsyslog-config-file-format.html
> 
> Doc about current capabilities is here:
>  
> http://www.rsyslog.com/module-Static_Docs-view-f-rsyslog_conf.html.phtml
> 
> search for "Filter Conditions" on that page.
> 
> The properties you need are documented here:
>  
> http://www.rsyslog.com/module-Static_Docs-view-f-property_replacer.html.
> phtml
> 
> Keep in mind they ARE CASE-SENSITIVE! (and don't ask me why I had that
> idea... ;))
> 
> As always, feedback is appreciated. I would be most interested to learn
> at which final config you arrived.

Awesome! This is TOTALLY workable for what we're doing. I'm trying
to set it up and getting a weird message:

------------------------------------------------------------------

Dec 21 08:43:17 green rsyslogd:error: extra characters in config
line ignored: '/var/log/cisco-dsl-%$NOW%.log'
Dec 21 08:43:17 green rsyslogd: Could not find template 'dsl' -
action disabled
Dec 21 08:43:17 green rsyslogd:the last error occured in
/etc/rsyslog.conf, line 37

------------------------------------------------------------------

The config I used it almost exactly what you provided...

$template dsl, /var/log/cisco-dsl-%$NOW%.log
:FROMHOST, startswith, "10.3."           -?dsl

Am I missing something basic?

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Fri Dec 21 17:53:32 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 21 Dec 2007 17:53:32 +0100
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <476BEE6C.50109@web-ster.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>	<476AE2B9.30002@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>	<476B076B.7090003@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086D0@grfint2.intern.adiscon.com>
	<476BEE6C.50109@web-ster.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086D8@grfint2.intern.adiscon.com>

[sniiiip]
> The config I used it almost exactly what you provided...
> 
> $template dsl, /var/log/cisco-dsl-%$NOW%.log
> :FROMHOST, startswith, "10.3."           -?dsl
> 
> Am I missing something basic?

I forgot the quotes around the template string ;) That should work (not
tested, though):

$template dsl, "/var/log/cisco-dsl-%$NOW%.log"
:FROMHOST, startswith, "10.3."           -?dsl

Rainer


From bakers at web-ster.com  Fri Dec 21 18:55:36 2007
From: bakers at web-ster.com (Scott Baker)
Date: Fri, 21 Dec 2007 09:55:36 -0800
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086D8@grfint2.intern.adiscon.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>	<476AE2B9.30002@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>	<476B076B.7090003@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086D0@grfint2.intern.adiscon.com>	<476BEE6C.50109@web-ster.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086D8@grfint2.intern.adiscon.com>
Message-ID: <476BFE18.4000008@web-ster.com>

Rainer Gerhards wrote:
> I forgot the quotes around the template string ;) That should work (not
> tested, though):
> 
> $template dsl, "/var/log/cisco-dsl-%$NOW%.log"
> :FROMHOST, startswith, "10.3."           -?dsl

That was it... now rsyslog starts without errors.

Howeever it's not logging to the file I have set to though, instead
it's going to /var/log/messages. I've verified with tcpdump that
it's sending from the correct IP address. I've tried both startswith
and isequal and neither seem to work.

I don't know how to troubleshoot any further? Is there some debug
mode or something that will give me more information about why it's
not logging the way I think it should?

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253



From rgerhards at hq.adiscon.com  Fri Dec 21 18:58:49 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 21 Dec 2007 18:58:49 +0100
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <476BFE18.4000008@web-ster.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>	<476AE2B9.30002@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>	<476B076B.7090003@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086D0@grfint2.intern.adiscon.com>	<476BEE6C.50109@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086D8@grfint2.intern.adiscon.com>
	<476BFE18.4000008@web-ster.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086D9@grfint2.intern.adiscon.com>

Debug mode is the keyword here ;)

Add -d -n to the command line and run it interactively with stdout
redirected to a file. Maybe I've made a mistake with the directives,
maybe we have another issue. Debug output can get quite large, please do
not send it to the list (I think mailman will bounce it anyhow). You can
email me privately (rgerhards at adiscon.com) if you like.

Just be warned: its 7p right now here and I can not promise to fix a
larger problem before xmas ;) 

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Scott Baker
> Sent: Friday, December 21, 2007 6:56 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Separating logs by host?
> 
> Rainer Gerhards wrote:
> > I forgot the quotes around the template string ;) That should work
> (not
> > tested, though):
> >
> > $template dsl, "/var/log/cisco-dsl-%$NOW%.log"
> > :FROMHOST, startswith, "10.3."           -?dsl
> 
> That was it... now rsyslog starts without errors.
> 
> Howeever it's not logging to the file I have set to though, instead
> it's going to /var/log/messages. I've verified with tcpdump that
> it's sending from the correct IP address. I've tried both startswith
> and isequal and neither seem to work.
> 
> I don't know how to troubleshoot any further? Is there some debug
> mode or something that will give me more information about why it's
> not logging the way I think it should?
> 
> --
> Scott Baker - Canby Telcom
> RHCE - System Administrator - 503.266.8253
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From bakers at web-ster.com  Fri Dec 21 19:13:33 2007
From: bakers at web-ster.com (Scott Baker)
Date: Fri, 21 Dec 2007 10:13:33 -0800
Subject: [rsyslog] Separating logs by host?
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086D9@grfint2.intern.adiscon.com>
References: <476ADEC3.5040506@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086C8@grfint2.intern.adiscon.com>	<476AE2B9.30002@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086CA@grfint2.intern.adiscon.com>	<476B076B.7090003@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086D0@grfint2.intern.adiscon.com>	<476BEE6C.50109@web-ster.com><577465F99B41C842AAFBE9ED71E70ABA3086D8@grfint2.intern.adiscon.com>	<476BFE18.4000008@web-ster.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086D9@grfint2.intern.adiscon.com>
Message-ID: <476C024D.30200@web-ster.com>

Rainer Gerhards wrote:
> Debug mode is the keyword here ;)
> 
> Add -d -n to the command line and run it interactively with stdout
> redirected to a file. Maybe I've made a mistake with the directives,
> maybe we have another issue. Debug output can get quite large, please do
> not send it to the list (I think mailman will bounce it anyhow). You can
> email me privately (rgerhards at adiscon.com) if you like.
> 
> Just be warned: its 7p right now here and I can not promise to fix a
> larger problem before xmas ;) 

You rock! Rsyslog Debug Mode = Good stuff! I was able to get it
working just fine with the BSD style selectors. The FROMHOST part is
still giving me grief though:

Filter: check for property 'FROMHOST' (value 'extirpate') isequal
'1.2.3.4': FALSE

So my question is... for the BSD style selectors and FROM it appears
that rsyslog is using just the hostname and not the FULL reverse dns
name: extirpate vs extirpate.web-ster.com. Is that by design? I
don't think it will be a problem but I could see where you would
want to separate.

mail.domain1.com
vs
mail.domain2.com

Anyway, FROMHOST is using the name and not the IP address to match
against. Is there another property I should use? I don't see
anything in the docs... In reality I could match about the hostname
with a regexp since all my DSL is named the same...

Again IP would be ideal and easier...

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Fri Dec 21 19:20:07 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 21 Dec 2007 19:20:07 +0100
Subject: [rsyslog] Separating logs by host?
Message-ID: <002e01c843fe$24521c22$060013ac@intern.adiscon.com>

Just out of my head and from the pda while being out of office: try the -x option. Will check mail later again :)

----- Urspr?ngliche Nachricht -----
Von: "Scott Baker" <bakers at web-ster.com>
An: "rsyslog-users" <rsyslog at lists.adiscon.com>
Gesendet: 21.12.07 19:13
Betreff: Re: [rsyslog] Separating logs by host?

Rainer Gerhards wrote:
> Debug mode is the keyword here ;)
> 
> Add -d -n to the command line and run it interactively with stdout
> redirected to a file. Maybe I've made a mistake with the directives,
> maybe we have another issue. Debug output can get quite large, please do
> not send it to the list (I think mailman will bounce it anyhow). You can
> email me privately (rgerhards at adiscon.com) if you like.
> 
> Just be warned: its 7p right now here and I can not promise to fix a
> larger problem before xmas ;) 

You rock! Rsyslog Debug Mode = Good stuff! I was able to get it
working just fine with the BSD style selectors. The FROMHOST part is
still giving me grief though:

Filter: check for property 'FROMHOST' (value 'extirpate') isequal
'1.2.3.4': FALSE

So my question is... for the BSD style selectors and FROM it appears
that rsyslog is using just the hostname and not the FULL reverse dns
name: extirpate vs extirpate.web-ster.com. Is that by design? I
don't think it will be a problem but I could see where you would
want to separate.

mail.domain1.com
vs
mail.domain2.com

Anyway, FROMHOST is using the name and not the IP address to match
against. Is there another property I should use? I don't see
anything in the docs... In reality I could match about the hostname
with a regexp since all my DSL is named the same...

Again IP would be ideal and easier...

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog


From bakers at web-ster.com  Sat Dec 22 01:13:01 2007
From: bakers at web-ster.com (Scott Baker)
Date: Fri, 21 Dec 2007 16:13:01 -0800
Subject: [rsyslog] Rsyslog is awesome
Message-ID: <476C568D.9050807@web-ster.com>

Rainer:

I spent the better part of today setting up my centralized rsyslog
today. It's logging for about 20 different hosts now and working well.

I do have some questions about hostname matching DNS versus hosts
files. I think I may have found a crash bug too. I had a host that
was rsyslog was logging from without reverse DNS. As soon as I would
add the reverse DNS record rsyslog would crash. As if that source
name changing caused rsyslog to get confused.

I'm very happy to go over this in more detail with you when you have
some more time. With Christmas coming up I'm sure we'll touch base
after. Overall I'm very impressed with rsyslog. Good work!

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From mbiebl at gmail.com  Sun Dec 23 15:46:26 2007
From: mbiebl at gmail.com (Michael Biebl)
Date: Sun, 23 Dec 2007 15:46:26 +0100
Subject: [rsyslog] Very Early Preview of rsyslog 3.0.0
In-Reply-To: <1198233572.2492.16.camel@localhost.localdomain>
References: <1198233572.2492.16.camel@localhost.localdomain>
Message-ID: <aa76cdac0712230646r273b2a1aj13fec13695a3013f@mail.gmail.com>

2007/12/21, Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Hi all,
>
> I have just reached a stage where everything in rsyslog 3.0.0 is a bit
> stable. I thought it would be good to release a preview before I begin
> to pull some other code and place it in its own module. There are
> already a number of input plugins, so I think looking at that code
> provides a very good idea of where we are heading.
>
> Please note that there is a doc/v3compatibility.html document. It tells
> about know compatibility issues with pre-v3 versions and other work that
> is currently in progress. At least, as far as I wrote it down ;)

I tested imuxsock and imklog (yay for getting rid of rklogd) and it
seems to work fine so far.
Nice work.

Michael
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Sun Dec 23 18:20:00 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Sun, 23 Dec 2007 18:20:00 +0100
Subject: [rsyslog] rsyslog 1.21.1 released
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086DE@grfint2.intern.adiscon.com>

Dear All,

I have just released rsyslog 1.21.1. It is a maintenance release to
1.21.0. Most importantly, it fixes a memory leak introduced in 1.21.0.
There are also some other bugfixes present. Users of 1.21.0 should
upgrade to this release.

If you run 1.21.0 and experience(d) the segfault previously, please let
me know your results with 1.21.0 before upgrading. This would be most
useful for me.

Download and Changelog can be found on the usual places of
www.rsyslog.com.

I would also like to wish a great holiday season to everyone! Thanks for
your interest in rsyslog and all the good feedback provided.

All the best,
Rainer


From bakers at web-ster.com  Mon Dec 24 17:33:11 2007
From: bakers at web-ster.com (Scott Baker)
Date: Mon, 24 Dec 2007 08:33:11 -0800
Subject: [rsyslog] rsyslog 1.21.1 released
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086DE@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA3086DE@grfint2.intern.adiscon.com>
Message-ID: <476FDF47.1060607@web-ster.com>

Rainer Gerhards wrote:
> Dear All,
> 
> I have just released rsyslog 1.21.1. It is a maintenance release to
> 1.21.0. Most importantly, it fixes a memory leak introduced in 1.21.0.
> There are also some other bugfixes present. Users of 1.21.0 should
> upgrade to this release.
> 
> If you run 1.21.0 and experience(d) the segfault previously, please let
> me know your results with 1.21.0 before upgrading. This would be most
> useful for me.
> 
> Download and Changelog can be found on the usual places of
> www.rsyslog.com.
> 
> I would also like to wish a great holiday season to everyone! Thanks for
> your interest in rsyslog and all the good feedback provided.

Rainer:

I just got to work and I checked my rsyslog server today and it's
running happily. After upgrading from 1.19.0 to 1.21.0 I didn't have
those segfaults that were happening about every 3 hours. In fact
rsyslog logged just over 1.3g of logs in 3 days.

I'll update to 1.21.1 right now and we'll see how she does.

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From bakers at web-ster.com  Mon Dec 24 17:57:02 2007
From: bakers at web-ster.com (Scott Baker)
Date: Mon, 24 Dec 2007 08:57:02 -0800
Subject: [rsyslog] BSD Style Selectors
Message-ID: <476FE4DE.1020201@web-ster.com>

It's been my experience that if I have a config like:

+server1
# The authpriv file has restricted access.
authpriv.*                          -/var/log/secure

# Log all the mail messages in one place.
mail.*                       -/var/log/maillog

# Log cron stuff
cron.*                        -/var/log/cron

-----------------------------------------------------

That +server1 setting applies to everything below it until it's
changed by another BSD style selector? In other words if server2
sent cron.* syslog in this config it would not log that in
/var/log/cron ??

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Mon Dec 24 18:35:52 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 24 Dec 2007 18:35:52 +0100
Subject: [rsyslog] BSD Style Selectors
In-Reply-To: <476FE4DE.1020201@web-ster.com>
References: <476FE4DE.1020201@web-ster.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086E1@grfint2.intern.adiscon.com>

Yes, that's right - and: merry xmas :)

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Scott Baker
> Sent: Monday, December 24, 2007 5:57 PM
> To: rsyslog-users
> Subject: [rsyslog] BSD Style Selectors
> 
> It's been my experience that if I have a config like:
> 
> +server1
> # The authpriv file has restricted access.
> authpriv.*                          -/var/log/secure
> 
> # Log all the mail messages in one place.
> mail.*                       -/var/log/maillog
> 
> # Log cron stuff
> cron.*                        -/var/log/cron
> 
> -----------------------------------------------------
> 
> That +server1 setting applies to everything below it until it's
> changed by another BSD style selector? In other words if server2
> sent cron.* syslog in this config it would not log that in
> /var/log/cron ??
> 
> --
> Scott Baker - Canby Telcom
> RHCE - System Administrator - 503.266.8253
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog


From bakers at web-ster.com  Mon Dec 24 19:10:35 2007
From: bakers at web-ster.com (Scott Baker)
Date: Mon, 24 Dec 2007 10:10:35 -0800
Subject: [rsyslog] Hostname matching with DNS
Message-ID: <476FF61B.3020006@web-ster.com>

I have a couple host on private IPs 10.x.x.x and thus they have no
DNS entries. So rather that log the IP in syslog I setup host
entries for them.

If I do something like

:FROMHOST, isequal, "foobar"                -?dialup

it doesn't match the /etc/hosts entry I have for foobar. If I setup
a catchall entry that goes to a test log I see the line

Dec 24 10:06:23 foobar [This is the message]

So it's logging the hostname like I would expect it to (rsyslog is
aware of the host entry) but I can't match against it? Unfortunately
my server is SUPER busy now and I can't put the server in debug mode
to check what's coming across. Is there another way I could check this?

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Mon Dec 24 19:59:18 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 24 Dec 2007 19:59:18 +0100
Subject: [rsyslog] Hostname matching with DNS
Message-ID: <002f01c8465f$209fa2ec$060013ac@intern.adiscon.com>

Really quick: check the HOSTNAME (or so ;)) property. 

----- Urspr?ngliche Nachricht -----
Von: "Scott Baker" <bakers at web-ster.com>
An: "rsyslog-users" <rsyslog at lists.adiscon.com>
Gesendet: 24.12.07 19:10
Betreff: [rsyslog] Hostname matching with DNS

I have a couple host on private IPs 10.x.x.x and thus they have no
DNS entries. So rather that log the IP in syslog I setup host
entries for them.

If I do something like

:FROMHOST, isequal, "foobar"                -?dialup

it doesn't match the /etc/hosts entry I have for foobar. If I setup
a catchall entry that goes to a test log I see the line

Dec 24 10:06:23 foobar [This is the message]

So it's logging the hostname like I would expect it to (rsyslog is
aware of the host entry) but I can't match against it? Unfortunately
my server is SUPER busy now and I can't put the server in debug mode
to check what's coming across. Is there another way I could check this?

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog


From rgerhards at hq.adiscon.com  Mon Dec 24 20:59:02 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 24 Dec 2007 20:59:02 +0100
Subject: [rsyslog] Hostname matching with DNS
In-Reply-To: <002f01c8465f$209fa2ec$060013ac@intern.adiscon.com>
References: <002f01c8465f$209fa2ec$060013ac@intern.adiscon.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086E2@grfint2.intern.adiscon.com>

Scott,

So now a bit more in-depth: the HOSTNAME is taken form the syslog message, while FROMHOST is the last hope. There is only a difference in relay scenarios - or, like here, based on DNS resolution. This is why you see different values. The point is to match against the same one that is used in the catchall rule.

However, I think the most appropriate thing to do is add a FROMHOST-IP property, which always has the IP address of the sender, no matter if the -x option is given or not.

Would that help?

Rainer 

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of 
> Rainer Gerhards
> Sent: Monday, December 24, 2007 7:59 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Hostname matching with DNS
> 
> Really quick: check the HOSTNAME (or so ;)) property. 
> 
> ----- Urspr?ngliche Nachricht -----
> Von: "Scott Baker" <bakers at web-ster.com>
> An: "rsyslog-users" <rsyslog at lists.adiscon.com>
> Gesendet: 24.12.07 19:10
> Betreff: [rsyslog] Hostname matching with DNS
> 
> I have a couple host on private IPs 10.x.x.x and thus they have no
> DNS entries. So rather that log the IP in syslog I setup host
> entries for them.
> 
> If I do something like
> 
> :FROMHOST, isequal, "foobar"                -?dialup
> 
> it doesn't match the /etc/hosts entry I have for foobar. If I setup
> a catchall entry that goes to a test log I see the line
> 
> Dec 24 10:06:23 foobar [This is the message]
> 
> So it's logging the hostname like I would expect it to (rsyslog is
> aware of the host entry) but I can't match against it? Unfortunately
> my server is SUPER busy now and I can't put the server in debug mode
> to check what's coming across. Is there another way I could 
> check this?
> 
> -- 
> Scott Baker - Canby Telcom
> RHCE - System Administrator - 503.266.8253
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> 


From rgerhards at hq.adiscon.com  Mon Dec 24 21:01:21 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 24 Dec 2007 21:01:21 +0100
Subject: [rsyslog] Hostname matching with DNS
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086E2@grfint2.intern.adiscon.com>
References: <002f01c8465f$209fa2ec$060013ac@intern.adiscon.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086E2@grfint2.intern.adiscon.com>
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086E3@grfint2.intern.adiscon.com>

> while FROMHOST is the last hope 


Lol... No, it is not my last *hope*, even though it may be for the time
being ;) That should have been "last hop". Nice typo ;)

Rainer


From bakers at web-ster.com  Mon Dec 24 21:06:48 2007
From: bakers at web-ster.com (Scott Baker)
Date: Mon, 24 Dec 2007 12:06:48 -0800
Subject: [rsyslog] Hostname matching with DNS
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086E2@grfint2.intern.adiscon.com>
References: <002f01c8465f$209fa2ec$060013ac@intern.adiscon.com>
	<577465F99B41C842AAFBE9ED71E70ABA3086E2@grfint2.intern.adiscon.com>
Message-ID: <47701158.9080506@web-ster.com>

Rainer Gerhards wrote:
> Scott,
> 
> So now a bit more in-depth: the HOSTNAME is taken form the syslog message, while FROMHOST is the last hope. There is only a difference in relay scenarios - or, like here, based on DNS resolution. This is why you see different values. The point is to match against the same one that is used in the catchall rule.
> 
> However, I think the most appropriate thing to do is add a FROMHOST-IP property, which always has the IP address of the sender, no matter if the -x option is given or not.
> 
> Would that help?

Ya I think a FROMHOST-IP property would be great. It'd make setting
up my rules easier. Tell me more about hosts resolution vs DNS
resolution?

I have some FROMHOST rules that work just fine, but as soon as I add
a hosts entry it stops logging. So somewhere it's using the host
entry but I can't figure out how/where or what precedence that has
over DNS.

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Mon Dec 24 21:23:58 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 24 Dec 2007 21:23:58 +0100
Subject: [rsyslog] Hostname matching with DNS
Message-ID: <003001c8466a$f111b332$060013ac@intern.adiscon.com>

Rsyslog doesn't do anything specific with the hosts file - it just uses the system resolver library and everything else is handled in there...

I'll have a look at hostname-ip as soon as i can.

rainer

----- Urspr?ngliche Nachricht -----
Von: "Scott Baker" <bakers at web-ster.com>
An: "rsyslog-users" <rsyslog at lists.adiscon.com>
Gesendet: 24.12.07 21:06
Betreff: Re: [rsyslog] Hostname matching with DNS

Rainer Gerhards wrote:
> Scott,
> 
> So now a bit more in-depth: the HOSTNAME is taken form the syslog message, while FROMHOST is the last hope. There is only a difference in relay scenarios - or, like here, based on DNS resolution. This is why you see different values. The point is to match against the same one that is used in the catchall rule.
> 
> However, I think the most appropriate thing to do is add a FROMHOST-IP property, which always has the IP address of the sender, no matter if the -x option is given or not.
> 
> Would that help?

Ya I think a FROMHOST-IP property would be great. It'd make setting
up my rules easier. Tell me more about hosts resolution vs DNS
resolution?

I have some FROMHOST rules that work just fine, but as soon as I add
a hosts entry it stops logging. So somewhere it's using the host
entry but I can't figure out how/where or what precedence that has
over DNS.

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog


From bakers at web-ster.com  Mon Dec 24 22:53:26 2007
From: bakers at web-ster.com (Scott Baker)
Date: Mon, 24 Dec 2007 13:53:26 -0800
Subject: [rsyslog] Log Permissions?
Message-ID: <47702A56.7030805@web-ster.com>

Is there a way to set the owner/group or read permissions of log
files created by rsyslog? I can't find much in the man pages.

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Mon Dec 24 23:06:52 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 24 Dec 2007 23:06:52 +0100
Subject: [rsyslog] Log Permissions?
Message-ID: <003101c84679$50a4fc86$060013ac@intern.adiscon.com>

Look at www.rsyslog.com/doc Not the man pages. It is too much for man (but the above url is in it ;)). Check the config directives, there are a number of settings for that.

rainer

----- Urspr?ngliche Nachricht -----
Von: "Scott Baker" <bakers at web-ster.com>
An: "rsyslog-users" <rsyslog at lists.adiscon.com>
Gesendet: 24.12.07 22:53
Betreff: [rsyslog] Log Permissions?

Is there a way to set the owner/group or read permissions of log
files created by rsyslog? I can't find much in the man pages.

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog


From janfrode at tanso.net  Tue Dec 25 00:34:02 2007
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Tue, 25 Dec 2007 00:34:02 +0100
Subject: [rsyslog] rsyslog 1.21.1 released
References: <577465F99B41C842AAFBE9ED71E70ABA3086DE@grfint2.intern.adiscon.com>
Message-ID: <slrnfn0gfa.j64.janfrode@lc4eb8045376502.ibm.com>

On 2007-12-23, Rainer Gerhards <rgerhards at hq.adiscon.com> wrote:
>
> If you run 1.21.0 and experience(d) the segfault previously, please let
> me know your results with 1.21.0 before upgrading. This would be most
> useful for me.

I upgraded to 1.21.0 december 19., and it's been up since (4h19m cputime, 
134h process uptime). No segfaults yet. It hasn't been eating any noticeable
amount of memory either, so I'll probably hold off on the upgrade a bit.



  -jf



From rgerhards at hq.adiscon.com  Tue Dec 25 10:21:07 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 25 Dec 2007 10:21:07 +0100
Subject: [rsyslog] rsyslog 1.21.1 released
Message-ID: <003201c846d7$84c7c4a0$060013ac@intern.adiscon.com>

Ah, thats good news! And, yes, if you don't see the mem leak it is probably better to hild off the update, so that we get some more information on the stability. The memleak occurs only when the internal message queue is full, which normally shouldn't happen. So there is a good chance you never see it...

Rainer

----- Urspr?ngliche Nachricht -----
Von: "Jan-Frode Myklebust" <janfrode at tanso.net>
An: "rsyslog at lists.adiscon.com" <rsyslog at lists.adiscon.com>
Gesendet: 25.12.07 00:34
Betreff: Re: [rsyslog] rsyslog 1.21.1 released

On 2007-12-23, Rainer Gerhards <rgerhards at hq.adiscon.com> wrote:
>
> If you run 1.21.0 and experience(d) the segfault previously, please let
> me know your results with 1.21.0 before upgrading. This would be most
> useful for me.

I upgraded to 1.21.0 december 19., and it's been up since (4h19m cputime, 
134h process uptime). No segfaults yet. It hasn't been eating any noticeable
amount of memory either, so I'll probably hold off on the upgrade a bit.



  -jf

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog


From janfrode at tanso.net  Wed Dec 26 17:42:53 2007
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Wed, 26 Dec 2007 17:42:53 +0100
Subject: [rsyslog] Log Permissions?
References: <47702A56.7030805@web-ster.com>
Message-ID: <slrnfn514d.7qs.janfrode@lc4eb8045376502.ibm.com>

On 2007-12-24, Scott Baker <bakers at web-ster.com> wrote:
> Is there a way to set the owner/group or read permissions of log
> files created by rsyslog? I can't find much in the man pages.
>

I use these in my /etc/rsyslog.conf to give group "logread" read
permissions of all files:

$DirGroup logread
$DirCreateMode 0750
$FileGroup logread
$umask 0027



  -jf



From rgerhards at hq.adiscon.com  Thu Dec 27 14:19:18 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 27 Dec 2007 14:19:18 +0100
Subject: [rsyslog] New rsyslog 3.0.0 preview - with new features
Message-ID: <1198761558.10564.3.camel@localhost.localdomain>

Hi all,

I have just uploaded a new preview of 3.0.0. It now has all input
modules (though they still have a lot of internal dependencies).
Finally, I also managed to get in some new features: the UDP listener
can now be bound to specific local interfaces and there can be multiple
UDP listeners active at the same time (on different interface and/or
different ports). The same feature for TCP (including GSS) is planned,
but I may need to do some more stage work before.

The preview is available at

http://download.rsyslog.com/rsyslog/rsyslog-3.0.0-pre.tar.gz
md5sum: 979763fe645ad6ee6fddf11737af7ebd 

This is a snapshot. Development continues. Feedback is appreciated.

Rainer


From rgerhards at hq.adiscon.com  Fri Dec 28 09:26:49 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 28 Dec 2007 09:26:49 +0100
Subject: [rsyslog] omfwd/gss-api in rsyslog v1/v2
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086F0@grfint2.intern.adiscon.com>

Hi all,

as you know, I am working on the modular structure of rsyslog v3. I am
currently revisiting gss-api support. I notice that with the current
omfwd, it will be extremely hard to separate gss-api support into its
own module. Doing so will break backward compatibility to the
configuration file.

GSS-API has been out only for a few days, and mostly over the holiday
period. So it is much less of a concern if we introduce now some changes
that will case rsyslog.conf format modifications. Much less trouble than
when we release v2, a release expected to be in wide use for at least
half a year, if not much longer. V2 released with the current syntax
would require me to do some tricks in v3 to keep compatibility. Quite
complex.

So I decided to create a omgssapi for v3 and extract the gss-api code
from omfwd. It looks like this can be done without too much code
duplication. There will be some duplicate code, but it will shrink as v3
continues to be developed. Once I have a good working version, which I
expect very soon, I will backport that to the v1/2 source tree. I'll
then do a new v1 release with a slightly incompatible gss-api config
file syntax. After this is out for a few days, I hope I can than finally
push out that version as v2.

I hope this is a good decision. I think it will save us major future
trouble at the expense of a relatively slight disturbance in the late v1
timeline. I guess most user's won't even notice there is a change.

As always, Feedback is appreciated.

Rainer 


From bakers at web-ster.com  Fri Dec 28 17:15:07 2007
From: bakers at web-ster.com (Scott Baker)
Date: Fri, 28 Dec 2007 08:15:07 -0800
Subject: [rsyslog] omfwd/gss-api in rsyslog v1/v2
In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA3086F0@grfint2.intern.adiscon.com>
References: <577465F99B41C842AAFBE9ED71E70ABA3086F0@grfint2.intern.adiscon.com>
Message-ID: <4775210B.804@web-ster.com>

Rainer Gerhards wrote:
> Hi all,
> 
> as you know, I am working on the modular structure of rsyslog v3. I am
> currently revisiting gss-api support. I notice that with the current
> omfwd, it will be extremely hard to separate gss-api support into its
> own module. Doing so will break backward compatibility to the
> configuration file.
> 
> GSS-API has been out only for a few days, and mostly over the holiday
> period. So it is much less of a concern if we introduce now some changes
> that will case rsyslog.conf format modifications. Much less trouble than
> when we release v2, a release expected to be in wide use for at least
> half a year, if not much longer. V2 released with the current syntax
> would require me to do some tricks in v3 to keep compatibility. Quite
> complex.
> 
> So I decided to create a omgssapi for v3 and extract the gss-api code
> from omfwd. It looks like this can be done without too much code
> duplication. There will be some duplicate code, but it will shrink as v3
> continues to be developed. Once I have a good working version, which I
> expect very soon, I will backport that to the v1/2 source tree. I'll
> then do a new v1 release with a slightly incompatible gss-api config
> file syntax. After this is out for a few days, I hope I can than finally
> push out that version as v2.
> 
> I hope this is a good decision. I think it will save us major future
> trouble at the expense of a relatively slight disturbance in the late v1
> timeline. I guess most user's won't even notice there is a change.
> 
> As always, Feedback is appreciated.

I think this is the best solution moving forward. I don't think
configuration changes moving to a new major version are a huge
headache, especially since GSS-API is so new.

-- 
Scott Baker - Canby Telcom
RHCE - System Administrator - 503.266.8253


From rgerhards at hq.adiscon.com  Fri Dec 28 22:39:59 2007
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 28 Dec 2007 22:39:59 +0100
Subject: [rsyslog] Rsyslog 1.21.2 released
Message-ID: <577465F99B41C842AAFBE9ED71E70ABA3086F7@grfint2.intern.adiscon.com>

Hi all,

just a quick note:

I have released rsyslog 1.21.2. It is a maintenance release. The most
important new feature is that GSS-API support for sending TCP messages
has been moved to a separate output plugin. This is an important
forward-compatibility option for rsyslog v3. This resulted in some
configuration file changes. This release also fixes a number of minor
bugs. This is a recommended update for all users.

Please also keep my message from earlier today in mind.

Rainer