From rgerhards at hq.adiscon.com Tue Jul 3 09:02:03 2007 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Jul 2007 09:02:03 +0200 Subject: [rsyslog] rsyslog 1.14.2 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA278478@grfint2.intern.adiscon.com> Hi all, I just released rsyslog 1.14.2. This version fixes all known nits with IPv6. 1.14.2 is the first version fully backwards-compatible with the 1.13.x series, so it now is an update I can generally recommend. This release also includes some cosmetic changes, including somewhat improved documentation. Change log: http://www.rsyslog.com/Article81.phtml Download: http://www.rsyslog.com/Downloads-req-getit-lid-37.phtml As always, feedback is appreciated. Rainer Gerhards From rgerhards at hq.adiscon.com Thu Jul 5 11:03:45 2007 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Jul 2007 11:03:45 +0200 Subject: [rsyslog] rsyslog 1.15.0 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA2784A1@grfint2.intern.adiscon.com> Hi all, currently a lot of work is going on with rsyslog. Thus, we already have another *major* feature introduced: rsyslog has now the capability to generate output file names based on templates. This feature was often discussed and is now implemented in a very high-performance way. What does this mean? If you would like to split log files based on the originator host, you can now do that with just one selector line. All you need to define is a template that has the host property in it and assign that template to the file action. A sample would be: $template DynaFile,"/var/log/system-%HOSTNAME%.log" *.* -?DynaFile That's it. And, of course, you can not only use HOSTNAME but any property that is available (see http://www.rsyslog.com/module-Static_Docs-view-f-property_replacer.html. phtml). Besides this major enhancement, control for output file permissions have been increased and some minor things be added/changed. Please see the change log for details. Change Log: http://www.rsyslog.com/Article83.phtml Download: http://www.rsyslog.com/Downloads-req-getit-lid-38.phtml I would appreciate feedback on the new DynaFile feature as well as experience/bug reports very much. Thanks, Rainer Gerhards From rgerhards at hq.adiscon.com Mon Jul 9 15:26:10 2007 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 9 Jul 2007 15:26:10 +0200 Subject: [rsyslog] rsyslog moving toward autotools - FEEDBACK REQUESTED Message-ID: <577465F99B41C842AAFBE9ED71E70ABA2784DA@grfint2.intern.adiscon.com> Hi all, rsyslog is moving towards autotools. In order to drive the decision process on how this is done, I would very much appreciate your feedback. Please visit http://www.rsyslog.com/index.php?name=PNphpBB2&file=viewtopic&p=593#593 for some information. I would appreciate if you could post any feedback you may have over there. Many thanks, Rainer Gerhards From rgerhards at hq.adiscon.com Tue Jul 10 22:45:23 2007 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 10 Jul 2007 22:45:23 +0200 Subject: [rsyslog] Rsyslog 1.15.1 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA2784FC@grfint2.intern.adiscon.com> Hi all, I have just released rsyslog 1.15.1. It is primarily a stability release. Most importantly, the "last message repeated n times" functionality found in stock sysklogd has been re-enabled. It was lost during some modifications. Also, a bug was fixed that caused a dynamic file name selector line to become inoperable when a file could not be opened. In order to better take care of dynamic files, a number of properties for the current date/time has been added (e.g. $YEAR, $MONTH). Some minor bugs were also fixed. Change Log: http://www.rsyslog.com/Article87.phtml Download: http://www.rsyslog.com/Downloads-req-getit-lid-40.phtml As always, feedback is appreciated. Rainer Gerhards From rgerhards at hq.adiscon.com Fri Jul 13 21:37:56 2007 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 13 Jul 2007 21:37:56 +0200 Subject: [rsyslog] rsyslog 1.16.0 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA278560@grfint2.intern.adiscon.com> Hi all, I am proud to announce the immediate availability of rsyslog 1.16.0. The build system was converted to GNU autotools. New properties have been added, command line option "-x" can now be used to disable DNS lookups. There are also some bugfixes and optimizations. Code-restructuring has begun in an effort to provide better modularization for rsyslogd. Users experiencing problems fixed are advised to update. Others may do so to enjoy the benefits of the new build environment. Change Log: http://www.rsyslog.com/Article88.phtml Download: http://www.rsyslog.com/Downloads-req-getit-lid-41.phtml As always, feedback is appreciated. Rainer Gerhards From skvidal at fedoraproject.org Tue Jul 17 00:41:55 2007 From: skvidal at fedoraproject.org (seth vidal) Date: Mon, 16 Jul 2007 18:41:55 -0400 Subject: [rsyslog] first time using rsyslog - some questions Message-ID: <1184625715.606.32.camel@cutter> Hi folks, Fedora 8 will most likely be including rsyslog as the default syslog daemon. As such I've been experimenting with it to see how well it works to replace some of my former uses of syslog-ng. I like how small rsyslog is and it looks pretty straightforward. I have a couple of usage issues and some other random questions: 1. Can rsyslog automatically create subdirs for specific log outputs? For example: $template myvarname, "/var/log/hosts/%HOSTNAME%/messages" right now I have to manually make /var/log/hosts/%HOSTNAME%/ as a dir before any logs will be outputted? Can there be an option to do that automatically? 2. related to 1 is there a way to set specific file ownerships not just modes on the logs? I'd like to set the logs to be owned by root:log_readers and to be 640, along with all directories that it autocreates. That way no one else can look at the logs and I don't have to do anything hurky like 'sg' before running rsyslog. 3. I understand the desire to remain config file format compatible with sysklogd, however it seems like with any more logic being added into the config you'll end up with progressively more difficult config files. *.* @@hostname:port is pretty baroque looking as-is. Would it be reasonable to have a newer config file format/language and do: PSEUDOCODE if [ -f /etc/$new-style-format.conf ]; then conffile=/etc/$new-style-format.conf else: conffile=/etc/syslog.conf fi ? That way you retain backward compat to the old config file format from sysklogd and you provide a migration path to a nicer file formats for the future? Thanks! -sv From halljer at auburn.edu Tue Jul 17 03:01:20 2007 From: halljer at auburn.edu (Dusty Hall) Date: Mon, 16 Jul 2007 20:01:20 -0500 Subject: [rsyslog] first time using rsyslog - some questions Message-ID: <469BCE900200003A000234AD@groupwise1.duc.auburn.edu> Seth, I think you have some very good ideas and I know I would benefit from those additional features in rsyslog. FWIW, I've been running rsyslog for a couple of years and it works flawlessly. Cheers, -Dusty >>> seth vidal 07/16/07 5:41 PM >>> Hi folks, Fedora 8 will most likely be including rsyslog as the default syslog daemon. As such I've been experimenting with it to see how well it works to replace some of my former uses of syslog-ng. I like how small rsyslog is and it looks pretty straightforward. I have a couple of usage issues and some other random questions: 1. Can rsyslog automatically create subdirs for specific log outputs? For example: $template myvarname, "/var/log/hosts/%HOSTNAME%/messages" right now I have to manually make /var/log/hosts/%HOSTNAME%/ as a dir before any logs will be outputted? Can there be an option to do that automatically? 2. related to 1 is there a way to set specific file ownerships not just modes on the logs? I'd like to set the logs to be owned by root:log_readers and to be 640, along with all directories that it autocreates. That way no one else can look at the logs and I don't have to do anything hurky like 'sg' before running rsyslog. 3. I understand the desire to remain config file format compatible with sysklogd, however it seems like with any more logic being added into the config you'll end up with progressively more difficult config files. *.* @@hostname:port is pretty baroque looking as-is. Would it be reasonable to have a newer config file format/language and do: PSEUDOCODE if [ -f /etc/$new-style-format.conf ]; then conffile=/etc/$new-style-format.conf else: conffile=/etc/syslog.conf fi ? That way you retain backward compat to the old config file format from sysklogd and you provide a migration path to a nicer file formats for the future? Thanks! -sv _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Tue Jul 17 08:15:27 2007 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 17 Jul 2007 08:15:27 +0200 Subject: [rsyslog] first time using rsyslog - some questions In-Reply-To: <1184625715.606.32.camel@cutter> References: <1184625715.606.32.camel@cutter> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA278588@grfint2.intern.adiscon.com> Hi Sv, > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of seth vidal > Sent: Tuesday, July 17, 2007 12:42 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] first time using rsyslog - some questions > > Hi folks, > Fedora 8 will most likely be including rsyslog as the default syslog > daemon. As such I've been experimenting with it to see how well it > works > to replace some of my former uses of syslog-ng. I like how small > rsyslog > is and it looks pretty straightforward. Great to have someone experienced with syslog-ng on the list. I really like to make rsyslog the best syslogd ever and it is most interesting to learn what syslog-ng users find the most important features. In my point of view, we have some things syslog-ng can not do, but others where syslog-ng is ahead. I'd like to change that, but in order of importance to the user ;) > > I have a couple of usage issues and some other random questions: > > > 1. Can rsyslog automatically create subdirs for specific log outputs? > For example: > $template myvarname, "/var/log/hosts/%HOSTNAME%/messages" > > right now I have to manually make /var/log/hosts/%HOSTNAME%/ as a dir > before any logs will be outputted? Can there be an option to do that > automatically? It's on the todo list, most probably to be implemented this week. All interested please see: https://sourceforge.net/tracker/index.php?func=detail&aid=1755143&group_ id=123448&atid=696552 > > 2. related to 1 is there a way to set specific file ownerships not just > modes on the logs? I'd like to set the logs to be owned by > root:log_readers and to be 640, along with all directories that it > autocreates. That way no one else can look at the logs and I don't have > to do anything hurky like 'sg' before running rsyslog. Not yet on the feature list, but definitely interesting. I've filed a new feature request: https://sourceforge.net/tracker/index.php?func=detail&aid=1755276&group_ id=123448&atid=696555 > > 3. I understand the desire to remain config file format compatible with > sysklogd, however it seems like with any more logic being added into > the > config you'll end up with progressively more difficult config files. > *.* > @@hostname:port is pretty baroque looking as-is. Would it be reasonable > to have a newer config file format/language and do: > PSEUDOCODE > if [ -f /etc/$new-style-format.conf ]; then > conffile=/etc/$new-style-format.conf > else: > conffile=/etc/syslog.conf > fi > ? > > That way you retain backward compat to the old config file format from > sysklogd and you provide a migration path to a nicer file formats for > the future? I've already some ideas for a new log file format. I won't elaborate right now, because I'd like to iron out 1.16.1 this morning and that's a priority. However, the log file will still be quite different from syslog-ng, following a more simple concept (at least I think so). I searched the web a lot and found numerous questions on the syslog-ng log file format. So I think for the next version of rsyslogs format, it should be very easy to grasp yet powerful. And, yes, I agree the current way of doing things is not optimal. OK, already written more than I wanted: my plan is to get feature-complete, then restructure the code to be finally modular (I inherited a non-modular design form syslogd) and then define output and input plugins. In the course of that, you'll see config file changes. I am thinking along these lines: (for a listener) $protocol tcp $port 514 $RepeatedMsgReduction on # already in 1.16.1 $EscapeControlCharactersOnReceive off # already in 1.16.1 $listen 127.0.0.1 Similar things for the output side... Looking forward to more comments. Rainer > > Thanks! > -sv > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From skvidal at fedoraproject.org Tue Jul 17 14:33:34 2007 From: skvidal at fedoraproject.org (seth vidal) Date: Tue, 17 Jul 2007 08:33:34 -0400 Subject: [rsyslog] first time using rsyslog - some questions In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA278588@grfint2.intern.adiscon.com> References: <1184625715.606.32.camel@cutter> <577465F99B41C842AAFBE9ED71E70ABA278588@grfint2.intern.adiscon.com> Message-ID: <1184675614.606.62.camel@cutter> On Tue, 2007-07-17 at 08:15 +0200, Rainer Gerhards wrote: > Great to have someone experienced with syslog-ng on the list. I really > like to make rsyslog the best syslogd ever and it is most interesting to > learn what syslog-ng users find the most important features. In my point > of view, we have some things syslog-ng can not do, but others where > syslog-ng is ahead. I'd like to change that, but in order of importance > to the user ;) Great! Featurewise, right now, rsyslog has a lot of the pieces in the right place to be a great syslog server. My primary uses of syslog-ng was: 1. large scale central file server (500ish systems logging everything centrally) 2. ssl-encrypting/tcp syslog server for communicating over wans. 3. invoking sec for specialized event behavior > It's on the todo list, most probably to be implemented this week. All > interested please see: > > https://sourceforge.net/tracker/index.php?func=detail&aid=1755143&group_ > id=123448&atid=696552 > Great to hear! Thanks. One of the use cases for me is: /var/log/hosts/%HOSTNAME%/%YEAR%/%MONTH%/%DAY%/messages /var/log/hosts/%HOSTNAME%/%YEAR%/%MONTH%/%DAY%/secure.log /var/log/hosts/%HOSTNAME%/%YEAR%/%MONTH%/%DAY%/kern.log etc so that for log cleanup I can simply do: tmpwatch -f 720 /var/log/hosts/ and that removes all logs older than 30 days. No need to run logrotate, no need to hup the server. It also makes logs rsyncable b/c the filenames/path change as they progress. This lets me rsync my logs to another box, process them for results/reports and not disrupt or bog down my primary syslog server. So if automatically-creating-dirs is made possible then I can do the above on my central syslog server. > > 2. related to 1 is there a way to set specific file ownerships not > just > > modes on the logs? I'd like to set the logs to be owned by > > root:log_readers and to be 640, along with all directories that it > > autocreates. That way no one else can look at the logs and I don't > have > > to do anything hurky like 'sg' before running rsyslog. > > Not yet on the feature list, but definitely interesting. I've filed a > new feature request: > > https://sourceforge.net/tracker/index.php?func=detail&aid=1755276&group_ > id=123448&atid=696555 great! monitoring that rfe now, thanks. > I've already some ideas for a new log file format. I won't elaborate > right now, because I'd like to iron out 1.16.1 this morning and that's a > priority. However, the log file will still be quite different from > syslog-ng, following a more simple concept (at least I think so). I > searched the web a lot and found numerous questions on the syslog-ng log > file format. So I think for the next version of rsyslogs format, it > should be very easy to grasp yet powerful. And, yes, I agree the current > way of doing things is not optimal. OK, already written more than I > wanted: my plan is to get feature-complete, then restructure the code to > be finally modular (I inherited a non-modular design form syslogd) and > then define output and input plugins. In the course of that, you'll see > config file changes. I am thinking along these lines: > > (for a listener) > > $protocol tcp > $port 514 > $RepeatedMsgReduction on # already in 1.16.1 > $EscapeControlCharactersOnReceive off # already in 1.16.1 > $listen 127.0.0.1 > > Similar things for the output side... > > Looking forward to more comments. What would you think of a config file format that's a bit closer to a language? So you could do more case-like statements for what to get logged where? for example: my_emerg_output = /var/log/mail/emergency my_generic_output = |/usr/bin/keep_count_of_messages my_normal_logs = /var/log/mail/normal if facility == 'mail': if priority == 'emerg': output == my_emerg_output elif priority < 'emerg' and priority > 'debug': filter_out(host='localhost') output == my_generic_output, my_normal_logs else: output == /dev/null So this would mean: if the facility is mail and priority emerg then send it to my mail emergency file if the facility is mail and the priority is between emerg and debug then remove all logs where the sending host is 'localhost' and then send it to my generic output AND to my normal logs otherwise, dump the output to /dev/null. Would something like that be reasonable? Alternatively, can you tell me a better way to do something like the above with what you have in mind for the config file? I'm really not wed to any one format - I just like it when config file examples are easy to put together and parse. For additional feature sets: Something that syslog-ng cannot do but I've always wanted a syslog daemon to do is store-and-forward remote logging and/or failover remote logging. ie: If you're using tcp-based logging from client to server you can determine that the server is no longer accepting your messages from the client. In that event, have the client start packing away all of the log entries it would normally send to the server into a tmpfile some place. When the server comes back push all of the logs over to it from earliest to latest. Yes, the logs will be out of order but that's the least of the worries, at least you have the data remotely. Setting a limit on the size of the tmpfile makes sense to me and when that limit is hit start discarding the EARLIER log entries for the later ones. As an option related to this. Allow a remote log server configuration option to include multiple hostnames/destinations in decreasing priority. So as long as the first one is up, you log to it, when it stops being up you log to the next one in the list and so on and so on. Thanks, -sv From skvidal at linux.duke.edu Tue Jul 17 14:28:24 2007 From: skvidal at linux.duke.edu (seth vidal) Date: Tue, 17 Jul 2007 08:28:24 -0400 Subject: [rsyslog] first time using rsyslog - some questions In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA278588@grfint2.intern.adiscon.com> References: <1184625715.606.32.camel@cutter> <577465F99B41C842AAFBE9ED71E70ABA278588@grfint2.intern.adiscon.com> Message-ID: <1184675304.606.61.camel@cutter> On Tue, 2007-07-17 at 08:15 +0200, Rainer Gerhards wrote: > Great to have someone experienced with syslog-ng on the list. I really > like to make rsyslog the best syslogd ever and it is most interesting to > learn what syslog-ng users find the most important features. In my point > of view, we have some things syslog-ng can not do, but others where > syslog-ng is ahead. I'd like to change that, but in order of importance > to the user ;) Great! Featurewise, right now, rsyslog has a lot of the pieces in the right place to be a great syslog server. My primary uses of syslog-ng was: 1. large scale central file server (500ish systems logging everything centrally) 2. ssl-encrypting/tcp syslog server for communicating over wans. 3. invoking sec for specialized event behavior > It's on the todo list, most probably to be implemented this week. All > interested please see: > > https://sourceforge.net/tracker/index.php?func=detail&aid=1755143&group_ > id=123448&atid=696552 > Great to hear! Thanks. One of the use cases for me is: /var/log/hosts/%HOSTNAME%/%YEAR%/%MONTH%/%DAY%/messages /var/log/hosts/%HOSTNAME%/%YEAR%/%MONTH%/%DAY%/secure.log /var/log/hosts/%HOSTNAME%/%YEAR%/%MONTH%/%DAY%/kern.log etc so that for log cleanup I can simply do: tmpwatch -f 720 /var/log/hosts/ and that removes all logs older than 30 days. No need to run logrotate, no need to hup the server. It also makes logs rsyncable b/c the filenames/path change as they progress. This lets me rsync my logs to another box, process them for results/reports and not disrupt or bog down my primary syslog server. So if automatically-creating-dirs is made possible then I can do the above on my central syslog server. > > 2. related to 1 is there a way to set specific file ownerships not > just > > modes on the logs? I'd like to set the logs to be owned by > > root:log_readers and to be 640, along with all directories that it > > autocreates. That way no one else can look at the logs and I don't > have > > to do anything hurky like 'sg' before running rsyslog. > > Not yet on the feature list, but definitely interesting. I've filed a > new feature request: > > https://sourceforge.net/tracker/index.php?func=detail&aid=1755276&group_ > id=123448&atid=696555 great! monitoring that rfe now, thanks. > I've already some ideas for a new log file format. I won't elaborate > right now, because I'd like to iron out 1.16.1 this morning and that's a > priority. However, the log file will still be quite different from > syslog-ng, following a more simple concept (at least I think so). I > searched the web a lot and found numerous questions on the syslog-ng log > file format. So I think for the next version of rsyslogs format, it > should be very easy to grasp yet powerful. And, yes, I agree the current > way of doing things is not optimal. OK, already written more than I > wanted: my plan is to get feature-complete, then restructure the code to > be finally modular (I inherited a non-modular design form syslogd) and > then define output and input plugins. In the course of that, you'll see > config file changes. I am thinking along these lines: > > (for a listener) > > $protocol tcp > $port 514 > $RepeatedMsgReduction on # already in 1.16.1 > $EscapeControlCharactersOnReceive off # already in 1.16.1 > $listen 127.0.0.1 > > Similar things for the output side... > > Looking forward to more comments. What would you think of a config file format that's a bit closer to a language? So you could do more case-like statements for what to get logged where? for example: my_emerg_output = /var/log/mail/emergency my_generic_output = |/usr/bin/keep_count_of_messages my_normal_logs = /var/log/mail/normal if facility == 'mail': if priority == 'emerg': output == my_emerg_output elif priority < 'emerg' and priority > 'debug': filter_out(host='localhost') output == my_generic_output, my_normal_logs else: output == /dev/null So this would mean: if the facility is mail and priority emerg then send it to my mail emergency file if the facility is mail and the priority is between emerg and debug then remove all logs where the sending host is 'localhost' and then send it to my generic output AND to my normal logs otherwise, dump the output to /dev/null. Would something like that be reasonable? Alternatively, can you tell me a better way to do something like the above with what you have in mind for the config file? I'm really not wed to any one format - I just like it when config file examples are easy to put together and parse. For additional feature sets: Something that syslog-ng cannot do but I've always wanted a syslog daemon to do is store-and-forward remote logging and/or failover remote logging. ie: If you're using tcp-based logging from client to server you can determine that the server is no longer accepting your messages from the client. In that event, have the client start packing away all of the log entries it would normally send to the server into a tmpfile some place. When the server comes back push all of the logs over to it from earliest to latest. Yes, the logs will be out of order but that's the least of the worries, at least you have the data remotely. Setting a limit on the size of the tmpfile makes sense to me and when that limit is hit start discarding the EARLIER log entries for the later ones. As an option related to this. Allow a remote log server configuration option to include multiple hostnames/destinations in decreasing priority. So as long as the first one is up, you log to it, when it stops being up you log to the next one in the list and so on and so on. Thanks, -sv From rgerhards at hq.adiscon.com Tue Jul 17 17:19:00 2007 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 17 Jul 2007 17:19:00 +0200 Subject: [rsyslog] Rsyslog 1.17.0 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA278597@grfint2.intern.adiscon.com> Hi all, I am pleased to announce the availability of release 1.17.0. This release offers a number of exciting new features. Directories can now be automatically generated for dynamic file names. Also, the owner and group can be set for created files and directories. Regular expressions are now supported in filters and $AllowedSenders does now fully support IPv6 addresses as well as domain wildcards. Also, control character replacement is now a user-configurable option (restoring backwards compatibility to sysklogd). There are a number of new config file directives. Change Log: http://www.rsyslog.com/Article89.phtml Download: http://www.rsyslog.com/Downloads-req-getit-lid-42.phtml As always, feedback is appreciated. Best regards, Rainer Gerhards From rgerhards at hq.adiscon.com Tue Jul 17 17:38:57 2007 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 17 Jul 2007 17:38:57 +0200 Subject: [rsyslog] first time using rsyslog - some questions In-Reply-To: <1184675304.606.61.camel@cutter> References: <1184625715.606.32.camel@cutter> <577465F99B41C842AAFBE9ED71E70ABA278588@grfint2.intern.adiscon.com> <1184675304.606.61.camel@cutter> Message-ID: <1184686737.6175.22.camel@localhost.localdomain> Hi, I finished 1.17.0 (too much went in it for 1.16.1 ;)), so I finally have time for an intelligent answer ;) See inline below... On Tue, 2007-07-17 at 08:28 -0400, seth vidal wrote: > On Tue, 2007-07-17 at 08:15 +0200, Rainer Gerhards wrote: > > > Great to have someone experienced with syslog-ng on the list. I really > > like to make rsyslog the best syslogd ever and it is most interesting to > > learn what syslog-ng users find the most important features. In my point > > of view, we have some things syslog-ng can not do, but others where > > syslog-ng is ahead. I'd like to change that, but in order of importance > > to the user ;) > > Great! Featurewise, right now, rsyslog has a lot of the pieces in the > right place to be a great syslog server. My primary uses of syslog-ng > was: > 1. large scale central file server (500ish systems logging everything > centrally) > 2. ssl-encrypting/tcp syslog server for communicating over wans. > 3. invoking sec for specialized event behavior > That sounds great. I would be really interested in your experience with 1.17.0 in this environment (now is the time to try to break it, feature-wise its more or less ready for fedora 8 ;)). > > > > It's on the todo list, most probably to be implemented this week. All > > interested please see: > > > > https://sourceforge.net/tracker/index.php?func=detail&aid=1755143&group_ > > id=123448&atid=696552 > > > > Great to hear! Thanks. I had a good time today, its in. > > One of the use cases for me is: > > /var/log/hosts/%HOSTNAME%/%YEAR%/%MONTH%/%DAY%/messages > /var/log/hosts/%HOSTNAME%/%YEAR%/%MONTH%/%DAY%/secure.log > /var/log/hosts/%HOSTNAME%/%YEAR%/%MONTH%/%DAY%/kern.log > etc > This is a sample of what you can do now: $umask 0000 # let default umask not get into our way $CreateDirs on $DynaCacheFileSize 50 # depending on number of hosts $FileOwner whoever $FileGroup loggroup $FileCreateMode 0600 $DirOwner whoever $DirGroup loggroup $DirCreateMode 0600 $FailOnChownFailure off # its more important for us that logs be written $template file-msgs,"/var/log/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY %/messages $RepeatedMessageReduction on # or off? *.* -?file-msgs It is important to keep in mind that all parameters keep in effect until they are either overwritten (by a new directive) or $ResetConfigVariables is specified. > " $ > so that for log cleanup I can simply do: > tmpwatch -f 720 /var/log/hosts/ > > and that removes all logs older than 30 days. > > No need to run logrotate, no need to hup the server. It also makes logs > rsyncable b/c the filenames/path change as they progress. This lets me > rsync my logs to another box, process them for results/reports and not > disrupt or bog down my primary syslog server. > > So if automatically-creating-dirs is made possible then I can do the > above on my central syslog server. Please check out - it's brand-new code, so probably some things go wrong at some place (of course, not in lab ;)). [snip] > > I've already some ideas for a new log file format. I won't elaborate > > right now, because I'd like to iron out 1.16.1 this morning and that's a > > priority. However, the log file will still be quite different from > > syslog-ng, following a more simple concept (at least I think so). I > > searched the web a lot and found numerous questions on the syslog-ng log > > file format. So I think for the next version of rsyslogs format, it > > should be very easy to grasp yet powerful. And, yes, I agree the current > > way of doing things is not optimal. OK, already written more than I > > wanted: my plan is to get feature-complete, then restructure the code to > > be finally modular (I inherited a non-modular design form syslogd) and > > then define output and input plugins. In the course of that, you'll see > > config file changes. I am thinking along these lines: > > [snip] > What would you think of a config file format that's a bit closer to a > language? So you could do more case-like statements for what to get > logged where? for example: > > my_emerg_output = /var/log/mail/emergency > my_generic_output = |/usr/bin/keep_count_of_messages > my_normal_logs = /var/log/mail/normal > > if facility == 'mail': > if priority == 'emerg': > output == my_emerg_output > elif priority < 'emerg' and priority > 'debug': > filter_out(host='localhost') > output == my_generic_output, my_normal_logs > else: > output == /dev/null > > So this would mean: > if the facility is mail and priority emerg then send it to my mail > emergency file > if the facility is mail and the priority is between emerg and debug > then remove all logs where the sending host is 'localhost' and then send > it to my generic output AND to my normal logs > otherwise, dump the output to /dev/null. > > > Would something like that be reasonable? Alternatively, can you tell me > a better way to do something like the above with what you have in mind > for the config file? I'm really not wed to any one format - I just like > it when config file examples are easy to put together and parse. > I have to admit I have not yet thought about the filtering side. The modules are input - filter - output. Whatever it is, it won't happen for 2.0.0, which shall be the result of the current development branch in the not so distant future (let's say 3 to 5 weeks from now). I'll consider your recommendation when I think about a new way to handle filters. But no promise ;) > > For additional feature sets: > Something that syslog-ng cannot do but I've always wanted a syslog > daemon to do is store-and-forward remote logging and/or failover remote > logging. > > ie: > If you're using tcp-based logging from client to server you can > determine that the server is no longer accepting your messages from the > client. In that event, have the client start packing away all of the log > entries it would normally send to the server into a tmpfile some place. > When the server comes back push all of the logs over to it from earliest > to latest. Yes, the logs will be out of order but that's the least of > the worries, at least you have the data remotely. Setting a limit on the > size of the tmpfile makes sense to me and when that limit is hit start > discarding the EARLIER log entries for the later ones. This *is* on the upcoming feature list. But to implement it, a number of core changes must be applied first, including a change of the threading model. I already have a rough idea on most of these things, so I wouldn't outrule that the whole thing materializes this year (but again, no promise ;)). > > As an option related to this. Allow a remote log server configuration > option to include multiple hostnames/destinations in decreasing > priority. > So as long as the first one is up, you log to it, when it stops being up > you log to the next one in the list and so on and so on. That sounds fairly easy. Let's see if I can fit that in without touching too much code. I've added a tracker: http://sourceforge.net/tracker/index.php?func=detail&aid=1755530&group_id=123448&atid=696555 Thanks for the comments, Rainer From skvidal at linux.duke.edu Tue Jul 17 22:22:42 2007 From: skvidal at linux.duke.edu (seth vidal) Date: Tue, 17 Jul 2007 16:22:42 -0400 Subject: [rsyslog] Rsyslog 1.17.0 released In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA278597@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA278597@grfint2.intern.adiscon.com> Message-ID: <1184703762.606.85.camel@cutter> On Tue, 2007-07-17 at 17:19 +0200, Rainer Gerhards wrote: > Hi all, > > I am pleased to announce the availability of release 1.17.0. > > This release offers a number of exciting new features. Directories can > now be automatically generated for dynamic file names. Also, the owner > and group can be set for created files and directories. Regular > expressions are now supported in filters and $AllowedSenders does now > fully support IPv6 addresses as well as domain wildcards. Also, control > character replacement is now a user-configurable option (restoring > backwards compatibility to sysklogd). There are a number of new config > file directives. > First and foremost: 1. the createdirs works great 2. the dir and file perms works great Thank you for responding so quickly. -sv From skvidal at linux.duke.edu Wed Jul 18 06:38:49 2007 From: skvidal at linux.duke.edu (seth vidal) Date: Wed, 18 Jul 2007 00:38:49 -0400 Subject: [rsyslog] first time using rsyslog - some questions In-Reply-To: <1184686737.6175.22.camel@localhost.localdomain> References: <1184625715.606.32.camel@cutter> <577465F99B41C842AAFBE9ED71E70ABA278588@grfint2.intern.adiscon.com> <1184675304.606.61.camel@cutter> <1184686737.6175.22.camel@localhost.localdomain> Message-ID: <1184733529.606.101.camel@cutter> On Tue, 2007-07-17 at 17:38 +0200, Rainer Gerhards wrote: > Hi, > > That sounds great. I would be really interested in your experience with > 1.17.0 in this environment (now is the time to try to break it, > feature-wise its more or less ready for fedora 8 ;)). right now it is doing what it should be doing, except for one thing. if I have a log defined like this: $template hostsecure, "/var/log/hosts/%HOSTNAME%/secure/%$NOW%" authpriv.* -?hostsecure I'm getting logs from all of the hosts in this path not just ones of % HOSTNAME% It seemed to be working before 1.17.0. > I had a good time today, its in. > Thank you for taking care of this so quickly. It is quite helpful. -sv From rgerhards at hq.adiscon.com Wed Jul 18 13:56:31 2007 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 18 Jul 2007 13:56:31 +0200 Subject: [rsyslog] first time using rsyslog - some questions In-Reply-To: <1184733529.606.101.camel@cutter> References: <1184625715.606.32.camel@cutter><577465F99B41C842AAFBE9ED71E70ABA278588@grfint2.intern.adiscon.com><1184675304.606.61.camel@cutter><1184686737.6175.22.camel@localhost.localdomain> <1184733529.606.101.camel@cutter> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA2785A3@grfint2.intern.adiscon.com> > right now it is doing what it should be doing, except for one thing. if > I have a log defined like this: > > $template hostsecure, "/var/log/hosts/%HOSTNAME%/secure/%$NOW%" > authpriv.* -?hostsecure > > > I'm getting logs from all of the hosts in this path not just ones of % > HOSTNAME% > > It seemed to be working before 1.17.0. I have to admit I do not fully understand. There is no hostname restriction (at least not one I can see from the excerpt), so it should be processed for all hosts. I am sure I overlook something, please provide a bit more detail. > > > I had a good time today, its in. > > > > Thank you for taking care of this so quickly. It is quite helpful. The question came at the right time, everything was already set. I can't guarantee this will be the case the next time ;) Rainer From rgerhards at hq.adiscon.com Fri Jul 20 15:08:40 2007 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 20 Jul 2007 15:08:40 +0200 Subject: [rsyslog] rsyslog 1.17.1 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA2785C8@grfint2.intern.adiscon.com> Hi all, rsyslog 1.17.1 has just been released. It is a cleanup and bug fixing release. IPv6 scopes are now correctly supported in $AllowedSenders. The capability to continue trying to write log files if the file system is full has been added. The binaries are now again named correctly. A few minor memory leaks on HUP have been fixed. Code cleanup and modularization continued. This is a recommended update for all users. Change Log: http://www.rsyslog.com/Article92.phtml Download: http://www.rsyslog.com/Downloads-req-getit-lid-43.phtml As always, feedback is appreciated. Rainer Gerhards From rgerhards at hq.adiscon.com Mon Jul 23 17:05:37 2007 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 23 Jul 2007 17:05:37 +0200 Subject: [rsyslog] rsyslog 1.17.2 Message-ID: <577465F99B41C842AAFBE9ED71E70ABA2785F0@grfint2.intern.adiscon.com> Hi all, Rsyslog 1.17.2 has been released. This is primarily a code cleanup release. Output modules have mostly been moved to their own .c/.h files. For those of you modifying the sources, please note that patch targets most probably have been totally moved around. Please use the new version as basis for future patches. As a side-note, modularization is far from being finished. I expect to shuffle lots of code the next few days. Some bugs were fixed. The system() call is now replaced with a more decent function. Added the $ModLoad MySQL and $DropTrailingLFOnReception config directives. This is a recommended update for all users. Change Log: http://www.rsyslog.com/Article94.phtml Download: http://www.rsyslog.com/Downloads-req-getit-lid-44.phtml As always, feedback is appreciated. Rainer Gerhards From rgerhards at hq.adiscon.com Tue Jul 24 18:34:07 2007 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 24 Jul 2007 18:34:07 +0200 Subject: [rsyslog] rsyslog error messages on startup - how and where to put? Message-ID: <577465F99B41C842AAFBE9ED71E70ABA278606@grfint2.intern.adiscon.com> Hello everyone, I am seeking input. I am restructuring the code, and this means that I will also revisit the way error messages are generated on startup. Those errors happen very early in processing and the normal syslog message processing is not yet available when it is carried out. These messages are mostly about errors in the configuration file. Rsyslog up to version 1.17.2 spits these messages to the terminal that is associated with stdin and the system console. For future releases, I think about sending them to the console, stderr and optionally a file specified on the command line. Also, I will stop sending a full syslog header and instead a canned format, most probably just consisting on a timestamp (is it needed) and the message itself. I am seeking feedback an what you would find the appropriate place to send such messages to and what format you would like to see. Comments are greatly appreciated. Many thanks, Rainer Gerhards From rgerhards at hq.adiscon.com Wed Jul 25 17:53:53 2007 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 25 Jul 2007 17:53:53 +0200 Subject: [rsyslog] rsyslog 1.17.3 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA278614@grfint2.intern.adiscon.com> Hi all, I have just released rsyslog 1.17.3. This version is a code cleanup release, implementing more of the future output module interface. It also fixes some minor bugs. The primary reason for release is to gather feedback. Lots of code has been changed, and there is a potential for new bugs. Please test this version and report back any problems you may experience. Installing this release is recommended for people interested in testing, only. As always, download and change log can be found on http://www.rsyslog.com. Please provide feedback. Rainer Gerhards From rgerhards at hq.adiscon.com Fri Jul 27 12:29:32 2007 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 27 Jul 2007 12:29:32 +0200 Subject: [rsyslog] rsyslog 1.17.4 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA278644@grfint2.intern.adiscon.com> Hi all, rsyslog 1.17.4 has been released today. This is part of the ongoing modularization effort. We now have a much cleaner interface between the main syslogd and output modules. The main purpose for release is to allow as much early testing as possible. So we would appreciate any feedback from reality. The release also contains some bug fixes. The update is only recommended to those interested in either testing or the fixes. Change Log: http://www.rsyslog.com/Article99.phtml Download: http://www.rsyslog.com/Downloads-req-getit-lid-46.phtml As always, feedback is appreciated. Rainer Gerhards From rgerhards at hq.adiscon.com Sat Jul 28 17:41:37 2007 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sat, 28 Jul 2007 17:41:37 +0200 Subject: [rsyslog] On the syslogd -h option Message-ID: <577465F99B41C842AAFBE9ED71E70ABA278651@grfint2.intern.adiscon.com> Hi folks, As I need to post this to several places, please see my blog for the actual message. http://rgerhards.blogspot.com/2007/07/on-syslogd-h-option.html Feedback appreciated. Thanks, Rainer Gerhards