[rsyslog] rsyslog error messages

Rainer Gerhards rgerhards at hq.adiscon.com
Fri Jul 25 17:16:12 CEST 2008 

On Fri, 2008-07-25 at 10:33 -0400, (private) HKS wrote:
> Hi Rainer,
> 
> Interesting post. The tendency to ignore errors/diagnostics is not
> limited to rsyslog, it's something that plagues pretty much every
> piece of software ever created anywhere.

I totally agree, but I think the situation is even worse for a syslogd.
Because it *is* the usual error reporting system. I think we are a bit
in the same situation (if you allow that analogy) like HIV which is
attacking the human immune system. We have a problem in the
troubleshooting system itself, and it may even prevent us from conveying
troubleshooting information :(

>  Some users are ignorant of
> basic diagnostic tools like syslog, --help flags, tcpdump, and man
> pages. Others just need a bit of prodding before they slap themselves
> and realize that they forgot some basic troubleshooting. Some are lazy
> slobs.
> 
> However, rsyslog lends itself to error reports that lack
> troubleshooting for one huge reason: errors are not reliably reported
> in the default configuration.


>  It's happened a few times that I start
> rsyslogd at the command line, and yet there's nothing in the process
> table, and nothing dumped to my logs. 

Could you elaborate a bit about above sentence - I have to admit I do
not fully get you...

> I've been able to track down the
> problem with -dn switches, but that involves weeding through a lot of
> irrelevant information.
> 
> There are three modifications I'd love to see that would help resolve this:
> 
> 1 - Configuration errors should be fatal
> 2 - Configuration and other runtime errors should be printed to STDERR

Does that really help? If so, that would be extremely easy to add. But
does somebody see them?

> 3 - By default, rsyslogd should log all of its own errors (fatal or
> not) to /var/log/rsyslog.log. This should also be user-configurable

This default comes into the way of some distros: they use different
pathes. I really don't like the idea of embedding any information about
the file system into the binary. Just think about embedded systems...

> 
> My reasoning:
> 
> 1 - If rsyslog doesn't understand your config file, obviously it won't
> be doing what you intend it to. There's little benefit in running with
> a horked configuration, but if you don't test thoroughly, the cost
> could be devastating. Some users won't know they're not logging things
> until they need them - then, if they keep their jobs, they probably
> won't keep rsyslog ("This software doesn't even log ssh attempts!").

That's quite controversal and and does not match rsyslog's root
philosophy. Let me explain:

The syslogd is an extremely important piece of software. If it is not
functioning, 

a) vital data can be lost
b) the system as whole may hang (due to the fact that the log 
   socket gets filled up and so other processes block on it)

As such, rsyslogd is written in a sense that it tries to continue to
work under almost all circumstances. It even tries to survive a
system-wide low memory condition, though I am not 100% convinced it will
manage to do that in all cases. At least it is coded careful to survive
as much as possible.

Now let's assume that a configuration error happens that prevents
rsyslogd from executing an otherwise perfect configuration. Should we
actually stop it from running? And live with the results? Or should we
do whatever we can do and do that as good as possible? What, if it is
not a configuration issue but a hard disk failure or an admin error that
make part of the config unavailable (e.g. not able to load module, not
able to write file)? Assuming we can carry out other actions - should we
really abort? This may even come at the price that we can NOT warn users
(via wall) of a serious problem. And what with warnings? Things that
look wrong, but may be OK. Reason to abort? Or continue run? Tough
decision. So I still think the right thing to do is to run as long as
there is limited sense in it, because everything else would actually be
worse. Please note that rsyslogd even comes with a hardcoded emergency
configuration which is used in case the real configuration file can not
be obtained.

What I could add, however, is an option to make it stop on any config
error (not warning). However, I fear that, if turned on, the results can
be fatal in some cases...

> 
> 2 - This will avoid a lot of silly questions without any supporting
> information. At least when you get a complaint like "Rsyslogd won't
> start!" it will generally include a "It just says 'configuration file
> invalid: broken syntax at line 135.'" For most users, this kind of
> error reporting will be enough to lead them to a solution. If nothing
> else, it removes the need to explain how to find the errors.
> 
> 3 - For those more subtle problems, and things you might miss upon
> bootup. This also gives you a simple, easily accessible debugging
> source with minimal security implications and additional code and no
> required user configuration. I'm not sure how the code is written, but
> I'd prefer to see this happen even if the configuration file can't be
> read. Perhaps an $RsyslogErrors directive that, by default, references
> a different module that just dumps directly to a file? Users can
> configure it to put rsyslog errors through the actual logging engine
> and then manage it with typical syslog selector/action lines.

All internally-generated messages are run through a specific interface.
For the "diag plugin", I will define a hook, where a diag server can
register to receive these messages. Maybe a good compromise would be to
add this $RsyslogErrors directive and make it point to a file. So,
still, we do not necessarily have it, but package maintainers would
hopefully include it and point it to a specific file (what still means I
do not necessarily know where to look). A problem is source tarball
install, in which case the directive may not be set. But with a proper
sample rsyslog.conf...

> The HTTP server is an interesting idea, but not something I would make
> use of. The potential security problems are enormous, and it would
> introduce too many additional factors to the debugging process: is the
> firewall right? Was there already a process listening on that port? Is
> the user's configuration file correct? These are the kinds of things
> you already have to deal with on rsyslog itself - why force the
> debugging process to go through them again?

I see (and share) much of your argument. This is why I stayed away for
such a long time. But if you think a bit more about it, it has *very
interesting* capabilities. Eg. I may also use it to gather some
real-time view of internal state, emit commands like HUP and do a number
of other nice things. Again, security is problematic.

In any case, I'll give it a try. I have started to write a small plugin
which will dump the error messages within 10 minutes after startup. It
will not be a real http server but rather a listener that response to
anyone who connects to it (so that nc can be used as a client ;)). This
is very rough, but it has two side-effects:

a) it provides something to actually experiment with (and to extend if
   we decide to keep that route)
b) it helps create the plumbing inside the internal interface that could
   also be used by any other method, so the potential waste of time is
   limited

So while I object some suggestions, I am very interested in a
discussion. The rsyslog "keep running" philosophy *is* a discussion
topic, just expect me to argue strongly in favor of it (pls think about
the HIV analogy to see why I think we have a unique situation in case of
a syslogd).

Thanks for the good post, and I guess I just wrote another long mail ;)

Rainer
> 
> Hope that helps, and sorry for the long email.
> -HKS
> 
> 
> 
> 
> On Fri, Jul 25, 2008 at 6:31 AM, Rainer Gerhards
> <rgerhards at hq.adiscon.com> wrote:
> > Thanks, HKS, for the good answers. There is one thing I would like to
> > bring to your attention: I often see that people do not check for
> > rsyslog error messages themselves. That complicates setup. I do not
> > blame anyone, but would like to make you aware of the problem.
> >
> > I have just blogged about a potential (partial) solution and will try to
> > implement it:
> >
> > http://blog.gerhards.net/2008/07/rsyslog-error-reporting-how-to-do-it.ht
> > ml
> >
> > Rainer
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of (private) HKS
> >> Sent: Thursday, July 24, 2008 11:36 PM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] Help with Ommail Configuration
> >>
> >> Oh - one other possibility. rsyslogd has to have mail enabled at
> >> compile time to work with it at runtime. check your catchall logfile -
> >> if it has messages like this, you didn't compile it in:
> >>
> >> rsyslogd: could not load module '/usr/local/lib/rsyslog/ommail.so',
> >> dlopen: /usr/local/lib/rsyslog/ommail.so: cannot open shared object
> >> file: No such file or directory
> >>
> >> To fix this, you'll need to reinstall. This time, when you run
> >> ./configure be sure to include --enable-mail. make install clean and
> >> you should be good to go...
> >>
> >> -HKS
> >>
> >> On Thu, Jul 24, 2008 at 5:01 PM, (private) HKS <hks.private at gmail.com>
> >> wrote:
> >> > A few things to try:
> >> >
> >> >  - You load ommail.so twice, once at the top and once right above
> >> your
> >> > $ActionMail... lines. I don't think this will break it, but it's
> >> > unnecessary - delete the second one.
> >> >
> >> >  - $ActionExecOnlyOnceEveryInterval 21600 means that it's only going
> >> > to attempt sending a message once every 6 hours. For testing
> >> purposes,
> >> > this will be obnoxious. Until you're ready for production, change it
> >> > to:
> >> > $ActionExecOnlyOnceEveryInterval 30
> >> >
> >> >  - Send everything to make sure you're  not missing it based on
> >> > something in the property-replacer/templates/whatever. Replace "if
> >> > $msg contains 'hard disk fatal failure' then :ommail:;mailBody"
> > with:
> >> > *.*  :ommail:;mailBody
> >> >
> >> > Try again. Try logging a few messages from the localhost first (with
> >> > RHEL you can just run "logger test" to log a user.notice message
> > with
> >> > content "test") and see if you get them.
> >> >
> >> > If you don't, check the mail logs on your mail server to see if it
> >> > ever received the message. If not, it's time to break out tcpdump
> > and
> >> > see if the packets are ever being generated.
> >> >
> >> > Hope that helps.
> >> >
> >> > -HKS
> >> >
> >> >
> >> >
> >> > On Thu, Jul 24, 2008 at 3:50 PM, Goutos, Kevin (DOB)
> >> > <Kevin.Goutos at budget.state.ny.us> wrote:
> >> >> Hello all,
> >> >>
> >> >> First off, I am not very Linux savvy. I don't have a lot of
> >> experience
> >> >> other then a basic course. This is probably way past my knowledge,
> >> but I
> >> >> really need to get it done. Appreciate any help you guys have to
> >> offer.
> >> >>
> >> >> I am working on a Red Hat Enterprise 4 box and I am running the
> >> latest
> >> >> edition of rsyslog. I currently have rsyslog configured to receive
> >> >> messages remotely via UDP. I am trying to set it up so that it will
> >> send
> >> >> out E-mail messages to the system Admin's based on the severity
> >> level of
> >> >> the log files I am receiving. I would like it so that any device
> >> that
> >> >> sends a log with a critical, alert, or emergency level facility
> > will
> >> >> send out an e-mail to a specific address.
> >> >>
> >> >> Here is my rsyslog.conf file. I used the sample code from Rainer
> >> >> Gerhards configuration page. I tried sending a test syslog with
> >> 'hard
> >> >> disk fatal failure' in it, but it is not sending out mail. Also
> >> tried
> >> >> without the templates below thinking it would just send me an email
> >> for
> >> >> every syslog that I received, but it doesn't appear to be sending
> >> mail.
> >> >> Any thoughts on what I am doing wrong. I'm sure there is a lot I
> >> need to
> >> >> do, so please let me know. Thanks!
> >> >>
> >> >> $template mailSubject,"disk problem on %hostname%"
> >> >> $template mailBody,"RSYSLOG Alert\r\nmsg='%msg%'"'
> >> >>
> >> >> I edited out the 3 lines below in the code for security reasons..
> >> >> $ActionMailSMTPServer <ip of smtp server>
> >> >> $ActionMailFrom <from address>
> >> >> $ActionMailTo <my email>
> >> >>
> >> >>
> >> >>
> >> >> Here is my code from rsyslog.conf below.
> >> >>
> >> >>
> >> >>
> >> >> # if you experience problems, check
> >> >> # http://www.rsyslog.com/troubleshoot for assistance
> >> >>
> >> >> # rsyslog v3: load input modules
> >> >> # If you do not load inputs, nothing happens!
> >> >> # You may need to set the module load path if modules are not
> > found.
> >> >>
> >> >> $ModLoad immark.so # provides --MARK-- message capability
> >> >> $ModLoad imuxsock.so # provides support for local system logging
> >> (e.g.
> >> >> via logger command)
> >> >> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
> >> >> $ModLoad ommail
> >> >>
> >> >> $template TraditionalFormatWithPRI,"%PRI-text%: %timegenerated%
> >> >> %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n"
> >> >>
> >> >> # Log all kernel messages to the console.
> >> >> # Logging much else clutters up the screen.
> >> >> #kern.*
> > /dev/console
> >> >>
> >> >> # Log anything (except mail) of level info or higher.
> >> >> # Don't log private authentication messages!
> >> >> *.info;mail.none;authpriv.none;cron.none
> >> >> -/var/log/messages
> >> >>
> >> >> # The authpriv file has restricted access.
> >> >> authpriv.*
> >> /var/log/secure
> >> >>
> >> >> # Log all the mail messages in one place.
> >> >> mail.*
> >> >> -/var/log/maillog
> >> >>
> >> >> # Log cron stuff
> >> >> cron.*                                                  -
> >> /var/log/cron
> >> >>
> >> >> # Everybody gets emergency messages
> >> >> *.emerg                                                 *
> >> >>
> >> >> # Save news errors of level crit and higher in a special file.
> >> >> uucp,news.crit
> >> >> -/var/log/spooler
> >> >>
> >> >> # Save boot messages also to boot.log
> >> >> local7.*
> >> >> /var/log/boot.log
> >> >>
> >> >> #Catch all incoming syslog messages
> >> >> *.*
> >> >> /var/log/catchall;TraditionalFormatWithPRI
> >> >>
> >> >> # Remote Logging (we use TCP for reliable delivery)
> >> >> # An on-disk queue is created for this action. If the remote host
> > is
> >> >> # down, messages are spooled to disk and sent when it is up again.
> >> >> $WorkDirectory /rsyslog/spool # where to place spool files
> >> >> $ActionQueueFileName uniqName # unique name prefix for spool files
> >> >> $ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as
> >> >> possible)
> >> >> $ActionQueueSaveOnShutdown on # save messages to disk on shutdown
> >> >> $ActionQueueType LinkedList   # run asynchronously
> >> >> $ActionResumeRetryCount -1    # infinite retries if host is down
> >> >> # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
> >> >> *.* @10.57.106.140:514
> >> >>
> >> >> $ModLoad ommail
> >> >> $ActionMailSMTPServer <ip of smtp server>
> >> >> $ActionMailFrom <from address>
> >> >> $ActionMailTo <my email>
> >> >> $template mailSubject,"disk problem on %hostname%"
> >> >> $template mailBody,"RSYSLOG Alert\r\nmsg='%msg%'"
> >> >> $ActionMailSubject mailSubject
> >> >> # make sure we receive a mail only once in six
> >> >> # hours (21,600 seconds ;))
> >> >> $ActionExecOnlyOnceEveryInterval 21600
> >> >> # the if ... then ... mailBody mus be on one line!
> >> >> if $msg contains 'hard disk fatal failure' then :ommail:;mailBody
> >> >>
> >> >>
> >> >> # ######### Receiving Messages from Remote Hosts ##########
> >> >> # TCP Syslog Server:
> >> >> # provides TCP syslog reception and GSS-API (if compiled to support
> >> it)
> >> >> $ModLoad imtcp.so  # load module
> >> >> $InputTCPServerRun 514 # start up TCP listener at port 514
> >> >>
> >> >> # UDP Syslog Server:
> >> >> $ModLoad imudp.so  # provides UDP syslog reception
> >> >> $UDPServerRun 514 # start a UDP syslog server at standard port
> >> >> --------------------------------------------------------
> >> >> This e-mail, including any attachments, may be confidential,
> >> privileged or otherwise legally protected. If you have received this
> > e-
> >> mail in error, or from someone who was not authorized to send it to
> >> you, do not disseminate, copy or otherwise use this e-mail or its
> >> attachments. Please notify the sender immediately if you have received
> >> this e-mail by mistake, and delete it from your system.
> >> >> _______________________________________________
> >> >> rsyslog mailing list
> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> >>
> >> >
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog

More information about the rsyslog mailing list