[rsyslog] rsyslog error messages
Rainer Gerhards
rgerhards at hq.adiscon.com
Fri Jul 25 17:25:31 CEST 2008
Hi Rio-san,
On Fri, 2008-07-25 at 23:56 +0900, Ryo Fujita wrote:
> Hi HKS-san,
>
> +1
>
> > 2 - This will avoid a lot of silly questions without any supporting
> > information. At least when you get a complaint like "Rsyslogd won't
> > start!" it will generally include a "It just says 'configuration file
> > invalid: broken syntax at line 135.'
>
> In addition,
> '... at line 135. See man 5 rsyslog.conf or $doc/rsyslog_conf.html.'
Same thought over here :)
The recent build even emits live links to troubleshooting information.
Log lines look like this:
2008-07-25T17:22:18.523900+02:00 rgf9dev rsyslogd-3003: invalid or
yet-unknown config file command - have you forgotten to load a module?
[try http://www.rsyslog.com/e/3003 ]
Note the part in the bracket at the end of the message. That repository
is far from being complete, but I hope more and more information will be
added.
I hope this is a useful addition. Together with the error number, we can
hopefully most often pinpoint the source of the problem (though the 3003
above is a good example of where it may not be possible to tell the
exact cause, but for sure give a good hint).
BTW: this is also one thing that makes me think about the scripting
engine. It shall provide better error reporting capabilties, but that's
harder than it initially sounds ;)
Rainer
>
> Any ideas?
>
> Best Rio.
>
> On 2008/07/25, at 23:33, (private) HKS wrote:
>
> > Hi Rainer,
> >
> > Interesting post. The tendency to ignore errors/diagnostics is not
> > limited to rsyslog, it's something that plagues pretty much every
> > piece of software ever created anywhere. Some users are ignorant of
> > basic diagnostic tools like syslog, --help flags, tcpdump, and man
> > pages. Others just need a bit of prodding before they slap themselves
> > and realize that they forgot some basic troubleshooting. Some are lazy
> > slobs.
> >
> > However, rsyslog lends itself to error reports that lack
> > troubleshooting for one huge reason: errors are not reliably reported
> > in the default configuration. It's happened a few times that I start
> > rsyslogd at the command line, and yet there's nothing in the process
> > table, and nothing dumped to my logs. I've been able to track down the
> > problem with -dn switches, but that involves weeding through a lot of
> > irrelevant information.
> >
> > There are three modifications I'd love to see that would help
> > resolve this:
> >
> > 1 - Configuration errors should be fatal
> > 2 - Configuration and other runtime errors should be printed to STDERR
> > 3 - By default, rsyslogd should log all of its own errors (fatal or
> > not) to /var/log/rsyslog.log. This should also be user-configurable
> >
> > My reasoning:
> >
> > 1 - If rsyslog doesn't understand your config file, obviously it won't
> > be doing what you intend it to. There's little benefit in running with
> > a horked configuration, but if you don't test thoroughly, the cost
> > could be devastating. Some users won't know they're not logging things
> > until they need them - then, if they keep their jobs, they probably
> > won't keep rsyslog ("This software doesn't even log ssh attempts!").
> >
> > 2 - This will avoid a lot of silly questions without any supporting
> > information. At least when you get a complaint like "Rsyslogd won't
> > start!" it will generally include a "It just says 'configuration file
> > invalid: broken syntax at line 135.'" For most users, this kind of
> > error reporting will be enough to lead them to a solution. If nothing
> > else, it removes the need to explain how to find the errors.
> >
> > 3 - For those more subtle problems, and things you might miss upon
> > bootup. This also gives you a simple, easily accessible debugging
> > source with minimal security implications and additional code and no
> > required user configuration. I'm not sure how the code is written, but
> > I'd prefer to see this happen even if the configuration file can't be
> > read. Perhaps an $RsyslogErrors directive that, by default, references
> > a different module that just dumps directly to a file? Users can
> > configure it to put rsyslog errors through the actual logging engine
> > and then manage it with typical syslog selector/action lines.
> >
> > The HTTP server is an interesting idea, but not something I would make
> > use of. The potential security problems are enormous, and it would
> > introduce too many additional factors to the debugging process: is the
> > firewall right? Was there already a process listening on that port? Is
> > the user's configuration file correct? These are the kinds of things
> > you already have to deal with on rsyslog itself - why force the
> > debugging process to go through them again?
> >
> > Hope that helps, and sorry for the long email.
> > -HKS
> >
> >
> >
> >
> > On Fri, Jul 25, 2008 at 6:31 AM, Rainer Gerhards
> > <rgerhards at hq.adiscon.com> wrote:
> >> Thanks, HKS, for the good answers. There is one thing I would like to
> >> bring to your attention: I often see that people do not check for
> >> rsyslog error messages themselves. That complicates setup. I do not
> >> blame anyone, but would like to make you aware of the problem.
> >>
> >> I have just blogged about a potential (partial) solution and will
> >> try to
> >> implement it:
> >>
> >> http://blog.gerhards.net/2008/07/rsyslog-error-reporting-how-to-do-it.ht
> >> ml
> >>
> >> Rainer
> >>
> >>> -----Original Message-----
> >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >>> bounces at lists.adiscon.com] On Behalf Of (private) HKS
> >>> Sent: Thursday, July 24, 2008 11:36 PM
> >>> To: rsyslog-users
> >>> Subject: Re: [rsyslog] Help with Ommail Configuration
> >>>
> >>> Oh - one other possibility. rsyslogd has to have mail enabled at
> >>> compile time to work with it at runtime. check your catchall
> >>> logfile -
> >>> if it has messages like this, you didn't compile it in:
> >>>
> >>> rsyslogd: could not load module '/usr/local/lib/rsyslog/ommail.so',
> >>> dlopen: /usr/local/lib/rsyslog/ommail.so: cannot open shared object
> >>> file: No such file or directory
> >>>
> >>> To fix this, you'll need to reinstall. This time, when you run
> >>> ./configure be sure to include --enable-mail. make install clean and
> >>> you should be good to go...
> >>>
> >>> -HKS
> >>>
> >>> On Thu, Jul 24, 2008 at 5:01 PM, (private) HKS <hks.private at gmail.com
> >>> >
> >>> wrote:
> >>>> A few things to try:
> >>>>
> >>>> - You load ommail.so twice, once at the top and once right above
> >>> your
> >>>> $ActionMail... lines. I don't think this will break it, but it's
> >>>> unnecessary - delete the second one.
> >>>>
> >>>> - $ActionExecOnlyOnceEveryInterval 21600 means that it's only going
> >>>> to attempt sending a message once every 6 hours. For testing
> >>> purposes,
> >>>> this will be obnoxious. Until you're ready for production, change
> >>>> it
> >>>> to:
> >>>> $ActionExecOnlyOnceEveryInterval 30
> >>>>
> >>>> - Send everything to make sure you're not missing it based on
> >>>> something in the property-replacer/templates/whatever. Replace "if
> >>>> $msg contains 'hard disk fatal failure' then :ommail:;mailBody"
> >> with:
> >>>> *.* :ommail:;mailBody
> >>>>
> >>>> Try again. Try logging a few messages from the localhost first
> >>>> (with
> >>>> RHEL you can just run "logger test" to log a user.notice message
> >> with
> >>>> content "test") and see if you get them.
> >>>>
> >>>> If you don't, check the mail logs on your mail server to see if it
> >>>> ever received the message. If not, it's time to break out tcpdump
> >> and
> >>>> see if the packets are ever being generated.
> >>>>
> >>>> Hope that helps.
> >>>>
> >>>> -HKS
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Jul 24, 2008 at 3:50 PM, Goutos, Kevin (DOB)
> >>>> <Kevin.Goutos at budget.state.ny.us> wrote:
> >>>>> Hello all,
> >>>>>
> >>>>> First off, I am not very Linux savvy. I don't have a lot of
> >>> experience
> >>>>> other then a basic course. This is probably way past my knowledge,
> >>> but I
> >>>>> really need to get it done. Appreciate any help you guys have to
> >>> offer.
> >>>>>
> >>>>> I am working on a Red Hat Enterprise 4 box and I am running the
> >>> latest
> >>>>> edition of rsyslog. I currently have rsyslog configured to receive
> >>>>> messages remotely via UDP. I am trying to set it up so that it
> >>>>> will
> >>> send
> >>>>> out E-mail messages to the system Admin's based on the severity
> >>> level of
> >>>>> the log files I am receiving. I would like it so that any device
> >>> that
> >>>>> sends a log with a critical, alert, or emergency level facility
> >> will
> >>>>> send out an e-mail to a specific address.
> >>>>>
> >>>>> Here is my rsyslog.conf file. I used the sample code from Rainer
> >>>>> Gerhards configuration page. I tried sending a test syslog with
> >>> 'hard
> >>>>> disk fatal failure' in it, but it is not sending out mail. Also
> >>> tried
> >>>>> without the templates below thinking it would just send me an
> >>>>> email
> >>> for
> >>>>> every syslog that I received, but it doesn't appear to be sending
> >>> mail.
> >>>>> Any thoughts on what I am doing wrong. I'm sure there is a lot I
> >>> need to
> >>>>> do, so please let me know. Thanks!
> >>>>>
> >>>>> $template mailSubject,"disk problem on %hostname%"
> >>>>> $template mailBody,"RSYSLOG Alert\r\nmsg='%msg%'"'
> >>>>>
> >>>>> I edited out the 3 lines below in the code for security reasons..
> >>>>> $ActionMailSMTPServer <ip of smtp server>
> >>>>> $ActionMailFrom <from address>
> >>>>> $ActionMailTo <my email>
> >>>>>
> >>>>>
> >>>>>
> >>>>> Here is my code from rsyslog.conf below.
> >>>>>
> >>>>>
> >>>>>
> >>>>> # if you experience problems, check
> >>>>> # http://www.rsyslog.com/troubleshoot for assistance
> >>>>>
> >>>>> # rsyslog v3: load input modules
> >>>>> # If you do not load inputs, nothing happens!
> >>>>> # You may need to set the module load path if modules are not
> >> found.
> >>>>>
> >>>>> $ModLoad immark.so # provides --MARK-- message capability
> >>>>> $ModLoad imuxsock.so # provides support for local system logging
> >>> (e.g.
> >>>>> via logger command)
> >>>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
> >>>>> $ModLoad ommail
> >>>>>
> >>>>> $template TraditionalFormatWithPRI,"%PRI-text%: %timegenerated%
> >>>>> %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n"
> >>>>>
> >>>>> # Log all kernel messages to the console.
> >>>>> # Logging much else clutters up the screen.
> >>>>> #kern.*
> >> /dev/console
> >>>>>
> >>>>> # Log anything (except mail) of level info or higher.
> >>>>> # Don't log private authentication messages!
> >>>>> *.info;mail.none;authpriv.none;cron.none
> >>>>> -/var/log/messages
> >>>>>
> >>>>> # The authpriv file has restricted access.
> >>>>> authpriv.*
> >>> /var/log/secure
> >>>>>
> >>>>> # Log all the mail messages in one place.
> >>>>> mail.*
> >>>>> -/var/log/maillog
> >>>>>
> >>>>> # Log cron stuff
> >>>>> cron.* -
> >>> /var/log/cron
> >>>>>
> >>>>> # Everybody gets emergency messages
> >>>>> *.emerg *
> >>>>>
> >>>>> # Save news errors of level crit and higher in a special file.
> >>>>> uucp,news.crit
> >>>>> -/var/log/spooler
> >>>>>
> >>>>> # Save boot messages also to boot.log
> >>>>> local7.*
> >>>>> /var/log/boot.log
> >>>>>
> >>>>> #Catch all incoming syslog messages
> >>>>> *.*
> >>>>> /var/log/catchall;TraditionalFormatWithPRI
> >>>>>
> >>>>> # Remote Logging (we use TCP for reliable delivery)
> >>>>> # An on-disk queue is created for this action. If the remote host
> >> is
> >>>>> # down, messages are spooled to disk and sent when it is up again.
> >>>>> $WorkDirectory /rsyslog/spool # where to place spool files
> >>>>> $ActionQueueFileName uniqName # unique name prefix for spool files
> >>>>> $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as
> >>>>> possible)
> >>>>> $ActionQueueSaveOnShutdown on # save messages to disk on shutdown
> >>>>> $ActionQueueType LinkedList # run asynchronously
> >>>>> $ActionResumeRetryCount -1 # infinite retries if host is down
> >>>>> # remote host is: name/ip:port, e.g. 192.168.0.1:514, port
> >>>>> optional
> >>>>> *.* @10.57.106.140:514
> >>>>>
> >>>>> $ModLoad ommail
> >>>>> $ActionMailSMTPServer <ip of smtp server>
> >>>>> $ActionMailFrom <from address>
> >>>>> $ActionMailTo <my email>
> >>>>> $template mailSubject,"disk problem on %hostname%"
> >>>>> $template mailBody,"RSYSLOG Alert\r\nmsg='%msg%'"
> >>>>> $ActionMailSubject mailSubject
> >>>>> # make sure we receive a mail only once in six
> >>>>> # hours (21,600 seconds ;))
> >>>>> $ActionExecOnlyOnceEveryInterval 21600
> >>>>> # the if ... then ... mailBody mus be on one line!
> >>>>> if $msg contains 'hard disk fatal failure' then :ommail:;mailBody
> >>>>>
> >>>>>
> >>>>> # ######### Receiving Messages from Remote Hosts ##########
> >>>>> # TCP Syslog Server:
> >>>>> # provides TCP syslog reception and GSS-API (if compiled to
> >>>>> support
> >>> it)
> >>>>> $ModLoad imtcp.so # load module
> >>>>> $InputTCPServerRun 514 # start up TCP listener at port 514
> >>>>>
> >>>>> # UDP Syslog Server:
> >>>>> $ModLoad imudp.so # provides UDP syslog reception
> >>>>> $UDPServerRun 514 # start a UDP syslog server at standard port
> >>>>> --------------------------------------------------------
> >>>>> This e-mail, including any attachments, may be confidential,
> >>> privileged or otherwise legally protected. If you have received this
> >> e-
> >>> mail in error, or from someone who was not authorized to send it to
> >>> you, do not disseminate, copy or otherwise use this e-mail or its
> >>> attachments. Please notify the sender immediately if you have
> >>> received
> >>> this e-mail by mistake, and delete it from your system.
> >>>>> _______________________________________________
> >>>>> rsyslog mailing list
> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>>
> >>>>
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>
> ########################################################################
> Ryo Fujita <rfujita at redhat.com>
> Senior Solution Architect, RHCE
> Red Hat K.K.
> TEL +81-3-5798-8500
> FAX +81-3-5798-8599
> Ebisu Neonato 8F
> 4-1-18 Ebisu, Shibuya-ku,
> Tokyo Japan 1500013
> ########################################################################
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
More information about the rsyslog
mailing list