[rsyslog] rsyslog error messages

[Rainer Gerhards]

inline...

On Fri, 2008-07-25 at 11:44 -0400, (private) HKS wrote:
> Responses inline.
> 
> > I totally agree, but I think the situation is even worse for a syslogd.
> > Because it *is* the usual error reporting system. I think we are a bit
> > in the same situation (if you allow that analogy) like HIV which is
> > attacking the human immune system. We have a problem in the
> > troubleshooting system itself, and it may even prevent us from conveying
> > troubleshooting information :(
> 
> That's a good way of putting it. This is one of the reasons I think
> it's so important to print errors to STDERR - if the logging
> mechanisms are broken for some reason, then at least they're going
> somewhere. And I sincerely hope that users have the sense to start it
> from the command line the first time rather than sticking it into
> their system startup scripts and rebooting.

If I look at some of the support calls I see, I doubt that ;) The
problem is that there are a lot of novices who have never ever done any
real system administration. But as the rsyslog homepage says: "Its
advanced features make it suitable for enterprise-class, encryption
protected syslog relay chains while at the same time being very easy to
setup for the novice user." Look at the end of the sentence. In any
case, I see the utility of what you say, if not in the novice case, then
with more experienced users.

> >>  It's happened a few times that I start
> >> rsyslogd at the command line, and yet there's nothing in the process
> >> table, and nothing dumped to my logs.
> >
> > Could you elaborate a bit about above sentence - I have to admit I do
> > not fully get you...
> 
> I've messed up my configs bad enough a couple times that when I start
> rsyslogd, nothing apparently happens. No errors to STDERR, no errors
> in my logs, and the process isn't running. Rewriting the config or
> running with -dn is the solution.

Ah, ok, now I understand. Thanks.

> 
> 
> 
> >> 2 - Configuration and other runtime errors should be printed to STDERR
> >
> > Does that really help? If so, that would be extremely easy to add. But
> > does somebody see them?
> 
> Absolutely. The first time I configure any daemon, I do it in a test
> environment and run it by hand until I have the configuration acting
> exactly as I want. 

I understand. That makes a lot of sense. As I said, it should be easy to
implement. I'll once again shuffle priorities a bit and make that the
top one. Should not take too long to implement. I'll probably have a
switch to disable it, but being enabled by default.

> Most users take the "run-by-hand" precaution even
> if they don't have the luxury of a test setup. These will also pop up
> when a user tries to restart the daemon after making a config change.
> I'm not sure if they'll be visible when a HUP signal is sent, but that
> would make me awfully happy.

So expect to be happy some time next week ;)

> 
> 
> >> 3 - By default, rsyslogd should log all of its own errors (fatal or
> >> not) to /var/log/rsyslog.log. This should also be user-configurable
> >
> > This default comes into the way of some distros: they use different
> > pathes. I really don't like the idea of embedding any information about
> > the file system into the binary. Just think about embedded systems...
> 
> Good point.
> 
> 
> >> 1 - If rsyslog doesn't understand your config file, obviously it won't
> >> be doing what you intend it to. There's little benefit in running with
> >> a horked configuration, but if you don't test thoroughly, the cost
> >> could be devastating. Some users won't know they're not logging things
> >> until they need them - then, if they keep their jobs, they probably
> >> won't keep rsyslog ("This software doesn't even log ssh attempts!").
> >
> > That's quite controversal and and does not match rsyslog's root
> > philosophy. Let me explain:
> >
> > The syslogd is an extremely important piece of software. If it is not
> > functioning,
> >
> > a) vital data can be lost
> > b) the system as whole may hang (due to the fact that the log
> >   socket gets filled up and so other processes block on it)
> >
> > As such, rsyslogd is written in a sense that it tries to continue to
> > work under almost all circumstances. It even tries to survive a
> > system-wide low memory condition, though I am not 100% convinced it will
> > manage to do that in all cases. At least it is coded careful to survive
> > as much as possible.
> >
> > Now let's assume that a configuration error happens that prevents
> > rsyslogd from executing an otherwise perfect configuration. Should we
> > actually stop it from running? And live with the results? Or should we
> > do whatever we can do and do that as good as possible? What, if it is
> > not a configuration issue but a hard disk failure or an admin error that
> > make part of the config unavailable (e.g. not able to load module, not
> > able to write file)? Assuming we can carry out other actions - should we
> > really abort? This may even come at the price that we can NOT warn users
> > (via wall) of a serious problem. And what with warnings? Things that
> > look wrong, but may be OK. Reason to abort? Or continue run? Tough
> > decision. So I still think the right thing to do is to run as long as
> > there is limited sense in it, because everything else would actually be
> > worse. Please note that rsyslogd even comes with a hardcoded emergency
> > configuration which is used in case the real configuration file can not
> > be obtained.
> >
> > What I could add, however, is an option to make it stop on any config
> > error (not warning). However, I fear that, if turned on, the results can
> > be fatal in some cases...
> 
> You make a lot of good points here. Does the hardcoded emergency
> configuration that rsyslogd uses actually log?

Yes, but in a very limited sense. It puts *.err onto the console, does a
wall for *.panic and writes everything else to the controlling tty. I
have to admit that this code isn't awfully well tested, usually only
when something is done at the config code (and sometimes not even
then...).

>  I only ask because it's
> easy (if you fail to read the documentation) to write a config file
> that doesn't do anything. Would it make sense for rsyslog to determine
> whether the passed configuration actually asks it to do anything, and
> complaining if not?

That's an interesting idea. I am not sure if this is already checked. I
can at least check if there is no action at all, which means there is no
useful work to be done. The question remains, as usual, where to
complain to. But that could be another situation where the hardcoded
config could be activated.

I have also thought about an implicit write user, with the user being
root by default (and being overwritable via command line and config
file).

> 
> Printing errors to STDERR would help with this. Another possibility
> might be adding a flag (-n is typical for a lot of systems, but that's
> already in use) that will tell rsyslog to parse the configuration,
> spit out any problems, and quit. That way, administrators can modify a
> config file on a live system and verify it's syntax without hosing the
> actual rsyslogd process.

That's also a good idea. Rsyslog is quite modular (not yet as much as I
would like it to be, but well enough). So I could even generate a
dedicated config check tool. That would not touch any running
configuration at all. But maybe the switch is easier ;) I need to think
on how this affects the testbed I have begun to work on. Looks like this
could also drive some automated tests.

> 
> 
> 
> >> 3 - For those more subtle problems, and things you might miss upon
> >> bootup. This also gives you a simple, easily accessible debugging
> >> source with minimal security implications and additional code and no
> >> required user configuration. I'm not sure how the code is written, but
> >> I'd prefer to see this happen even if the configuration file can't be
> >> read. Perhaps an $RsyslogErrors directive that, by default, references
> >> a different module that just dumps directly to a file? Users can
> >> configure it to put rsyslog errors through the actual logging engine
> >> and then manage it with typical syslog selector/action lines.
> >
> > All internally-generated messages are run through a specific interface.
> > For the "diag plugin", I will define a hook, where a diag server can
> > register to receive these messages. Maybe a good compromise would be to
> > add this $RsyslogErrors directive and make it point to a file. So,
> > still, we do not necessarily have it, but package maintainers would
> > hopefully include it and point it to a specific file (what still means I
> > do not necessarily know where to look). A problem is source tarball
> > install, in which case the directive may not be set. But with a proper
> > sample rsyslog.conf...
> 
> I think a locally installed sample rsyslog.conf is a solid workaround
> for the "different distros, different locations" problem.
> 
> >
> >> The HTTP server is an interesting idea, but not something I would make
> >> use of. The potential security problems are enormous, and it would
> >> introduce too many additional factors to the debugging process: is the
> >> firewall right? Was there already a process listening on that port? Is
> >> the user's configuration file correct? These are the kinds of things
> >> you already have to deal with on rsyslog itself - why force the
> >> debugging process to go through them again?
> >
> > I see (and share) much of your argument. This is why I stayed away for
> > such a long time. But if you think a bit more about it, it has *very
> > interesting* capabilities. Eg. I may also use it to gather some
> > real-time view of internal state, emit commands like HUP and do a number
> > of other nice things. Again, security is problematic.
> >
> > In any case, I'll give it a try. I have started to write a small plugin
> > which will dump the error messages within 10 minutes after startup. It
> > will not be a real http server but rather a listener that response to
> > anyone who connects to it (so that nc can be used as a client ;)). This
> > is very rough, but it has two side-effects:
> >
> > a) it provides something to actually experiment with (and to extend if
> >   we decide to keep that route)
> > b) it helps create the plumbing inside the internal interface that could
> >   also be used by any other method, so the potential waste of time is
> >   limited
> 
> It sounds interesting, and I agree that there's a lot of potential
> there. Among them, the possibility of adding an XML-RPC API to allow
> for greater interface flexibility...but frankly, that scares the hell
> out of me. ;) I'll wait to see what you come up with.

hehe... We'll see, but I'd say its faaar down the road. One interesting
aspect, though, could be a realtime view of syslog data as it is flowing
through the system. That could even be integrated into phpLogCon.

In any case, I do NOT want to add much (http) application logic /
presentation to such a system. But a few html (or xml ;)) pages
generated at appropriate times could be really useful. Just think about
things like a "granular HUP", like being able to tell to close some
files specifically etc etc. But that's too far advanced. For me
personally, it would be quite useful to get access to state variables
(like mem utilization, object instances, error count, frame rate) of the
running system - not under a debugger but in a real live system. From
the developers (and troubleshooter's point of view), that would be very
interesting. I could even hold the last 2000 or so messages of the debug
log in memory, plus the initial first 1000. Together, I guess, they can
solve over 90% of all things I have used debug logs for. But this can
only be done if I have dynamic runtime access to the live instance. And
this obviously also means there is a big security issue ;)

Please keep the thoughts flowing, we make good progess :)
Rainer
> 
> -HKS
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog