[rsyslog] rsyslog error messages

[(private) HKS]

Responses inline.

> I totally agree, but I think the situation is even worse for a syslogd.
> Because it *is* the usual error reporting system. I think we are a bit
> in the same situation (if you allow that analogy) like HIV which is
> attacking the human immune system. We have a problem in the
> troubleshooting system itself, and it may even prevent us from conveying
> troubleshooting information :(

That's a good way of putting it. This is one of the reasons I think
it's so important to print errors to STDERR - if the logging
mechanisms are broken for some reason, then at least they're going
somewhere. And I sincerely hope that users have the sense to start it
from the command line the first time rather than sticking it into
their system startup scripts and rebooting.


>>  It's happened a few times that I start
>> rsyslogd at the command line, and yet there's nothing in the process
>> table, and nothing dumped to my logs.
>
> Could you elaborate a bit about above sentence - I have to admit I do
> not fully get you...

I've messed up my configs bad enough a couple times that when I start
rsyslogd, nothing apparently happens. No errors to STDERR, no errors
in my logs, and the process isn't running. Rewriting the config or
running with -dn is the solution.



>> 2 - Configuration and other runtime errors should be printed to STDERR
>
> Does that really help? If so, that would be extremely easy to add. But
> does somebody see them?

Absolutely. The first time I configure any daemon, I do it in a test
environment and run it by hand until I have the configuration acting
exactly as I want. Most users take the "run-by-hand" precaution even
if they don't have the luxury of a test setup. These will also pop up
when a user tries to restart the daemon after making a config change.
I'm not sure if they'll be visible when a HUP signal is sent, but that
would make me awfully happy.


>> 3 - By default, rsyslogd should log all of its own errors (fatal or
>> not) to /var/log/rsyslog.log. This should also be user-configurable
>
> This default comes into the way of some distros: they use different
> pathes. I really don't like the idea of embedding any information about
> the file system into the binary. Just think about embedded systems...

Good point.


>> 1 - If rsyslog doesn't understand your config file, obviously it won't
>> be doing what you intend it to. There's little benefit in running with
>> a horked configuration, but if you don't test thoroughly, the cost
>> could be devastating. Some users won't know they're not logging things
>> until they need them - then, if they keep their jobs, they probably
>> won't keep rsyslog ("This software doesn't even log ssh attempts!").
>
> That's quite controversal and and does not match rsyslog's root
> philosophy. Let me explain:
>
> The syslogd is an extremely important piece of software. If it is not
> functioning,
>
> a) vital data can be lost
> b) the system as whole may hang (due to the fact that the log
>   socket gets filled up and so other processes block on it)
>
> As such, rsyslogd is written in a sense that it tries to continue to
> work under almost all circumstances. It even tries to survive a
> system-wide low memory condition, though I am not 100% convinced it will
> manage to do that in all cases. At least it is coded careful to survive
> as much as possible.
>
> Now let's assume that a configuration error happens that prevents
> rsyslogd from executing an otherwise perfect configuration. Should we
> actually stop it from running? And live with the results? Or should we
> do whatever we can do and do that as good as possible? What, if it is
> not a configuration issue but a hard disk failure or an admin error that
> make part of the config unavailable (e.g. not able to load module, not
> able to write file)? Assuming we can carry out other actions - should we
> really abort? This may even come at the price that we can NOT warn users
> (via wall) of a serious problem. And what with warnings? Things that
> look wrong, but may be OK. Reason to abort? Or continue run? Tough
> decision. So I still think the right thing to do is to run as long as
> there is limited sense in it, because everything else would actually be
> worse. Please note that rsyslogd even comes with a hardcoded emergency
> configuration which is used in case the real configuration file can not
> be obtained.
>
> What I could add, however, is an option to make it stop on any config
> error (not warning). However, I fear that, if turned on, the results can
> be fatal in some cases...

You make a lot of good points here. Does the hardcoded emergency
configuration that rsyslogd uses actually log? I only ask because it's
easy (if you fail to read the documentation) to write a config file
that doesn't do anything. Would it make sense for rsyslog to determine
whether the passed configuration actually asks it to do anything, and
complaining if not?

Printing errors to STDERR would help with this. Another possibility
might be adding a flag (-n is typical for a lot of systems, but that's
already in use) that will tell rsyslog to parse the configuration,
spit out any problems, and quit. That way, administrators can modify a
config file on a live system and verify it's syntax without hosing the
actual rsyslogd process.



>> 3 - For those more subtle problems, and things you might miss upon
>> bootup. This also gives you a simple, easily accessible debugging
>> source with minimal security implications and additional code and no
>> required user configuration. I'm not sure how the code is written, but
>> I'd prefer to see this happen even if the configuration file can't be
>> read. Perhaps an $RsyslogErrors directive that, by default, references
>> a different module that just dumps directly to a file? Users can
>> configure it to put rsyslog errors through the actual logging engine
>> and then manage it with typical syslog selector/action lines.
>
> All internally-generated messages are run through a specific interface.
> For the "diag plugin", I will define a hook, where a diag server can
> register to receive these messages. Maybe a good compromise would be to
> add this $RsyslogErrors directive and make it point to a file. So,
> still, we do not necessarily have it, but package maintainers would
> hopefully include it and point it to a specific file (what still means I
> do not necessarily know where to look). A problem is source tarball
> install, in which case the directive may not be set. But with a proper
> sample rsyslog.conf...

I think a locally installed sample rsyslog.conf is a solid workaround
for the "different distros, different locations" problem.

>
>> The HTTP server is an interesting idea, but not something I would make
>> use of. The potential security problems are enormous, and it would
>> introduce too many additional factors to the debugging process: is the
>> firewall right? Was there already a process listening on that port? Is
>> the user's configuration file correct? These are the kinds of things
>> you already have to deal with on rsyslog itself - why force the
>> debugging process to go through them again?
>
> I see (and share) much of your argument. This is why I stayed away for
> such a long time. But if you think a bit more about it, it has *very
> interesting* capabilities. Eg. I may also use it to gather some
> real-time view of internal state, emit commands like HUP and do a number
> of other nice things. Again, security is problematic.
>
> In any case, I'll give it a try. I have started to write a small plugin
> which will dump the error messages within 10 minutes after startup. It
> will not be a real http server but rather a listener that response to
> anyone who connects to it (so that nc can be used as a client ;)). This
> is very rough, but it has two side-effects:
>
> a) it provides something to actually experiment with (and to extend if
>   we decide to keep that route)
> b) it helps create the plumbing inside the internal interface that could
>   also be used by any other method, so the potential waste of time is
>   limited

It sounds interesting, and I agree that there's a lot of potential
there. Among them, the possibility of adding an XML-RPC API to allow
for greater interface flexibility...but frankly, that scares the hell
out of me. ;) I'll wait to see what you come up with.

-HKS