[rsyslog] omfile redesign / $OutChannel will go away!

[david at lang.hm]

On Tue, 30 Jun 2009, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>
>> On Tue, 30 Jun 2009, Sergei Zhirikov wrote:
>>
>>> Rainer Gerhards wrote:
>>>> Hi all,
>>>>
>>>> The most prominent use case is to size-limit files. $OutChannel provides
>> the
>>>> ability to execute a command when a certain file size is reached. The
>> command
>>>> may then rotate the file. After command execution, omfile checks the
> file
>>>> size again. If it still is above the configured limit, this file will be
>>>> disabled (no longer be written to until restart).
>>>>
>>>> If you use this for size-limiting, please let me know how you do it.
> Please
>>>> also let me know if you would like to see things handled differently.
>>>>
>>>> Looking forward to your feedback.
>>>>
>>>> Thanks,
>>>> Rainer
>>>>
>>>
>>> Hi Rainer,
>>>
>>> I do use $OutChannel for size-limiting and it would be a disappointment
>>> to me if that feature disappeared. As a matter of fact, it was the main
>>> reason why I chose to use rsyslog in the first place.
>>>
>>> I find the "classic" way of using cron+logrotate+SIGHUP quite ugly and
>>> in some cases unacceptable. I have a small home gateway with read-only
>>> main file system, so all the log files are written into a tmpfs mount.
>>> It happened a couple of times that some misconfigured or malfunctioning
>>> software produced a lot of logging and filled in most of the available
>>> RAM before the next scheduled run of logrotate. In this situation the
>>> size limiting feature of rsyslog is exactly what I need in order to be
>>> able to avoid such this kind of problems.
>>
>> this is a very legitimate use. I suspect the biggest issue is how to give
>> you this capability without slowing down the people who don't use it.
>>
>> I wonder if rainerscript could be extended to where it could be used to do
>> this?
>>
>> something like:
>>
>> for each message logged, add the length of the message to a counter, and
>> have an if statement that calls the external program to roll the log and
>> reset the counter.
>
> I probably should have phrased the initial question less agressive, but than
> I'd probably not received any feedback. From the feedback I received so far,
> there seem to be two use cases:
>
> a) circular logs
> b) stop logging when file hits max size (was that really what this post
> described?)

no, this is a third case
c) limit the size of the logfile that is being added to, and when it hits 
this limit roll it off to permanent storage (at which point the file 
becomes read-only)

this is a need at both the low-end and the high-end

for this poster, he needs it at the low end as his 'working' files are in 
ram (his /var/log/messages), but then he rolls it off to network storage.

at this high end you can have large volumes of logs arriving, and want to 
have ver fast (but expensive) storage to handle the high file modification 
rate, but want to make sure that you don't run out of space there by 
rolling the logs off to slower, but larger disks so that they don't 
fillthe fast drive and cause you to loose logs.

> In any case, these two seem to be the only things that are done with
> outchannel. So when $outchannel goes away (and it will at some time), the
> "regular" file writer will have options to support these use cases - probably
> more elegant than before. For example, for case a), circular logs, I do not
> need to involve an external process, so I can remove that feature without
> harm, as long as I replace it with something suitable (and think that omfile
> now uses the stream writer, which *already does* circular "logging" for disk
> queues).

that sounds reasonable

> So functionality in actual need will not go away, as long as people tell what
> they need.
>
> If I look at the b) case, I suspect that this is not even the intent of the
> original poster. I think he would be much better served with a circular log
> that records as many log records as fit into a finite, operator-specified log
> file size. Or am I wrong here? (If nobody tells me I am wrong, I'll probably
> remove the b) use case in the long term!).

no, it's not that total space for logs is limited to a fixed amount, it's 
that you have two spaces for logs, a small fast space and a large slow 
space, and you need to not run out of the fast space by moving logs to the 
slow space 'soon enough'


from a security point of view there are two arguments about option b

on the one hand, if a bad guy can run you out of space he can then do new 
things that aren't logged.

on the other hand, if you have a circular buffer, the bad buy can generate 
enough logs to cause you to overwrite the evidence of him breaking in to 
your system

there are places for both.

David Lang