[rsyslog] rsyslog and holding on to dyna (daily) files

[Rainer Gerhards]

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Jeffrey 'jf' Lim
> Sent: Monday, April 27, 2009 9:37 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog and holding on to dyna (daily) files
> 
> On Mon, Apr 27, 2009 at 2:49 PM, Rainer Gerhards
> <rgerhards at hq.adiscon.com> wrote:
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Jeffrey 'jf' Lim
> >> Sent: Monday, April 27, 2009 8:45 AM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] rsyslog and holding on to dyna (daily) files
> >>
> >> On Mon, Apr 27, 2009 at 2:32 PM, Rainer Gerhards
> >> <rgerhards at hq.adiscon.com> wrote:
> >> >
> >> > That has nothing to do with RELP. The issue here is that the file
> >> output
> >> > writer (in v3) uses the sysklogd concept of "if I can't write it,
> >> I'll throw
> >> > it away". This is another issue that was "fixed" in v4 (not really
> a
> >> fix, but
> >> > a conceptual change).
> >> >
> >>
> >> that's good to know. So how does it deal with this situation now?
> >
> >
> > I v3 ... that's the way it is...
> >
> 
> yeah, I mean, how does it work now in v4?

Just like the database output: it puts itself onto hold, core queues messages
and retries action. Once it succeeds (disk space available again), writing
resumes. Note that other than databases partial records maybe written at the
point of failure, resulting in potential mild message duplication. There are
ways to prevent this, but I did not yet have time to do that.

> 
> >> >
> >> > One approach (which I could implement and have nearly done a few
> >> times when
> >> > something else distracted me) is the ability to configure the
> >> dynafile cache
> >> > size. Let's say you know you have 5 hosts and write daily files,
> so
> >> you then
> >> > could set the cache size to 5 what means that the "older" files
> will
> >> be
> >> > closed as soon as a message from each host is received the next
> days.
> >> Not a
> >> > real timeout, but probably a useful work-around.
> >> >
> >>
> >> the timeout is preferable (otherwise you'd have to keep a proper
> count
> >> (?) of the number of dynafiles that you need). But sure, as a
> >> workaround, it definitely is useful.
> >>
> >
> > I agree, but someone must implement it. If I look at my current
> activity
> > schedule, I'd say in 2 month at earliest ;) If we do timeouts, we
> should do
> > them "right", that is provide core functionality for them. Quite some
> work,
> > but with benefit for many more than just dynafiles.
> >
> 
> right. Perhaps i'll see what i can do to implement the dynafile count
> in my own version of rsyslog, and then let you have a look? Or is
> there another way to do this that's better than the cache size
> workaround, and not so complicated as the timeout? Perhaps have some
> way to mark the template as a "daily or time-based template" - and
> then have rsyslog keep at most only n (configurable) copies of this
> dynafile? (yes, this is similar to your cache size - but at least this
> is per template - rather than rsyslog-wide).

My config setting would be action-specific, which is even more granular than
on a per-template basis. It is also far easier to do, so I'd still vote for
that.

Regarding the timeout, if you provide a sufficiently clean patch, I'll gladly
look at it. I have to admit that I currently have limited interest in
enhancing the file output as it is scheduled for total revamping and from my
POV (definitely not yours, I understand that), every effort I put into it
before this large re-design is actually wasted... Sorry to bet blunt, but I
think it is best if folks understand the motivation behind my moves.

Rainer
> 
> -jf
> 
> --
> In the meantime, here is your PSA:
> "It's so hard to write a graphics driver that open-sourcing it would
> not help."
>     -- Andrew Fear, Software Product Manager, NVIDIA Corporation
> http://kerneltrap.org/node/7228
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com