[rsyslog] Disappearing buffered remote logs on reboot/boot

[david at lang.hm]

My understanding is that rsyslog attempts to save all the data to disk 
when it shuts down, but it also has a timeout to keep it from sitting 
forever if it is having trouble doing so.

you may want to try setting this timeout longer.

you may also need to do the disk-assist version of the queue rather than 
one of the pure memory queues.

this is an aread I haven't worked with much.

RELP would help get the logs to the queue, but doesn't have any effect on 
the shutdown behavior of rsyslog.

David Lang

  On Fri, 3 Dec 2010, Bgs wrote:

> New info from tests:
>
> I switched to disk from linked list. This solved the logs lost on rsyslogd 
> shutdown. These are normal clients so the performance impact is negligible. 
> I'd spare unneeded IO, but I can live with it. I also set the retry timer to 
> 10 seconds which seems to bring efficiency up a lot. After reboot, logs are 
> not all lost. But there still are some. It looks like the first portion of 
> the logs (written into the spool file but probably also still in memory) 
> waiting to be sent is handled as ok, while other logs in the spool file are 
> handled correctly as unsent and sent in after boot.
>
> For example I did the following test: I pulled the plug to simulate a network 
> outage. Made 1500 log entries using logger and numbered them.  All entries 
> appeared in the local 'messages' and also in to spool file. I rebooted the 
> system which came up with network connection. I got a bunch of logs on the 
> server, starting with number 756 and all test entries up to number 1500. The 
> first 755 entries were lost.
>
> So the questions that remain:
>
> 1) Are the lost entries related to tcp socket buffers or log entry tagging?
> 2) If the later, is there a way to keep the unsent messages with the 
> appropriate status?
> 3) If it's socket related, can anything be done about it? Is this the 
> described loss possibility for tcp syslog?
> 4) Would RELP solve this problem? (Reliably handling the entries).
> 5) Is there a way in linkedlist mode to tell rsyslogd to write out all 
> buffered data to the spool? With file logs this can beg done with a HUP, but 
> that does not force rsyslogd to write out the queue too...
>
> Regards
> Bgs
>
>
> On 12/02/2010 06:22 PM, Bgs wrote:
>>
>>  Hi all,
>> 
>> I ran into a strange problem. My setup:
>> 
>> Ubuntu, rsyslog with remote tcp logging to central syslog server. 
>> Connection is periodic, so I need the buffer to file ability of rsyslog.
>> Ubuntu 10.04LTS, rsyslog 4.2.0-2ubuntu8, using imuxsock module for remote 
>> logging.
>> Config:
>>  queue type = linkedlist
>>  workdirectory/filename/maxdiskpace set
>>  saveonshutdown on
>>  retrycount -1
>> 
>> What I would expect when there is no connection at reboot is: save in 
>> memory buffer to file (create if there was nothing there yet), at boot open 
>> it and send it in at the first opportunity.
>> 
>> What actually happens: A portion of the buffer is saved to file (checked by 
>> booting from outside and looking at the rsyslog spool) and when the system 
>> boots up, the file disappears without any content showing up on the remote 
>> side. After the last message before connection break, the next one that 
>> gets through are the boot logs.
>> 
>> During normal operation, connection breaks are handled without any problem.
>> 
>> What might the problem be? Configuration problem? Ubuntu upstart related? 
>> rsyslog bug?
>> 
>> Regards
>> Bgs
>> 
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>> 
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>