[rsyslog-notify] Forum Thread: Re: How to Determine Message Drops ? - (Mode 'reply')

[noreply at adiscon.com]

User: dlang 
Forumlink: http://kb.monitorware.com/viewtopic.php?p=25951#p25951

Message: 
----------
[quote="jtsai.cp":10r699nz]Great explanation, thank you.

1. I need to ship logs for 900 servers/devices total.  Current test, I have
110 Windows servers shipping logs to RSyslog each with it's own file a CIFS
Share (our Senior Admin demanded writing to Windows share each server per
files, no batch copies but direct writes) there are over 100 open files, we
are writing to Windows 2012 R2 share.  On top of that, the Windows box
reboot weekly (another issue that is killing RSyslog, seems autofs not too
good or badly configured).   
[/quote:10r699nz]
Ok, if you have this many systems logging (and each ending up in a
different file) with the default settings, it's going to be dog slow.

either configure $DynaFileCacheSize > # systems (probably significantly
greater to be on the safe side), or set it as part of the action()
statement in the new style config, see <!-- m --><a class="postlink"
href="http://www.rsyslog.com/doc/master/configuration/modules/omfile.html">http://www.rsyslog.com/doc/master/confi
... mfile.html</a><!-- m -->

[quote:10r699nz]
2. Performance wise, the SAN, ESXi, VM never gets too heavily used in terms
of IOPS, transfer, CPU, and memory.  We are running new 3par, sas/ssd,
fibre, HP gen9, 10Gb etc.... I can give it as much resources as it is
needed to handle the load it is capable of.
[/quote:10r699nz]
except that if you serialize things, you can't get a disk fast enough
[quote:10r699nz]
3. Moving the "*.* ?RemoteLogs" down toward the bottom causes all logs goes
into log files under /var/logs/messages and bootlog and it never gets
processed, one big log file locally instead of specified one for each host
on CIFS share.
[/quote:10r699nz]

It's not the ?RemoteLogs, it's the line after that that throws all the logs
away that causes everything after that to be ignored.

[quote:10r699nz]

4. I should move all the following above the "*.* ?Remote....." ?
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down
$ActionQueueWorkerThreadMinimumMessages 1000
$ActionQueueWorkerThreads 5
$ActionQueueWorkerTimeoutThreadShutdown 0
$ActionQueueDequeueBatchSize 5000
$ActionQueueSize="100000"
[/quote:10r699nz]
you should not need an action queue, in fact an action queue is likely to
slow things down rather than help. You need to set the dequeue batch size
on the main queue (and again, don't use multiple threads, it's going to
hurt rather than help in almost all cases)

[quote:10r699nz]
5. I did set the OMBuffer, maybe need to be larger...
$OMFileIOBufferSize 256k[/quote:10r699nz]
[/quote]
no, that's not the problem

With your current config, you only allow rsyslog to have 10 of the
RemoteLog files open at once. When the 11th server delivers a message,
rsyslog has to close one of the 10 files, which means flushing all pending
writes, and then open the file for the 11th server. Repeat for the 12th
server, etc

set that one parameter to 1000+ and you will see things speed up
drastically.

David Lang