[rsyslog-notify] Forum Thread: Time lag issue of forwarding vs available I/O at 6600 mps - (Mode 'edit_topic')

[noreply at adiscon.com]

User: rajatg 
Forumlink: http://kb.monitorware.com/viewtopic.php?p=26779#p26779

Message: 
----------
Hi,

I have setup a rsyslog centralized remote logging server to which 5 clients
forward their logs.

On the client there is a radosgw process which logs to /var/log/syslog
which inturns forward the logs through TCP to centralized server. For a use
case of generating nearly 1 Mbps logs (1500 messages per second), with a
default in-memory mainmsgqueuesize, I am observing that rsyslog is not
leaving any I/O available for radosgw process, because of which the
throughput of radosgw goes down (request serving time increases). Is there
any other alternative so that the radosgw process need not wait for the
I/O? Client config for the above I/O race scenario is below-:
[code:2b2kh2am]
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog   # provides kernel logging support (previously done by
rklogd)
$RepeatedMsgReduction on
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirOwner syslog
$DirGroup adm
$DirCreateMode 0755
$PrivDropToUser syslog
$PrivDropToGroup syslog
$WorkDirectory /var/spool/rsyslog
$EscapeControlCharactersOnReceive off

$WorkDirectory /var/spool/rsyslog    # where to place spool files
$ActionQueueFileName queue      # unique name prefix for spool files
$ActionQueueSaveOnShutdown on   # save messages to disk on shutdown
$ActionQueueType LinkedList     # run asynchronously, enables a FixedArray
in-memory queue
$ActionResumeRetryCount -1      # infinety retries
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat


if $programname == 'lt-radosgw' or $programname == 'radosgw' or $msg
contains 'radosgw' then @@10.140.222.240:514;

 # Custom fragment
 undef
[/code:2b2kh2am]




To solve the above I/O race issue, I tried to implement a disk-assisted
in-memory MainMsgQueue. This solved the I/O problem. But now, too many
unprocessed messages are found in the main queue (using impstats), i.e.
processing of the messages is not sufficiently fast as compared to incoming
requests. This results in a very long lag in dumping to the /var/log/syslog
as compared to the real-time log generation of radosgw.log. I have tried to
use both batch processing(DequeueBatchSize) and multi-processing(increasing
worker threads) , hoping that they would improve the message processing
speed, but I have not found any success till now. 
Parameters which I added to the Client Configurations for a large queue are
below-:
[code:2b2kh2am]
$MainMsgQueueSize 100000
$MainMsgQueueType LinkedList
$MainMsgQueueFileName mainqueue
$MainMsgQueueHighWatermark 80000
$MainMsgQueueLowWatermark 60000
$MainMsgQueueWorkerThreads 20
$MainMsgQueueDequeueBatchSize 50000
[/code:2b2kh2am]

Central server configuration are-:

[code:2b2kh2am]
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog   # provides kernel logging support
$ModLoad imtcp
$InputTCPServerRun 514
$KLogPermitNonKernelFacility on
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$RepeatedMsgReduction on
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog

[/code:2b2kh2am]

I have spent 3 long days on this problem, but I am not figure out the
solution in spite of knowing the actual bottleneck. I know that according
to <!-- m --><a class="postlink"
href="http://www.gerhards.net/download/LinuxKongress2010rsyslog.pdf">http://www.gerhards.net/download/LinuxK
... syslog.pdf</a><!-- m --> that rsyslog can easily handle the amount of
messages I am generating, but I don't know what am I missing in the config
right now.

Thanks,
Rajat