[rsyslog] 100% cpu lockup rsyslog 8.29 on FreeBSD 11.0Rp12 amd64 - bad config?

David Lang david at lang.hm
Fri Sep 22 02:43:23 CEST 2017


On Thu, 21 Sep 2017, Dave Cottlehuber wrote:

> hi folks,
>
> for the last few weeks I have been seeing rsyslog locking up a single
> cpu at 100% on multiple servers. All are running FreeBSD 11.0R amd64p12,
> and rsyslog 8.29, although the lockup has been seen with prior patch
> levels and rsyslog versions.
>
> The general architecture is as follows:
>
> - rsyslog replaces normal freebsd syslog
> - listens on UDP & TCP both on jails and on main host
> - does basic template tidyups to correct hostnames
> - forwards these over a spiped[1] secure connection rather than inbuilt
> TLS to a centralised rsyslog logging host
> - central rsyslog sticks these in a file and also throws them into
> graylog[2]
> - the local rsyslog is supposed to be resilient to reboots of the
> central system, and to buffer locally via queue any messages in case of
> e.g. reboots or transient network issues, but I don't believe this is
> working as expected
>
> Is it possible my config[3] is somehow incorrect, and somehow causing
> this lock up?
>
> [1]: http://www.tarsnap.com/spiped.html
> [2]: https://graylog.org/
> [3]:
> https://gist.github.com/dch/c5dcb7b6159dc2b070aefe89207c3a92#file-rsyslog-conf
>
>
> # /usr/local/etc/rsyslog.conf
> # central logging server is largely the same except has an additional
> stanza for graylog
> # Load Modules
> module(load="imtcp")
> module(load="imklog")
> module(load="imudp")
> module(load="imrelp")
> module(load="omrelp")
> module(load="imuxsock")
> module(load="immark")
> module(load="impstats")
> module(load="imfile" PollingInterval="5")
>
> # Disk Queues
> # zfs create -o canmount=off zroot/var/spool
> # zfs create zroot/var/spool/rsyslog
> # mkdir -m0700 /var/log/rsyslog
> $WorkDirectory /var/spool/rsyslog
> $MaxMessageSize 64k

one thing that's wrong is the following ActionQueue lines, they aren't changing 
settings on the main queue, they are instead setting up a queue for the next 
legacy action (which if I am reading things corectly is the NSSWITCH test)

As a result, the main queue is running on it's default settings (any queue 
settings really should be done with the new syntax as it makes errors like this 
much more obvious)

> $ActionQueueFileName mainq
> $ActionQueueType LinkedList
> $ActionQueueSaveOnShutDown on
> $ActionQueueMaxDiskSpace 10g
> $ActionSendResendLastMsgOnReconnect on
> $ActionResumeRetryCount -1
>
> # FreeBSD kernel / console logging
> $KLogPermitNonKernelFacility on
> $KlogConsoleLogLevel 10
>
> # rsyslog templates
>
> ## handle multiline messages (e.g. from various APIs safely)
> $EscapeControlCharactersOnReceive="off"
>
> ## ensure hostnames make sense when coming via UDP / jails
> template(name="normalised" type="string"
>    string="<%pri%>%protocol-version% %timestamp:::date-rfc3339%
>    example.org
>    %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n")
> # rsyslog input modules
> input(type="imtcp"
>    address="127.0.0.1"
>    port="514")
> input(type="imudp"
>    rcvbufSize="1m"
>    address="127.0.0.1"
>    port="514")
> # allow logging via jail hosts
> input(type="imtcp"
>    address="10.241.0.0"
>    port="514")
> input(type="imudp"
>    rcvbufSize="1m"
>    address="10.241.0.0"
>    port="514")
>
> # filters
>
> :msg, contains, "NSSWITCH(_nsdispatch)"  ~
>
> # rulesets
>
> # rsyslog output modules
>
> # base servers forward logs using the hostname-fixing template
> action(type="omrelp"
>    target="127.0.0.1"
>    template="normalised"
>    port="14514"
> )
>
> # log locally just like normal syslog with hostnames adjusted
> action(type="omfile"
>    queue.type="linkedlist"

here you define a queue type on a local file output, this is very unlikely to be 
a win and instead is very likely to eat up a lot of CPU.

>    template="normalised"
>    file="/var/log/messages"
> )
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>


More information about the rsyslog mailing list