[rsyslog-notify] Forum Thread: Re: messages are missing from the log - (Mode 'reply')

[noreply at adiscon.com]

User: dlang 
Forumlink: http://kb.monitorware.com/viewtopic.php?p=24505#p24505

Message: 
----------
for this testing, it's probably easiest if you just bypass the syslog()
calls and write directly

open a unix socket (/dev/log or /dev/reallog) and write something with the
following format
<42>Apr  8 21:15:01 syslogtag: message content\n

42 is the facility and severity, for testing, it really doesn't matter what
the value is

syslogtag is the programname (followed by :) and then your content after
that

in the timestamp, the day takes two columns, so in this example there are
two spaces between Apr and 8

or use logger, if you do

cat file |logger -t programname -u /dev/reallog

it will take all the lines in file, use the programname you specify, and
write the logs to /dev/realllog

get some large file that takes a while to dump like this (10 seconds+) and
try to capture the total system load (assuming the system is otherwise
idle), because you have multiple moving parts here

1. the log generation, which takes little cpu, but may be blocked if the
thing it's logging to can't read fast enough

2. (sometimes) journald reading the data from #1, looking up additional
information, saving the data, and then making it available to rsyslog

3. rsyslog reading the data (either directly, from it's /dev/log from the
journal, or via imjournal) and putting it on it's main memory queue

4. rsyslog going through the logs in it's queue and writing them out to
disk

if the logging is slow, you may see the wall clock on #1 take longer,
without using much more CPU, but in the meantime, a log of extra CPU could
be taken by #2 and #3 (depending on the modes involved)

If we see the queue backing up, we may want to simplify the output for the
test to make sure #4 isn't the bottleneck. although if it is a bottleneck
because the journal is also writing the data to disk, but not if the
journal is bypassed, that's useful information.