[rsyslog-notify] Forum Thread: ompipe(action failed) or omfile( queues are filling) - (Mode 'post')
noreply at adiscon.com
noreply at adiscon.com
Mon May 30 18:18:24 CEST 2016
User: printul77700
Forumlink: http://kb.monitorware.com/viewtopic.php?p=26549#p26549
Message:
----------
Hello, I am glad there are some guys out there I can write to because I
encounter some issues while using rsyslog.
I will try to describe them as short as possible while giving as many
details as I have.
It might be that some things to be in other questions /threads but I have
an entire setup that I would like to check with you
Also attached everything if it helps: [attachment=0:1r3wvbha]<!-- ia0
-->Adiscon case rsyslog (2).zip<!-- ia0 -->[/attachment:1r3wvbha]
>From the beginning note that I am not a programmer so working with a code
is something I did not do while I read a lot about the rsyslog
configurations;if needed my friend/colleague can help me with this at one
point.
• Setup:
Rsyslog listens on tcp/udp:514 filters messages through rulesets and sends
them to pipes,always to pipes even if I use ompipe or omfile, see next
setup
Readers take messages from pipe, parse it, make it STREAM format and sends
it back to rsyslog on 10514
Rsyslog forwards these messages to remote servers.
End of story.
If needed I can provide some output from debug, but I tried to add for the
beginning the stuff I thought worth taking a look into.
Rsyslog version:
rsyslogd 7.6.7, compiled with:
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
Runtime Instrumentation (slow code): No
uuid support: No
Number of Bits in RainerScript integers: 64
See <!-- m --><a class="postlink"
href="http://www.rsyslog.com">http://www.rsyslog.com</a><!-- m --> for more
information.
OS version:
Red Hat Enterprise Linux Server release 5.11
The issue is we lose packets on rsyslog flow on both following
configurations/cases
1. Omfile in the action:we enqueue messages and drop them because we can
not process them in time (even if I take out the queue in front of the
action and use only the queue from the ruleset I still drop,of course I can
go higher and higher until I find the right size, but then I might have
other performance issues I guess and then nothing it will be real time)
action_syslog_pipe_X queue | size=1000 enqueued=588554 full=36
discarded.full=35 discarded.nf=0 maxqsize=1000
2. Ompipe in the action and the actions that write the messages to pipes
fail
action_syslog_pipe_Y | processed=6200 failed=308701 suspended=31
suspended.duration=2430 resumed=0
the ruleset queue is fine, as packet go fast thorugh it but fail in the
action(above)
syslog_pipe_Y ruleset |size=0 enqueued=1621079 full=0 discarded.full=0
discarded.nf=0 maxqsize=1177
I have also implemented impstats so we can have some references and points
to talk on.
To have a starting point:
• We receive about 1 million events/ hour according to impstats, but we
send to multiple pipes:
• Configuration is using ruleset binded to tcp/udp and under that ruleset,
we have several rulesets each for an action
• Each action writes to a pipe (though in the configurations sometimes we
use the omfile module in case 1 respectively ompipe module in case 2.
• I have queues before the actions because I want each action to have it’s
queue;maybe I will add other actions under same ruleset so then I don’t
want that one action filling the ruleset queue and affecting my second
action ( I read that this might be slower, but not sure why would be slower
and anyway, given the fact I have issues with my queues I would keep it
until I solve the problem)
• I have timeout=0 for ruleset because I don’t want this queue once is full
to affect all other ruleset once rsyslog will start to queue to main queue
(aka ruleset x)
/etc/rsyslog.conf
module(
load="impstats"
interval="7200" # how often to generate stats
resetCounters="on" # to get deltas (e.g. # of messages submitted
in the last 10 seconds)
# log.file="/tmp/stats" # file to write those stats to
log.file="/var/log/pipe_syslog_queues_impstats" # pipe to write stats to
log.syslog="off" # don't send stats through the normal
processing pipeline. More on that in a bit
)
# Load the modules we need
# Omfile for regular, local logging
module(load="builtin:omfile" Template="RSYSLOG_FileFormat")
# Omfwd for.. forwarding.
module(load="builtin:omfwd")
# imptcp for the local ZZZ receiver, and YYY receiver
module(load="imptcp")
# imudp for secmon receiver
module(load="imudp")
module(load="imklog")
# imuxsock for regular syslog routing
module(load="imuxsock")
ruleset(name="ruleset x"
# queue.type="Direct"
queue.type="LinkedList"
# queue.timeoutenqueue="0"
) {
# Pipes, heavier to lighter streams
# TODO: Validate ordering
call syslog_pipe_x
call syslog_pipe_y
# Leave unix at the bottom of the list
# This allows for non-Unix events to be stripped out before hitting
this pipe
stop
}
# Set syslog_local as the default policy
$DefaultRuleset syslog_pipe_local
input(type="imptcp" port="514" ruleset="ruleset x")
input(type="imudp" port="514" ruleset="ruleset x")
90-syslog_pipe_X.conf
ruleset(name="syslog_pipe_X"
# queue.type="Direct"
queue.type="LinkedList"
queue.timeoutenqueue="0"
) {
if condition
then {
action(type="omfile"
file="/var/log/solution/syslog_pipe_X"
name="action_syslog_pipe_X"
template="RSYSLOG_TraditionalFileFormat"
# queue.type="LinkedList"
)
stop
}
}
Or
ruleset(name="syslog_pipe_X"
# queue.type="Direct"
queue.type="LinkedList"
queue.timeoutenqueue="0"
) {
if condition
then {
action(type="ompipe"
pipe="/var/log/solution/syslog_pipe_X"
name="action_syslog_pipe_X"
template="RSYSLOG_TraditionalFileFormat"
# queue.type="LinkedList"
)
stop
}
}
All the other configurations of rulesets are similar.
10-streamForwarder.conf
global(
workDirectory="/opt/solution/rsyslog/queues"
)
ruleset(name="name"
queue.workerThreads="2"
queue.FileName="stream.queue"
queue.MaxDiskSpace="10G"
queue.HighWatermark="100000"
queue.SaveOnShutdown="on"
queue.Size="120000"
queue.maxfilesize="100M"
queue.type="LinkedList"
queue.spoolDirectory="/opt/solution/rsyslog/queues"
) {
if ($rawmsg contains 'STREAM:') then {
action(type="omfwd"
name="action_stream_forwarder_system1"
Target="IP"
Port="514"
Protocol="tcp"
ZipLevel="9"
compression.mode="stream:always"
StreamDriverMode="1"
StreamDriver="gtls"
StreamDriverAuthMode="anon"
Template="Raw"
queue.type="LinkedList"
queue.FileName="system1.queue"
queue.MaxDiskSpace="10G"
queue.HighWatermark="100000"
queue.SaveOnShutdown="on"
queue.Size="120000"
queue.maxfilesize="100M"
queue.spoolDirectory="/opt/solution/rsyslog/queues"
action.resumeRetryCount="-1"
)
=====
#ls -ahl /var/log | grep pipe
prw-r--r-- 1 root root 0 May 13 18:00 syslog_pipe_X
Questions:
1. Why do the actions fail when we have configured ompipe? I suppose you
could say that because the pipe is full, but the issue is sometimes this
failures happens even when only a few packets are sent to the pipe
a. Can we go deeper with the debugging to see more output from syslog when
tries to write to pipe/ or see the status of the pipe in real time ( flags,
size etc. )- I want to see if the pipe is really full or maybe there is
some issues of Input/Output on the system
2. Why the queues are filling when omfile is used, is the speed of writing
higher when using ompipe than when using omfile so when we have omfile
rsyslog needs to queue?
a. I suppose that it can be because of to many messages, but maybe you can
help with more options to avoid this
3. Do you recommend me to change the configurations, can you recommend me
some tunnings or some best practices so either I use ompipe or omfile not
to loose messages and also not to enqueuer so much, so I avoid non real
time messages and also avoid loading the memory of the system to much?
4. Is it any way I can debug rsyslog :
a. configuration ( there is some rsyslog –d, anything else? )
b. at running time/live ( to see exactly why some events are failing to
be written etc.)
1. Maybe you know that redhat logs the full pipes or pipe messages
somewhere?
More information about the rsyslog-notify
mailing list