[rsyslog-notify] Forum Thread: Re: best way to trigger an omprog action - (Mode 'reply')

[noreply at adiscon.com]

User: robertws8 
Forumlink: http://kb.monitorware.com/viewtopic.php?p=26672#p26672

Message: 
----------
After some more research and trial/error I got this working the way I
wanted.  But it is not immediately obvious so I thought I would write it up
for anybody else to use without having to go through my problems.

My initial attempt just executed the kvlogger script every ten minutes but
not in sync with the sysstat sa1 cron job.  This meant that stats could be
up to almost ten minutes late.  The following two stanzas work by watching
for changes to the /var/log/sa/sa[0-9]* files and then triggering the
sysstat ruleset.

[code:3l2ldqzk]ruleset(name="sysstat"){
  action(type="omprog"
  action.execonlyonceeveryinterval="61"
  binary="/etc/rsyslog.d/scripts/kvlogger")
}

input(type="imfile"
  file="/var/log/sa/sa[0-9]*"
  tag="sysstat"
  ruleset="sysstat"
)[/code:3l2ldqzk]

In my environment sysstat is set to capture metrics every ten minutes. 
Every time it does it writes multiple records to the current sa[0-9]* file.
 Consequently, the imfile module triggers repeatedly for every write to the
sa* file.  This caused a barrage of repeated stats being logged!  Not what
I wanted.

The solution was to create a ruleset to essentially throttle the omprog
module by putting a 61 second re-trigger delay.  Now, just the first write
to an sa* file triggers the kvlogger script.

Here, as an fyi is the kvlogger script (I ultimately pass the resulting
logs through a logstash kv filter to elasticsearch and view them in
kibana)...

[code:3l2ldqzk]#!/usr/bin/env bash
# kvlogger - sysstat sar data logger
sleep 5

# sadf syntax changes between version 9 and 10

SADF="/usr/bin/sadf -d -U -s"
read -a SADF_VERSION < <( sadf -V2 2>&1 )
if (( ${SADF_VERSION[2]%%.*} < 10 )); then
  SADF="/usr/bin/sadf -D -s"
fi
declare -A types=(
  [disk]="-d -p"
  [swap]="-S"
  [memory]="-r"
  [runqueue]="-q"
  [network]="-n DEV"
  [cpu]="-P ALL" )

for type in disk swap memory runqueue network cpu; do
  ${SADF} $(date -d"20 min ago" +%T) -- ${types[${type}]} 1 1 |
  while read myline; do
    if [[ ! ${hdr} ]]; then
      hdr=( ${myline//;/ } )
      continue
    fi
    mystat=( ${myline//;/ } )
    OUT1="type=${type} logsource=${mystat[0]%%.*}
event_time=${mystat[2]}000"
    for ((x=3;x<=${#hdr[*]}-2;x++)); do
      OUT1="${OUT1} ${hdr[$((${x}+1))]}=${mystat[${x}]}"
    done
    logger -t sysstat "${OUT1}"
    sleep 0.5
  done
done[/code:3l2ldqzk]


Enjoy!