[rsyslog-notify] Forum Thread: IMPstats into Elasticsearch - (Mode 'post')

[noreply at adiscon.com]

User: snorman1483 
Forumlink: http://kb.monitorware.com/viewtopic.php?p=26652#p26652

Message: 
----------
Hello;

    I have been reading up on piping Rsyslog impstats into Elasticsearch
with Kibana and Logstash. I have read all the post on rsyslog.com and
sematext, but it seems that I am missing something to get Rsyslog to
forward. I am able to pipe the stats to a file but when I remove the
log.file="/tmp/stats" and then enable my actions to pipe the stats over
LogStash; that I do not see anything. I did have enable one of my actions
to stream it's logs to LogStash and it seems to work fine. 

[username at originhost final]# wget "http://LogStash Server:5044"
--2016-06-22 23:54:26--  <!-- m --><a class="postlink"
href="http://Logstash">http://Logstash</a><!-- m --> Server:5044/
Connecting to LogStash Server:5044... connected.
HTTP request sent, awaiting response... ^C
[user at originhost final]# telnet "IP of LS" 5044
Trying "IP of LS"...
Connected to "IP of LS".
Escape character is '^]'.
^]
telnet> quit
Connection closed.


Logs Hitting the Stat File
******************************
Wed Jun 22 23:40:40 2016: @cee: {"name":"main
Q","size":2,"enqueued":8284,"full":0,"discarded.full":0,"discarded.nf":0,"maxqsize":36}
Wed Jun 22 23:40:45 2016: @cee:
{"name":"imuxsock","submitted":74,"ratelimit.discarded":0,"ratelimit.numratelimiters":74}
Wed Jun 22 23:40:45 2016: @cee:
{"name":"omelasticsearch","submitted":0,"failed.http":0,"failed.httprequests":0,"failed.es":0}
Wed Jun 22 23:40:45 2016: @cee:
{"name":"parse_impstats","processed":0,"failed":0}
Wed Jun 22 23:40:45 2016: @cee:
{"name":"rsyslog_stats","processed":0,"failed":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"dynafile cache
message","requests":8226,"level0":8225,"missed":1,"evicted":0,"maxused":1}
Wed Jun 22 23:40:45 2016: @cee: {"name":"action
3","processed":8226,"failed":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"dynafile cache
secure","requests":19,"level0":18,"missed":1,"evicted":0,"maxused":1}
Wed Jun 22 23:40:45 2016: @cee: {"name":"action
4","processed":19,"failed":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"dynafile cache
maillog","requests":0,"level0":0,"missed":0,"evicted":0,"maxused":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"action
5","processed":0,"failed":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"dynafile cache
cron","requests":40,"level0":39,"missed":1,"evicted":0,"maxused":1}
Wed Jun 22 23:40:45 2016: @cee: {"name":"action
6","processed":40,"failed":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"action
7","processed":0,"failed":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"dynafile cache
spooler","requests":0,"level0":0,"missed":0,"evicted":0,"maxused":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"action
8","processed":0,"failed":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"dynafile cache
boot","requests":14,"level0":13,"missed":1,"evicted":0,"maxused":1}
Wed Jun 22 23:40:45 2016: @cee: {"name":"action
9","processed":14,"failed":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"****","processed":0,"failed":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"****","processed":0,"failed":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"****","processed":0,"failed":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"****","processed":0,"failed":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"****","processed":0,"failed":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"****","processed":0,"failed":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"****","processed":0,"failed":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"****","processed":0,"failed":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"****","processed":0,"failed":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"****","processed":0,"failed":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"****","processed":0,"failed":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"****","processed":0,"failed":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"****","processed":0,"failed":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"imtcp(514)","submitted":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"imudp(*:514)","submitted":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"imudp(*:514)","submitted":0}
Wed Jun 22 23:40:45 2016: @cee:
{"name":"****[DA]","size":0,"enqueued":0,"full":0,"discarded.full":0,"discarded.nf":0,"maxqsize":0}
Wed Jun 22 23:40:45 2016: @cee:
{"name":"","size":0,"enqueued":0,"full":0,"discarded.full":0,"discarded.nf":0,"maxqsize":0}
Wed Jun 22 23:40:45 2016: @cee: {"name":"main
Q","size":1,"enqueued":8320,"full":0,"discarded.full":0,"discarded.nf":0,"maxqsize":36}


My Configuration

********************************************************************************************************

#################
#### MODULES ####
#################

module(load="imtcp")
module(load="imudp")
module(load="imuxsock")
module(load="mmjsonparse")
module(load="omelasticsearch")

#############################
#### Performance Counter ####
#############################

module(
        load="impstats"
        interval="5"
        resetCounters="on"
        format="cee"
        ruleset="monitoring"
        log.file="/tmp/stats"
)

template(name="stats" type="list") {
constant(value="{")
property(name="timereported" dateFormat="rfc3339" format="jsonf"
outname="@timestamp")  # the timestamp
constant(value=",")
property(name="hostname" format="jsonf" outname="host")
constant(value=",\"source\":\"impstats\",")
#property(name="$!all-json" position.from="2")
}

ruleset(name="monitoring"){

action(
  name="parse_impstats"        # parse the
  type="mmjsonparse"           # JSON stats
)
action(
name="rsyslog_stats"
type="omelasticsearch"
server="LogStash Server"
serverport="5044"
template="stats"
)
}


Here is LogStash conf
*******************************************

input {
  tcp {
   port => 5044
   type => syslog
  }
  udp {
	port => 5044
	type => syslog
  }
  tcp {
		port => 5044
		type => json
  }
  udp {
		port => 5044
		type => json
  }
}

output {
  elasticsearch {
    hosts => "ES Server:9200"
	index => 'logstash-%{+YYYY.MM.dd}'        
  }
}