[rsyslog-notify] Forum Thread: mongodb problems, no entries in mongodb - (Mode 'post')

Thu Mar 12 23:45:24 CET 2015

User: superbob 
Forumlink: http://kb.monitorware.com/viewtopic.php?p=25319#p25319

Message: 
----------
All,

I've inherited a logging server from a no-longer-here admin guy.  It's
broke, of course, and I'm having an 
unfortunate time trying to get the rsyslog/mongodb/loganalyzer combo to
work.

I'm mostly following this guide I found:

<!-- m --><a class="postlink"
href="http://loganalyzer.adiscon.com/articles/using-mongodb-with-rsyslog-and-loganalyzer/">http://loganalyzer.adiscon.com/articles
... ganalyzer/</a><!-- m -->

The "box" I will be referring to is the syslog collector for the whole rest
of the system.
I've got the syslogs that deal with the local box -only- being recorded in
/var/log/messages, etc.
I've got the syslogs that other systems send to this collector being
recorded in /var/log/hosts/$HOSTNAME/

What I don't have (so far) is all of the syslog messages from whatever
source being stored in mongodb.  

I've let it run overnight, thinking maybe there was some cache flushing
that I just needed to wait on, but this 
morning loganalyzer was empty.

One side note: the adiscon writeup above says that rsyslog will
automatically generate the mongodb database, but I
didn't see that happening (with or without uid and pwd), so I used the
rsyslog v7 /mongodb writeup on the rsyslog site
instructions to go into mongo shell and generate the syslog database.

I know this is not quantum mechanics so I'm sure it's something
insignificant I'm missing.  Can you kind folks 
take a look and see if a configuration anything jumps out at you?

Thanks!

========================================

/etc/rsyslog.conf

# rsyslog configuration file
# note that most of this config file uses old-style format,
# because it is well-known AND quite suitable for simple cases
# like we have with the default config. For more advanced 
# things, RainerScript configuration is suggested.

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see <snip>

#### MODULES ####

module(load="imuxsock") # provides support for local system logging (e.g.
via logger command)
module(load="imklog")   # provides kernel logging support (previously done
by rklogd)
module(load"immark")    # provides --MARK-- message capability

# Provides UDP syslog reception
# for parameters see <snip>
$ModLoad imudp # needs to be done just once

# Provides TCP syslog reception
# for parameters see <snip>
$ModLoad imtcp # needs to be done just once

#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

$FileGroup syslog
$FileOwner syslog
$DirGroup syslog
$DirOwner syslog
$DirCreateMode 0750
$FileCreateMode 0644

############################################################################################################
# This next set of templates will place all remote logging files in their
own subdirectory under /var/log/hosts

############################################################################################################

$template
RemoteHosts,"/var/log/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/syslog.log

#### LOCAL RULES ####

$RuleSet local 

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  /var/log/maillog

# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

# use the local RuleSet as default if not specified otherwise
$DefaultRuleset local

## Logs for remote hosts

$RuleSet remote
*.*  ?RemoteHosts

# Bind the remote host RuleSet to the tcp listener
$InputTCPServerBindRuleset remote

# now activate it for the tcp listener
$InputTCPServerRun 514

# Bind the remote host RuleSet to the udp listener
$InputUDPServerBindRuleset remote

# now activate it for the udp listener
$UDPServerRun 514

===============================================

/etc/rsyslog.d/mongodb.conf

$ cat /etc/rsyslog.d/mongodb.conf

# Provides mongodb database functionality
# for parameters see <snip>
module(load="mmjsonparse")
module(load="ommongodb") # needs to be done just once
input(type="imtcp" port="514" RuleSet="mongodb")

template(name="mongodball" type="subtree" subtree="$!")

ruleset(name="mongodb") {
	action(type="mmjsonparse")
	if $parsesuccess == "OK" then {
		set $!time = $timestamp;
		set $!sys = $hostname;
		set $!procid = $syslogtag;
		set $!syslog_fac = $syslogfacility;
		set $!syslogsever = $syslogpriority;
		set $!pid = $procid;
		action(type="ommongodb" server="127.0.0.1" db="syslog"
collection="systemevents" uid="rsyslog" pwd="rsyslog"
template="mongodball")
		}
	}

===============================================

Logs:

$ sudo service rsyslog restart

Mar 12 12:28:11 syslog-collector rsyslogd: [origin software="rsyslogd"
swVersion="8.8.0.ad1" x-pid="30298" x-info="<snip>"] exiting on signal 15.
Mar 12 12:28:11 syslog-collector rsyslogd: [origin software="rsyslogd"
swVersion="8.8.0.ad1" x-pid="30395" x-info="<snip>"] start

$ sudo service mongod restart

Thu Mar 12 12:29:53.006 [signalProcessingThread] got signal 15
(Terminated), will terminate after current cmd ends
Thu Mar 12 12:29:53.006 [signalProcessingThread] now exiting
Thu Mar 12 12:29:53.006 dbexit: 
Thu Mar 12 12:29:53.006 [signalProcessingThread] shutdown: going to close
listening sockets...
Thu Mar 12 12:29:53.006 [signalProcessingThread] closing listening socket:
9
Thu Mar 12 12:29:53.006 [signalProcessingThread] closing listening socket:
10
Thu Mar 12 12:29:53.006 [signalProcessingThread] removing socket file:
/tmp/mongodb-27017.sock
Thu Mar 12 12:29:53.007 [signalProcessingThread] shutdown: going to flush
diaglog...
Thu Mar 12 12:29:53.007 [signalProcessingThread] shutdown: going to close
sockets...
Thu Mar 12 12:29:53.009 [signalProcessingThread] shutdown: waiting for fs
preallocator...
Thu Mar 12 12:29:53.009 [signalProcessingThread] shutdown: lock for final
commit...
Thu Mar 12 12:29:53.009 [signalProcessingThread] shutdown: final commit...
Thu Mar 12 12:29:53.014 [signalProcessingThread] shutdown: closing all
files...
Thu Mar 12 12:29:53.014 [signalProcessingThread] closeAllFiles() finished
Thu Mar 12 12:29:53.014 [signalProcessingThread] journalCleanup...
Thu Mar 12 12:29:53.014 [signalProcessingThread] removeJournalFiles
Thu Mar 12 12:29:53.034 [signalProcessingThread] shutdown: removing fs
lock...
Thu Mar 12 12:29:53.034 dbexit: really exiting now
about to fork child process, waiting until server is ready for connections.
forked process: 30555
all output going to: /var/log/mongodb/mongodb.log

===============================================

versions:

Scientific Linux release 6.6 (Carbon)

rsyslogd: [origin software="rsyslogd" swVersion="8.8.0.ad1" 

mongodb: 
$ mongod --version
db version v2.4.12
Thu Mar 12 12:07:56.517 git version: nogitversion

loganalyzer: 4.1.2

apache:  

$ httpd -V
Server version: Apache/2.2.15 (Unix)
Server built:   Oct 16 2014 10:08:27
Server's Module Magic Number: 20051115:25
Server loaded:  APR 1.3.9, APR-Util 1.3.9
Compiled using: APR 1.3.9, APR-Util 1.3.9
Architecture:   64-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/etc/httpd"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="run/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="logs/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

$ php -v
PHP 5.3.3 (cli) (built: Nov  3 2014 10:27:01) 
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2010 Zend Technologies