From friedl at hq.adiscon.com Thu Sep 1 13:16:36 2011 From: friedl at hq.adiscon.com (Florian Riedl) Date: Thu, 1 Sep 2011 13:16:36 +0200 Subject: [rsyslog] rsyslog multiple buxfixes released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7281184@GRFEXC.intern.adiscon.com> Hi all, There has a security issue been identified that can potentially lead to DoS. It is triggered by malformed RFC3164 messages. An abort only happens under very specific environmental trigger factors. Full details can be found in our security advisory here: http://www.rsyslog.com/potential-dos-with-malformed-tag/ We would like to thank the Red Hat security team for finding this issue and working with us to resolve it. As a consequence, we have updated all currently active versions. Please note that they do not only contain the fix for the security issue mentioned above but also other stability updates. For obvious reasons, updating to these versions is recommended. For details, please see the relevant ChangeLog. v4-stable: 4.6.8 v4-beta: 4.7.5 v5-stable: 5.8.5 v5-devel: 5.9.3 v6-beta: 6.1.12 v6-devel: 6.3.5 All versions are available right now. If you do not want to update, you should consider applying an update to older versions. The fix is trivial, so it should apply to all vulnerable versions without problems (but we have not checked the myriad of versions out there). The security advisory contains the details. The Changelogs and Download Links can be found below: v4-stable: 4.6.8 ChangeLog: http://www.rsyslog.com/changelog-for-4-6-8-v4-stable/ Download: http://www.rsyslog.com/rsyslog-4-6-8-v4-stable/ v4-beta: 4.7.5 ChangeLog: http://www.rsyslog.com/changelog-for-4-7-5-v4-beta/ Download: http://www.rsyslog.com/rsyslog-4-7-5-v4-beta/ v5-stable: 5.8.5 ChangeLog: http://www.rsyslog.com/changelog-for-5-8-5-v5-stable/ Download: http://www.rsyslog.com/rsyslog-5-8-5-v5-stable/ v5-devel: 5.9.3 ChangeLog: http://www.rsyslog.com/changelog-for-5-9-3-v5-devel/ Download: http://www.rsyslog.com/rsyslog-5-9-3-v5-devel/ v6-beta: 6.1.12 ChangeLog: http://www.rsyslog.com/changelog-for-6-1-12-v6-beta/ Download: http://www.rsyslog.com/rsyslog-6-1-12-v6-beta/ v6-devel: 6.3.5 ChangeLog: http://www.rsyslog.com/changelog-for-6-3-5-v6-devel/ Download: http://www.rsyslog.com/rsyslog-6-3-5-v6-devel/ As always, feedback is appreciated. Best regards, Florian Riedl From a.piesk at gmx.net Thu Sep 1 19:41:01 2011 From: a.piesk at gmx.net (Andreas Piesk) Date: Thu, 01 Sep 2011 19:41:01 +0200 Subject: [rsyslog] rsyslog 5.8.4: no MARK messages In-Reply-To: <4E5E4495.4090607@gmx.net> References: <4E5D395C.2070603@gmx.net><4E5D4886.1040205@mejor.pl><4E5D4EFD.5070400@gmx.net><4E5D56E3.9060801@mejor.pl><4E5D5B38.6010807@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA728116E@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA728116F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7281170@GRFEXC.intern.adiscon.com> <4E5DFEBC.7010007@mejor.pl> <4E5E4495.4090607@gmx.net> Message-ID: <4E5FC3AD.6040605@gmx.net> On 31.08.2011 16:26, Andreas Piesk wrote: > > OK, i' ll try it and report back. > the patch fixes the problem, thanks. time to deploy 5.8.5. regards, -ap From a.piesk at gmx.net Thu Sep 1 19:57:25 2011 From: a.piesk at gmx.net (Andreas Piesk) Date: Thu, 01 Sep 2011 19:57:25 +0200 Subject: [rsyslog] rsyslog 5.8.4 hangs at startup In-Reply-To: <4E5D3D6B.2020009@gmx.net> References: <4E5D3D6B.2020009@gmx.net> Message-ID: <4E5FC785.5080708@gmx.net> On 30.08.2011 21:43, Andreas Piesk wrote: > sometimes one of the servers hangs at reboot because of rsyslog. the version in use is 5.8.4 64bit. > > i beieve, the hangs are caused by 0-length spool files. as soon as i remove these files and restart > again, rsyslog starts normally. my current workaround is an additional check in the start skript to > remove and 0-length spool files because a server hanging in the boot process for ever is a nasty thing. > > the config uses DA queues and forwards all messages to central logservers. > > has anyone had this problem too? > i probably know how it happens. one of the machines having the issue had incorrect runlevels for rsyslog. at shutdown the network will be shutted down before rsyslog, so rsyslog can no longer forward any messages to central logservers and starts to spool to disk, thus the files in rsyslog's spool directory. why they are 0-length i don't know, maybe shutdown kills rsyslog before it can write anything to the spool file. i changed the runlevels to sane values and haven't had the issue again since then. regards, -ap From tracy.felts at etrade.com Fri Sep 2 17:28:05 2011 From: tracy.felts at etrade.com (Felts, Tracy) Date: Fri, 2 Sep 2011 15:28:05 +0000 Subject: [rsyslog] rsyslog (v 5.8.3) multi-line message support Message-ID: Does rsyslog support multi-line log messaging such as Tomcat stack traces or kernel messages? During my testing I'm seeing a single multi-line message broken up in to single line messages on the rsyslog receiver/collector (see samples below). I've read through the various posts in the forum and mailing lists on this topic but haven't come away with a clear answer. I understand multi-line message support is not a trivial solution, broken framing, trying to determine what constitutes the beginning and end of a multi-line message. I've tried several things read from various posts such as octet-frame counting (http://kb.monitorware.com/multiline-messages-t10184.html), disabling control character escaping ($EscapeControlCharactersOnReceive) with little success. I've read David Lang has been working on multi-line logging but I'm not sure the status today. I've included the configurations for two servers I'm using for rsyslog testing. You can find a multi-line log message sample below the configurations. Receiver/Collector/Server rsyslog.conf: ------------------------------------------------- $EscapeControlCharactersOnReceive off # Load desired modules. # for TCP use: $modload imtcp # for UDP use: #$modload imudp # Provides kernel logging support (previously done by rklogd) $ModLoad imklog # Provides support for local system logging (e.g. via logger command) $ModLoad imuxsock # Reliable Event Logging Protocol module $ModLoad imrelp # Use traditional timestamp format $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format $template rawfmt,"%rawmsg%\n" #$InputTCPServerBindRuleset remote5140 $InputTCPServerRun 5140 $InputRELPServerRun 10514 #$UDPServerRun 514 # Maximum rsyslog message size $MaxMessageSize 32k $template AccessLog,"/var/log/rsyslog/%HOSTNAME%/%$YEAR%-%$MONTH%-%$DAY%/access.log" $template SecureLog,"/var/log/rsyslog/%HOSTNAME%/%$YEAR%-%$MONTH%-%$DAY%/secure.log" $template MessagesLog,"/var/log/rsyslog/%HOSTNAME%/%$YEAR%-%$MONTH%-%$DAY%/messages.log" $template CatalinaLog,"/var/log/rsyslog/%HOSTNAME%/%$YEAR%-%$MONTH%-%$DAY%/catalina.log" # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;kern.notice;mail.none;authpriv.none;cron.none -/var/log/messages # The authpriv file has restricted access. #authpriv.* /var/log/secure authpriv.*;auth.info /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log local6.* -/var/opt/silvertail/log/silvertail.log ### RSYSLOG Rules if $syslogfacility-text == 'local5' and $programname == 'apache' then -?AccessLog & ~ if $syslogfacility-text == 'local0' then -?CatalinaLog;rawfmt & ~ #if $syslogfacility-text == 'local5' and $programname == 'logview' then -?MessagesLog #& ~ if $syslogfacility-text == 'authpriv' then -?SecureLog Client rsyslog.conf: --------------------------- $EscapeControlCharactersOnReceive off # Provides kernel logging support (previously done by rklogd) $ModLoad imklog # Provides MARK support. #$ModLoad immark # Provides support for local system logging (e.g. via logger command) $ModLoad imuxsock # File Monitor configs $ModLoad imfile # Reliable Event Logging Protocol module $ModLoad omrelp # Reliable Event Logging Protocol module #$ModLoad omhdfs $RepeatedMsgReduction on $MaxMessageSize 32k # Use traditional timestamp format $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format $WorkDirectory /var/log/rsyslog/work # default location for work (spool) files $ActionQueueType LinkedList # use asynchronous processing $ActionQueueFileName spool_data # set file name, also enables disk mode $ActionResumeRetryCount -1 # infinite retries on insert failure $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down $ActionSendResendLastMsgOnReconnect on # File Monitor configs #$InputFileName /var/log/messages $InputFileName /adm/web/logview/logs/catalina.log $InputFileTag logview: $InputFileStateFile /var/log/rsyslog-messages.stat $InputFileSeverity info $InputFileFacility local0 $InputRunFileMonitor # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none /var/log/messages local5.* :omrelp:10.152.106.24:10514;RSYSLOG_ForwardFormat &~ local0.* @@(o)10.152.106.24:5140 &~ authpriv.* :omrelp:10.152.106.24:10514;RSYSLOG_ForwardFormat # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log Tomcat Message Sample (Original): -------------------------------------------- 2011-08-15 17:24:38,888 [:TP-Processor3] ERROR org.apache.jk.common.ChannelSocket - Error, processing connection java.lang.IndexOutOfBoundsException at java.io.BufferedInputStream.read(BufferedInputStream.java:310) at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:620) at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:577) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:685) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690) at java.lang.Thread.run(Thread.java:619) Tomcat Message Sample (from rsyslog server/collector): ---------------------------------------------------------------------- <134>Sep 1 17:31:57 appserver1 logview: 2011-08-15 17:24:38,888[:TP-Processor3] ERROR org.apache.jk.common.ChannelSocket - Error, processing connection <134>Sep 1 17:31:57 appserver1 logview: java.lang.IndexOutOfBoundsException <134>Sep 1 17:31:57 appserver1 logview: at java.io.BufferedInputStream.read(BufferedInputStream.java:310) <134>Sep 1 17:31:57 appserver1 logview: at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:620) <134>Sep 1 17:31:57 appserver1 logview: at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:577) <134>Sep 1 17:31:57 appserver1 logview: at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:685) <134>Sep 1 17:31:57 appserver1 logview: at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889) <134>Sep 1 17:31:57 appserver1 logview: at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690) <134>Sep 1 17:31:57 appserver1 logview: at java.lang.Thread.run(Thread.java:619) Please let me know if you need additional information or details from me. Thanks in advance for your assistance. Regards, Tracy From david at lang.hm Fri Sep 2 18:06:16 2011 From: david at lang.hm (david at lang.hm) Date: Fri, 2 Sep 2011 09:06:16 -0700 (PDT) Subject: [rsyslog] rsyslog (v 5.8.3) multi-line message support In-Reply-To: References: Message-ID: Yes, I submitted a patch to imfile that allows you to specify what the log separator is. default, each line is a new message indented, like your tomcat messages, a new log entry starts at the beginning of a line, if a line starts with a space it's part of the log message before it paragraph, there is a blank line between log messages. I don't knwo if this patch was applied to the 5.x series or only the 6.1 series. when this combines the mulitline messages, it will replace the newline with a #xxx number, but everything should come out as one line. If this does not work, please let me know so that we can fix it (I think you are the first person to as for this since I submitted it) David Lang On Fri, 2 Sep 2011, Felts, Tracy wrote: > Date: Fri, 2 Sep 2011 15:28:05 +0000 > From: "Felts, Tracy" > Reply-To: rsyslog-users > To: "rsyslog at lists.adiscon.com" > Subject: [rsyslog] rsyslog (v 5.8.3) multi-line message support > > Does rsyslog support multi-line log messaging such as Tomcat stack traces or kernel messages? During my testing I'm seeing a single multi-line message broken up in to single line messages on the rsyslog receiver/collector (see samples below). I've read through the various posts in the forum and mailing lists on this topic but haven't come away with a clear answer. I understand multi-line message support is not a trivial solution, broken framing, trying to determine what constitutes the beginning and end of a multi-line message. > > I've tried several things read from various posts such as octet-frame counting (http://kb.monitorware.com/multiline-messages-t10184.html), disabling control character escaping ($EscapeControlCharactersOnReceive) with little success. I've read David Lang has been working on multi-line logging but I'm not sure the status today. > > I've included the configurations for two servers I'm using for rsyslog testing. You can find a multi-line log message sample below the configurations. > > > Receiver/Collector/Server rsyslog.conf: > ------------------------------------------------- > $EscapeControlCharactersOnReceive off > > # Load desired modules. > # for TCP use: > $modload imtcp > # for UDP use: > #$modload imudp > > # Provides kernel logging support (previously done by rklogd) > $ModLoad imklog > # Provides support for local system logging (e.g. via logger command) > $ModLoad imuxsock > # Reliable Event Logging Protocol module > $ModLoad imrelp > > # Use traditional timestamp format > $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format > $template rawfmt,"%rawmsg%\n" > > #$InputTCPServerBindRuleset remote5140 > $InputTCPServerRun 5140 > $InputRELPServerRun 10514 > #$UDPServerRun 514 > > # Maximum rsyslog message size > $MaxMessageSize 32k > > $template AccessLog,"/var/log/rsyslog/%HOSTNAME%/%$YEAR%-%$MONTH%-%$DAY%/access.log" > $template SecureLog,"/var/log/rsyslog/%HOSTNAME%/%$YEAR%-%$MONTH%-%$DAY%/secure.log" > $template MessagesLog,"/var/log/rsyslog/%HOSTNAME%/%$YEAR%-%$MONTH%-%$DAY%/messages.log" > $template CatalinaLog,"/var/log/rsyslog/%HOSTNAME%/%$YEAR%-%$MONTH%-%$DAY%/catalina.log" > > # Log all kernel messages to the console. > # Logging much else clutters up the screen. > #kern.* /dev/console > > # Log anything (except mail) of level info or higher. > # Don't log private authentication messages! > *.info;kern.notice;mail.none;authpriv.none;cron.none -/var/log/messages > > # The authpriv file has restricted access. > #authpriv.* /var/log/secure > authpriv.*;auth.info /var/log/secure > > # Log all the mail messages in one place. > mail.* -/var/log/maillog > > # Log cron stuff > cron.* /var/log/cron > > # Everybody gets emergency messages > *.emerg * > > # Save news errors of level crit and higher in a special file. > uucp,news.crit /var/log/spooler > > # Save boot messages also to boot.log > local7.* /var/log/boot.log > > local6.* -/var/opt/silvertail/log/silvertail.log > > ### RSYSLOG Rules > if $syslogfacility-text == 'local5' and $programname == 'apache' then -?AccessLog > & ~ > if $syslogfacility-text == 'local0' then -?CatalinaLog;rawfmt > & ~ > #if $syslogfacility-text == 'local5' and $programname == 'logview' then -?MessagesLog > #& ~ > if $syslogfacility-text == 'authpriv' then -?SecureLog > > > Client rsyslog.conf: > --------------------------- > $EscapeControlCharactersOnReceive off > > # Provides kernel logging support (previously done by rklogd) > $ModLoad imklog > # Provides MARK support. > #$ModLoad immark > # Provides support for local system logging (e.g. via logger command) > $ModLoad imuxsock > # File Monitor configs > $ModLoad imfile > # Reliable Event Logging Protocol module > $ModLoad omrelp > # Reliable Event Logging Protocol module > #$ModLoad omhdfs > > $RepeatedMsgReduction on > $MaxMessageSize 32k > > # Use traditional timestamp format > $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format > > $WorkDirectory /var/log/rsyslog/work # default location for work (spool) files > $ActionQueueType LinkedList # use asynchronous processing > $ActionQueueFileName spool_data # set file name, also enables disk mode > $ActionResumeRetryCount -1 # infinite retries on insert failure > $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down > $ActionSendResendLastMsgOnReconnect on > > # File Monitor configs > #$InputFileName /var/log/messages > $InputFileName /adm/web/logview/logs/catalina.log > $InputFileTag logview: > $InputFileStateFile /var/log/rsyslog-messages.stat > $InputFileSeverity info > $InputFileFacility local0 > $InputRunFileMonitor > > # Log all kernel messages to the console. > # Logging much else clutters up the screen. > #kern.* /dev/console > > # Log anything (except mail) of level info or higher. > # Don't log private authentication messages! > *.info;mail.none;authpriv.none;cron.none /var/log/messages > > local5.* :omrelp:10.152.106.24:10514;RSYSLOG_ForwardFormat > &~ > local0.* @@(o)10.152.106.24:5140 > &~ > > authpriv.* :omrelp:10.152.106.24:10514;RSYSLOG_ForwardFormat > > # Log all the mail messages in one place. > mail.* -/var/log/maillog > > # Log cron stuff > cron.* /var/log/cron > > # Everybody gets emergency messages > *.emerg * > > # Save news errors of level crit and higher in a special file. > uucp,news.crit /var/log/spooler > > # Save boot messages also to boot.log > local7.* /var/log/boot.log > > > Tomcat Message Sample (Original): > -------------------------------------------- > 2011-08-15 17:24:38,888 [:TP-Processor3] ERROR org.apache.jk.common.ChannelSocket - Error, processing connection > java.lang.IndexOutOfBoundsException > at java.io.BufferedInputStream.read(BufferedInputStream.java:310) > at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:620) > at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:577) > at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:685) > at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889) > at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690) > at java.lang.Thread.run(Thread.java:619) > > > Tomcat Message Sample (from rsyslog server/collector): > ---------------------------------------------------------------------- > <134>Sep 1 17:31:57 appserver1 logview: 2011-08-15 17:24:38,888[:TP-Processor3] ERROR org.apache.jk.common.ChannelSocket - Error, processing connection > <134>Sep 1 17:31:57 appserver1 logview: java.lang.IndexOutOfBoundsException > <134>Sep 1 17:31:57 appserver1 logview: at java.io.BufferedInputStream.read(BufferedInputStream.java:310) > <134>Sep 1 17:31:57 appserver1 logview: at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:620) > <134>Sep 1 17:31:57 appserver1 logview: at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:577) > <134>Sep 1 17:31:57 appserver1 logview: at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:685) > <134>Sep 1 17:31:57 appserver1 logview: at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889) > <134>Sep 1 17:31:57 appserver1 logview: at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690) > <134>Sep 1 17:31:57 appserver1 logview: at java.lang.Thread.run(Thread.java:619) > > > > Please let me know if you need additional information or details from me. Thanks in advance for your assistance. > > Regards, > > Tracy > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Sat Sep 3 04:56:20 2011 From: david at lang.hm (david at lang.hm) Date: Fri, 2 Sep 2011 19:56:20 -0700 (PDT) Subject: [rsyslog] rsyslog (v 5.8.3) multi-line message support In-Reply-To: References: Message-ID: I went looking for the documentation on this and couldn't find it. the setting is $InputFileReadMode 0 = defult, line based 1 = indented 2 = paragraph please let me know how it works (or fails) for you. David Lang On Fri, 2 Sep 2011, david at lang.hm wrote: > Date: Fri, 2 Sep 2011 09:06:16 -0700 (PDT) > From: david at lang.hm > Reply-To: rsyslog-users > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog (v 5.8.3) multi-line message support > > Yes, I submitted a patch to imfile that allows you to specify what the log > separator is. > > default, each line is a new message > > indented, like your tomcat messages, a new log entry starts at the beginning > of a line, if a line starts with a space it's part of the log message before > it > > paragraph, there is a blank line between log messages. > > I don't knwo if this patch was applied to the 5.x series or only the 6.1 > series. > > when this combines the mulitline messages, it will replace the newline with a > #xxx number, but everything should come out as one line. > > If this does not work, please let me know so that we can fix it (I think you > are the first person to as for this since I submitted it) > > David Lang > > On Fri, 2 Sep 2011, Felts, Tracy wrote: > >> Date: Fri, 2 Sep 2011 15:28:05 +0000 >> From: "Felts, Tracy" >> Reply-To: rsyslog-users >> To: "rsyslog at lists.adiscon.com" >> Subject: [rsyslog] rsyslog (v 5.8.3) multi-line message support >> >> Does rsyslog support multi-line log messaging such as Tomcat stack traces >> or kernel messages? During my testing I'm seeing a single multi-line >> message broken up in to single line messages on the rsyslog >> receiver/collector (see samples below). I've read through the various >> posts in the forum and mailing lists on this topic but haven't come away >> with a clear answer. I understand multi-line message support is not a >> trivial solution, broken framing, trying to determine what constitutes the >> beginning and end of a multi-line message. >> >> I've tried several things read from various posts such as octet-frame >> counting (http://kb.monitorware.com/multiline-messages-t10184.html), >> disabling control character escaping ($EscapeControlCharactersOnReceive) >> with little success. I've read David Lang has been working on multi-line >> logging but I'm not sure the status today. >> >> I've included the configurations for two servers I'm using for rsyslog >> testing. You can find a multi-line log message sample below the >> configurations. >> >> >> Receiver/Collector/Server rsyslog.conf: >> ------------------------------------------------- >> $EscapeControlCharactersOnReceive off >> >> # Load desired modules. >> # for TCP use: >> $modload imtcp >> # for UDP use: >> #$modload imudp >> >> # Provides kernel logging support (previously done by rklogd) >> $ModLoad imklog >> # Provides support for local system logging (e.g. via logger command) >> $ModLoad imuxsock >> # Reliable Event Logging Protocol module >> $ModLoad imrelp >> >> # Use traditional timestamp format >> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format >> $template rawfmt,"%rawmsg%\n" >> >> #$InputTCPServerBindRuleset remote5140 >> $InputTCPServerRun 5140 >> $InputRELPServerRun 10514 >> #$UDPServerRun 514 >> >> # Maximum rsyslog message size >> $MaxMessageSize 32k >> >> $template >> AccessLog,"/var/log/rsyslog/%HOSTNAME%/%$YEAR%-%$MONTH%-%$DAY%/access.log" >> $template >> SecureLog,"/var/log/rsyslog/%HOSTNAME%/%$YEAR%-%$MONTH%-%$DAY%/secure.log" >> $template >> MessagesLog,"/var/log/rsyslog/%HOSTNAME%/%$YEAR%-%$MONTH%-%$DAY%/messages.log" >> $template >> CatalinaLog,"/var/log/rsyslog/%HOSTNAME%/%$YEAR%-%$MONTH%-%$DAY%/catalina.log" >> >> # Log all kernel messages to the console. >> # Logging much else clutters up the screen. >> #kern.* /dev/console >> >> # Log anything (except mail) of level info or higher. >> # Don't log private authentication messages! >> *.info;kern.notice;mail.none;authpriv.none;cron.none -/var/log/messages >> >> # The authpriv file has restricted access. >> #authpriv.* /var/log/secure >> authpriv.*;auth.info /var/log/secure >> >> # Log all the mail messages in one place. >> mail.* -/var/log/maillog >> >> # Log cron stuff >> cron.* /var/log/cron >> >> # Everybody gets emergency messages >> *.emerg * >> >> # Save news errors of level crit and higher in a special file. >> uucp,news.crit /var/log/spooler >> >> # Save boot messages also to boot.log >> local7.* /var/log/boot.log >> >> local6.* >> -/var/opt/silvertail/log/silvertail.log >> >> ### RSYSLOG Rules >> if $syslogfacility-text == 'local5' and $programname == 'apache' then >> -?AccessLog >> & ~ >> if $syslogfacility-text == 'local0' then -?CatalinaLog;rawfmt >> & ~ >> #if $syslogfacility-text == 'local5' and $programname == 'logview' then >> -?MessagesLog >> #& ~ >> if $syslogfacility-text == 'authpriv' then -?SecureLog >> >> >> Client rsyslog.conf: >> --------------------------- >> $EscapeControlCharactersOnReceive off >> >> # Provides kernel logging support (previously done by rklogd) >> $ModLoad imklog >> # Provides MARK support. >> #$ModLoad immark >> # Provides support for local system logging (e.g. via logger command) >> $ModLoad imuxsock >> # File Monitor configs >> $ModLoad imfile >> # Reliable Event Logging Protocol module >> $ModLoad omrelp >> # Reliable Event Logging Protocol module >> #$ModLoad omhdfs >> >> $RepeatedMsgReduction on >> $MaxMessageSize 32k >> >> # Use traditional timestamp format >> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format >> >> $WorkDirectory /var/log/rsyslog/work # default location for work (spool) >> files >> $ActionQueueType LinkedList # use asynchronous processing >> $ActionQueueFileName spool_data # set file name, also enables disk mode >> $ActionResumeRetryCount -1 # infinite retries on insert failure >> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down >> $ActionSendResendLastMsgOnReconnect on >> >> # File Monitor configs >> #$InputFileName /var/log/messages >> $InputFileName /adm/web/logview/logs/catalina.log >> $InputFileTag logview: >> $InputFileStateFile /var/log/rsyslog-messages.stat >> $InputFileSeverity info >> $InputFileFacility local0 >> $InputRunFileMonitor >> >> # Log all kernel messages to the console. >> # Logging much else clutters up the screen. >> #kern.* /dev/console >> >> # Log anything (except mail) of level info or higher. >> # Don't log private authentication messages! >> *.info;mail.none;authpriv.none;cron.none /var/log/messages >> >> local5.* :omrelp:10.152.106.24:10514;RSYSLOG_ForwardFormat >> &~ >> local0.* @@(o)10.152.106.24:5140 >> &~ >> >> authpriv.* :omrelp:10.152.106.24:10514;RSYSLOG_ForwardFormat >> >> # Log all the mail messages in one place. >> mail.* -/var/log/maillog >> >> # Log cron stuff >> cron.* /var/log/cron >> >> # Everybody gets emergency messages >> *.emerg * >> >> # Save news errors of level crit and higher in a special file. >> uucp,news.crit /var/log/spooler >> >> # Save boot messages also to boot.log >> local7.* /var/log/boot.log >> >> >> Tomcat Message Sample (Original): >> -------------------------------------------- >> 2011-08-15 17:24:38,888 [:TP-Processor3] ERROR >> org.apache.jk.common.ChannelSocket - Error, processing connection >> java.lang.IndexOutOfBoundsException >> at java.io.BufferedInputStream.read(BufferedInputStream.java:310) >> at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:620) >> at >> org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:577) >> at >> org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:685) >> at >> org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889) >> at >> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690) >> at java.lang.Thread.run(Thread.java:619) >> >> >> Tomcat Message Sample (from rsyslog server/collector): >> ---------------------------------------------------------------------- >> <134>Sep 1 17:31:57 appserver1 logview: 2011-08-15 >> 17:24:38,888[:TP-Processor3] ERROR org.apache.jk.common.ChannelSocket - >> Error, processing connection >> <134>Sep 1 17:31:57 appserver1 logview: >> java.lang.IndexOutOfBoundsException >> <134>Sep 1 17:31:57 appserver1 logview: at >> java.io.BufferedInputStream.read(BufferedInputStream.java:310) >> <134>Sep 1 17:31:57 appserver1 logview: at >> org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:620) >> <134>Sep 1 17:31:57 appserver1 logview: at >> org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:577) >> <134>Sep 1 17:31:57 appserver1 logview: at >> org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:685) >> <134>Sep 1 17:31:57 appserver1 logview: at >> org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889) >> <134>Sep 1 17:31:57 appserver1 logview: at >> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690) >> <134>Sep 1 17:31:57 appserver1 logview: at >> java.lang.Thread.run(Thread.java:619) >> >> >> >> Please let me know if you need additional information or details from me. >> Thanks in advance for your assistance. >> >> Regards, >> >> Tracy >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rodney.mckee at gmail.com Mon Sep 5 05:17:55 2011 From: rodney.mckee at gmail.com (Rodney McKee) Date: Mon, 05 Sep 2011 13:17:55 +1000 (EST) Subject: [rsyslog] file permissions In-Reply-To: Message-ID: <3257465d-2d5f-42ee-a83a-3b8dc9ccde98@wsrmckee> I've got file permissions working but was wondering if it is possible to log a separate set of files with different permissions to those set with the global options for $DirGroup and $FileGroup. Using setgid on the directory does not appear to work as the files still get created with the global $DirGroup and $FileGroup. From rgerhards at hq.adiscon.com Mon Sep 5 10:43:31 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 5 Sep 2011 10:43:31 +0200 Subject: [rsyslog] file permissions In-Reply-To: <3257465d-2d5f-42ee-a83a-3b8dc9ccde98@wsrmckee> References: <3257465d-2d5f-42ee-a83a-3b8dc9ccde98@wsrmckee> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA728118E@GRFEXC.intern.adiscon.com> Sequence is important. You can use $(dir/file)Group multiple times. It affects all files until the next directive is given. HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rodney McKee > Sent: Monday, September 05, 2011 5:18 AM > To: rsyslog-users > Subject: [rsyslog] file permissions > > I've got file permissions working but was wondering if it is possible to log a > separate set of files with different permissions to those set with the global > options for $DirGroup and $FileGroup. Using setgid on the directory does not > appear to work as the files still get created with the global $DirGroup and > $FileGroup. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mark at thermeon.com Tue Sep 6 12:07:39 2011 From: mark at thermeon.com (Mark Olliver) Date: Tue, 6 Sep 2011 11:07:39 +0100 Subject: [rsyslog] Error message Message-ID: <00a601cc6c7c$d2f2be10$78d83a30$@thermeon.com> Hi All, We are getting a lot of the following message appear in our syslogs: Sep 6 10:01:04 aumelcpve1 rsyslogd: recvfrom UNIX: Socket operation on non-socket Sep 6 10:01:35 aumelcpve1 rsyslogd: last message repeated 648639 times Sep 6 10:02:36 aumelcpve1 rsyslogd: last message repeated 1294456 times Sep 6 10:03:37 aumelcpve1 rsyslogd: last message repeated 1299098 times When this happens the process seems to be using around 137% of the CPU and then after a while stops logging to files altogether although continues to keep running. A restart solves the problem. This condition does not start straight away. I am running on RHEL6.1 with rsyslogd, rsyslogd 4.6.8, compiled with: FEATURE_REGEXP: Yes FEATURE_LARGEFILE: No FEATURE_NETZIP (message compression): Yes GSSAPI Kerberos 5 support: No FEATURE_DEBUG (debug build, slow code): No Atomic operations supported: Yes Runtime Instrumentation (slow code): No Any ideas or help would be greatly appreciated. Thanks Mark From rgerhards at hq.adiscon.com Tue Sep 6 12:19:50 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 6 Sep 2011 12:19:50 +0200 Subject: [rsyslog] Error message In-Reply-To: <00a601cc6c7c$d2f2be10$78d83a30$@thermeon.com> References: <00a601cc6c7c$d2f2be10$78d83a30$@thermeon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72811A3@GRFEXC.intern.adiscon.com> A debug log would be useful to have. I know it can get pretty large, but it still is the best source of information. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mark Olliver > Sent: Tuesday, September 06, 2011 12:08 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Error message > > Hi All, > > > > We are getting a lot of the following message appear in our syslogs: > > Sep 6 10:01:04 aumelcpve1 rsyslogd: recvfrom UNIX: Socket operation on > non-socket > > Sep 6 10:01:35 aumelcpve1 rsyslogd: last message repeated 648639 times > > Sep 6 10:02:36 aumelcpve1 rsyslogd: last message repeated 1294456 times > > Sep 6 10:03:37 aumelcpve1 rsyslogd: last message repeated 1299098 times > > > > When this happens the process seems to be using around 137% of the CPU > and then after a while stops logging to files altogether although continues to > keep running. A restart solves the problem. This condition does not start > straight away. > > > > I am running on RHEL6.1 with rsyslogd, > > rsyslogd 4.6.8, compiled with: > > FEATURE_REGEXP: Yes > > FEATURE_LARGEFILE: No > > FEATURE_NETZIP (message compression): Yes > > GSSAPI Kerberos 5 support: No > > FEATURE_DEBUG (debug build, slow code): No > > Atomic operations supported: Yes > > Runtime Instrumentation (slow code): No > > > > Any ideas or help would be greatly appreciated. > > > > Thanks > > > > Mark > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mark at thermeon.com Tue Sep 6 12:35:56 2011 From: mark at thermeon.com (Mark Olliver) Date: Tue, 6 Sep 2011 11:35:56 +0100 Subject: [rsyslog] Error message In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA72811A3@GRFEXC.intern.adiscon.com> References: <00a601cc6c7c$d2f2be10$78d83a30$@thermeon.com> <9B6E2A8877C38245BFB15CC491A11DA72811A3@GRFEXC.intern.adiscon.com> Message-ID: <00c301cc6c80$c5dc61f0$519525d0$@thermeon.com> Hi Rainer, I am not sure I can give a debug log as there is too much PCI compliant information in there, on my other hosts which are Ubuntu I am not having this issue. I have run in debug mode but so far no found the issue there as yet If you have any ideas on how to isolate it then I am happy to try that and if I can get a clean debug then I am happy to send that over. Mark From rgerhards at hq.adiscon.com Tue Sep 6 12:37:59 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 6 Sep 2011 12:37:59 +0200 Subject: [rsyslog] Error message In-Reply-To: <00c301cc6c80$c5dc61f0$519525d0$@thermeon.com> References: <00a601cc6c7c$d2f2be10$78d83a30$@thermeon.com><9B6E2A8877C38245BFB15CC491A11DA72811A3@GRFEXC.intern.adiscon.com> <00c301cc6c80$c5dc61f0$519525d0$@thermeon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72811A4@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mark Olliver > Sent: Tuesday, September 06, 2011 12:36 PM > To: 'rsyslog-users' > Subject: Re: [rsyslog] Error message > > Hi Rainer, > > I am not sure I can give a debug log as there is too much PCI compliant > information in there, on my other hosts which are Ubuntu I am not having > this issue. > > I have run in debug mode but so far no found the issue there as yet If you > have any ideas on how to isolate it then I am happy to try that and if I can get > a clean debug then I am happy to send that over. Quite honestly, this is too generic to have any decent clue. You may want to review the debug log yourself and check if there are any messages related to the socket or its state. Especially between the point where it worked vs. where it does not work. I am unable to diagnose the problem without instructions to either reproduce or a complete debug log -- sorry. Rainer > > > Mark > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From jpoling at moody.edu Tue Sep 6 14:49:11 2011 From: jpoling at moody.edu (Jeff Poling) Date: Tue, 6 Sep 2011 07:49:11 -0500 Subject: [rsyslog] New User Help Message-ID: <9599A350A0A5884DB4E50D83F9287D0F01F305D0E8@exchmbx01.moody.edu> I am new to rsyslog (and Linux, really) and now I am responsible for all of our Linux servers which are setup to use rsyslog. We have been seeing the rsyslog service on our syslog collector stop. When this happens, the servers sending data to the collector spool their data and eventually /opt fills up. I would like to upgrade to the latest stable version, but I am not sure what the best way to do that is? We are using rsyslog 5.4.0. How do I go about upgrading to the latest version? Thanks! Jeff Jeffrey Poling System Administrator | Information Systems Moody Bible Institute 820 N. LaSalle Blvd., Chicago, IL 60610 312-329-8968 www.moodyministries.net >From the Word. To Life. From tracy.felts at etrade.com Tue Sep 6 15:42:03 2011 From: tracy.felts at etrade.com (Felts, Tracy) Date: Tue, 6 Sep 2011 13:42:03 +0000 Subject: [rsyslog] rsyslog (v 5.8.3) multi-line message support In-Reply-To: References: Message-ID: Thanks David. I will try it out. -----Original Message----- From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm Sent: Friday, September 02, 2011 10:56 PM To: rsyslog-users Subject: Re: [rsyslog] rsyslog (v 5.8.3) multi-line message support I went looking for the documentation on this and couldn't find it. the setting is $InputFileReadMode 0 = defult, line based 1 = indented 2 = paragraph please let me know how it works (or fails) for you. David Lang On Fri, 2 Sep 2011, david at lang.hm wrote: > Date: Fri, 2 Sep 2011 09:06:16 -0700 (PDT) > From: david at lang.hm > Reply-To: rsyslog-users > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog (v 5.8.3) multi-line message support > > Yes, I submitted a patch to imfile that allows you to specify what the > log separator is. > > default, each line is a new message > > indented, like your tomcat messages, a new log entry starts at the > beginning of a line, if a line starts with a space it's part of the > log message before it > > paragraph, there is a blank line between log messages. > > I don't knwo if this patch was applied to the 5.x series or only the > 6.1 series. > > when this combines the mulitline messages, it will replace the newline > with a #xxx number, but everything should come out as one line. > > If this does not work, please let me know so that we can fix it (I > think you are the first person to as for this since I submitted it) > > David Lang > > On Fri, 2 Sep 2011, Felts, Tracy wrote: > >> Date: Fri, 2 Sep 2011 15:28:05 +0000 >> From: "Felts, Tracy" >> Reply-To: rsyslog-users >> To: "rsyslog at lists.adiscon.com" >> Subject: [rsyslog] rsyslog (v 5.8.3) multi-line message support >> >> Does rsyslog support multi-line log messaging such as Tomcat stack >> traces or kernel messages? During my testing I'm seeing a single >> multi-line message broken up in to single line messages on the >> rsyslog receiver/collector (see samples below). I've read through >> the various posts in the forum and mailing lists on this topic but >> haven't come away with a clear answer. I understand multi-line >> message support is not a trivial solution, broken framing, trying to >> determine what constitutes the beginning and end of a multi-line message. >> >> I've tried several things read from various posts such as octet-frame >> counting (http://kb.monitorware.com/multiline-messages-t10184.html), >> disabling control character escaping ($EscapeControlCharactersOnReceive) >> with little success. I've read David Lang has been working on multi-line >> logging but I'm not sure the status today. >> >> I've included the configurations for two servers I'm using for >> rsyslog testing. You can find a multi-line log message sample below >> the configurations. >> >> >> Receiver/Collector/Server rsyslog.conf: >> ------------------------------------------------- >> $EscapeControlCharactersOnReceive off >> >> # Load desired modules. >> # for TCP use: >> $modload imtcp >> # for UDP use: >> #$modload imudp >> >> # Provides kernel logging support (previously done by rklogd) >> $ModLoad imklog # Provides support for local system logging (e.g. via >> logger command) $ModLoad imuxsock # Reliable Event Logging Protocol >> module $ModLoad imrelp >> >> # Use traditional timestamp format >> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format $template >> rawfmt,"%rawmsg%\n" >> >> #$InputTCPServerBindRuleset remote5140 $InputTCPServerRun 5140 >> $InputRELPServerRun 10514 #$UDPServerRun 514 >> >> # Maximum rsyslog message size >> $MaxMessageSize 32k >> >> $template >> AccessLog,"/var/log/rsyslog/%HOSTNAME%/%$YEAR%-%$MONTH%-%$DAY%/access.log" >> $template >> SecureLog,"/var/log/rsyslog/%HOSTNAME%/%$YEAR%-%$MONTH%-%$DAY%/secure.log" >> $template >> MessagesLog,"/var/log/rsyslog/%HOSTNAME%/%$YEAR%-%$MONTH%-%$DAY%/messages.log" >> $template >> CatalinaLog,"/var/log/rsyslog/%HOSTNAME%/%$YEAR%-%$MONTH%-%$DAY%/catalina.log" >> >> # Log all kernel messages to the console. >> # Logging much else clutters up the screen. >> #kern.* /dev/console >> >> # Log anything (except mail) of level info or higher. >> # Don't log private authentication messages! >> *.info;kern.notice;mail.none;authpriv.none;cron.none -/var/log/messages >> >> # The authpriv file has restricted access. >> #authpriv.* /var/log/secure >> authpriv.*;auth.info /var/log/secure >> >> # Log all the mail messages in one place. >> mail.* -/var/log/maillog >> >> # Log cron stuff >> cron.* /var/log/cron >> >> # Everybody gets emergency messages >> *.emerg * >> >> # Save news errors of level crit and higher in a special file. >> uucp,news.crit /var/log/spooler >> >> # Save boot messages also to boot.log >> local7.* /var/log/boot.log >> >> local6.* >> -/var/opt/silvertail/log/silvertail.log >> >> ### RSYSLOG Rules >> if $syslogfacility-text == 'local5' and $programname == 'apache' then >> -?AccessLog & ~ if $syslogfacility-text == 'local0' then >> -?CatalinaLog;rawfmt & ~ #if $syslogfacility-text == 'local5' and >> $programname == 'logview' then -?MessagesLog #& ~ if >> $syslogfacility-text == 'authpriv' then -?SecureLog >> >> >> Client rsyslog.conf: >> --------------------------- >> $EscapeControlCharactersOnReceive off >> >> # Provides kernel logging support (previously done by rklogd) >> $ModLoad imklog # Provides MARK support. >> #$ModLoad immark >> # Provides support for local system logging (e.g. via logger command) >> $ModLoad imuxsock # File Monitor configs $ModLoad imfile # Reliable >> Event Logging Protocol module $ModLoad omrelp # Reliable Event >> Logging Protocol module #$ModLoad omhdfs >> >> $RepeatedMsgReduction on >> $MaxMessageSize 32k >> >> # Use traditional timestamp format >> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format >> >> $WorkDirectory /var/log/rsyslog/work # default location for work >> (spool) files >> $ActionQueueType LinkedList # use asynchronous processing >> $ActionQueueFileName spool_data # set file name, also enables disk mode >> $ActionResumeRetryCount -1 # infinite retries on insert failure >> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts >> down $ActionSendResendLastMsgOnReconnect on >> >> # File Monitor configs >> #$InputFileName /var/log/messages >> $InputFileName /adm/web/logview/logs/catalina.log >> $InputFileTag logview: >> $InputFileStateFile /var/log/rsyslog-messages.stat $InputFileSeverity >> info $InputFileFacility local0 $InputRunFileMonitor >> >> # Log all kernel messages to the console. >> # Logging much else clutters up the screen. >> #kern.* /dev/console >> >> # Log anything (except mail) of level info or higher. >> # Don't log private authentication messages! >> *.info;mail.none;authpriv.none;cron.none /var/log/messages >> >> local5.* :omrelp:10.152.106.24:10514;RSYSLOG_ForwardFormat >> &~ >> local0.* @@(o)10.152.106.24:5140 >> &~ >> >> authpriv.* :omrelp:10.152.106.24:10514;RSYSLOG_ForwardFormat >> >> # Log all the mail messages in one place. >> mail.* -/var/log/maillog >> >> # Log cron stuff >> cron.* /var/log/cron >> >> # Everybody gets emergency messages >> *.emerg * >> >> # Save news errors of level crit and higher in a special file. >> uucp,news.crit /var/log/spooler >> >> # Save boot messages also to boot.log >> local7.* /var/log/boot.log >> >> >> Tomcat Message Sample (Original): >> -------------------------------------------- >> 2011-08-15 17:24:38,888 [:TP-Processor3] ERROR >> org.apache.jk.common.ChannelSocket - Error, processing connection >> java.lang.IndexOutOfBoundsException >> at java.io.BufferedInputStream.read(BufferedInputStream.java:310) >> at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:620) >> at >> org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:577) >> at >> org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:685) >> at >> org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889) >> at >> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690) >> at java.lang.Thread.run(Thread.java:619) >> >> >> Tomcat Message Sample (from rsyslog server/collector): >> --------------------------------------------------------------------- >> - <134>Sep 1 17:31:57 appserver1 logview: 2011-08-15 >> 17:24:38,888[:TP-Processor3] ERROR org.apache.jk.common.ChannelSocket >> - Error, processing connection <134>Sep 1 17:31:57 appserver1 >> logview: >> java.lang.IndexOutOfBoundsException >> <134>Sep 1 17:31:57 appserver1 logview: at >> java.io.BufferedInputStream.read(BufferedInputStream.java:310) >> <134>Sep 1 17:31:57 appserver1 logview: at >> org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:620) >> <134>Sep 1 17:31:57 appserver1 logview: at >> org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:577) >> <134>Sep 1 17:31:57 appserver1 logview: at >> org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:685) >> <134>Sep 1 17:31:57 appserver1 logview: at >> org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889) >> <134>Sep 1 17:31:57 appserver1 logview: at >> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690) >> <134>Sep 1 17:31:57 appserver1 logview: at >> java.lang.Thread.run(Thread.java:619) >> >> >> >> Please let me know if you need additional information or details from me. >> Thanks in advance for your assistance. >> >> Regards, >> >> Tracy >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From david at lang.hm Tue Sep 6 21:24:03 2011 From: david at lang.hm (david at lang.hm) Date: Tue, 6 Sep 2011 12:24:03 -0700 (PDT) Subject: [rsyslog] New User Help In-Reply-To: <9599A350A0A5884DB4E50D83F9287D0F01F305D0E8@exchmbx01.moody.edu> References: <9599A350A0A5884DB4E50D83F9287D0F01F305D0E8@exchmbx01.moody.edu> Message-ID: On Tue, 6 Sep 2011, Jeff Poling wrote: > I am new to rsyslog (and Linux, really) and now I am responsible for all of our Linux servers which are setup to use rsyslog. We have been seeing the rsyslog service on our syslog collector stop. When this happens, the servers sending data to the collector spool their data and eventually /opt fills up. > > I would like to upgrade to the latest stable version, but I am not sure what the best way to do that is? We are using rsyslog 5.4.0. How do I go about upgrading to the latest version? The key question is if you are willing to compile your own version, or do are you limited to running packages from your distro? (and if so, what distro?) if you are compiling it yourself, I would recommend using something like checkinstall to make .deb or .rpm packages that you then deploy out to your systems. David Lang From jpoling at moody.edu Tue Sep 6 21:48:51 2011 From: jpoling at moody.edu (Jeff Poling) Date: Tue, 6 Sep 2011 14:48:51 -0500 Subject: [rsyslog] New User Help In-Reply-To: References: <9599A350A0A5884DB4E50D83F9287D0F01F305D0E8@exchmbx01.moody.edu> Message-ID: <9599A350A0A5884DB4E50D83F9287D0F01F305D233@exchmbx01.moody.edu> David, > The key question is if you are willing to compile your own version, or do are > you limited to running packages from your distro? (and if so, what > distro?) > > if you are compiling it yourself, I would recommend using something like > checkinstall to make .deb or .rpm packages that you then deploy out to your > systems. [] Thanks for your reply. We are using a mix of RHEL4 and RHEL5 systems. The collector is RHEL 4. My preference would be to have an RPM that I can deploy, but if I have to compile my own, I can probably muddle my way through that. I found the following help from rsyslog: http://www.rsyslog.com/doc/install.html Is that pretty much all that is needed to compile it for my systems (besides the steps to make it an rpm)? Thanks, Jeff From david at lang.hm Tue Sep 6 21:59:52 2011 From: david at lang.hm (david at lang.hm) Date: Tue, 6 Sep 2011 12:59:52 -0700 (PDT) Subject: [rsyslog] New User Help In-Reply-To: <9599A350A0A5884DB4E50D83F9287D0F01F305D233@exchmbx01.moody.edu> References: <9599A350A0A5884DB4E50D83F9287D0F01F305D0E8@exchmbx01.moody.edu> <9599A350A0A5884DB4E50D83F9287D0F01F305D233@exchmbx01.moody.edu> Message-ID: On Tue, 6 Sep 2011, Jeff Poling wrote: > David, > >> The key question is if you are willing to compile your own version, or do are >> you limited to running packages from your distro? (and if so, what >> distro?) >> >> if you are compiling it yourself, I would recommend using something like >> checkinstall to make .deb or .rpm packages that you then deploy out to your >> systems. > > [] Thanks for your reply. We are using a mix of RHEL4 and RHEL5 > systems. The collector is RHEL 4. My preference would be to have an > RPM that I can deploy, but if I have to compile my own, I can probably > muddle my way through that. I found the following help from rsyslog: > http://www.rsyslog.com/doc/install.html Is that pretty much all that is > needed to compile it for my systems (besides the steps to make it an > rpm)? that will work if you compile it manually on each server, but if you create a .rpm, then you do something like this on the build server and then deply the .rpm on each of your other servers. with checkinstall you replace the 'make install' step with 'checkinstall' and then it creates the .rpm file that you can install. you may be able to make one .rpm package that will work for both RHEL4 and RHEL5 sytems, but I would recommend doing a separate package for each one as you may run into library version issues otherwise. by the way, RHEL4 is _very_ near the point where there will be no more security patches so you should be looking at migrating off of it rather suddenly. David Lang From jason at jasonantman.com Tue Sep 6 22:05:48 2011 From: jason at jasonantman.com (Jason Antman) Date: Tue, 06 Sep 2011 16:05:48 -0400 Subject: [rsyslog] imrelp and ruleset? Message-ID: <4E667D1C.50702@jasonantman.com> Hello, I'm planning to finally start moving some of our client (log sending) devices from stock sysklogd (UDP) to rsyslog/RELP (central server is rsyslog 5.6.0 currently, planning 5.8.5 soon). Right now we have a default ruleset for localhost on the central log server, and bind imudp and imtcp to a separate ruleset for remote hosts. I notice that there is no $InputRELPServerBindRuleset directive. Should I file a bug for this, or is there a reason why it hasn't been implemented the way it has for the other remote input modules? Thanks, Jason Antman Rutgers University From s.pilypenko at nimble.com Wed Sep 7 00:15:50 2011 From: s.pilypenko at nimble.com (Sergey Pilypenko) Date: Wed, 7 Sep 2011 01:15:50 +0300 Subject: [rsyslog] rsyslog tcp forwarding of stacktraces In-Reply-To: References: Message-ID: Hi! I trying to make logging from my python scripts to loggly (loggly.com) via rsyslog tcp forwarding. Rsyslog receiving log message from unix log socket and sends it to loggly via tcp. The problem is when my python scripts generate exception and send stacktrace to rsyslog. Rsyslog replace all spectial characters from trace and I see it at loggly like one ugly row. When I use "$EscapeControlCharactersOnReceive off" directive rsyslog split trace to separate messages and add for each one date, host, etc. Then I see that trace like set of separate messages on loggly. So what I can do to see normal traces on loggly service? From s.pilypenko at nimble.com Wed Sep 7 00:17:03 2011 From: s.pilypenko at nimble.com (Sergey Pilypenko) Date: Wed, 7 Sep 2011 01:17:03 +0300 Subject: [rsyslog] rsyslog tcp forwarding of stacktraces Message-ID: Hi! I trying to make logging from my python scripts to loggly (loggly.com) via rsyslog tcp forwarding. Rsyslog receiving log message from unix log socket and sends it to loggly via tcp. The problem is when my python scripts generate exception and send stacktrace to rsyslog. Rsyslog replace all spectial characters from trace and I see it at loggly like one ugly row. When I use "$EscapeControlCharactersOnReceive off" directive rsyslog split trace to separate messages and add for each one date, host, etc. Then I see that trace like set of separate messages on loggly. So what I can do to see normal traces on loggly service? From david at lang.hm Wed Sep 7 00:21:39 2011 From: david at lang.hm (david at lang.hm) Date: Tue, 6 Sep 2011 15:21:39 -0700 (PDT) Subject: [rsyslog] rsyslog tcp forwarding of stacktraces In-Reply-To: References: Message-ID: On Wed, 7 Sep 2011, Sergey Pilypenko wrote: > Hi! I trying to make logging from my python scripts to loggly (loggly.com) > via rsyslog tcp forwarding. Rsyslog receiving log message from unix log > socket and sends it to loggly via tcp. The problem is when my python scripts > generate exception and send stacktrace to rsyslog. Rsyslog replace all > spectial characters from trace and I see it at loggly like one ugly row. > When I use "$EscapeControlCharactersOnReceive off" directive rsyslog split > trace to separate messages and add for each one date, host, etc. Then I see > that trace like set of separate messages on loggly. So what I can do to see > normal traces on loggly service? unfortunantly you need to choose one or the other. standard syslog says that each line sent to syslog is a separate log (the behavior you see in the second case), a newline is not a valid character inside a message. rsyslog has the default of replacing control characters in a single entry that are sent to it with escaped representations of those characters. This makes sure that everything upstream of you treats the log message as a single entry, but it makes it look ugly as you are seeing in the first case. remember that the #nnn represent specific characters, so you can run the log entry through sed to replace #011 with a tab, etc. David Lang From mark.a.reidenbach at gmail.com Wed Sep 7 05:23:59 2011 From: mark.a.reidenbach at gmail.com (Mark Reidenbach) Date: Tue, 6 Sep 2011 22:23:59 -0500 Subject: [rsyslog] Log avaya ip office phone system - how to prepend a tcp message Message-ID: I have an avaya phone system that logs call records via tcp, but the format of the message does not include a timestampm hostname or tag. It sends a line of csv data like the following line (the lines end in a CR/LF according to tcpdump): 2011/09/06 16:01:34,00:00:08,2,918176574009,I,550,2477,,0,1000237,0,E101,connie,T9001,Line 1.1,0,0,,,,,,,,,,,,,mark, This causes debug messages like: 2903.519225673:7f6457fff700: TCP Message with octet-counter, size 2011. 2903.519236548:7f6457fff700: Called LogError, msg: Framing Error in received TCP message: delimiter is not SP but has ASCII value 47. Using telnet to send the same data causes the same error, but if the data is sent prefixed with "Sep 6 15:36:19 server avaya: " the message is logged as hoped: Sep 6 15:36:19 server avaya: 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark, I had hoped to use a template to add the generated timestamp, host and tag, but it seems like the message is discarded in the input phase before using a template to add the missing info is possible. Is it possible to prefix tcp messages without writing a custom input module in C? From ctoomey at gmail.com Wed Sep 7 08:13:59 2011 From: ctoomey at gmail.com (Chris Toomey) Date: Tue, 6 Sep 2011 23:13:59 -0700 Subject: [rsyslog] NFS log files not re-opened after rotation Message-ID: We've recently switched our Apache 2.x configuration to log indirectly via rsyslog rather than directly to the log files, in order to get the benefits of asynchronous logging (we see periodic latency spikes when logging to our NFS filer). When the rsyslog-generated Apache log files are on a local disk, the nightly log rotation (which triggers an rsyslog reload == HUP signal) works fine. But when the log files are on an NFS filer, for some reason rsyslogd doesn't re-open them after the HUP -- lsof on the rsyslogd process shows the log files open before but not after the HUP. Doing a restart of rsyslogd does open them again. We're using rsyslog 4.2.0, which is the version distributed with our Ubuntu 10.10. Any ideas on why rsyslog is responding differently to the HUP for local vs. NFS log files and what to do to make it work for NFS log files? I ran rsyslogd with -dn and captured the debug output across the HUP processing both when using local and NFS files, but didn't see anything obvious about why the log files weren't reopened when over NFS. Thanks for any help. Chris From david at lang.hm Wed Sep 7 08:43:45 2011 From: david at lang.hm (david at lang.hm) Date: Tue, 6 Sep 2011 23:43:45 -0700 (PDT) Subject: [rsyslog] Log avaya ip office phone system - how to prepend a tcp message In-Reply-To: References: Message-ID: On Tue, 6 Sep 2011, Mark Reidenbach wrote: > I have an avaya phone system that logs call records via tcp, but the format > of the message does not include a timestampm hostname or tag. It sends a > line of csv data like the following line (the lines end in a CR/LF according > to tcpdump): > 2011/09/06 > 16:01:34,00:00:08,2,918176574009,I,550,2477,,0,1000237,0,E101,connie,T9001,Line > 1.1,0,0,,,,,,,,,,,,,mark, > > This causes debug messages like: > 2903.519225673:7f6457fff700: TCP Message with octet-counter, size 2011. > 2903.519236548:7f6457fff700: Called LogError, msg: Framing Error in received > TCP message: delimiter is not SP but has ASCII value 47. > > Using telnet to send the same data causes the same error, but if the data is > sent prefixed with "Sep 6 15:36:19 server avaya: " the message is logged as > hoped: > Sep 6 15:36:19 server avaya: 2011/09/06 > 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line > 1.22,0,0,,,,,,,,,,,U,mark, > > I had hoped to use a template to add the generated timestamp, host and tag, > but it seems like the message is discarded in the input phase before using a > template to add the missing info is possible. > > Is it possible to prefix tcp messages without writing a custom input module > in C? I think this needs the custom parser module (much simpler than a custom imput module), but still a bit of C programming so is that two lines, or three (line wrapping makes it hard to tell) David Lang From david at lang.hm Wed Sep 7 08:46:08 2011 From: david at lang.hm (david at lang.hm) Date: Tue, 6 Sep 2011 23:46:08 -0700 (PDT) Subject: [rsyslog] NFS log files not re-opened after rotation In-Reply-To: References: Message-ID: On Tue, 6 Sep 2011, Chris Toomey wrote: > We've recently switched our Apache 2.x configuration to log indirectly via > rsyslog rather than directly to the log files, in order to get the benefits > of asynchronous logging (we see periodic latency spikes when logging to our > NFS filer). > > When the rsyslog-generated Apache log files are on a local disk, the nightly > log rotation (which triggers an rsyslog reload == HUP signal) works fine. > But when the log files are on an NFS filer, for some reason rsyslogd doesn't > re-open them after the HUP -- lsof on the rsyslogd process shows the log > files open before but not after the HUP. Doing a restart of rsyslogd does > open them again. > > We're using rsyslog 4.2.0, which is the version distributed with our Ubuntu > 10.10. > > Any ideas on why rsyslog is responding differently to the HUP for local vs. > NFS log files and what to do to make it work for NFS log files? > > I ran rsyslogd with -dn and captured the debug output across the HUP > processing both when using local and NFS files, but didn't see anything > obvious about why the log files weren't reopened when over NFS. rsyslog 4.2 is ancient at this point, and there has been a lot of work, (I believe including changes in how HUP works) since that point. Since 4.2 is no longer supported by adiscon, I think your options are to upgrade to a newer version or ask Ubuntu for help. David Lang From rgerhards at hq.adiscon.com Wed Sep 7 09:53:50 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 7 Sep 2011 09:53:50 +0200 Subject: [rsyslog] NFS log files not re-opened after rotation In-Reply-To: References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72811B6@GRFEXC.intern.adiscon.com> This sounds like another incarnation of a problem with the Ubuntu package. See this bug tracker: https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/407862 If you think this is the reason, please make yourself heard. It would be really great if Ubuntu finally fixed the issue... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Chris Toomey > Sent: Wednesday, September 07, 2011 8:14 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] NFS log files not re-opened after rotation > > We've recently switched our Apache 2.x configuration to log indirectly > via > rsyslog rather than directly to the log files, in order to get the > benefits > of asynchronous logging (we see periodic latency spikes when logging to > our > NFS filer). > > When the rsyslog-generated Apache log files are on a local disk, the > nightly > log rotation (which triggers an rsyslog reload == HUP signal) works > fine. > But when the log files are on an NFS filer, for some reason rsyslogd > doesn't > re-open them after the HUP -- lsof on the rsyslogd process shows the > log > files open before but not after the HUP. Doing a restart of rsyslogd > does > open them again. > > We're using rsyslog 4.2.0, which is the version distributed with our > Ubuntu > 10.10. > > Any ideas on why rsyslog is responding differently to the HUP for local > vs. > NFS log files and what to do to make it work for NFS log files? > > I ran rsyslogd with -dn and captured the debug output across the HUP > processing both when using local and NFS files, but didn't see anything > obvious about why the log files weren't reopened when over NFS. > > Thanks for any help. > Chris > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Sep 7 09:56:37 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 7 Sep 2011 09:56:37 +0200 Subject: [rsyslog] Log avaya ip office phone system - how to prepend a tcp message In-Reply-To: References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72811B7@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, September 07, 2011 8:44 AM > To: rsyslog-users > Subject: Re: [rsyslog] Log avaya ip office phone system - how to > prepend a tcp message > > On Tue, 6 Sep 2011, Mark Reidenbach wrote: > > > I have an avaya phone system that logs call records via tcp, but the > format > > of the message does not include a timestampm hostname or tag. It > sends a > > line of csv data like the following line (the lines end in a CR/LF > according > > to tcpdump): > > 2011/09/06 > > > 16:01:34,00:00:08,2,918176574009,I,550,2477,,0,1000237,0,E101,connie,T9 > 001,Line > > 1.1,0,0,,,,,,,,,,,,,mark, > > > > This causes debug messages like: > > 2903.519225673:7f6457fff700: TCP Message with octet-counter, size > 2011. > > 2903.519236548:7f6457fff700: Called LogError, msg: Framing Error in > received > > TCP message: delimiter is not SP but has ASCII value 47. > > > > Using telnet to send the same data causes the same error, but if the > data is > > sent prefixed with "Sep 6 15:36:19 server avaya: " the message is > logged as > > hoped: > > Sep 6 15:36:19 server avaya: 2011/09/06 > > > 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T900 > 1,Line > > 1.22,0,0,,,,,,,,,,,U,mark, > > > > I had hoped to use a template to add the generated timestamp, host > and tag, > > but it seems like the message is discarded in the input phase before > using a > > template to add the missing info is possible. > > > > Is it possible to prefix tcp messages without writing a custom input > module > > in C? > > I think this needs the custom parser module (much simpler than a custom > imput module), but still a bit of C programming The main problem is that the message looks like it uses TCP octect-counted framing, but does not do so. This would require at least changes the the tcp input, or a new input. A parser can not handle that part of the problem, as it is not just a malformed message, but also a protocol violation. > so is that two lines, or three (line wrapping makes it hard to tell) If it is multiple lines, that would be even more complicated and requie a dedicated input in any case. Rainer > > David Lang > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Sep 7 10:02:27 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 7 Sep 2011 10:02:27 +0200 Subject: [rsyslog] imrelp and ruleset? In-Reply-To: <4E667D1C.50702@jasonantman.com> References: <4E667D1C.50702@jasonantman.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72811BA@GRFEXC.intern.adiscon.com> The short answer is that this feature is not yet implemented for imrelp. However, I'll have a look and check if it can be sufficiently quick implemented. Please ping me if you have not heard back by next Monday. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Jason Antman > Sent: Tuesday, September 06, 2011 10:06 PM > To: rsyslog-users > Subject: [rsyslog] imrelp and ruleset? > > Hello, > > I'm planning to finally start moving some of our client (log sending) > devices from stock sysklogd (UDP) to rsyslog/RELP (central server is > rsyslog 5.6.0 currently, planning 5.8.5 soon). Right now we have a > default ruleset for localhost on the central log server, and bind imudp > and imtcp to a separate ruleset for remote hosts. I notice that there > is > no $InputRELPServerBindRuleset directive. Should I file a bug for this, > or is there a reason why it hasn't been implemented the way it has for > the other remote input modules? > > Thanks, > Jason Antman > Rutgers University > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From oliver at obeattie.com Wed Sep 7 10:32:19 2011 From: oliver at obeattie.com (Oliver Beattie) Date: Wed, 7 Sep 2011 09:32:19 +0100 Subject: [rsyslog] imrelp and ruleset? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA72811BA@GRFEXC.intern.adiscon.com> References: <4E667D1C.50702@jasonantman.com> <9B6E2A8877C38245BFB15CC491A11DA72811BA@GRFEXC.intern.adiscon.com> Message-ID: Just weighing in, this was something I was looking for a little while back and was surprised not to find, too. Would be very useful :) ?Oliver On 7 September 2011 09:02, Rainer Gerhards wrote: > The short answer is that this feature is not yet implemented for imrelp. > However, I'll have a look and check if it can be sufficiently quick > implemented. Please ping me if you have not heard back by next Monday. > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Jason Antman > > Sent: Tuesday, September 06, 2011 10:06 PM > > To: rsyslog-users > > Subject: [rsyslog] imrelp and ruleset? > > > > Hello, > > > > I'm planning to finally start moving some of our client (log sending) > > devices from stock sysklogd (UDP) to rsyslog/RELP (central server is > > rsyslog 5.6.0 currently, planning 5.8.5 soon). Right now we have a > > default ruleset for localhost on the central log server, and bind imudp > > and imtcp to a separate ruleset for remote hosts. I notice that there > > is > > no $InputRELPServerBindRuleset directive. Should I file a bug for this, > > or is there a reason why it hasn't been implemented the way it has for > > the other remote input modules? > > > > Thanks, > > Jason Antman > > Rutgers University > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Sep 7 15:56:24 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 7 Sep 2011 15:56:24 +0200 Subject: [rsyslog] imrelp and ruleset? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA72811BA@GRFEXC.intern.adiscon.com> References: <4E667D1C.50702@jasonantman.com> <9B6E2A8877C38245BFB15CC491A11DA72811BA@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72811C8@GRFEXC.intern.adiscon.com> Quick update: it's not entirely trivial, as librelp is missing some plumbing in order to shuffle the ruleset information back to the message-receiving callback. Changing all that is beyond the time frame I currently have. I think a compromise is that I permit to set a single replacement ruleset for *all* relp listeners (with tcp, you can assign different ones to different listeners). Does this sound useful enough? Also I'd like to note that I will enhance the current devel, only. I guess the patch will also apply to older versions, but I will not check that (doing new development in older versions is very time consuming for me, so I usually do that only under paid contracts - sorry for that). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Wednesday, September 07, 2011 10:02 AM > To: rsyslog-users > Subject: Re: [rsyslog] imrelp and ruleset? > > The short answer is that this feature is not yet implemented for imrelp. > However, I'll have a look and check if it can be sufficiently quick > implemented. Please ping me if you have not heard back by next Monday. > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Jason Antman > > Sent: Tuesday, September 06, 2011 10:06 PM > > To: rsyslog-users > > Subject: [rsyslog] imrelp and ruleset? > > > > Hello, > > > > I'm planning to finally start moving some of our client (log sending) > > devices from stock sysklogd (UDP) to rsyslog/RELP (central server is > > rsyslog 5.6.0 currently, planning 5.8.5 soon). Right now we have a > > default ruleset for localhost on the central log server, and bind > > imudp and imtcp to a separate ruleset for remote hosts. I notice that > > there is no $InputRELPServerBindRuleset directive. Should I file a bug > > for this, or is there a reason why it hasn't been implemented the way > > it has for the other remote input modules? > > > > Thanks, > > Jason Antman > > Rutgers University > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From friedl at hq.adiscon.com Wed Sep 7 16:03:28 2011 From: friedl at hq.adiscon.com (Florian Riedl) Date: Wed, 7 Sep 2011 16:03:28 +0200 Subject: [rsyslog] rsyslog 4.8.0 (v4-stable) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72811C9@GRFEXC.intern.adiscon.com> There are no changes compared to 4.7.5, just a re-release with the new version number as new v4-stable. The most important new feature (for the v4-stable branch!) is Solaris support. Note: major new development to v4 is concluded and will only be done for custom projects. ChangeLog: http://www.rsyslog.com/changelog-for-4-8-0-v4-stable/ Download: http://www.rsyslog.com/rsyslog-4-8-0-v4-stable/ As always, feedback is appreciated. Best regards, Tim Eifler From jason at jasonantman.com Wed Sep 7 18:05:05 2011 From: jason at jasonantman.com (Jason Antman) Date: Wed, 07 Sep 2011 12:05:05 -0400 Subject: [rsyslog] imrelp and ruleset? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA72811C8@GRFEXC.intern.adiscon.com> References: <4E667D1C.50702@jasonantman.com> <9B6E2A8877C38245BFB15CC491A11DA72811BA@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA72811C8@GRFEXC.intern.adiscon.com> Message-ID: <4E679631.5020202@jasonantman.com> For my use at least, I can say that: 1) My need for this has dropped significantly in the past 24 hours, as my load has gone beyond the point that RELP will help (i.e. UDP is no longer the bottleneck). 2) Your suggestion would be fine; as is, I simply bind all remote inputs (currently imudp and imtcp) to a "remote" ruleset, and default everything else to a "localhost" ruleset. Thanks, Jason Rainer Gerhards wrote: > Quick update: it's not entirely trivial, as librelp is missing some plumbing > in order to shuffle the ruleset information back to the message-receiving > callback. Changing all that is beyond the time frame I currently have. I > think a compromise is that I permit to set a single replacement ruleset for > *all* relp listeners (with tcp, you can assign different ones to different > listeners). Does this sound useful enough? > > Also I'd like to note that I will enhance the current devel, only. I guess > the patch will also apply to older versions, but I will not check that (doing > new development in older versions is very time consuming for me, so I usually > do that only under paid contracts - sorry for that). > > Rainer > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Wednesday, September 07, 2011 10:02 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] imrelp and ruleset? >> >> The short answer is that this feature is not yet implemented for imrelp. >> However, I'll have a look and check if it can be sufficiently quick >> implemented. Please ping me if you have not heard back by next Monday. >> Rainer >> >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Jason Antman >>> Sent: Tuesday, September 06, 2011 10:06 PM >>> To: rsyslog-users >>> Subject: [rsyslog] imrelp and ruleset? >>> >>> Hello, >>> >>> I'm planning to finally start moving some of our client (log sending) >>> devices from stock sysklogd (UDP) to rsyslog/RELP (central server is >>> rsyslog 5.6.0 currently, planning 5.8.5 soon). Right now we have a >>> default ruleset for localhost on the central log server, and bind >>> imudp and imtcp to a separate ruleset for remote hosts. I notice that >>> there is no $InputRELPServerBindRuleset directive. Should I file a bug >>> for this, or is there a reason why it hasn't been implemented the way >>> it has for the other remote input modules? >>> >>> Thanks, >>> Jason Antman >>> Rutgers University >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > > From mark.a.reidenbach at gmail.com Wed Sep 7 18:57:16 2011 From: mark.a.reidenbach at gmail.com (Mark Reidenbach) Date: Wed, 7 Sep 2011 11:57:16 -0500 Subject: [rsyslog] Log avaya ip office phone system - how to prepend a tcp message In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA72811B7@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA72811B7@GRFEXC.intern.adiscon.com> Message-ID: I won't argue that the phone system may have issues with octect-counted framing, but the same symptom occurs when using telnet. If telnet is a valid test then the actual message text seems to be causing an issue. The debug log from the test is available here: http://www.everytruckjob.com/rsyslog.log My telnet test sent the following lines: Sep 6 15:36:19 server avaya: 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark, Sep 6 15:36:19 server avaya: 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark, Sep 6 15:36:19 server avaya: 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark, Sep 6 15:36:19 server avaya: 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark, Sep 6 15:36:19 server avaya: 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark, 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark, 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark, 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark, 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark, 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark, The first 5 lines are logged ok, but the 6th line causes the octet-counter framing error: 3309.688933361:7ffe54db5700: TCP Message with octet-counter, size 2011. 3309.688958730:7ffe54db5700: Called LogError, msg: Framing Error in received TCP message: delimiter is not SP but has ASCII value 47. The last 4 lines result in only: 3310.828859835:7ffe54db5700: poll returned with i 0, pUsr 0x7ffe4c002190 3310.828889679:7ffe54db5700: netstream 0x7ffe4c0020c0 with new data 3311.738870777:7ffe54db5700: poll returned with i 0, pUsr 0x7ffe4c002190 3311.738906355:7ffe54db5700: netstream 0x7ffe4c0020c0 with new data 3312.608796839:7ffe54db5700: poll returned with i 0, pUsr 0x7ffe4c002190 3312.608825078:7ffe54db5700: netstream 0x7ffe4c0020c0 with new data 3314.018822814:7ffe54db5700: poll returned with i 0, pUsr 0x7ffe4c002190 3314.018867595:7ffe54db5700: netstream 0x7ffe4c0020c0 with new data The log file contains the following after the test (first 5 lines sent via telnet only): Sep 7 11:34:59 192.168.13.151 Sep 6 15:36:19 server avaya: 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark,#015 Sep 7 11:35:00 192.168.13.151 Sep 6 15:36:19 server avaya: 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark,#015 Sep 7 11:35:01 192.168.13.151 Sep 6 15:36:19 server avaya: 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark,#015 Sep 7 11:35:02 192.168.13.151 Sep 6 15:36:19 server avaya: 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark,#015 Sep 7 11:35:03 192.168.13.151 Sep 6 15:36:19 server avaya: 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark,#015 On Wed, Sep 7, 2011 at 2:56 AM, Rainer Gerhards wrote: > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Wednesday, September 07, 2011 8:44 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] Log avaya ip office phone system - how to > > prepend a tcp message > > > > On Tue, 6 Sep 2011, Mark Reidenbach wrote: > > > > > I have an avaya phone system that logs call records via tcp, but the > > format > > > of the message does not include a timestampm hostname or tag. It > > sends a > > > line of csv data like the following line (the lines end in a CR/LF > > according > > > to tcpdump): > > > 2011/09/06 > > > > > 16:01:34,00:00:08,2,918176574009,I,550,2477,,0,1000237,0,E101,connie,T9 > > 001,Line > > > 1.1,0,0,,,,,,,,,,,,,mark, > > > > > > This causes debug messages like: > > > 2903.519225673:7f6457fff700: TCP Message with octet-counter, size > > 2011. > > > 2903.519236548:7f6457fff700: Called LogError, msg: Framing Error in > > received > > > TCP message: delimiter is not SP but has ASCII value 47. > > > > > > Using telnet to send the same data causes the same error, but if the > > data is > > > sent prefixed with "Sep 6 15:36:19 server avaya: " the message is > > logged as > > > hoped: > > > Sep 6 15:36:19 server avaya: 2011/09/06 > > > > > 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T900 > > 1,Line > > > 1.22,0,0,,,,,,,,,,,U,mark, > > > > > > I had hoped to use a template to add the generated timestamp, host > > and tag, > > > but it seems like the message is discarded in the input phase before > > using a > > > template to add the missing info is possible. > > > > > > Is it possible to prefix tcp messages without writing a custom input > > module > > > in C? > > > > I think this needs the custom parser module (much simpler than a custom > > imput module), but still a bit of C programming > > The main problem is that the message looks like it uses TCP octect-counted > framing, but does not do so. This would require at least changes the the > tcp > input, or a new input. A parser can not handle that part of the problem, as > it is not just a malformed message, but also a protocol violation. > > > so is that two lines, or three (line wrapping makes it hard to tell) > > If it is multiple lines, that would be even more complicated and requie a > dedicated input in any case. > > Rainer > > > > David Lang > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Sep 7 20:01:51 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 07 Sep 2011 20:01:51 +0200 Subject: [rsyslog] Log avaya ip office phone system - how to prepend atcp message Message-ID: <002601cc6d88$17c67852$100013ac@intern.adiscon.com> >From phone: If the message starts with a number, it uses octet counting framing and the number is the number of octets in the frame. Now look at your failing messages ;-) Rainer Mark Reidenbach hat geschrieben:I won't argue that the phone system may have issues with octect-counted framing, but the same symptom occurs when using telnet. If telnet is a valid test then the actual message text seems to be causing an issue. The debug log from the test is available here: http://www.everytruckjob.com/rsyslog.log My telnet test sent the following lines: Sep 6 15:36:19 server avaya: 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark, Sep 6 15:36:19 server avaya: 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark, Sep 6 15:36:19 server avaya: 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark, Sep 6 15:36:19 server avaya: 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark, Sep 6 15:36:19 server avaya: 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark, 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark, 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark, 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark, 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark, 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark, The first 5 lines are logged ok, but the 6th line causes the octet-counter framing error: 3309.688933361:7ffe54db5700: TCP Message with octet-counter, size 2011. 3309.688958730:7ffe54db5700: Called LogError, msg: Framing Error in received TCP message: delimiter is not SP but has ASCII value 47. The last 4 lines result in only: 3310.828859835:7ffe54db5700: poll returned with i 0, pUsr 0x7ffe4c002190 3310.828889679:7ffe54db5700: netstream 0x7ffe4c0020c0 with new data 3311.738870777:7ffe54db5700: poll returned with i 0, pUsr 0x7ffe4c002190 3311.738906355:7ffe54db5700: netstream 0x7ffe4c0020c0 with new data 3312.608796839:7ffe54db5700: poll returned with i 0, pUsr 0x7ffe4c002190 3312.608825078:7ffe54db5700: netstream 0x7ffe4c0020c0 with new data 3314.018822814:7ffe54db5700: poll returned with i 0, pUsr 0x7ffe4c002190 3314.018867595:7ffe54db5700: netstream 0x7ffe4c0020c0 with new data The log file contains the following after the test (first 5 lines sent via telnet only): Sep 7 11:34:59 192.168.13.151 Sep 6 15:36:19 server avaya: 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark,#015 Sep 7 11:35:00 192.168.13.151 Sep 6 15:36:19 server avaya: 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark,#015 Sep 7 11:35:01 192.168.13.151 Sep 6 15:36:19 server avaya: 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark,#015 Sep 7 11:35:02 192.168.13.151 Sep 6 15:36:19 server avaya: 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark,#015 Sep 7 11:35:03 192.168.13.151 Sep 6 15:36:19 server avaya: 2011/09/06 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line 1.22,0,0,,,,,,,,,,,U,mark,#015 On Wed, Sep 7, 2011 at 2:56 AM, Rainer Gerhards wrote: > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Wednesday, September 07, 2011 8:44 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] Log avaya ip office phone system - how to > > prepend a tcp message > > > > On Tue, 6 Sep 2011, Mark Reidenbach wrote: > > > > > I have an avaya phone system that logs call records via tcp, but the > > format > > > of the message does not include a timestampm hostname or tag. It > > sends a > > > line of csv data like the following line (the lines end in a CR/LF > > according > > > to tcpdump): > > > 2011/09/06 > > > > > 16:01:34,00:00:08,2,918176574009,I,550,2477,,0,1000237,0,E101,connie,T9 > > 001,Line > > > 1.1,0,0,,,,,,,,,,,,,mark, > > > > > > This causes debug messages like: > > > 2903.519225673:7f6457fff700: TCP Message with octet-counter, size > > 2011. > > > 2903.519236548:7f6457fff700: Called LogError, msg: Framing Error in > > received > > > TCP message: delimiter is not SP but has ASCII value 47. > > > > > > Using telnet to send the same data causes the same error, but if the > > data is > > > sent prefixed with "Sep 6 15:36:19 server avaya: " the message is > > logged as > > > hoped: > > > Sep 6 15:36:19 server avaya: 2011/09/06 > > > > > 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T900 > > 1,Line > > > 1.22,0,0,,,,,,,,,,,U,mark, > > > > > > I had hoped to use a template to add the generated timestamp, host > > and tag, > > > but it seems like the message is discarded in the input phase before > > using a > > > template to add the missing info is possible. > > > > > > Is it possible to prefix tcp messages without writing a custom input > > module > > > in C? > > > > I think this needs the custom parser module (much simpler than a custom > > imput module), but still a bit of C programming > > The main problem is that the message looks like it uses TCP octect-counted > framing, but does not do so. This would require at least changes the the > tcp > input, or a new input. A parser can not handle that part of the problem, as > it is not just a malformed message, but also a protocol violation. > > > so is that two lines, or three (line wrapping makes it hard to tell) > > If it is multiple lines, that would be even more complicated and requie a > dedicated input in any case. > > Rainer > > > > David Lang > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From ctoomey at gmail.com Wed Sep 7 21:04:34 2011 From: ctoomey at gmail.com (Chris Toomey) Date: Wed, 7 Sep 2011 12:04:34 -0700 Subject: [rsyslog] NFS log files not re-opened after rotation Message-ID: I upgraded to rsyslog 4.6.4-2ubuntu4, which is what comes w/ Ubuntu 11.04, and that fixes the problem. I'm not sure how this relates to https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/407862 , maybe my problem was fixed but other cases are still not handled. Thanks David and Rainer for your help. Chris From mark.a.reidenbach at gmail.com Wed Sep 7 21:09:28 2011 From: mark.a.reidenbach at gmail.com (Mark Reidenbach) Date: Wed, 7 Sep 2011 14:09:28 -0500 Subject: [rsyslog] Log avaya ip office phone system - how to prepend atcp message In-Reply-To: <002601cc6d88$17c67852$100013ac@intern.adiscon.com> References: <002601cc6d88$17c67852$100013ac@intern.adiscon.com> Message-ID: Your explanation of what designates octet-counting definitely clears up why the octet framing issue is occurring. Removing the octet-counting check in imptcp allows me to log the call records but obviously isn't the correct way of doing things. Is there a secret rsyslog config parameter like "$bMessagesStartWithNumber" I could specify for a port that only the phone system talks to? Would such a configuration parameter be a useful option for any other dump devices that insist on sending messages starting with a number? --- plugins/imptcp/imptcp.c 2011-08-31 05:13:51.000000000 -0500 +++ plugins/imptcp/imptcp.c.new 2011-09-07 14:05:11.000000000 -0500 @@ -541,7 +541,7 @@ DEFiRet; if(pThis->inputState == eAtStrtFram) { - if(isdigit((int) c)) { + if(!cs.bMessagesStartWithNumber && isdigit((int) c)) { pThis->inputState = eInOctetCnt; pThis->iOctetsRemain = 0; pThis->eFraming = TCP_FRAMING_OCTET_COUNTING; Might it be possible to add such a configuration option for those of us unlucky enough to own such a device? On Wed, Sep 7, 2011 at 1:01 PM, Rainer Gerhards wrote: > From phone: > > If the message starts with a number, it uses octet counting framing and the > number is the number of octets in the frame. Now look at your failing > messages ;-) > > Rainer > Mark Reidenbach hat geschrieben:I won't > argue that the phone system may have issues with octect-counted > framing, but the same symptom occurs when using telnet. If telnet is a > valid test then the actual message text seems to be causing an issue. The > debug log from the test is available here: > http://www.everytruckjob.com/rsyslog.log > > My telnet test sent the following lines: > Sep 6 15:36:19 server avaya: 2011/09/06 > > 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line > 1.22,0,0,,,,,,,,,,,U,mark, > Sep 6 15:36:19 server avaya: 2011/09/06 > > 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line > 1.22,0,0,,,,,,,,,,,U,mark, > Sep 6 15:36:19 server avaya: 2011/09/06 > > 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line > 1.22,0,0,,,,,,,,,,,U,mark, > Sep 6 15:36:19 server avaya: 2011/09/06 > > 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line > 1.22,0,0,,,,,,,,,,,U,mark, > Sep 6 15:36:19 server avaya: 2011/09/06 > > 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line > 1.22,0,0,,,,,,,,,,,U,mark, > 2011/09/06 > > 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line > 1.22,0,0,,,,,,,,,,,U,mark, > 2011/09/06 > > 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line > 1.22,0,0,,,,,,,,,,,U,mark, > 2011/09/06 > > 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line > 1.22,0,0,,,,,,,,,,,U,mark, > 2011/09/06 > > 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line > 1.22,0,0,,,,,,,,,,,U,mark, > 2011/09/06 > > 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line > 1.22,0,0,,,,,,,,,,,U,mark, > > The first 5 lines are logged ok, but the 6th line causes the octet-counter > framing error: > 3309.688933361:7ffe54db5700: TCP Message with octet-counter, size > 2011. > 3309.688958730:7ffe54db5700: Called LogError, msg: Framing Error in > received > TCP message: delimiter is not SP but has ASCII value 47. > > The last 4 lines result in only: > 3310.828859835:7ffe54db5700: poll returned with i 0, pUsr 0x7ffe4c002190 > 3310.828889679:7ffe54db5700: netstream 0x7ffe4c0020c0 with new data > 3311.738870777:7ffe54db5700: poll returned with i 0, pUsr 0x7ffe4c002190 > 3311.738906355:7ffe54db5700: netstream 0x7ffe4c0020c0 with new data > 3312.608796839:7ffe54db5700: poll returned with i 0, pUsr 0x7ffe4c002190 > 3312.608825078:7ffe54db5700: netstream 0x7ffe4c0020c0 with new data > 3314.018822814:7ffe54db5700: poll returned with i 0, pUsr 0x7ffe4c002190 > 3314.018867595:7ffe54db5700: netstream 0x7ffe4c0020c0 with new data > > > The log file contains the following after the test (first 5 lines sent via > telnet only): > Sep 7 11:34:59 192.168.13.151 Sep 6 15:36:19 server avaya: 2011/09/06 > > 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line > 1.22,0,0,,,,,,,,,,,U,mark,#015 > Sep 7 11:35:00 192.168.13.151 Sep 6 15:36:19 server avaya: 2011/09/06 > > 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line > 1.22,0,0,,,,,,,,,,,U,mark,#015 > Sep 7 11:35:01 192.168.13.151 Sep 6 15:36:19 server avaya: 2011/09/06 > > 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line > 1.22,0,0,,,,,,,,,,,U,mark,#015 > Sep 7 11:35:02 192.168.13.151 Sep 6 15:36:19 server avaya: 2011/09/06 > > 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line > 1.22,0,0,,,,,,,,,,,U,mark,#015 > Sep 7 11:35:03 192.168.13.151 Sep 6 15:36:19 server avaya: 2011/09/06 > > 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T9001,Line > 1.22,0,0,,,,,,,,,,,U,mark,#015 > > > > On Wed, Sep 7, 2011 at 2:56 AM, Rainer Gerhards >wrote: > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > > Sent: Wednesday, September 07, 2011 8:44 AM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] Log avaya ip office phone system - how to > > > prepend a tcp message > > > > > > On Tue, 6 Sep 2011, Mark Reidenbach wrote: > > > > > > > I have an avaya phone system that logs call records via tcp, but the > > > format > > > > of the message does not include a timestampm hostname or tag. It > > > sends a > > > > line of csv data like the following line (the lines end in a CR/LF > > > according > > > > to tcpdump): > > > > 2011/09/06 > > > > > > > 16:01:34,00:00:08,2,918176574009,I,550,2477,,0,1000237,0,E101,connie,T9 > > > 001,Line > > > > 1.1,0,0,,,,,,,,,,,,,mark, > > > > > > > > This causes debug messages like: > > > > 2903.519225673:7f6457fff700: TCP Message with octet-counter, size > > > 2011. > > > > 2903.519236548:7f6457fff700: Called LogError, msg: Framing Error in > > > received > > > > TCP message: delimiter is not SP but has ASCII value 47. > > > > > > > > Using telnet to send the same data causes the same error, but if the > > > data is > > > > sent prefixed with "Sep 6 15:36:19 server avaya: " the message is > > > logged as > > > > hoped: > > > > Sep 6 15:36:19 server avaya: 2011/09/06 > > > > > > > 14:53:04,00:00:07,0,107,O,92422131,92422131,,0,1000191,0,E107,mark,T900 > > > 1,Line > > > > 1.22,0,0,,,,,,,,,,,U,mark, > > > > > > > > I had hoped to use a template to add the generated timestamp, host > > > and tag, > > > > but it seems like the message is discarded in the input phase before > > > using a > > > > template to add the missing info is possible. > > > > > > > > Is it possible to prefix tcp messages without writing a custom input > > > module > > > > in C? > > > > > > I think this needs the custom parser module (much simpler than a custom > > > imput module), but still a bit of C programming > > > > The main problem is that the message looks like it uses TCP > octect-counted > > framing, but does not do so. This would require at least changes the the > > tcp > > input, or a new input. A parser can not handle that part of the problem, > as > > it is not just a malformed message, but also a protocol violation. > > > > > so is that two lines, or three (line wrapping makes it hard to tell) > > > > If it is multiple lines, that would be even more complicated and requie a > > dedicated input in any case. > > > > Rainer > > > > > > David Lang > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From andreas+rsyslog at majestyk.de Thu Sep 8 12:10:36 2011 From: andreas+rsyslog at majestyk.de (Andreas Grosse) Date: Thu, 8 Sep 2011 12:10:36 +0200 Subject: [rsyslog] netstream errors when logging with a high message rate Message-ID: <20110908101036.GD16242@majestyk.de> Hi, I am running rsyslog 5.8.3 as a central log collector which then sends the log messages to an archive using tls encryption. When the log message rate increases, I start seeing log messages like this: rsyslog: netstream session 0x8b05ef0 will be closed due to error [try http://www.rsyslog.com/e/2165 ] The logging does not stop and the tcp connection to the remote archive does not break, though - it just starts spewing out these messages up to two times per minute. I enabled additional debugging, and the logfile contained this: 9275.065615635:40976b70: netstream 0x41000c50 with new data 9275.065635173:40976b70: error during recv on NSD 0x41000b88: Connection reset by peer 9275.065641798:40976b70: gtlsRcv return. nsd 0x413fff98, iRet -2165, lenRcvBuf 0, ptrRcvBuf 0 9275.065648064:40976b70: Called LogError, msg: netstream session 0x41000c50 will be closed due to error 9275.065675043:40976b70: main Q: entry added, size now log 6919, phys 6951 entries 9275.065682225:40976b70: main Q: EnqueueMsg advised worker start 9275.065707944:40976b70: -------- calling select, active fds (max 19): 14 15 16 19 On the receiving end I get no notification of an error happening at all. Following is the configuration the I use: $MaxMessageSize 64k $RepeatedMsgReduction off $EscapeControlCharactersOnReceive off $WorkDirectory /var/rsyslog # default location for work (spool) files $ModLoad imtcp $ModLoad imudp $ModLoad imptcp $ModLoad omuxsock $ModLoad impstats $InputPTCPServerListenIP 127.0.0.1 $InputPTCPServerRun 10100 $PStatsInterval 300 # log local syslog messages back to syslog-ng $OMUxSockSocket /dev/tosyslog if $programname startswith 'rsyslog' then :omuxsock: if $programname startswith 'rsyslog' then ~ $ActionQueueType LinkedList $ActionQueueFileName srvrfwd $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on $ActionQueueMaxDiskSpace 819200 $DefaultNetstreamDriver gtls $DefaultNetstreamDriverCAFile /etc/ca/cacert.pem $DefaultNetstreamDriverCertFile /etc/client.pem $DefaultNetstreamDriverKeyFile /etc/client.key $ActionSendStreamDriverMode 1 $ActionSendStreamDriverAuthMode x509/certvalid $InputTCPServerStreamDriverMode 0 $InputTCPServerRun 10101 $UDPServerRun 10101 *.* @@(o,z0)loghost:5077;RSYSLOG_SyslogProtocol23Format If you need more data from the debug log, just ask. This is also easy to reproduce, therefore I am able to try some things if you come up with suggestions what happens there and how to get rid of those error messages. Thank you for your help! Best regards, Andreas Grosse From jpoling at moody.edu Thu Sep 8 12:48:14 2011 From: jpoling at moody.edu (Jeff Poling) Date: Thu, 8 Sep 2011 05:48:14 -0500 Subject: [rsyslog] New User Help In-Reply-To: References: <9599A350A0A5884DB4E50D83F9287D0F01F305D0E8@exchmbx01.moody.edu> <9599A350A0A5884DB4E50D83F9287D0F01F305D233@exchmbx01.moody.edu> Message-ID: <9599A350A0A5884DB4E50D83F9287D0F01F305D4B2@exchmbx01.moody.edu> > that will work if you compile it manually on each server, but if you create a > .rpm, then you do something like this on the build server and then deply the > .rpm on each of your other servers. [] Thanks, David. I ran through the compile and install process on a test system, but found that our current install of rsyslog is not in the default location. Ours is in /opt/rsyslog. How do I change the default install location? Is that part of the configure script? Thanks, Jeff From rgerhards at hq.adiscon.com Thu Sep 8 13:10:18 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 8 Sep 2011 13:10:18 +0200 Subject: [rsyslog] New User Help In-Reply-To: <9599A350A0A5884DB4E50D83F9287D0F01F305D4B2@exchmbx01.moody.edu> References: <9599A350A0A5884DB4E50D83F9287D0F01F305D0E8@exchmbx01.moody.edu><9599A350A0A5884DB4E50D83F9287D0F01F305D233@exchmbx01.moody.edu> <9599A350A0A5884DB4E50D83F9287D0F01F305D4B2@exchmbx01.moody.edu> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72811D8@GRFEXC.intern.adiscon.com> For all params, use ./configure --help I think what you need is --bindir=... HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Jeff Poling > Sent: Thursday, September 08, 2011 12:48 PM > To: rsyslog-users > Subject: Re: [rsyslog] New User Help > > > that will work if you compile it manually on each server, but if you > create a > > .rpm, then you do something like this on the build server and then > deply the > > .rpm on each of your other servers. > > [] Thanks, David. I ran through the compile and install process on a > test system, but found that our current install of rsyslog is not in > the default location. Ours is in /opt/rsyslog. How do I change the > default install location? Is that part of the configure script? > > Thanks, > > Jeff > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Sep 8 13:11:40 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 8 Sep 2011 13:11:40 +0200 Subject: [rsyslog] netstream errors when logging with a high message rate In-Reply-To: <20110908101036.GD16242@majestyk.de> References: <20110908101036.GD16242@majestyk.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72811D9@GRFEXC.intern.adiscon.com> You should update to the latest stable version and see if the problem persists. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Andreas Grosse > Sent: Thursday, September 08, 2011 12:11 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] netstream errors when logging with a high message > rate > > Hi, > I am running rsyslog 5.8.3 as a central log collector which then sends > the log messages to an archive using tls encryption. When the log > message rate increases, I start seeing log messages like this: > > rsyslog: netstream session 0x8b05ef0 will be closed due to error [try > http://www.rsyslog.com/e/2165 ] > > The logging does not stop and the tcp connection to the remote archive > does not break, though - it just starts spewing out these messages up > to > two times per minute. > I enabled additional debugging, and the logfile contained this: > > 9275.065615635:40976b70: netstream 0x41000c50 with new data > 9275.065635173:40976b70: error during recv on NSD 0x41000b88: > Connection reset by peer > 9275.065641798:40976b70: gtlsRcv return. nsd 0x413fff98, iRet -2165, > lenRcvBuf 0, ptrRcvBuf 0 > 9275.065648064:40976b70: Called LogError, msg: netstream session > 0x41000c50 will be closed due to error > > 9275.065675043:40976b70: main Q: entry added, size now log 6919, phys > 6951 entries > 9275.065682225:40976b70: main Q: EnqueueMsg advised worker start > 9275.065707944:40976b70: -------- calling select, active > fds (max 19): 14 15 16 19 > > On the receiving end I get no notification of an error happening at > all. Following is the configuration the I use: > > $MaxMessageSize 64k > $RepeatedMsgReduction off > $EscapeControlCharactersOnReceive off > $WorkDirectory /var/rsyslog # default location for work (spool) files > > $ModLoad imtcp > $ModLoad imudp > $ModLoad imptcp > $ModLoad omuxsock > $ModLoad impstats > > $InputPTCPServerListenIP 127.0.0.1 > $InputPTCPServerRun 10100 > > $PStatsInterval 300 > > # log local syslog messages back to syslog-ng > $OMUxSockSocket /dev/tosyslog > if $programname startswith 'rsyslog' then :omuxsock: > if $programname startswith 'rsyslog' then ~ > > $ActionQueueType LinkedList > $ActionQueueFileName srvrfwd > $ActionResumeRetryCount -1 > $ActionQueueSaveOnShutdown on > $ActionQueueMaxDiskSpace 819200 > > $DefaultNetstreamDriver gtls > > $DefaultNetstreamDriverCAFile /etc/ca/cacert.pem > $DefaultNetstreamDriverCertFile /etc/client.pem > $DefaultNetstreamDriverKeyFile /etc/client.key > > $ActionSendStreamDriverMode 1 > $ActionSendStreamDriverAuthMode x509/certvalid > > $InputTCPServerStreamDriverMode 0 > $InputTCPServerRun 10101 > $UDPServerRun 10101 > > *.* @@(o,z0)loghost:5077;RSYSLOG_SyslogProtocol23Format > > > If you need more data from the debug log, just ask. This is also easy > to > reproduce, therefore I am able to try some things if you come up with > suggestions what happens there and how to get rid of those error > messages. Thank you for your help! > > Best regards, > Andreas Grosse > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From andreas+rsyslog at majestyk.de Thu Sep 8 14:32:29 2011 From: andreas+rsyslog at majestyk.de (Andreas Grosse) Date: Thu, 8 Sep 2011 14:32:29 +0200 Subject: [rsyslog] netstream errors when logging with a high message rate In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA72811D9@GRFEXC.intern.adiscon.com> References: <20110908101036.GD16242@majestyk.de> <9B6E2A8877C38245BFB15CC491A11DA72811D9@GRFEXC.intern.adiscon.com> Message-ID: <20110908123229.GE16242@majestyk.de> Hi Rainer, thanks for the quick response. I updated to the latest version 5.8.5 and still have the same error messages. Two examples for this happening can be seen here: Error message on console/logfile: rsyslogd: netstream session 0x4100cab0 will be closed due to error Debug output of the thread managing this netstream when the error occured: 4442.720793561:40976b70: netstream 0x4100cab0 with new data 4442.720833000:40976b70: error during recv on NSD 0x413d1ae8: Connection reset by peer 4442.720840716:40976b70: gtlsRcv return. nsd 0x41079868, iRet -2165, lenRcvBuf 0, ptrRcvBuf 0 4442.720848795:40976b70: Called LogError, msg: netstream session 0x4100cab0 will be closed due to error 4442.720863318:40976b70: MsgSetTAG in: len 14, pszBuf: rsyslogd-2165: 4442.720869899:40976b70: MsgSetTAG exit: pMsg->iLenTAG 14, pMsg->TAG.szBuf: rsyslogd-2165: 4442.720884625:40976b70: main Q: entry added, size now log 6340, phys 6372 entries 4442.720893681:40976b70: main Q: EnqueueMsg advised worker start 4442.720920645:40976b70: -------- calling select, active fds (max 19): 14 15 19 Second occurence, different netstream: rsyslogd: netstream session 0x8640fc0 will be closed due to error 4438.033367497:40976b70: main Q: entry added, size now log 6809, phys 6841 entries 4438.033388122:40976b70: main Q: entry added, size now log 6810, phys 6842 entries 4438.033396264:40976b70: main Q: entry added, size now log 6811, phys 6843 entries 4438.033404087:40976b70: main Q: entry added, size now log 6812, phys 6844 entries 4438.033412176:40976b70: main Q: entry added, size now log 6813, phys 6845 entries 4438.033420108:40976b70: main Q: entry added, size now log 6814, phys 6846 entries 4438.033427975:40976b70: main Q: entry added, size now log 6815, phys 6847 entries 4438.033436020:40976b70: main Q: entry added, size now log 6816, phys 6848 entries 4438.033443886:40976b70: main Q: entry added, size now log 6817, phys 6849 entries 4438.033451823:40976b70: main Q: entry added, size now log 6818, phys 6850 entries 4438.033459819:40976b70: main Q: entry added, size now log 6819, phys 6851 entries 4438.033467757:40976b70: main Q: entry added, size now log 6820, phys 6852 entries 4438.033475728:40976b70: main Q: entry added, size now log 6821, phys 6853 entries 4438.033483613:40976b70: main Q: entry added, size now log 6822, phys 6854 entries 4438.033491751:40976b70: main Q: entry added, size now log 6823, phys 6855 entries 4438.033499760:40976b70: main Q: entry added, size now log 6824, phys 6856 entries 4438.033507589:40976b70: main Q: entry added, size now log 6825, phys 6857 entries 4438.033515679:40976b70: main Q: entry added, size now log 6826, phys 6858 entries 4438.033523711:40976b70: main Q: entry added, size now log 6827, phys 6859 entries 4438.033531526:40976b70: main Q: entry added, size now log 6828, phys 6860 entries 4438.033539457:40976b70: main Q: entry added, size now log 6829, phys 6861 entries 4438.033547373:40976b70: main Q: entry added, size now log 6830, phys 6862 entries 4438.033555150:40976b70: main Q: entry added, size now log 6831, phys 6863 entries 4438.033563201:40976b70: main Q: entry added, size now log 6832, phys 6864 entries 4438.033571121:40976b70: main Q: entry added, size now log 6833, phys 6865 entries 4438.033578976:40976b70: main Q: entry added, size now log 6834, phys 6866 entries 4438.033586987:40976b70: main Q: entry added, size now log 6835, phys 6867 entries 4438.033594920:40976b70: main Q: entry added, size now log 6836, phys 6868 entries 4438.033602860:40976b70: main Q: entry added, size now log 6837, phys 6869 entries 4438.033610750:40976b70: main Q: entry added, size now log 6838, phys 6870 entries 4438.033618581:40976b70: main Q: entry added, size now log 6839, phys 6871 entries 4438.033626573:40976b70: main Q: entry added, size now log 6840, phys 6872 entries 4438.033635275:40976b70: main Q: entry added, size now log 6841, phys 6873 entries 4438.033644155:40976b70: main Q: entry added, size now log 6842, phys 6874 entries 4438.033652187:40976b70: main Q: entry added, size now log 6843, phys 6875 entries 4438.033660012:40976b70: main Q: entry added, size now log 6844, phys 6876 entries 4438.033667977:40976b70: main Q: entry added, size now log 6845, phys 6877 entries 4438.033675943:40976b70: main Q: entry added, size now log 6846, phys 6878 entries 4438.033683820:40976b70: main Q: entry added, size now log 6847, phys 6879 entries 4438.033691877:40976b70: main Q: entry added, size now log 6848, phys 6880 entries 4438.033699825:40976b70: main Q: entry added, size now log 6849, phys 6881 entries 4438.033707743:40976b70: main Q: entry added, size now log 6850, phys 6882 entries 4438.033716401:40976b70: main Q: MultiEnqObj advised worker start 4438.033735637:40976b70: -------- calling select, active fds (max 19): 14 15 16 18 19 4438.033772666:40976b70: netstream 0x8640fc0 with new data 4438.033792922:40976b70: error during recv on NSD 0x8612ca0: Connection reset by peer 4438.033800723:40976b70: gtlsRcv return. nsd 0x863fd58, iRet -2165, lenRcvBuf 0, ptrRcvBuf 0 4438.033808563:40976b70: Called LogError, msg: netstream session 0x8640fc0 will be closed due to error 4438.033821043:40976b70: MsgSetTAG in: len 14, pszBuf: rsyslogd-2165: 4438.033827708:40976b70: MsgSetTAG exit: pMsg->iLenTAG 14, pMsg->TAG.szBuf: rsyslogd-2165: 4438.033850214:40976b70: main Q: entry added, size now log 6851, phys 6883 entries 4438.033859170:40976b70: main Q: EnqueueMsg advised worker start 4438.033881307:40976b70: -------- calling select, active fds (max 19): 14 15 18 19 Because of the gtlsRcv I assume this happens on the outgoing connection since this is the only encrypted communication channel, but I am not sure even this assumption is correct. If you have anything you want me to provide or try, I'd be happy to do so. Many thanks in advance! Best regards, Andreas Rainer Gerhards [08:09:11 13:11] wrote: > You should update to the latest stable version and see if the problem > persists. > > Rainer > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Andreas Grosse > > Sent: Thursday, September 08, 2011 12:11 PM > > To: rsyslog at lists.adiscon.com > > Subject: [rsyslog] netstream errors when logging with a high message > > rate > > > > Hi, > > I am running rsyslog 5.8.3 as a central log collector which then sends > > the log messages to an archive using tls encryption. When the log > > message rate increases, I start seeing log messages like this: > > > > rsyslog: netstream session 0x8b05ef0 will be closed due to error [try > > http://www.rsyslog.com/e/2165 ] > > > > The logging does not stop and the tcp connection to the remote archive > > does not break, though - it just starts spewing out these messages up > > to > > two times per minute. > > I enabled additional debugging, and the logfile contained this: > > > > 9275.065615635:40976b70: netstream 0x41000c50 with new data > > 9275.065635173:40976b70: error during recv on NSD 0x41000b88: > > Connection reset by peer > > 9275.065641798:40976b70: gtlsRcv return. nsd 0x413fff98, iRet -2165, > > lenRcvBuf 0, ptrRcvBuf 0 > > 9275.065648064:40976b70: Called LogError, msg: netstream session > > 0x41000c50 will be closed due to error > > > > 9275.065675043:40976b70: main Q: entry added, size now log 6919, phys > > 6951 entries > > 9275.065682225:40976b70: main Q: EnqueueMsg advised worker start > > 9275.065707944:40976b70: -------- calling select, active > > fds (max 19): 14 15 16 19 > > > > On the receiving end I get no notification of an error happening at > > all. Following is the configuration the I use: > > > > $MaxMessageSize 64k > > $RepeatedMsgReduction off > > $EscapeControlCharactersOnReceive off > > $WorkDirectory /var/rsyslog # default location for work (spool) files > > > > $ModLoad imtcp > > $ModLoad imudp > > $ModLoad imptcp > > $ModLoad omuxsock > > $ModLoad impstats > > > > $InputPTCPServerListenIP 127.0.0.1 > > $InputPTCPServerRun 10100 > > > > $PStatsInterval 300 > > > > # log local syslog messages back to syslog-ng > > $OMUxSockSocket /dev/tosyslog > > if $programname startswith 'rsyslog' then :omuxsock: > > if $programname startswith 'rsyslog' then ~ > > > > $ActionQueueType LinkedList > > $ActionQueueFileName srvrfwd > > $ActionResumeRetryCount -1 > > $ActionQueueSaveOnShutdown on > > $ActionQueueMaxDiskSpace 819200 > > > > $DefaultNetstreamDriver gtls > > > > $DefaultNetstreamDriverCAFile /etc/ca/cacert.pem > > $DefaultNetstreamDriverCertFile /etc/client.pem > > $DefaultNetstreamDriverKeyFile /etc/client.key > > > > $ActionSendStreamDriverMode 1 > > $ActionSendStreamDriverAuthMode x509/certvalid > > > > $InputTCPServerStreamDriverMode 0 > > $InputTCPServerRun 10101 > > $UDPServerRun 10101 > > > > *.* @@(o,z0)loghost:5077;RSYSLOG_SyslogProtocol23Format > > > > > > If you need more data from the debug log, just ask. This is also easy > > to > > reproduce, therefore I am able to try some things if you come up with > > suggestions what happens there and how to get rid of those error > > messages. Thank you for your help! > > > > Best regards, > > Andreas Grosse > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Thu Sep 8 14:34:09 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 8 Sep 2011 14:34:09 +0200 Subject: [rsyslog] netstream errors when logging with a high message rate In-Reply-To: <20110908123229.GE16242@majestyk.de> References: <20110908101036.GD16242@majestyk.de><9B6E2A8877C38245BFB15CC491A11DA72811D9@GRFEXC.intern.adiscon.com> <20110908123229.GE16242@majestyk.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72811DA@GRFEXC.intern.adiscon.com> Ok, at least we know we have a new issue. Can you please provide me the complete debug log. You can mail it to me privately (do NOT mail to the list except if you want to broadcast the contents to the world ;)). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Andreas Grosse > Sent: Thursday, September 08, 2011 2:32 PM > To: rsyslog-users > Subject: Re: [rsyslog] netstream errors when logging with a high message > rate > > Hi Rainer, > thanks for the quick response. I updated to the latest version 5.8.5 and still > have the same error messages. Two examples for this happening can be > seen here: > > Error message on console/logfile: > rsyslogd: netstream session 0x4100cab0 will be closed due to error > > Debug output of the thread managing this netstream when the error > occured: > 4442.720793561:40976b70: netstream 0x4100cab0 with new data > 4442.720833000:40976b70: error during recv on NSD 0x413d1ae8: Connection > reset by peer > 4442.720840716:40976b70: gtlsRcv return. nsd 0x41079868, iRet -2165, > lenRcvBuf 0, ptrRcvBuf 0 > 4442.720848795:40976b70: Called LogError, msg: netstream session > 0x4100cab0 will be closed due to error > > 4442.720863318:40976b70: MsgSetTAG in: len 14, pszBuf: rsyslogd-2165: > 4442.720869899:40976b70: MsgSetTAG exit: pMsg->iLenTAG 14, pMsg- > >TAG.szBuf: rsyslogd-2165: > 4442.720884625:40976b70: main Q: entry added, size now log 6340, phys 6372 > entries > 4442.720893681:40976b70: main Q: EnqueueMsg advised worker start > 4442.720920645:40976b70: -------- calling select, active fds > (max 19): 14 15 19 > > Second occurence, different netstream: > rsyslogd: netstream session 0x8640fc0 will be closed due to error > > 4438.033367497:40976b70: main Q: entry added, size now log 6809, phys 6841 > entries > 4438.033388122:40976b70: main Q: entry added, size now log 6810, phys 6842 > entries > 4438.033396264:40976b70: main Q: entry added, size now log 6811, phys 6843 > entries > 4438.033404087:40976b70: main Q: entry added, size now log 6812, phys 6844 > entries > 4438.033412176:40976b70: main Q: entry added, size now log 6813, phys 6845 > entries > 4438.033420108:40976b70: main Q: entry added, size now log 6814, phys 6846 > entries > 4438.033427975:40976b70: main Q: entry added, size now log 6815, phys 6847 > entries > 4438.033436020:40976b70: main Q: entry added, size now log 6816, phys 6848 > entries > 4438.033443886:40976b70: main Q: entry added, size now log 6817, phys 6849 > entries > 4438.033451823:40976b70: main Q: entry added, size now log 6818, phys 6850 > entries > 4438.033459819:40976b70: main Q: entry added, size now log 6819, phys 6851 > entries > 4438.033467757:40976b70: main Q: entry added, size now log 6820, phys 6852 > entries > 4438.033475728:40976b70: main Q: entry added, size now log 6821, phys 6853 > entries > 4438.033483613:40976b70: main Q: entry added, size now log 6822, phys 6854 > entries > 4438.033491751:40976b70: main Q: entry added, size now log 6823, phys 6855 > entries > 4438.033499760:40976b70: main Q: entry added, size now log 6824, phys 6856 > entries > 4438.033507589:40976b70: main Q: entry added, size now log 6825, phys 6857 > entries > 4438.033515679:40976b70: main Q: entry added, size now log 6826, phys 6858 > entries > 4438.033523711:40976b70: main Q: entry added, size now log 6827, phys 6859 > entries > 4438.033531526:40976b70: main Q: entry added, size now log 6828, phys 6860 > entries > 4438.033539457:40976b70: main Q: entry added, size now log 6829, phys 6861 > entries > 4438.033547373:40976b70: main Q: entry added, size now log 6830, phys 6862 > entries > 4438.033555150:40976b70: main Q: entry added, size now log 6831, phys 6863 > entries > 4438.033563201:40976b70: main Q: entry added, size now log 6832, phys 6864 > entries > 4438.033571121:40976b70: main Q: entry added, size now log 6833, phys 6865 > entries > 4438.033578976:40976b70: main Q: entry added, size now log 6834, phys 6866 > entries > 4438.033586987:40976b70: main Q: entry added, size now log 6835, phys 6867 > entries > 4438.033594920:40976b70: main Q: entry added, size now log 6836, phys 6868 > entries > 4438.033602860:40976b70: main Q: entry added, size now log 6837, phys 6869 > entries > 4438.033610750:40976b70: main Q: entry added, size now log 6838, phys 6870 > entries > 4438.033618581:40976b70: main Q: entry added, size now log 6839, phys 6871 > entries > 4438.033626573:40976b70: main Q: entry added, size now log 6840, phys 6872 > entries > 4438.033635275:40976b70: main Q: entry added, size now log 6841, phys 6873 > entries > 4438.033644155:40976b70: main Q: entry added, size now log 6842, phys 6874 > entries > 4438.033652187:40976b70: main Q: entry added, size now log 6843, phys 6875 > entries > 4438.033660012:40976b70: main Q: entry added, size now log 6844, phys 6876 > entries > 4438.033667977:40976b70: main Q: entry added, size now log 6845, phys 6877 > entries > 4438.033675943:40976b70: main Q: entry added, size now log 6846, phys 6878 > entries > 4438.033683820:40976b70: main Q: entry added, size now log 6847, phys 6879 > entries > 4438.033691877:40976b70: main Q: entry added, size now log 6848, phys 6880 > entries > 4438.033699825:40976b70: main Q: entry added, size now log 6849, phys 6881 > entries > 4438.033707743:40976b70: main Q: entry added, size now log 6850, phys 6882 > entries > 4438.033716401:40976b70: main Q: MultiEnqObj advised worker start > 4438.033735637:40976b70: -------- calling select, active fds > (max 19): 14 15 16 18 19 > 4438.033772666:40976b70: netstream 0x8640fc0 with new data > 4438.033792922:40976b70: error during recv on NSD 0x8612ca0: Connection > reset by peer > 4438.033800723:40976b70: gtlsRcv return. nsd 0x863fd58, iRet -2165, > lenRcvBuf 0, ptrRcvBuf 0 > 4438.033808563:40976b70: Called LogError, msg: netstream session 0x8640fc0 > will be closed due to error > > 4438.033821043:40976b70: MsgSetTAG in: len 14, pszBuf: rsyslogd-2165: > 4438.033827708:40976b70: MsgSetTAG exit: pMsg->iLenTAG 14, pMsg- > >TAG.szBuf: rsyslogd-2165: > 4438.033850214:40976b70: main Q: entry added, size now log 6851, phys 6883 > entries > 4438.033859170:40976b70: main Q: EnqueueMsg advised worker start > 4438.033881307:40976b70: -------- calling select, active fds > (max 19): 14 15 18 19 > > Because of the gtlsRcv I assume this happens on the outgoing connection > since this is the only encrypted communication channel, but I am not sure > even this assumption is correct. > If you have anything you want me to provide or try, I'd be happy to do so. > Many thanks in advance! > Best regards, > Andreas > > > Rainer Gerhards [08:09:11 13:11] wrote: > > You should update to the latest stable version and see if the problem > > persists. > > > > Rainer > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Andreas Grosse > > > Sent: Thursday, September 08, 2011 12:11 PM > > > To: rsyslog at lists.adiscon.com > > > Subject: [rsyslog] netstream errors when logging with a high message > > > rate > > > > > > Hi, > > > I am running rsyslog 5.8.3 as a central log collector which then > > > sends the log messages to an archive using tls encryption. When the > > > log message rate increases, I start seeing log messages like this: > > > > > > rsyslog: netstream session 0x8b05ef0 will be closed due to error > > > [try > > > http://www.rsyslog.com/e/2165 ] > > > > > > The logging does not stop and the tcp connection to the remote > > > archive does not break, though - it just starts spewing out these > > > messages up to two times per minute. > > > I enabled additional debugging, and the logfile contained this: > > > > > > 9275.065615635:40976b70: netstream 0x41000c50 with new data > > > 9275.065635173:40976b70: error during recv on NSD 0x41000b88: > > > Connection reset by peer > > > 9275.065641798:40976b70: gtlsRcv return. nsd 0x413fff98, iRet -2165, > > > lenRcvBuf 0, ptrRcvBuf 0 > > > 9275.065648064:40976b70: Called LogError, msg: netstream session > > > 0x41000c50 will be closed due to error > > > > > > 9275.065675043:40976b70: main Q: entry added, size now log 6919, > > > phys > > > 6951 entries > > > 9275.065682225:40976b70: main Q: EnqueueMsg advised worker start > > > 9275.065707944:40976b70: -------- calling select, > > > active fds (max 19): 14 15 16 19 > > > > > > On the receiving end I get no notification of an error happening at > > > all. Following is the configuration the I use: > > > > > > $MaxMessageSize 64k > > > $RepeatedMsgReduction off > > > $EscapeControlCharactersOnReceive off > > > $WorkDirectory /var/rsyslog # default location for work (spool) files > > > > > > $ModLoad imtcp > > > $ModLoad imudp > > > $ModLoad imptcp > > > $ModLoad omuxsock > > > $ModLoad impstats > > > > > > $InputPTCPServerListenIP 127.0.0.1 > > > $InputPTCPServerRun 10100 > > > > > > $PStatsInterval 300 > > > > > > # log local syslog messages back to syslog-ng $OMUxSockSocket > > > /dev/tosyslog if $programname startswith 'rsyslog' then :omuxsock: > > > if $programname startswith 'rsyslog' then ~ > > > > > > $ActionQueueType LinkedList > > > $ActionQueueFileName srvrfwd > > > $ActionResumeRetryCount -1 > > > $ActionQueueSaveOnShutdown on > > > $ActionQueueMaxDiskSpace 819200 > > > > > > $DefaultNetstreamDriver gtls > > > > > > $DefaultNetstreamDriverCAFile /etc/ca/cacert.pem > > > $DefaultNetstreamDriverCertFile /etc/client.pem > > > $DefaultNetstreamDriverKeyFile /etc/client.key > > > > > > $ActionSendStreamDriverMode 1 > > > $ActionSendStreamDriverAuthMode x509/certvalid > > > > > > $InputTCPServerStreamDriverMode 0 > > > $InputTCPServerRun 10101 > > > $UDPServerRun 10101 > > > > > > *.* @@(o,z0)loghost:5077;RSYSLOG_SyslogProtocol23Format > > > > > > > > > If you need more data from the debug log, just ask. This is also > > > easy to reproduce, therefore I am able to try some things if you > > > come up with suggestions what happens there and how to get rid of > > > those error messages. Thank you for your help! > > > > > > Best regards, > > > Andreas Grosse > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Sep 8 14:35:24 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 8 Sep 2011 14:35:24 +0200 Subject: [rsyslog] netstream errors when logging with a high message rate In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA72811DA@GRFEXC.intern.adiscon.com> References: <20110908101036.GD16242@majestyk.de><9B6E2A8877C38245BFB15CC491A11DA72811D9@GRFEXC.intern.adiscon.com><20110908123229.GE16242@majestyk.de> <9B6E2A8877C38245BFB15CC491A11DA72811DA@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72811DB@GRFEXC.intern.adiscon.com> Maybe I do not need the complete log, because... > > Debug output of the thread managing this netstream when the error > > occured: > > 4442.720793561:40976b70: netstream 0x4100cab0 with new data > > 4442.720833000:40976b70: error during recv on NSD 0x413d1ae8: > > Connection reset by peer It looks like the remote peer has closed the session. Any idea why? Rainer > > 4442.720840716:40976b70: gtlsRcv return. nsd 0x41079868, iRet -2165, > > lenRcvBuf 0, ptrRcvBuf 0 > > 4442.720848795:40976b70: Called LogError, msg: netstream session > > 0x4100cab0 will be closed due to error > > > > 4442.720863318:40976b70: MsgSetTAG in: len 14, pszBuf: rsyslogd-2165: > > 4442.720869899:40976b70: MsgSetTAG exit: pMsg->iLenTAG 14, pMsg- > > >TAG.szBuf: rsyslogd-2165: > > 4442.720884625:40976b70: main Q: entry added, size now log 6340, phys > > 6372 entries > > 4442.720893681:40976b70: main Q: EnqueueMsg advised worker start > > 4442.720920645:40976b70: -------- calling select, active > > fds (max 19): 14 15 19 > > > > Second occurence, different netstream: > > rsyslogd: netstream session 0x8640fc0 will be closed due to error > > > > 4438.033367497:40976b70: main Q: entry added, size now log 6809, phys > > 6841 entries > > 4438.033388122:40976b70: main Q: entry added, size now log 6810, phys > > 6842 entries > > 4438.033396264:40976b70: main Q: entry added, size now log 6811, phys > > 6843 entries > > 4438.033404087:40976b70: main Q: entry added, size now log 6812, phys > > 6844 entries > > 4438.033412176:40976b70: main Q: entry added, size now log 6813, phys > > 6845 entries > > 4438.033420108:40976b70: main Q: entry added, size now log 6814, phys > > 6846 entries > > 4438.033427975:40976b70: main Q: entry added, size now log 6815, phys > > 6847 entries > > 4438.033436020:40976b70: main Q: entry added, size now log 6816, phys > > 6848 entries > > 4438.033443886:40976b70: main Q: entry added, size now log 6817, phys > > 6849 entries > > 4438.033451823:40976b70: main Q: entry added, size now log 6818, phys > > 6850 entries > > 4438.033459819:40976b70: main Q: entry added, size now log 6819, phys > > 6851 entries > > 4438.033467757:40976b70: main Q: entry added, size now log 6820, phys > > 6852 entries > > 4438.033475728:40976b70: main Q: entry added, size now log 6821, phys > > 6853 entries > > 4438.033483613:40976b70: main Q: entry added, size now log 6822, phys > > 6854 entries > > 4438.033491751:40976b70: main Q: entry added, size now log 6823, phys > > 6855 entries > > 4438.033499760:40976b70: main Q: entry added, size now log 6824, phys > > 6856 entries > > 4438.033507589:40976b70: main Q: entry added, size now log 6825, phys > > 6857 entries > > 4438.033515679:40976b70: main Q: entry added, size now log 6826, phys > > 6858 entries > > 4438.033523711:40976b70: main Q: entry added, size now log 6827, phys > > 6859 entries > > 4438.033531526:40976b70: main Q: entry added, size now log 6828, phys > > 6860 entries > > 4438.033539457:40976b70: main Q: entry added, size now log 6829, phys > > 6861 entries > > 4438.033547373:40976b70: main Q: entry added, size now log 6830, phys > > 6862 entries > > 4438.033555150:40976b70: main Q: entry added, size now log 6831, phys > > 6863 entries > > 4438.033563201:40976b70: main Q: entry added, size now log 6832, phys > > 6864 entries > > 4438.033571121:40976b70: main Q: entry added, size now log 6833, phys > > 6865 entries > > 4438.033578976:40976b70: main Q: entry added, size now log 6834, phys > > 6866 entries > > 4438.033586987:40976b70: main Q: entry added, size now log 6835, phys > > 6867 entries > > 4438.033594920:40976b70: main Q: entry added, size now log 6836, phys > > 6868 entries > > 4438.033602860:40976b70: main Q: entry added, size now log 6837, phys > > 6869 entries > > 4438.033610750:40976b70: main Q: entry added, size now log 6838, phys > > 6870 entries > > 4438.033618581:40976b70: main Q: entry added, size now log 6839, phys > > 6871 entries > > 4438.033626573:40976b70: main Q: entry added, size now log 6840, phys > > 6872 entries > > 4438.033635275:40976b70: main Q: entry added, size now log 6841, phys > > 6873 entries > > 4438.033644155:40976b70: main Q: entry added, size now log 6842, phys > > 6874 entries > > 4438.033652187:40976b70: main Q: entry added, size now log 6843, phys > > 6875 entries > > 4438.033660012:40976b70: main Q: entry added, size now log 6844, phys > > 6876 entries > > 4438.033667977:40976b70: main Q: entry added, size now log 6845, phys > > 6877 entries > > 4438.033675943:40976b70: main Q: entry added, size now log 6846, phys > > 6878 entries > > 4438.033683820:40976b70: main Q: entry added, size now log 6847, phys > > 6879 entries > > 4438.033691877:40976b70: main Q: entry added, size now log 6848, phys > > 6880 entries > > 4438.033699825:40976b70: main Q: entry added, size now log 6849, phys > > 6881 entries > > 4438.033707743:40976b70: main Q: entry added, size now log 6850, phys > > 6882 entries > > 4438.033716401:40976b70: main Q: MultiEnqObj advised worker start > > 4438.033735637:40976b70: -------- calling select, active > > fds (max 19): 14 15 16 18 19 > > 4438.033772666:40976b70: netstream 0x8640fc0 with new data > > 4438.033792922:40976b70: error during recv on NSD 0x8612ca0: > > Connection reset by peer > > 4438.033800723:40976b70: gtlsRcv return. nsd 0x863fd58, iRet -2165, > > lenRcvBuf 0, ptrRcvBuf 0 > > 4438.033808563:40976b70: Called LogError, msg: netstream session > > 0x8640fc0 will be closed due to error > > > > 4438.033821043:40976b70: MsgSetTAG in: len 14, pszBuf: rsyslogd-2165: > > 4438.033827708:40976b70: MsgSetTAG exit: pMsg->iLenTAG 14, pMsg- > > >TAG.szBuf: rsyslogd-2165: > > 4438.033850214:40976b70: main Q: entry added, size now log 6851, phys > > 6883 entries > > 4438.033859170:40976b70: main Q: EnqueueMsg advised worker start > > 4438.033881307:40976b70: -------- calling select, active > > fds (max 19): 14 15 18 19 > > > > Because of the gtlsRcv I assume this happens on the outgoing > > connection since this is the only encrypted communication channel, but > > I am not sure even this assumption is correct. > > If you have anything you want me to provide or try, I'd be happy to do so. > > Many thanks in advance! > > Best regards, > > Andreas > > > > > > Rainer Gerhards [08:09:11 13:11] wrote: > > > You should update to the latest stable version and see if the > > > problem persists. > > > > > > Rainer > > > > -----Original Message----- > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > > bounces at lists.adiscon.com] On Behalf Of Andreas Grosse > > > > Sent: Thursday, September 08, 2011 12:11 PM > > > > To: rsyslog at lists.adiscon.com > > > > Subject: [rsyslog] netstream errors when logging with a high > > > > message rate > > > > > > > > Hi, > > > > I am running rsyslog 5.8.3 as a central log collector which then > > > > sends the log messages to an archive using tls encryption. When > > > > the log message rate increases, I start seeing log messages like this: > > > > > > > > rsyslog: netstream session 0x8b05ef0 will be closed due to error > > > > [try > > > > http://www.rsyslog.com/e/2165 ] > > > > > > > > The logging does not stop and the tcp connection to the remote > > > > archive does not break, though - it just starts spewing out these > > > > messages up to two times per minute. > > > > I enabled additional debugging, and the logfile contained this: > > > > > > > > 9275.065615635:40976b70: netstream 0x41000c50 with new data > > > > 9275.065635173:40976b70: error during recv on NSD 0x41000b88: > > > > Connection reset by peer > > > > 9275.065641798:40976b70: gtlsRcv return. nsd 0x413fff98, iRet > > > > -2165, lenRcvBuf 0, ptrRcvBuf 0 > > > > 9275.065648064:40976b70: Called LogError, msg: netstream session > > > > 0x41000c50 will be closed due to error > > > > > > > > 9275.065675043:40976b70: main Q: entry added, size now log 6919, > > > > phys > > > > 6951 entries > > > > 9275.065682225:40976b70: main Q: EnqueueMsg advised worker start > > > > 9275.065707944:40976b70: -------- calling select, > > > > active fds (max 19): 14 15 16 19 > > > > > > > > On the receiving end I get no notification of an error happening > > > > at all. Following is the configuration the I use: > > > > > > > > $MaxMessageSize 64k > > > > $RepeatedMsgReduction off > > > > $EscapeControlCharactersOnReceive off > > > > $WorkDirectory /var/rsyslog # default location for work (spool) files > > > > > > > > $ModLoad imtcp > > > > $ModLoad imudp > > > > $ModLoad imptcp > > > > $ModLoad omuxsock > > > > $ModLoad impstats > > > > > > > > $InputPTCPServerListenIP 127.0.0.1 $InputPTCPServerRun 10100 > > > > > > > > $PStatsInterval 300 > > > > > > > > # log local syslog messages back to syslog-ng $OMUxSockSocket > > > > /dev/tosyslog if $programname startswith 'rsyslog' then :omuxsock: > > > > if $programname startswith 'rsyslog' then ~ > > > > > > > > $ActionQueueType LinkedList > > > > $ActionQueueFileName srvrfwd > > > > $ActionResumeRetryCount -1 > > > > $ActionQueueSaveOnShutdown on > > > > $ActionQueueMaxDiskSpace 819200 > > > > > > > > $DefaultNetstreamDriver gtls > > > > > > > > $DefaultNetstreamDriverCAFile /etc/ca/cacert.pem > > > > $DefaultNetstreamDriverCertFile /etc/client.pem > > > > $DefaultNetstreamDriverKeyFile /etc/client.key > > > > > > > > $ActionSendStreamDriverMode 1 > > > > $ActionSendStreamDriverAuthMode x509/certvalid > > > > > > > > $InputTCPServerStreamDriverMode 0 > > > > $InputTCPServerRun 10101 > > > > $UDPServerRun 10101 > > > > > > > > *.* @@(o,z0)loghost:5077;RSYSLOG_SyslogProtocol23Format > > > > > > > > > > > > If you need more data from the debug log, just ask. This is also > > > > easy to reproduce, therefore I am able to try some things if you > > > > come up with suggestions what happens there and how to get rid of > > > > those error messages. Thank you for your help! > > > > > > > > Best regards, > > > > Andreas Grosse > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From andreas+rsyslog at majestyk.de Thu Sep 8 14:47:26 2011 From: andreas+rsyslog at majestyk.de (Andreas Grosse) Date: Thu, 8 Sep 2011 14:47:26 +0200 Subject: [rsyslog] netstream errors when logging with a high message rate In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA72811DB@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA72811DA@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA72811DB@GRFEXC.intern.adiscon.com> Message-ID: <20110908124726.GF16242@majestyk.de> Rainer Gerhards [08:09:11 14:35] wrote: > Maybe I do not need the complete log, because... > > > > Debug output of the thread managing this netstream when the error > > > occured: > > > 4442.720793561:40976b70: netstream 0x4100cab0 with new data > > > 4442.720833000:40976b70: error during recv on NSD 0x413d1ae8: > > > Connection reset by peer > > It looks like the remote peer has closed the session. Any idea why? It logs this message, but we have no indication on the receiving end that there is an error or that the connection is closed. There is also nothing in the rsyslog debug log that shows that a tcp reconnect is necessary; the thread just continues to receive data after that error. I'll just send you the complete debug log in a separate mail. Best regards, Andreas From rgerhards at hq.adiscon.com Thu Sep 8 14:48:37 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 8 Sep 2011 14:48:37 +0200 Subject: [rsyslog] netstream errors when logging with a high message rate In-Reply-To: <20110908124726.GF16242@majestyk.de> References: <9B6E2A8877C38245BFB15CC491A11DA72811DA@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA72811DB@GRFEXC.intern.adiscon.com> <20110908124726.GF16242@majestyk.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72811DC@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Andreas Grosse > Sent: Thursday, September 08, 2011 2:47 PM > To: rsyslog-users > Subject: Re: [rsyslog] netstream errors when logging with a high message > rate > > Rainer Gerhards [08:09:11 14:35] wrote: > > Maybe I do not need the complete log, because... > > > > > > Debug output of the thread managing this netstream when the error > > > > occured: > > > > 4442.720793561:40976b70: netstream 0x4100cab0 with new data > > > > 4442.720833000:40976b70: error during recv on NSD 0x413d1ae8: > > > > Connection reset by peer > > > > It looks like the remote peer has closed the session. Any idea why? > > It logs this message, but we have no indication on the receiving end that > there is an error or that the connection is closed. There is also nothing in the > rsyslog debug log that shows that a tcp reconnect is necessary; the thread > just continues to receive data after that error. > I'll just send you the complete debug log in a separate mail. Please also include a log of the receiving end, as the problem is probably rooted there. Rainer From rgerhards at hq.adiscon.com Thu Sep 8 15:07:42 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 8 Sep 2011 15:07:42 +0200 Subject: [rsyslog] imrelp and ruleset? In-Reply-To: <4E679631.5020202@jasonantman.com> References: <4E667D1C.50702@jasonantman.com> <9B6E2A8877C38245BFB15CC491A11DA72811BA@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA72811C8@GRFEXC.intern.adiscon.com> <4E679631.5020202@jasonantman.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72811DD@GRFEXC.intern.adiscon.com> I now have added support for specifying a ruleset in imrelp: http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=c55997638f8833daec34d7f5 1b9ff6694620f6f8 It's only lightly tested, let me know if there are issues with it. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Jason Antman > Sent: Wednesday, September 07, 2011 6:05 PM > To: rsyslog-users > Subject: Re: [rsyslog] imrelp and ruleset? > > For my use at least, I can say that: > 1) My need for this has dropped significantly in the past 24 hours, as my load > has gone beyond the point that RELP will help (i.e. UDP is no longer the > bottleneck). > 2) Your suggestion would be fine; as is, I simply bind all remote inputs > (currently imudp and imtcp) to a "remote" ruleset, and default everything > else to a "localhost" ruleset. > > Thanks, > Jason > > Rainer Gerhards wrote: > > Quick update: it's not entirely trivial, as librelp is missing some > > plumbing in order to shuffle the ruleset information back to the > > message-receiving callback. Changing all that is beyond the time frame > > I currently have. I think a compromise is that I permit to set a > > single replacement ruleset for > > *all* relp listeners (with tcp, you can assign different ones to > > different listeners). Does this sound useful enough? > > > > Also I'd like to note that I will enhance the current devel, only. I > > guess the patch will also apply to older versions, but I will not > > check that (doing new development in older versions is very time > > consuming for me, so I usually do that only under paid contracts - sorry for > that). > > > > Rainer > > > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > >> Sent: Wednesday, September 07, 2011 10:02 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] imrelp and ruleset? > >> > >> The short answer is that this feature is not yet implemented for imrelp. > >> However, I'll have a look and check if it can be sufficiently quick > >> implemented. Please ping me if you have not heard back by next > Monday. > >> Rainer > >> > >> > >>> -----Original Message----- > >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>> bounces at lists.adiscon.com] On Behalf Of Jason Antman > >>> Sent: Tuesday, September 06, 2011 10:06 PM > >>> To: rsyslog-users > >>> Subject: [rsyslog] imrelp and ruleset? > >>> > >>> Hello, > >>> > >>> I'm planning to finally start moving some of our client (log > >>> sending) devices from stock sysklogd (UDP) to rsyslog/RELP (central > >>> server is rsyslog 5.6.0 currently, planning 5.8.5 soon). Right now > >>> we have a default ruleset for localhost on the central log server, > >>> and bind imudp and imtcp to a separate ruleset for remote hosts. I > >>> notice that there is no $InputRELPServerBindRuleset directive. > >>> Should I file a bug for this, or is there a reason why it hasn't > >>> been implemented the way it has for the other remote input modules? > >>> > >>> Thanks, > >>> Jason Antman > >>> Rutgers University > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Sep 8 15:43:50 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 8 Sep 2011 15:43:50 +0200 Subject: [rsyslog] netstream errors when logging with a high message rate In-Reply-To: <20110908124726.GF16242@majestyk.de> References: <9B6E2A8877C38245BFB15CC491A11DA72811DA@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA72811DB@GRFEXC.intern.adiscon.com> <20110908124726.GF16242@majestyk.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72811DE@GRFEXC.intern.adiscon.com> Thanks for the debug log. Unfortunately, it points to the remote peer as well. I have quickly checked the code. The message "Connection reset by peer" is directly generated by strerror() and thus represents the proper cause of the error as the OS sees it. All in all I conclude that the remote machine has closed the session for some reason. I suggest to run both instances under debug log and check both logs when the problem occurs (or forward them to me). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Andreas Grosse > Sent: Thursday, September 08, 2011 2:47 PM > To: rsyslog-users > Subject: Re: [rsyslog] netstream errors when logging with a high message > rate > > Rainer Gerhards [08:09:11 14:35] wrote: > > Maybe I do not need the complete log, because... > > > > > > Debug output of the thread managing this netstream when the error > > > > occured: > > > > 4442.720793561:40976b70: netstream 0x4100cab0 with new data > > > > 4442.720833000:40976b70: error during recv on NSD 0x413d1ae8: > > > > Connection reset by peer > > > > It looks like the remote peer has closed the session. Any idea why? > > It logs this message, but we have no indication on the receiving end that > there is an error or that the connection is closed. There is also nothing in the > rsyslog debug log that shows that a tcp reconnect is necessary; the thread > just continues to receive data after that error. > I'll just send you the complete debug log in a separate mail. > Best regards, > Andreas > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From jason at jasonantman.com Thu Sep 8 18:45:02 2011 From: jason at jasonantman.com (Jason Antman) Date: Thu, 08 Sep 2011 12:45:02 -0400 Subject: [rsyslog] imrelp and ruleset? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA72811DD@GRFEXC.intern.adiscon.com> References: <4E667D1C.50702@jasonantman.com> <9B6E2A8877C38245BFB15CC491A11DA72811BA@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA72811C8@GRFEXC.intern.adiscon.com> <4E679631.5020202@jasonantman.com> <9B6E2A8877C38245BFB15CC491A11DA72811DD@GRFEXC.intern.adiscon.com> Message-ID: <4E68F10E.3090007@jasonantman.com> Thanks. Probably next week, as I now have a more pressing issue (post to follow). -Jason Rainer Gerhards wrote: > I now have added support for specifying a ruleset in imrelp: > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=c55997638f8833daec34d7f5 > 1b9ff6694620f6f8 > > It's only lightly tested, let me know if there are issues with it. > > Rainer > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Jason Antman >> Sent: Wednesday, September 07, 2011 6:05 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] imrelp and ruleset? >> >> For my use at least, I can say that: >> 1) My need for this has dropped significantly in the past 24 hours, as my >> > load > >> has gone beyond the point that RELP will help (i.e. UDP is no longer the >> bottleneck). >> 2) Your suggestion would be fine; as is, I simply bind all remote inputs >> (currently imudp and imtcp) to a "remote" ruleset, and default everything >> else to a "localhost" ruleset. >> >> Thanks, >> Jason >> >> Rainer Gerhards wrote: >> >>> Quick update: it's not entirely trivial, as librelp is missing some >>> plumbing in order to shuffle the ruleset information back to the >>> message-receiving callback. Changing all that is beyond the time frame >>> I currently have. I think a compromise is that I permit to set a >>> single replacement ruleset for >>> *all* relp listeners (with tcp, you can assign different ones to >>> different listeners). Does this sound useful enough? >>> >>> Also I'd like to note that I will enhance the current devel, only. I >>> guess the patch will also apply to older versions, but I will not >>> check that (doing new development in older versions is very time >>> consuming for me, so I usually do that only under paid contracts - sorry >>> > for > >> that). >> >>> Rainer >>> >>> >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >>>> Sent: Wednesday, September 07, 2011 10:02 AM >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] imrelp and ruleset? >>>> >>>> The short answer is that this feature is not yet implemented for imrelp. >>>> However, I'll have a look and check if it can be sufficiently quick >>>> implemented. Please ping me if you have not heard back by next >>>> >> Monday. >> >>>> Rainer >>>> >>>> >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of Jason Antman >>>>> Sent: Tuesday, September 06, 2011 10:06 PM >>>>> To: rsyslog-users >>>>> Subject: [rsyslog] imrelp and ruleset? >>>>> >>>>> Hello, >>>>> >>>>> I'm planning to finally start moving some of our client (log >>>>> sending) devices from stock sysklogd (UDP) to rsyslog/RELP (central >>>>> server is rsyslog 5.6.0 currently, planning 5.8.5 soon). Right now >>>>> we have a default ruleset for localhost on the central log server, >>>>> and bind imudp and imtcp to a separate ruleset for remote hosts. I >>>>> notice that there is no $InputRELPServerBindRuleset directive. >>>>> Should I file a bug for this, or is there a reason why it hasn't >>>>> been implemented the way it has for the other remote input modules? >>>>> >>>>> Thanks, >>>>> Jason Antman >>>>> Rutgers University >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > > From oliver at obeattie.com Thu Sep 8 18:48:50 2011 From: oliver at obeattie.com (Oliver Beattie) Date: Thu, 8 Sep 2011 17:48:50 +0100 Subject: [rsyslog] imrelp and ruleset? In-Reply-To: <4E68F10E.3090007@jasonantman.com> References: <4E667D1C.50702@jasonantman.com> <9B6E2A8877C38245BFB15CC491A11DA72811BA@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA72811C8@GRFEXC.intern.adiscon.com> <4E679631.5020202@jasonantman.com> <9B6E2A8877C38245BFB15CC491A11DA72811DD@GRFEXC.intern.adiscon.com> <4E68F10E.3090007@jasonantman.com> Message-ID: 'Fraid I'm bound to work off OS packages, so it'll be a little while before I can get around to playing with this one. Thanks though. ?Oliver On 8 September 2011 17:45, Jason Antman wrote: > Thanks. Probably next week, as I now have a more pressing issue (post to > follow). > > -Jason > > > Rainer Gerhards wrote: > >> I now have added support for specifying a ruleset in imrelp: >> >> http://git.adiscon.com/?p=**rsyslog.git;a=commitdiff;h=** >> c55997638f8833daec34d7f5 >> 1b9ff6694620f6f8 >> >> It's only lightly tested, let me know if there are issues with it. >> >> Rainer >> >> >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.**com[mailto: >>> rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Jason Antman >>> Sent: Wednesday, September 07, 2011 6:05 PM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] imrelp and ruleset? >>> >>> For my use at least, I can say that: >>> 1) My need for this has dropped significantly in the past 24 hours, as my >>> >>> >> load >> >> >>> has gone beyond the point that RELP will help (i.e. UDP is no longer the >>> bottleneck). >>> 2) Your suggestion would be fine; as is, I simply bind all remote inputs >>> (currently imudp and imtcp) to a "remote" ruleset, and default everything >>> else to a "localhost" ruleset. >>> >>> Thanks, >>> Jason >>> >>> Rainer Gerhards wrote: >>> >>> >>>> Quick update: it's not entirely trivial, as librelp is missing some >>>> plumbing in order to shuffle the ruleset information back to the >>>> message-receiving callback. Changing all that is beyond the time frame >>>> I currently have. I think a compromise is that I permit to set a >>>> single replacement ruleset for >>>> *all* relp listeners (with tcp, you can assign different ones to >>>> different listeners). Does this sound useful enough? >>>> >>>> Also I'd like to note that I will enhance the current devel, only. I >>>> guess the patch will also apply to older versions, but I will not >>>> check that (doing new development in older versions is very time >>>> consuming for me, so I usually do that only under paid contracts - sorry >>>> >>>> >>> for >> >> >>> that). >>> >>> >>>> Rainer >>>> >>>> >>>> >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.**com[mailto: >>>>> rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >>>>> Sent: Wednesday, September 07, 2011 10:02 AM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] imrelp and ruleset? >>>>> >>>>> The short answer is that this feature is not yet implemented for >>>>> imrelp. >>>>> However, I'll have a look and check if it can be sufficiently quick >>>>> implemented. Please ping me if you have not heard back by next >>>>> >>>>> >>>> Monday. >>> >>> >>>> Rainer >>>>> >>>>> >>>>> >>>>> >>>>>> -----Original Message----- >>>>>> From: rsyslog-bounces at lists.adiscon.**com[mailto: >>>>>> rsyslog- >>>>>> bounces at lists.adiscon.com] On Behalf Of Jason Antman >>>>>> Sent: Tuesday, September 06, 2011 10:06 PM >>>>>> To: rsyslog-users >>>>>> Subject: [rsyslog] imrelp and ruleset? >>>>>> >>>>>> Hello, >>>>>> >>>>>> I'm planning to finally start moving some of our client (log >>>>>> sending) devices from stock sysklogd (UDP) to rsyslog/RELP (central >>>>>> server is rsyslog 5.6.0 currently, planning 5.8.5 soon). Right now >>>>>> we have a default ruleset for localhost on the central log server, >>>>>> and bind imudp and imtcp to a separate ruleset for remote hosts. I >>>>>> notice that there is no $InputRELPServerBindRuleset directive. >>>>>> Should I file a bug for this, or is there a reason why it hasn't >>>>>> been implemented the way it has for the other remote input modules? >>>>>> >>>>>> Thanks, >>>>>> Jason Antman >>>>>> Rutgers University >>>>>> ______________________________**_________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/**mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>>> >>>>>> >>>>> ______________________________**_________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/**mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>>> >>>>> >>>> ______________________________**_________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/**mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>>> >>>> >>>> >>> ______________________________**_________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/**mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> ______________________________**_________________ >> rsyslog mailing list >> http://lists.adiscon.net/**mailman/listinfo/rsyslog >> http://www.rsyslog.com >> >> >> > > ______________________________**_________________ > rsyslog mailing list > http://lists.adiscon.net/**mailman/listinfo/rsyslog > http://www.rsyslog.com > From jason at jasonantman.com Thu Sep 8 18:55:08 2011 From: jason at jasonantman.com (Jason Antman) Date: Thu, 08 Sep 2011 12:55:08 -0400 Subject: [rsyslog] rsyslog 5.8.5 hanging on CentOS 5.5 x86_64 Message-ID: <4E68F36C.6000408@jasonantman.com> Hello, I'm experiencing an issue with intermittent rsyslog hangs requiring a restart. I'm running 5.8.5. I have ommysql running, and am doing a fair amount of parsing (regex) into MySQL. I have a peak load of approx. 43 msgs/sec. according to impstats (which is running every 120 seconds at the moment). The hangs are happening anywhere from 30 minutes to 6 hours, and don't appear to have any correlation to either message rate (as per impstats output) or other processes on the box (at least that I can tell). I'm attempting to run on-demand debugging, and have followed http://www.rsyslog.com/doc/troubleshoot.html but even when running without problems, the process doesn't appear to respond to SIGUSR1 or SIGUSR2. in my init script: export RSYSLOG_DEBUGLOG="/var/log/rsyslog-debuglog" export RSYSLOG_DEBUG="DebugOnDemand NoStdOut LogFuncFlow PrintFuncDB PrintMutexAction" called as: /sbin/rsyslogd -c 5 -d [root at css-dhcp log]# rsyslogd -v rsyslogd 5.8.5, compiled with: FEATURE_REGEXP: Yes FEATURE_LARGEFILE: No GSSAPI Kerberos 5 support: Yes FEATURE_DEBUG (debug build, slow code): No 32bit Atomic operations supported: Yes 64bit Atomic operations supported: No Runtime Instrumentation (slow code): No When I restart rsyslog, I see on console: 0699.281483000:2b18292632d0: Note: debug on demand turned on via configuraton file, use USR1 signal to activate. I have attached my /var/log/rsyslog-debuglog. As far as I can tell, it will write to it on startup, always ending with the "Checking pidfile" line, and then never write to it again no matter how many times I send USR1 or USR2. This is a *production* box, and I no longer have test hardware. I can deal with some short losses of syslog (as that's happening already when it hangs) but can't feasibly replace with a debug-enabled version. At the moment, I have a cron job running every 2 minutes that restarts rsyslog if my busiest log file is >= 2 minutes old. Any advice (on the debug log, or even better, the hang) would be greatly appreciated. Thanks, Jason Antman Rutgers University -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: rsyslog-debuglog URL: From jpoling at moody.edu Thu Sep 8 19:36:05 2011 From: jpoling at moody.edu (Jeff Poling) Date: Thu, 8 Sep 2011 12:36:05 -0500 Subject: [rsyslog] New User Help In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA72811D8@GRFEXC.intern.adiscon.com> References: <9599A350A0A5884DB4E50D83F9287D0F01F305D0E8@exchmbx01.moody.edu><9599A350A0A5884DB4E50D83F9287D0F01F305D233@exchmbx01.moody.edu> <9599A350A0A5884DB4E50D83F9287D0F01F305D4B2@exchmbx01.moody.edu> <9B6E2A8877C38245BFB15CC491A11DA72811D8@GRFEXC.intern.adiscon.com> Message-ID: <9599A350A0A5884DB4E50D83F9287D0F01F305D5BB@exchmbx01.moody.edu> > For all params, use ./configure --help > > I think what you need is --bindir=... Thanks, Rainer. That is very helpful. Jeff From david at lang.hm Thu Sep 8 20:04:21 2011 From: david at lang.hm (david at lang.hm) Date: Thu, 8 Sep 2011 11:04:21 -0700 (PDT) Subject: [rsyslog] netstream errors when logging with a high message rate In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA72811DE@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA72811DA@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA72811DB@GRFEXC.intern.adiscon.com> <20110908124726.GF16242@majestyk.de> <9B6E2A8877C38245BFB15CC491A11DA72811DE@GRFEXC.intern.adiscon.com> Message-ID: is there a firewall or stateful router (including NAT) between the two devices? it could be terminating the connection in the middle and both instances of rsyslog think the other is closing it. David Lang On Thu, 8 Sep 2011, Rainer Gerhards wrote: > Date: Thu, 8 Sep 2011 15:43:50 +0200 > From: Rainer Gerhards > Reply-To: rsyslog-users > To: rsyslog-users > Subject: Re: [rsyslog] netstream errors when logging with a high message rate > > Thanks for the debug log. Unfortunately, it points to the remote peer as > well. I have quickly checked the code. The message "Connection reset by peer" > is directly generated by strerror() and thus represents the proper cause of > the error as the OS sees it. All in all I conclude that the remote machine > has closed the session for some reason. I suggest to run both instances under > debug log and check both logs when the problem occurs (or forward them to > me). > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Andreas Grosse >> Sent: Thursday, September 08, 2011 2:47 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] netstream errors when logging with a high message >> rate >> >> Rainer Gerhards [08:09:11 14:35] wrote: >>> Maybe I do not need the complete log, because... >>> >>>>> Debug output of the thread managing this netstream when the error >>>>> occured: >>>>> 4442.720793561:40976b70: netstream 0x4100cab0 with new data >>>>> 4442.720833000:40976b70: error during recv on NSD 0x413d1ae8: >>>>> Connection reset by peer >>> >>> It looks like the remote peer has closed the session. Any idea why? >> >> It logs this message, but we have no indication on the receiving end that >> there is an error or that the connection is closed. There is also nothing > in the >> rsyslog debug log that shows that a tcp reconnect is necessary; the thread >> just continues to receive data after that error. >> I'll just send you the complete debug log in a separate mail. >> Best regards, >> Andreas >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From jpoling at moody.edu Fri Sep 9 06:02:00 2011 From: jpoling at moody.edu (Jeff Poling) Date: Thu, 8 Sep 2011 23:02:00 -0500 Subject: [rsyslog] 5.8.5 Upgrade Issue Message-ID: <9599A350A0A5884DB4E50D83F9287D0F01F305D691@exchmbx01.moody.edu> I installed rsyslog 5.8.5 on a system that already had 5.4.x on it. A symbolic link was used previously to point to the rsyslog install directory. After installing the new version, I adjusted to link to point to the new version. Here is what the file system looks like: drwxr-xr-x 2 root root 1024 Sep 14 2010 etc drwxr-xr-x 4 root root 1024 Sep 14 2010 relp lrwxrwxrwx 1 root root 26 Sep 8 13:27 rsyslog -> /opt/rsyslog/rsyslog-5.8.5 drwxr-xr-x 5 root root 1024 Sep 14 2010 rsyslog-5.4.0 drwxr-xr-x 5 root root 1024 Sep 8 13:24 rsyslog-5.8.5 drwxr-xr-x 2 root root 1024 Sep 14 2010 spool When I try to start rsyslog with the start script that was used previously (/etc/init.d/rsyslog), it hangs. After a reboot, rsyslog simply does not start. So, I am missing something :) Is there a way to debug this? Is there a local log that rsyslog creates with info that will help me determine what is wrong? Any insight is greatly appreciated. If I did not explain it well, please let me know that also. Thanks, Jeff Jeffrey Poling System Administrator | Information Systems Moody Bible Institute 820 N. LaSalle Blvd., Chicago, IL 60610 312-329-8968 www.moodyministries.net >From the Word. To Life. From andreas+rsyslog at majestyk.de Fri Sep 9 10:46:26 2011 From: andreas+rsyslog at majestyk.de (Andreas Grosse) Date: Fri, 9 Sep 2011 10:46:26 +0200 Subject: [rsyslog] netstream errors when logging with a high message rate In-Reply-To: References: <20110908124726.GF16242@majestyk.de> <9B6E2A8877C38245BFB15CC491A11DA72811DE@GRFEXC.intern.adiscon.com> Message-ID: <20110909084626.GM16242@majestyk.de> Hi, thanks for taking the time to look through the debug log! I did a tcpdump while reproducing that issue and it confirms that the TCP connection is not closed. There are no RESET or FIN packets which would indicate a teardown of the TCP connection, and no SYN packets which would be required for a new TCP connection to be established. Although that error is reported, packets just keep on going over that connection. So if the rsyslog code just passes on the OS error, and the error occurs although the connection is not actually reset, what makes the OS think that the connection was reset? Best regards, Andreas david at lang.hm [08:09:11 20:04] wrote: > is there a firewall or stateful router (including NAT) between the two > devices? it could be terminating the connection in the middle and both > instances of rsyslog think the other is closing it. > > David Lang > > On Thu, 8 Sep 2011, Rainer Gerhards wrote: > >> Date: Thu, 8 Sep 2011 15:43:50 +0200 >> From: Rainer Gerhards >> Reply-To: rsyslog-users >> To: rsyslog-users >> Subject: Re: [rsyslog] netstream errors when logging with a high message rate >> >> Thanks for the debug log. Unfortunately, it points to the remote peer as >> well. I have quickly checked the code. The message "Connection reset by peer" >> is directly generated by strerror() and thus represents the proper cause of >> the error as the OS sees it. All in all I conclude that the remote machine >> has closed the session for some reason. I suggest to run both instances under >> debug log and check both logs when the problem occurs (or forward them to >> me). >> >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Andreas Grosse >>> Sent: Thursday, September 08, 2011 2:47 PM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] netstream errors when logging with a high message >>> rate >>> >>> Rainer Gerhards [08:09:11 14:35] wrote: >>>> Maybe I do not need the complete log, because... >>>> >>>>>> Debug output of the thread managing this netstream when the error >>>>>> occured: >>>>>> 4442.720793561:40976b70: netstream 0x4100cab0 with new data >>>>>> 4442.720833000:40976b70: error during recv on NSD 0x413d1ae8: >>>>>> Connection reset by peer >>>> >>>> It looks like the remote peer has closed the session. Any idea why? >>> >>> It logs this message, but we have no indication on the receiving end that >>> there is an error or that the connection is closed. There is also nothing >> in the >>> rsyslog debug log that shows that a tcp reconnect is necessary; the thread >>> just continues to receive data after that error. >>> I'll just send you the complete debug log in a separate mail. >>> Best regards, >>> Andreas >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From jpoling at moody.edu Fri Sep 9 17:06:43 2011 From: jpoling at moody.edu (Jeff Poling) Date: Fri, 9 Sep 2011 10:06:43 -0500 Subject: [rsyslog] 5.8.5 Upgrade Issue In-Reply-To: <9599A350A0A5884DB4E50D83F9287D0F01F305D691@exchmbx01.moody.edu> References: <9599A350A0A5884DB4E50D83F9287D0F01F305D691@exchmbx01.moody.edu> Message-ID: <9599A350A0A5884DB4E50D83F9287D0F01F315BD2C@exchmbx01.moody.edu> I posted last night about an issue with rsyslog hanging after I started using the 5.8.5 version. Today I have some more info. The startup process actually fails and produces the following error: "Starting system logger (rsyslog): rsyslogd: $AbortOnUncleanConfig is set, and config is not clean. Check error log for details, fix errors and restart. As a last resort, you may want to remove $AbortOnUncleanConfig to permit a startup with a dirty config." I am using the same config file from the previous version of rsyslog. Is there something that I need to change in the config file to get it to work with 5.8.5? Below is the config we are using: ## $ModLoad immark # provides --MARK-- message capability $ModLoad imuxsock # provides support for local system logging (e.g. via logger command) $ModLoad imklog # kernel logging (formerly provided by rklogd) #### $ModLoad imfile # Input module for text files #$ModLoad imrelp # Input RELP protocol #$InputRELPServerRun 1514 $ModLoad omrelp $WorkDirectory /opt/rsyslog/spool $MarkMessagePeriod 600 $MaxMessageSize 64k #$OptimizeForUniprocessor on $RepeatedMsgReduction off #$MainMsgQueueFileName mailQ_ #$MainMsgQueueSaveOnShutdown on #$MainMsgQueueType LinkedList #$MainMsgQueueWorkerThreads 8 $ActionWriteAllMarkMessages on $ActionQueueFileName actionQ_ $ActionQueueSaveOnShutdown on $ActionQueueType LinkedList $ActionResumeRetryCount -1 #$ActionQueueWorkerThreads 8 #$ActionQueueDequeueBatchSize 32 #$ActionQueueHighWaterMark 12000 $AbortOnUncleanConfig on #$template HostFile,"/opt/var/log/messages.%FROMHOST%.%%%%%%%%" #$template TraditionalFormat,"%% %timegenerated:::date-rfc3164% %HOSTNAME% [%syslogfacility-text%.%syslogpriority-text%] %syslogtag%%msg%\n" *.* :omrelp:mysyslogserver.myorg.com:601 Thanks for any insight. Jeff From rgerhards at hq.adiscon.com Sun Sep 11 17:16:14 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 11 Sep 2011 17:16:14 +0200 Subject: [rsyslog] 5.8.5 Upgrade Issue Message-ID: <002d01cc7095$91073030$100013ac@intern.adiscon.com> Can you pls provide a debug log of the startup? One reason may be that the work directory does not exist. This was previously undetected. Rainer Jeff Poling hat geschrieben:I posted last night about an issue with rsyslog hanging after I started using the 5.8.5 version. Today I have some more info. The startup process actually fails and produces the following error: "Starting system logger (rsyslog): rsyslogd: $AbortOnUncleanConfig is set, and config is not clean. Check error log for details, fix errors and restart. As a last resort, you may want to remove $AbortOnUncleanConfig to permit a startup with a dirty config." I am using the same config file from the previous version of rsyslog. Is there something that I need to change in the config file to get it to work with 5.8.5? Below is the config we are using: ## $ModLoad immark # provides --MARK-- message capability $ModLoad imuxsock # provides support for local system logging (e.g. via logger command) $ModLoad imklog # kernel logging (formerly provided by rklogd) #### $ModLoad imfile # Input module for text files #$ModLoad imrelp # Input RELP protocol #$InputRELPServerRun 1514 $ModLoad omrelp $WorkDirectory /opt/rsyslog/spool $MarkMessagePeriod 600 $MaxMessageSize 64k #$OptimizeForUniprocessor on $RepeatedMsgReduction off #$MainMsgQueueFileName mailQ_ #$MainMsgQueueSaveOnShutdown on #$MainMsgQueueType LinkedList #$MainMsgQueueWorkerThreads 8 $ActionWriteAllMarkMessages on $ActionQueueFileName actionQ_ $ActionQueueSaveOnShutdown on $ActionQueueType LinkedList $ActionResumeRetryCount -1 #$ActionQueueWorkerThreads 8 #$ActionQueueDequeueBatchSize 32 #$ActionQueueHighWaterMark 12000 $AbortOnUncleanConfig on #$template HostFile,"/opt/var/log/messages.%FROMHOST%.%%%%%%%%" #$template TraditionalFormat,"%% %timegenerated:::date-rfc3164% %HOSTNAME% [%syslogfacility-text%.%syslogpriority-text%] %syslogtag%%msg%\n" *.* :omrelp:mysyslogserver.myorg.com:601 Thanks for any insight. Jeff _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From igor.sverkos at googlemail.com Sun Sep 11 21:38:44 2011 From: igor.sverkos at googlemail.com (Igor Sverkos) Date: Sun, 11 Sep 2011 21:38:44 +0200 Subject: [rsyslog] No mark messages are getting written Message-ID: <4E6D0E44.8000807@googlemail.com> Hi, My problem: =========== I want --MARK-- messages. :) But currently, rsyslog doesn't generate any --MARK-- message. My configuration: ================= Just a minimal test configuration: $ModLoad immark $ModLoad imuxsock $ModLoad imklog # Setting the --MARK-- frequency $MarkMessagePeriod 60 *.* /var/logTest/test.log My expectation: =============== I expect that rsyslog logs a --MARK-- message every 60 seconds. Current result: =============== Every syslog message seems to be written as expected to /var/logTest/test.log. But I don't see any --MARK-- message. System details: =============== OS: Debian GNU/Linux 6.0.2 (squeeze) rsyslog: v5.8.3-1 (from Debian testing) Debug log: ========== http://pastebin.com/PYnLnkaR -- Regards, Igor From marcin at mejor.pl Sun Sep 11 22:02:12 2011 From: marcin at mejor.pl (=?ISO-8859-2?Q?Marcin_Miros=B3aw?=) Date: Sun, 11 Sep 2011 22:02:12 +0200 Subject: [rsyslog] No mark messages are getting written In-Reply-To: <4E6D0E44.8000807@googlemail.com> References: <4E6D0E44.8000807@googlemail.com> Message-ID: <4E6D13C4.9070400@mejor.pl> W dniu 2011-09-11 21:38, Igor Sverkos pisze: > Hi, > > My problem: > =========== > I want --MARK-- messages. :) Hello! This problem is fixed: http://lists.adiscon.net/pipermail/rsyslog/2011-August/013724.html Regards. From igor.sverkos at googlemail.com Sun Sep 11 22:12:20 2011 From: igor.sverkos at googlemail.com (Igor Sverkos) Date: Sun, 11 Sep 2011 22:12:20 +0200 Subject: [rsyslog] No mark messages are getting written In-Reply-To: <4E6D13C4.9070400@mejor.pl> References: <4E6D0E44.8000807@googlemail.com> <4E6D13C4.9070400@mejor.pl> Message-ID: <4E6D1624.5020205@googlemail.com> Hi, Marcin Miros?aw wrote:: > This problem is fixed: > http://lists.adiscon.net/pipermail/rsyslog/2011-August/013724.html ...and I thought I searched before ;) Thank you! -- Regards, Igor From steve.chupack at dealer.com Sun Sep 11 22:50:57 2011 From: steve.chupack at dealer.com (Steve Chupack) Date: Sun, 11 Sep 2011 16:50:57 -0400 Subject: [rsyslog] Unable to implement on-disk queuing Message-ID: <20110911165057.77049b4d@stchupack-m4300> ?Hi, I have rsyslogd up and running and logging to mysql with loganalyzer as a front end. Very cool. However, I can't seem to get on-disk queuing working, which would be nice if mysql goes down or a table is locked because I'm purging records during a nightly maintenance cron. I never see any queue files created. I dropped the size of the memory queue to something ridiculously small, but still no queue files even after 20 minutes. I followed the instructions given here: http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along with a couple of other tutorials on rsyslogd and queuing. I'm quite sure I'm just completely missing something obvious. Details below... And thanks to anyone who might be able to point out what I'm doing wrong. -Steve MY CONFIG, please, no flames re the horrendously complex rules ;-) # if you experience problems, check # http://www.rsyslog.com/troubleshoot for assistance # rsyslog v3: load input modules # If you do not load inputs, nothing happens! # You may need to set the module load path if modules are not found. $ModLoad immark # provides --MARK-- message capability $ModLoad imuxsock # provides support for local system logging (e.g. via logger command) $ModLoad imklog # kernel logging (formerly provided by rklogd) $ModLoad ommysql # SC 2010.11.11: configure disk caching in case mysql is unavailable $MainMsgQueueSize 5 $WorkDirectory /var/log/rsyslogq # default location for work (spool) files $ActionQueueType LinkedList # use asynchronous processing $ActionQueueFileName dbq # set file name, also enables disk mode $ActionResumeRetryCount -1 # infinite retries on insert failure # send snmpd INFO messages to the dustbin if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd' then ~ $template vtfw,"insert into vtfw (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL if ($fromhost-ip contains '10.128.255') then :ommysql:localhost,Syslog,root,mysqldb44;vtfw & ~ #if ($fromhost-ip contains '10.128.255') then ~ $template vt1hs1_switches,"insert into vt1hs1_switches (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL if $fromhost-ip contains '10.128.0' or $hostname contains_i 'core01' or $hostname contains_i 'core02' then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches #if ($fromhost-ip contains '10.128.0' or $source=='vt1hs1-dc216-core01' or $source=='vt1hs1-dc216-core02') then /var/log/switches & ~ #if $fromhost-ip contains '10.128.0' or $hostname contains_i 'core01' or $hostname contains_i 'core02' then ~ $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL if $fromhost-ip contains '10.128.244' then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi & ~ #if $fromhost-ip contains '10.128.244' then ~ $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == '10.128.24.44' or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' or $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') and ($syslogseverity-text == 'warning' or $syslogseverity-text == 'warn' or $syslogseverity-text == 'err' or $syslogseverity-text == 'error' or $syslogseverity-text == 'crit' or $syslogseverity-text == 'alert' or $syslogseverity-text == 'emergency' or $syslogseverity-text == 'panic') then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere & ~ if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == '10.128.24.44' or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' or $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') and ($syslogseverity-text == 'notice') then ~ #*.* :ommysql:localhost,Syslog,root,mysqldb44 *.notice :ommysql:localhost,Syslog,root,mysqldb44 # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none -/var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* -/var/log/cron # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit -/var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log # Remote Logging (we use TCP for reliable delivery) # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. #$WorkDirectory /rsyslog/spool # where to place spool files #$ActionQueueFileName uniqName # unique name prefix for spool files #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown #$ActionQueueType LinkedList # run asynchronously #$ActionResumeRetryCount -1 # infinite retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional #*.* @@remote-host:514 # ######### Receiving Messages from Remote Hosts ########## # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) $ModLoad imtcp.so # load module $InputTCPServerRun 1470 # start up TCP listener at port 514 # UDP Syslog Server: $ModLoad imudp.so # provides UDP syslog reception $UDPServerRun 514 # start a UDP syslog server at standard port 514 SOME DEBUG OUTPUT: (see attached) Steve Chupack | IT Systems Administrator V: 877.327.8422 x 1242 Steve.Chupack at dealer.com | www.dealer.com -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: rsyslog-debug.txt URL: From steve.chupack at dealer.com Sun Sep 11 23:10:31 2011 From: steve.chupack at dealer.com (Steve Chupack) Date: Sun, 11 Sep 2011 17:10:31 -0400 Subject: [rsyslog] Unable to implement on-disk queuing In-Reply-To: <20110911165057.77049b4d@stchupack-m4300> References: <20110911165057.77049b4d@stchupack-m4300> Message-ID: <20110911171031.25bf90cc@stchupack-m4300> Ooops... previous message's attachment was the config, not the debug output. Debug attached. On Sun, 11 Sep 2011 16:50:57 -0400 Steve Chupack wrote: > Hi, > > I have rsyslogd up and running and logging to mysql with loganalyzer as a front end. Very cool. > > However, I can't seem to get on-disk queuing working, which would be nice if mysql goes down or a table is locked because I'm purging records during a nightly maintenance cron. > > I never see any queue files created. I dropped the size of the memory queue to something ridiculously small, but still no queue files even after 20 minutes. > > I followed the instructions given here: http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along with a couple of other tutorials on rsyslogd and queuing. > > I'm quite sure I'm just completely missing something obvious. Details below... And thanks to anyone who might be able to point out what I'm doing wrong. > > -Steve > > > > MY CONFIG, please, no flames re the horrendously complex rules ;-) > > # if you experience problems, check > # http://www.rsyslog.com/troubleshoot for assistance > > # rsyslog v3: load input modules > # If you do not load inputs, nothing happens! > # You may need to set the module load path if modules are not found. > > $ModLoad immark # provides --MARK-- message capability > $ModLoad imuxsock # provides support for local system logging (e.g. via logger command) > $ModLoad imklog # kernel logging (formerly provided by rklogd) > $ModLoad ommysql > > # SC 2010.11.11: configure disk caching in case mysql is unavailable > > $MainMsgQueueSize 5 > > $WorkDirectory /var/log/rsyslogq # default location for work (spool) files > > $ActionQueueType LinkedList # use asynchronous processing > $ActionQueueFileName dbq # set file name, also enables disk mode > $ActionResumeRetryCount -1 # infinite retries on insert failure > > # send snmpd INFO messages to the dustbin > > if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd' then ~ > > $template vtfw,"insert into vtfw (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > if ($fromhost-ip contains '10.128.255') then :ommysql:localhost,Syslog,root,mysqldb44;vtfw > & ~ > #if ($fromhost-ip contains '10.128.255') then ~ > > $template vt1hs1_switches,"insert into vt1hs1_switches (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > if $fromhost-ip contains '10.128.0' or $hostname contains_i 'core01' or $hostname contains_i 'core02' then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches > #if ($fromhost-ip contains '10.128.0' or $source=='vt1hs1-dc216-core01' or $source=='vt1hs1-dc216-core02') then /var/log/switches > & ~ > #if $fromhost-ip contains '10.128.0' or $hostname contains_i 'core01' or $hostname contains_i 'core02' then ~ > > $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > if $fromhost-ip contains '10.128.244' then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi > & ~ > #if $fromhost-ip contains '10.128.244' then ~ > > $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == '10.128.24.44' or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' or $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') and ($syslogseverity-text == 'warning' or $syslogseverity-text == 'warn' or $syslogseverity-text == 'err' or $syslogseverity-text == 'error' or $syslogseverity-text == 'crit' or $syslogseverity-text == 'alert' or $syslogseverity-text == 'emergency' or $syslogseverity-text == 'panic') then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere > & ~ > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == '10.128.24.44' or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' or $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') and ($syslogseverity-text == 'notice') then ~ > > > #*.* :ommysql:localhost,Syslog,root,mysqldb44 > *.notice :ommysql:localhost,Syslog,root,mysqldb44 > > # Log all kernel messages to the console. > # Logging much else clutters up the screen. > #kern.* /dev/console > > # Log anything (except mail) of level info or higher. > # Don't log private authentication messages! > *.info;mail.none;authpriv.none;cron.none -/var/log/messages > > # The authpriv file has restricted access. > authpriv.* /var/log/secure > > # Log all the mail messages in one place. > mail.* -/var/log/maillog > > > # Log cron stuff > cron.* -/var/log/cron > > # Everybody gets emergency messages > *.emerg * > > # Save news errors of level crit and higher in a special file. > uucp,news.crit -/var/log/spooler > > # Save boot messages also to boot.log > local7.* /var/log/boot.log > > # Remote Logging (we use TCP for reliable delivery) > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to disk and sent when it is up again. > #$WorkDirectory /rsyslog/spool # where to place spool files > #$ActionQueueFileName uniqName # unique name prefix for spool files > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown > #$ActionQueueType LinkedList # run asynchronously > #$ActionResumeRetryCount -1 # infinite retries if host is down > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > #*.* @@remote-host:514 > > > # ######### Receiving Messages from Remote Hosts ########## > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support it) > $ModLoad imtcp.so # load module > $InputTCPServerRun 1470 # start up TCP listener at port 514 > > # UDP Syslog Server: > $ModLoad imudp.so # provides UDP syslog reception > $UDPServerRun 514 # start a UDP syslog server at standard port 514 > > > SOME DEBUG OUTPUT: > > (see attached) > > > > > > > > > > Steve Chupack | IT Systems Administrator > V: 877.327.8422 x 1242 > Steve.Chupack at dealer.com | www.dealer.com > > -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: rsyslog-debug.txt URL: From mark at thermeon.com Mon Sep 12 09:43:31 2011 From: mark at thermeon.com (Mark Olliver) Date: Mon, 12 Sep 2011 08:43:31 +0100 Subject: [rsyslog] Error message In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA72811A4@GRFEXC.intern.adiscon.com> References: <00a601cc6c7c$d2f2be10$78d83a30$@thermeon.com><9B6E2A8877C38245BFB15CC491A11DA72811A3@GRFEXC.intern.adiscon.com> <00c301cc6c80$c5dc61f0$519525d0$@thermeon.com> <9B6E2A8877C38245BFB15CC491A11DA72811A4@GRFEXC.intern.adiscon.com> Message-ID: <009001cc711f$aec54330$0c4fc990$@thermeon.com> Hi Rainer, Well after a bit longer suffering from this issue, it seems that by turning on debug rsyslog works perfectly and does not consume most of the CPU however, as soon as I take of the debug flag the CPU usage starts to go up again and also some log messages don't get logged anymore once the daemon has been HUP'd. Do you have any ideas on what this could be, or why setting the debug flag makes everything work ok, obviously I cant really leave the debug flag on all the time as it generates around a Gb of data a day. (and doesn't redirect to a new log upon being HUP'd) Regards Mark > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: 06 September 2011 11:38 > To: rsyslog-users > Subject: Re: [rsyslog] Error message > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Mark Olliver > > Sent: Tuesday, September 06, 2011 12:36 PM > > To: 'rsyslog-users' > > Subject: Re: [rsyslog] Error message > > > > Hi Rainer, > > > > I am not sure I can give a debug log as there is too much PCI > > compliant information in there, on my other hosts which are Ubuntu I > > am not having this issue. > > > > I have run in debug mode but so far no found the issue there as yet If > > you have any ideas on how to isolate it then I am happy to try that > > and if I > can get > > a clean debug then I am happy to send that over. > > Quite honestly, this is too generic to have any decent clue. You may want to > review the debug log yourself and check if there are any messages related to > the socket or its state. Especially between the point where it worked vs. > where it does not work. > > I am unable to diagnose the problem without instructions to either > reproduce or a complete debug log -- sorry. > > Rainer > > > > > > Mark > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Sep 12 11:07:23 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 12 Sep 2011 11:07:23 +0200 Subject: [rsyslog] netstream errors when logging with a high message rate In-Reply-To: <20110909084626.GM16242@majestyk.de> References: <20110908124726.GF16242@majestyk.de><9B6E2A8877C38245BFB15CC491A11DA72811DE@GRFEXC.intern.adiscon.com> <20110909084626.GM16242@majestyk.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72811EC@GRFEXC.intern.adiscon.com> To rule out any intermediate layer, please run an strace on both rsyslogs. So we get the result codes directly from the OS. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Andreas Grosse > Sent: Friday, September 09, 2011 10:46 AM > To: rsyslog-users > Subject: Re: [rsyslog] netstream errors when logging with a high > message rate > > Hi, > thanks for taking the time to look through the debug log! > I did a tcpdump while reproducing that issue and it confirms that the > TCP > connection is not closed. There are no RESET or FIN packets which would > indicate a teardown of the TCP connection, and no SYN packets which > would > be required for a new TCP connection to be established. Although that > error is reported, packets just keep on going over that connection. > > So if the rsyslog code just passes on the OS error, and the error > occurs although the connection is not actually reset, what makes the OS > think that the connection was reset? > > Best regards, > Andreas > > david at lang.hm [08:09:11 20:04] wrote: > > is there a firewall or stateful router (including NAT) between the > two > > devices? it could be terminating the connection in the middle and > both > > instances of rsyslog think the other is closing it. > > > > David Lang > > > > On Thu, 8 Sep 2011, Rainer Gerhards wrote: > > > >> Date: Thu, 8 Sep 2011 15:43:50 +0200 > >> From: Rainer Gerhards > >> Reply-To: rsyslog-users > >> To: rsyslog-users > >> Subject: Re: [rsyslog] netstream errors when logging with a high > message rate > >> > >> Thanks for the debug log. Unfortunately, it points to the remote > peer as > >> well. I have quickly checked the code. The message "Connection reset > by peer" > >> is directly generated by strerror() and thus represents the proper > cause of > >> the error as the OS sees it. All in all I conclude that the remote > machine > >> has closed the session for some reason. I suggest to run both > instances under > >> debug log and check both logs when the problem occurs (or forward > them to > >> me). > >> > >> Rainer > >> > >>> -----Original Message----- > >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>> bounces at lists.adiscon.com] On Behalf Of Andreas Grosse > >>> Sent: Thursday, September 08, 2011 2:47 PM > >>> To: rsyslog-users > >>> Subject: Re: [rsyslog] netstream errors when logging with a high > message > >>> rate > >>> > >>> Rainer Gerhards [08:09:11 14:35] wrote: > >>>> Maybe I do not need the complete log, because... > >>>> > >>>>>> Debug output of the thread managing this netstream when the > error > >>>>>> occured: > >>>>>> 4442.720793561:40976b70: netstream 0x4100cab0 with new data > >>>>>> 4442.720833000:40976b70: error during recv on NSD 0x413d1ae8: > >>>>>> Connection reset by peer > >>>> > >>>> It looks like the remote peer has closed the session. Any idea > why? > >>> > >>> It logs this message, but we have no indication on the receiving > end that > >>> there is an error or that the connection is closed. There is also > nothing > >> in the > >>> rsyslog debug log that shows that a tcp reconnect is necessary; the > thread > >>> just continues to receive data after that error. > >>> I'll just send you the complete debug log in a separate mail. > >>> Best regards, > >>> Andreas > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Sep 12 11:10:43 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 12 Sep 2011 11:10:43 +0200 Subject: [rsyslog] Error message In-Reply-To: <009001cc711f$aec54330$0c4fc990$@thermeon.com> References: <00a601cc6c7c$d2f2be10$78d83a30$@thermeon.com><9B6E2A8877C38245BFB15CC491A11DA72811A3@GRFEXC.intern.adiscon.com> <00c301cc6c80$c5dc61f0$519525d0$@thermeon.com><9B6E2A8877C38245BFB15CC491A11DA72811A4@GRFEXC.intern.adiscon.com> <009001cc711f$aec54330$0c4fc990$@thermeon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72811ED@GRFEXC.intern.adiscon.com> Debug log changes timing, so if there is a bug that relies on a specific timing, debug log activation can (unfortunately) make it disappear. One thing you could try is use debug on demand and activate debug only after it enters an invalid state. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mark Olliver > Sent: Monday, September 12, 2011 9:44 AM > To: 'rsyslog-users' > Subject: Re: [rsyslog] Error message > > Hi Rainer, > > Well after a bit longer suffering from this issue, it seems that by > turning > on debug rsyslog works perfectly and does not consume most of the CPU > however, as soon as I take of the debug flag the CPU usage starts to > go up > again and also some log messages don't get logged anymore once the > daemon > has been HUP'd. > Do you have any ideas on what this could be, or why setting the debug > flag > makes everything work ok, obviously I cant really leave the debug flag > on > all the time as it generates around a Gb of data a day. (and doesn't > redirect to a new log upon being HUP'd) > > Regards > > Mark > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > > Sent: 06 September 2011 11:38 > > To: rsyslog-users > > Subject: Re: [rsyslog] Error message > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Mark Olliver > > > Sent: Tuesday, September 06, 2011 12:36 PM > > > To: 'rsyslog-users' > > > Subject: Re: [rsyslog] Error message > > > > > > Hi Rainer, > > > > > > I am not sure I can give a debug log as there is too much PCI > > > compliant information in there, on my other hosts which are Ubuntu > I > > > am not having this issue. > > > > > > I have run in debug mode but so far no found the issue there as yet > If > > > you have any ideas on how to isolate it then I am happy to try that > > > and if I > > can get > > > a clean debug then I am happy to send that over. > > > > Quite honestly, this is too generic to have any decent clue. You may > want > to > > review the debug log yourself and check if there are any messages > related > to > > the socket or its state. Especially between the point where it worked > vs. > > where it does not work. > > > > I am unable to diagnose the problem without instructions to either > > reproduce or a complete debug log -- sorry. > > > > Rainer > > > > > > > > > Mark > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Sep 12 11:13:28 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 12 Sep 2011 11:13:28 +0200 Subject: [rsyslog] Unable to implement on-disk queuing In-Reply-To: <20110911171031.25bf90cc@stchupack-m4300> References: <20110911165057.77049b4d@stchupack-m4300> <20110911171031.25bf90cc@stchupack-m4300> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72811EE@GRFEXC.intern.adiscon.com> Queue files are only created if necessary. The debug log does not contain any such situation. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Steve Chupack > Sent: Sunday, September 11, 2011 11:11 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > Ooops... previous message's attachment was the config, not the debug > output. Debug attached. > > > > On Sun, 11 Sep 2011 16:50:57 -0400 > Steve Chupack wrote: > > > Hi, > > > > I have rsyslogd up and running and logging to mysql with loganalyzer > as a front end. Very cool. > > > > However, I can't seem to get on-disk queuing working, which would be > nice if mysql goes down or a table is locked because I'm purging > records during a nightly maintenance cron. > > > > I never see any queue files created. I dropped the size of the memory > queue to something ridiculously small, but still no queue files even > after 20 minutes. > > > > I followed the instructions given here: > http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along with a > couple of other tutorials on rsyslogd and queuing. > > > > I'm quite sure I'm just completely missing something obvious. Details > below... And thanks to anyone who might be able to point out what I'm > doing wrong. > > > > -Steve > > > > > > > > MY CONFIG, please, no flames re the horrendously complex rules ;-) > > > > # if you experience problems, check > > # http://www.rsyslog.com/troubleshoot for assistance > > > > # rsyslog v3: load input modules > > # If you do not load inputs, nothing happens! > > # You may need to set the module load path if modules are not found. > > > > $ModLoad immark # provides --MARK-- message capability > > $ModLoad imuxsock # provides support for local system logging (e.g. > via logger command) > > $ModLoad imklog # kernel logging (formerly provided by rklogd) > > $ModLoad ommysql > > > > # SC 2010.11.11: configure disk caching in case mysql is unavailable > > > > $MainMsgQueueSize 5 > > > > $WorkDirectory /var/log/rsyslogq # default location for work (spool) > files > > > > $ActionQueueType LinkedList # use asynchronous processing > > $ActionQueueFileName dbq # set file name, also enables disk mode > > $ActionResumeRetryCount -1 # infinite retries on insert failure > > > > # send snmpd INFO messages to the dustbin > > > > if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd' > then ~ > > > > $template vtfw,"insert into vtfw (Message, Facility, FromHost, > Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values > ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, > '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, > '%syslogtag%')",SQL > > > > if ($fromhost-ip contains '10.128.255') then > :ommysql:localhost,Syslog,root,mysqldb44;vtfw > > & ~ > > #if ($fromhost-ip contains '10.128.255') then ~ > > > > $template vt1hs1_switches,"insert into vt1hs1_switches (Message, > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', > %syslogpriority%, '%timereported:::date-mysql%', > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > if $fromhost-ip contains '10.128.0' or $hostname contains_i 'core01' > or $hostname contains_i 'core02' then > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches > > #if ($fromhost-ip contains '10.128.0' or $source=='vt1hs1-dc216- > core01' or $source=='vt1hs1-dc216-core02') then /var/log/switches > > & ~ > > #if $fromhost-ip contains '10.128.0' or $hostname contains_i 'core01' > or $hostname contains_i 'core02' then ~ > > > > $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message, Facility, > FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, > SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', > %syslogpriority%, '%timereported:::date-mysql%', > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > if $fromhost-ip contains '10.128.244' then > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi > > & ~ > > #if $fromhost-ip contains '10.128.244' then ~ > > > > $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message, > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', > %syslogpriority%, '%timereported:::date-mysql%', > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == '10.128.24.44' > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' or > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or > $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') and > ($syslogseverity-text == 'warning' or $syslogseverity-text == 'warn' or > $syslogseverity-text == 'err' or $syslogseverity-text == 'error' or > $syslogseverity-text == 'crit' or $syslogseverity-text == 'alert' or > $syslogseverity-text == 'emergency' or $syslogseverity-text == 'panic') > then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere > > & ~ > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == '10.128.24.44' > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' or > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or > $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') and > ($syslogseverity-text == 'notice') then ~ > > > > > > #*.* :ommysql:localhost,Syslog,root,mysqldb44 > > *.notice :ommysql:localhost,Syslog,root,mysqldb44 > > > > # Log all kernel messages to the console. > > # Logging much else clutters up the screen. > > #kern.* /dev/console > > > > # Log anything (except mail) of level info or higher. > > # Don't log private authentication messages! > > *.info;mail.none;authpriv.none;cron.none - > /var/log/messages > > > > # The authpriv file has restricted access. > > authpriv.* > /var/log/secure > > > > # Log all the mail messages in one place. > > mail.* - > /var/log/maillog > > > > > > # Log cron stuff > > cron.* - > /var/log/cron > > > > # Everybody gets emergency messages > > *.emerg * > > > > # Save news errors of level crit and higher in a special file. > > uucp,news.crit - > /var/log/spooler > > > > # Save boot messages also to boot.log > > local7.* > /var/log/boot.log > > > > # Remote Logging (we use TCP for reliable delivery) > > # An on-disk queue is created for this action. If the remote host is > > # down, messages are spooled to disk and sent when it is up again. > > #$WorkDirectory /rsyslog/spool # where to place spool files > > #$ActionQueueFileName uniqName # unique name prefix for spool files > > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as > possible) > > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown > > #$ActionQueueType LinkedList # run asynchronously > > #$ActionResumeRetryCount -1 # infinite retries if host is down > > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > > #*.* @@remote-host:514 > > > > > > # ######### Receiving Messages from Remote Hosts ########## > > # TCP Syslog Server: > > # provides TCP syslog reception and GSS-API (if compiled to support > it) > > $ModLoad imtcp.so # load module > > $InputTCPServerRun 1470 # start up TCP listener at port 514 > > > > # UDP Syslog Server: > > $ModLoad imudp.so # provides UDP syslog reception > > $UDPServerRun 514 # start a UDP syslog server at standard port 514 > > > > > > SOME DEBUG OUTPUT: > > > > (see attached) > > > > > > > > > > > > > > > > > > > > Steve Chupack | IT Systems Administrator > > V: 877.327.8422 x 1242 > > Steve.Chupack at dealer.com | www.dealer.com > > > > From steve.chupack at dealer.com Mon Sep 12 15:02:09 2011 From: steve.chupack at dealer.com (Steve Chupack) Date: Mon, 12 Sep 2011 09:02:09 -0400 Subject: [rsyslog] Unable to implement on-disk queuing In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA72811EE@GRFEXC.intern.adiscon.com> References: <20110911165057.77049b4d@stchupack-m4300> <20110911171031.25bf90cc@stchupack-m4300> <9B6E2A8877C38245BFB15CC491A11DA72811EE@GRFEXC.intern.adiscon.com> Message-ID: <20110912090209.4d864ec6@stchupack-m4300> Hmmmm... the thing is, my $MainMsgQueueSize is only 5, and I had mysql turned off for 20 minutes... And my message flow rate is up around 1 per second. So I would have expected some queuing to happen. Is there anything else I should look at or try? On Mon, 12 Sep 2011 11:13:28 +0200 Rainer Gerhards wrote: > Queue files are only created if necessary. The debug log does not contain any > such situation. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Steve Chupack > > Sent: Sunday, September 11, 2011 11:11 PM > > To: rsyslog at lists.adiscon.com > > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > > > Ooops... previous message's attachment was the config, not the debug > > output. Debug attached. > > > > > > > > On Sun, 11 Sep 2011 16:50:57 -0400 > > Steve Chupack wrote: > > > > > Hi, > > > > > > I have rsyslogd up and running and logging to mysql with loganalyzer > > as a front end. Very cool. > > > > > > However, I can't seem to get on-disk queuing working, which would be > > nice if mysql goes down or a table is locked because I'm purging > > records during a nightly maintenance cron. > > > > > > I never see any queue files created. I dropped the size of the memory > > queue to something ridiculously small, but still no queue files even > > after 20 minutes. > > > > > > I followed the instructions given here: > > http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along with a > > couple of other tutorials on rsyslogd and queuing. > > > > > > I'm quite sure I'm just completely missing something obvious. Details > > below... And thanks to anyone who might be able to point out what I'm > > doing wrong. > > > > > > -Steve > > > > > > > > > > > > MY CONFIG, please, no flames re the horrendously complex rules ;-) > > > > > > # if you experience problems, check > > > # http://www.rsyslog.com/troubleshoot for assistance > > > > > > # rsyslog v3: load input modules > > > # If you do not load inputs, nothing happens! > > > # You may need to set the module load path if modules are not found. > > > > > > $ModLoad immark # provides --MARK-- message capability > > > $ModLoad imuxsock # provides support for local system logging (e.g. > > via logger command) > > > $ModLoad imklog # kernel logging (formerly provided by rklogd) > > > $ModLoad ommysql > > > > > > # SC 2010.11.11: configure disk caching in case mysql is unavailable > > > > > > $MainMsgQueueSize 5 > > > > > > $WorkDirectory /var/log/rsyslogq # default location for work (spool) > > files > > > > > > $ActionQueueType LinkedList # use asynchronous processing > > > $ActionQueueFileName dbq # set file name, also enables disk mode > > > $ActionResumeRetryCount -1 # infinite retries on insert failure > > > > > > # send snmpd INFO messages to the dustbin > > > > > > if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd' > > then ~ > > > > > > $template vtfw,"insert into vtfw (Message, Facility, FromHost, > > Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values > > ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, > > '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, > > '%syslogtag%')",SQL > > > > > > if ($fromhost-ip contains '10.128.255') then > > :ommysql:localhost,Syslog,root,mysqldb44;vtfw > > > & ~ > > > #if ($fromhost-ip contains '10.128.255') then ~ > > > > > > $template vt1hs1_switches,"insert into vt1hs1_switches (Message, > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', > > %syslogpriority%, '%timereported:::date-mysql%', > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > if $fromhost-ip contains '10.128.0' or $hostname contains_i 'core01' > > or $hostname contains_i 'core02' then > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches > > > #if ($fromhost-ip contains '10.128.0' or $source=='vt1hs1-dc216- > > core01' or $source=='vt1hs1-dc216-core02') then /var/log/switches > > > & ~ > > > #if $fromhost-ip contains '10.128.0' or $hostname contains_i 'core01' > > or $hostname contains_i 'core02' then ~ > > > > > > $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message, Facility, > > FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, > > SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', > > %syslogpriority%, '%timereported:::date-mysql%', > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > if $fromhost-ip contains '10.128.244' then > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi > > > & ~ > > > #if $fromhost-ip contains '10.128.244' then ~ > > > > > > $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message, > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', > > %syslogpriority%, '%timereported:::date-mysql%', > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == '10.128.24.44' > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' or > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or > > $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') and > > ($syslogseverity-text == 'warning' or $syslogseverity-text == 'warn' or > > $syslogseverity-text == 'err' or $syslogseverity-text == 'error' or > > $syslogseverity-text == 'crit' or $syslogseverity-text == 'alert' or > > $syslogseverity-text == 'emergency' or $syslogseverity-text == 'panic') > > then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere > > > & ~ > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == '10.128.24.44' > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' or > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or > > $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') and > > ($syslogseverity-text == 'notice') then ~ > > > > > > > > > #*.* :ommysql:localhost,Syslog,root,mysqldb44 > > > *.notice :ommysql:localhost,Syslog,root,mysqldb44 > > > > > > # Log all kernel messages to the console. > > > # Logging much else clutters up the screen. > > > #kern.* /dev/console > > > > > > # Log anything (except mail) of level info or higher. > > > # Don't log private authentication messages! > > > *.info;mail.none;authpriv.none;cron.none - > > /var/log/messages > > > > > > # The authpriv file has restricted access. > > > authpriv.* > > /var/log/secure > > > > > > # Log all the mail messages in one place. > > > mail.* - > > /var/log/maillog > > > > > > > > > # Log cron stuff > > > cron.* - > > /var/log/cron > > > > > > # Everybody gets emergency messages > > > *.emerg * > > > > > > # Save news errors of level crit and higher in a special file. > > > uucp,news.crit - > > /var/log/spooler > > > > > > # Save boot messages also to boot.log > > > local7.* > > /var/log/boot.log > > > > > > # Remote Logging (we use TCP for reliable delivery) > > > # An on-disk queue is created for this action. If the remote host is > > > # down, messages are spooled to disk and sent when it is up again. > > > #$WorkDirectory /rsyslog/spool # where to place spool files > > > #$ActionQueueFileName uniqName # unique name prefix for spool files > > > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as > > possible) > > > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown > > > #$ActionQueueType LinkedList # run asynchronously > > > #$ActionResumeRetryCount -1 # infinite retries if host is down > > > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > > > #*.* @@remote-host:514 > > > > > > > > > # ######### Receiving Messages from Remote Hosts ########## > > > # TCP Syslog Server: > > > # provides TCP syslog reception and GSS-API (if compiled to support > > it) > > > $ModLoad imtcp.so # load module > > > $InputTCPServerRun 1470 # start up TCP listener at port 514 > > > > > > # UDP Syslog Server: > > > $ModLoad imudp.so # provides UDP syslog reception > > > $UDPServerRun 514 # start a UDP syslog server at standard port 514 > > > > > > > > > SOME DEBUG OUTPUT: > > > > > > (see attached) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Steve Chupack | IT Systems Administrator > > > V: 877.327.8422 x 1242 > > > Steve.Chupack at dealer.com | www.dealer.com > > > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Sep 12 15:18:32 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 12 Sep 2011 15:18:32 +0200 Subject: [rsyslog] Unable to implement on-disk queuing In-Reply-To: <20110912090209.4d864ec6@stchupack-m4300> References: <20110911165057.77049b4d@stchupack-m4300><20110911171031.25bf90cc@stchupack-m4300><9B6E2A8877C38245BFB15CC491A11DA72811EE@GRFEXC.intern.adiscon.com> <20110912090209.4d864ec6@stchupack-m4300> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72811F1@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Steve Chupack > Sent: Monday, September 12, 2011 3:02 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > Hmmmm... the thing is, my $MainMsgQueueSize is only 5, and I had mysql > turned off for 20 minutes... And my message flow rate is up around 1 > per second. So I would have expected some queuing to happen. Is there > anything else I should look at or try? I think the unusual extremely low queue sizes results in some watermarks set to 0 and inhibiting DA queue startup. Set the main queue size to, say, 500 or 1000 and try again. Note that you can use logger or the tcpflood tool to injects lots of message in a short period. Rainer > > On Mon, 12 Sep 2011 11:13:28 +0200 > Rainer Gerhards wrote: > > > Queue files are only created if necessary. The debug log does not > contain any > > such situation. > > > > Rainer > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Steve Chupack > > > Sent: Sunday, September 11, 2011 11:11 PM > > > To: rsyslog at lists.adiscon.com > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > > > > > Ooops... previous message's attachment was the config, not the > debug > > > output. Debug attached. > > > > > > > > > > > > On Sun, 11 Sep 2011 16:50:57 -0400 > > > Steve Chupack wrote: > > > > > > > Hi, > > > > > > > > I have rsyslogd up and running and logging to mysql with > loganalyzer > > > as a front end. Very cool. > > > > > > > > However, I can't seem to get on-disk queuing working, which would > be > > > nice if mysql goes down or a table is locked because I'm purging > > > records during a nightly maintenance cron. > > > > > > > > I never see any queue files created. I dropped the size of the > memory > > > queue to something ridiculously small, but still no queue files > even > > > after 20 minutes. > > > > > > > > I followed the instructions given here: > > > http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along > with a > > > couple of other tutorials on rsyslogd and queuing. > > > > > > > > I'm quite sure I'm just completely missing something obvious. > Details > > > below... And thanks to anyone who might be able to point out what > I'm > > > doing wrong. > > > > > > > > -Steve > > > > > > > > > > > > > > > > MY CONFIG, please, no flames re the horrendously complex rules ;- > ) > > > > > > > > # if you experience problems, check > > > > # http://www.rsyslog.com/troubleshoot for assistance > > > > > > > > # rsyslog v3: load input modules > > > > # If you do not load inputs, nothing happens! > > > > # You may need to set the module load path if modules are not > found. > > > > > > > > $ModLoad immark # provides --MARK-- message capability > > > > $ModLoad imuxsock # provides support for local system logging > (e.g. > > > via logger command) > > > > $ModLoad imklog # kernel logging (formerly provided by rklogd) > > > > $ModLoad ommysql > > > > > > > > # SC 2010.11.11: configure disk caching in case mysql is > unavailable > > > > > > > > $MainMsgQueueSize 5 > > > > > > > > $WorkDirectory /var/log/rsyslogq # default location for work > (spool) > > > files > > > > > > > > $ActionQueueType LinkedList # use asynchronous processing > > > > $ActionQueueFileName dbq # set file name, also enables disk > mode > > > > $ActionResumeRetryCount -1 # infinite retries on insert failure > > > > > > > > # send snmpd INFO messages to the dustbin > > > > > > > > if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd' > > > then ~ > > > > > > > > $template vtfw,"insert into vtfw (Message, Facility, FromHost, > > > Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) > values > > > ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, > > > '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', > %iut%, > > > '%syslogtag%')",SQL > > > > > > > > if ($fromhost-ip contains '10.128.255') then > > > :ommysql:localhost,Syslog,root,mysqldb44;vtfw > > > > & ~ > > > > #if ($fromhost-ip contains '10.128.255') then ~ > > > > > > > > $template vt1hs1_switches,"insert into vt1hs1_switches (Message, > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, > '%HOSTNAME%', > > > %syslogpriority%, '%timereported:::date-mysql%', > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > if $fromhost-ip contains '10.128.0' or $hostname contains_i > 'core01' > > > or $hostname contains_i 'core02' then > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches > > > > #if ($fromhost-ip contains '10.128.0' or $source=='vt1hs1-dc216- > > > core01' or $source=='vt1hs1-dc216-core02') then /var/log/switches > > > > & ~ > > > > #if $fromhost-ip contains '10.128.0' or $hostname contains_i > 'core01' > > > or $hostname contains_i 'core02' then ~ > > > > > > > > $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message, > Facility, > > > FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, > > > SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', > > > %syslogpriority%, '%timereported:::date-mysql%', > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > if $fromhost-ip contains '10.128.244' then > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi > > > > & ~ > > > > #if $fromhost-ip contains '10.128.244' then ~ > > > > > > > > $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message, > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, > '%HOSTNAME%', > > > %syslogpriority%, '%timereported:::date-mysql%', > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == > '10.128.24.44' > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' > or > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or > > > $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') > and > > > ($syslogseverity-text == 'warning' or $syslogseverity-text == > 'warn' or > > > $syslogseverity-text == 'err' or $syslogseverity-text == 'error' or > > > $syslogseverity-text == 'crit' or $syslogseverity-text == 'alert' > or > > > $syslogseverity-text == 'emergency' or $syslogseverity-text == > 'panic') > > > then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere > > > > & ~ > > > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == > '10.128.24.44' > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' > or > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or > > > $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') > and > > > ($syslogseverity-text == 'notice') then ~ > > > > > > > > > > > > #*.* :ommysql:localhost,Syslog,root,mysqldb44 > > > > *.notice :ommysql:localhost,Syslog,root,mysqldb44 > > > > > > > > # Log all kernel messages to the console. > > > > # Logging much else clutters up the screen. > > > > #kern.* > /dev/console > > > > > > > > # Log anything (except mail) of level info or higher. > > > > # Don't log private authentication messages! > > > > *.info;mail.none;authpriv.none;cron.none - > > > /var/log/messages > > > > > > > > # The authpriv file has restricted access. > > > > authpriv.* > > > /var/log/secure > > > > > > > > # Log all the mail messages in one place. > > > > mail.* - > > > /var/log/maillog > > > > > > > > > > > > # Log cron stuff > > > > cron.* - > > > /var/log/cron > > > > > > > > # Everybody gets emergency messages > > > > *.emerg * > > > > > > > > # Save news errors of level crit and higher in a special file. > > > > uucp,news.crit - > > > /var/log/spooler > > > > > > > > # Save boot messages also to boot.log > > > > local7.* > > > /var/log/boot.log > > > > > > > > # Remote Logging (we use TCP for reliable delivery) > > > > # An on-disk queue is created for this action. If the remote host > is > > > > # down, messages are spooled to disk and sent when it is up > again. > > > > #$WorkDirectory /rsyslog/spool # where to place spool files > > > > #$ActionQueueFileName uniqName # unique name prefix for spool > files > > > > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as > > > possible) > > > > #$ActionQueueSaveOnShutdown on # save messages to disk on > shutdown > > > > #$ActionQueueType LinkedList # run asynchronously > > > > #$ActionResumeRetryCount -1 # infinite retries if host is down > > > > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port > optional > > > > #*.* @@remote-host:514 > > > > > > > > > > > > # ######### Receiving Messages from Remote Hosts ########## > > > > # TCP Syslog Server: > > > > # provides TCP syslog reception and GSS-API (if compiled to > support > > > it) > > > > $ModLoad imtcp.so # load module > > > > $InputTCPServerRun 1470 # start up TCP listener at port 514 > > > > > > > > # UDP Syslog Server: > > > > $ModLoad imudp.so # provides UDP syslog reception > > > > $UDPServerRun 514 # start a UDP syslog server at standard port > 514 > > > > > > > > > > > > SOME DEBUG OUTPUT: > > > > > > > > (see attached) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Steve Chupack | IT Systems Administrator > > > > V: 877.327.8422 x 1242 > > > > Steve.Chupack at dealer.com | www.dealer.com > > > > > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From steve.chupack at dealer.com Mon Sep 12 16:39:18 2011 From: steve.chupack at dealer.com (Steve Chupack) Date: Mon, 12 Sep 2011 10:39:18 -0400 Subject: [rsyslog] Unable to implement on-disk queuing In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA72811F1@GRFEXC.intern.adiscon.com> References: <20110911165057.77049b4d@stchupack-m4300> <20110911171031.25bf90cc@stchupack-m4300> <9B6E2A8877C38245BFB15CC491A11DA72811EE@GRFEXC.intern.adiscon.com> <20110912090209.4d864ec6@stchupack-m4300> <9B6E2A8877C38245BFB15CC491A11DA72811F1@GRFEXC.intern.adiscon.com> Message-ID: <20110912103918.4a41fd18@stchupack-m4300> Thanks for taking the time to respond, I really appreciate it. OK, I think I can better describe this now. You are correct, rsyslog is not seeing a need to queue. My debug log is filled with the following entries, which clearly indicate it sees no need to queue or spool. So again, I think I am missing something very basic here... 7878.780426000:455e1940: main Q: entry added, size now 1 entries 7878.780439000:455e1940: wtpAdviseMaxWorkers signals busy 7878.780566000:455e1940: main Q: EnqueueMsg advised worker start 7878.780585000:423dc940: main Q: entry deleted, state 0, size now 0 entries I used the very basic config from your article on buffering. $MainMsgQueueSize 500 $WorkDirectory /var/log/rsyslogq # default location for work (spool) files $MainMsgQueueFileName mainq # set file name, also enables disk mode $ActionResumeRetryCount -1 # infinite retries on insert failure Mysql is definitely not running, as show by the following: 8220.440850000:41537940: Called LogError, msg: db error (2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) rsyslogd: db error (2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) 8220.440865000:41537940: logmsg: flags 1, from 'vt1hs1-netservices01', msg db error (2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) 8220.441040000:41537940: Called action, logging to ommysql 8220.573450000:41537940: Called action, logging to ommysql 8220.574801000:41537940: Called action, logging to ommysql 8220.602406000:41537940: Called action, logging to ommysql 8220.604330000:41537940: Called action, logging to ommysql On Mon, 12 Sep 2011 15:18:32 +0200 Rainer Gerhards wrote: > > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Steve Chupack > > Sent: Monday, September 12, 2011 3:02 PM > > To: rsyslog at lists.adiscon.com > > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > > > Hmmmm... the thing is, my $MainMsgQueueSize is only 5, and I had mysql > > turned off for 20 minutes... And my message flow rate is up around 1 > > per second. So I would have expected some queuing to happen. Is there > > anything else I should look at or try? > > I think the unusual extremely low queue sizes results in some watermarks set > to 0 and inhibiting DA queue startup. Set the main queue size to, say, 500 > or 1000 and try again. Note that you can use logger or the tcpflood tool to > injects lots of message in a short period. > > Rainer > > > > > On Mon, 12 Sep 2011 11:13:28 +0200 > > Rainer Gerhards wrote: > > > > > Queue files are only created if necessary. The debug log does not > > contain any > > > such situation. > > > > > > Rainer > > > > > > > -----Original Message----- > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > > bounces at lists.adiscon.com] On Behalf Of Steve Chupack > > > > Sent: Sunday, September 11, 2011 11:11 PM > > > > To: rsyslog at lists.adiscon.com > > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > > > > > > > Ooops... previous message's attachment was the config, not the > > debug > > > > output. Debug attached. > > > > > > > > > > > > > > > > On Sun, 11 Sep 2011 16:50:57 -0400 > > > > Steve Chupack wrote: > > > > > > > > > Hi, > > > > > > > > > > I have rsyslogd up and running and logging to mysql with > > loganalyzer > > > > as a front end. Very cool. > > > > > > > > > > However, I can't seem to get on-disk queuing working, which would > > be > > > > nice if mysql goes down or a table is locked because I'm purging > > > > records during a nightly maintenance cron. > > > > > > > > > > I never see any queue files created. I dropped the size of the > > memory > > > > queue to something ridiculously small, but still no queue files > > even > > > > after 20 minutes. > > > > > > > > > > I followed the instructions given here: > > > > http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along > > with a > > > > couple of other tutorials on rsyslogd and queuing. > > > > > > > > > > I'm quite sure I'm just completely missing something obvious. > > Details > > > > below... And thanks to anyone who might be able to point out what > > I'm > > > > doing wrong. > > > > > > > > > > -Steve > > > > > > > > > > > > > > > > > > > > MY CONFIG, please, no flames re the horrendously complex rules ;- > > ) > > > > > > > > > > # if you experience problems, check > > > > > # http://www.rsyslog.com/troubleshoot for assistance > > > > > > > > > > # rsyslog v3: load input modules > > > > > # If you do not load inputs, nothing happens! > > > > > # You may need to set the module load path if modules are not > > found. > > > > > > > > > > $ModLoad immark # provides --MARK-- message capability > > > > > $ModLoad imuxsock # provides support for local system logging > > (e.g. > > > > via logger command) > > > > > $ModLoad imklog # kernel logging (formerly provided by rklogd) > > > > > $ModLoad ommysql > > > > > > > > > > # SC 2010.11.11: configure disk caching in case mysql is > > unavailable > > > > > > > > > > $MainMsgQueueSize 5 > > > > > > > > > > $WorkDirectory /var/log/rsyslogq # default location for work > > (spool) > > > > files > > > > > > > > > > $ActionQueueType LinkedList # use asynchronous processing > > > > > $ActionQueueFileName dbq # set file name, also enables disk > > mode > > > > > $ActionResumeRetryCount -1 # infinite retries on insert failure > > > > > > > > > > # send snmpd INFO messages to the dustbin > > > > > > > > > > if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd' > > > > then ~ > > > > > > > > > > $template vtfw,"insert into vtfw (Message, Facility, FromHost, > > > > Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) > > values > > > > ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, > > > > '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', > > %iut%, > > > > '%syslogtag%')",SQL > > > > > > > > > > if ($fromhost-ip contains '10.128.255') then > > > > :ommysql:localhost,Syslog,root,mysqldb44;vtfw > > > > > & ~ > > > > > #if ($fromhost-ip contains '10.128.255') then ~ > > > > > > > > > > $template vt1hs1_switches,"insert into vt1hs1_switches (Message, > > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, > > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, > > '%HOSTNAME%', > > > > %syslogpriority%, '%timereported:::date-mysql%', > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > > > if $fromhost-ip contains '10.128.0' or $hostname contains_i > > 'core01' > > > > or $hostname contains_i 'core02' then > > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches > > > > > #if ($fromhost-ip contains '10.128.0' or $source=='vt1hs1-dc216- > > > > core01' or $source=='vt1hs1-dc216-core02') then /var/log/switches > > > > > & ~ > > > > > #if $fromhost-ip contains '10.128.0' or $hostname contains_i > > 'core01' > > > > or $hostname contains_i 'core02' then ~ > > > > > > > > > > $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message, > > Facility, > > > > FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, > > > > SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', > > > > %syslogpriority%, '%timereported:::date-mysql%', > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > > > if $fromhost-ip contains '10.128.244' then > > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi > > > > > & ~ > > > > > #if $fromhost-ip contains '10.128.244' then ~ > > > > > > > > > > $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message, > > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, > > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, > > '%HOSTNAME%', > > > > %syslogpriority%, '%timereported:::date-mysql%', > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == > > '10.128.24.44' > > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' > > or > > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or > > > > $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') > > and > > > > ($syslogseverity-text == 'warning' or $syslogseverity-text == > > 'warn' or > > > > $syslogseverity-text == 'err' or $syslogseverity-text == 'error' or > > > > $syslogseverity-text == 'crit' or $syslogseverity-text == 'alert' > > or > > > > $syslogseverity-text == 'emergency' or $syslogseverity-text == > > 'panic') > > > > then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere > > > > > & ~ > > > > > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == > > '10.128.24.44' > > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' > > or > > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or > > > > $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') > > and > > > > ($syslogseverity-text == 'notice') then ~ > > > > > > > > > > > > > > > #*.* :ommysql:localhost,Syslog,root,mysqldb44 > > > > > *.notice :ommysql:localhost,Syslog,root,mysqldb44 > > > > > > > > > > # Log all kernel messages to the console. > > > > > # Logging much else clutters up the screen. > > > > > #kern.* > > /dev/console > > > > > > > > > > # Log anything (except mail) of level info or higher. > > > > > # Don't log private authentication messages! > > > > > *.info;mail.none;authpriv.none;cron.none - > > > > /var/log/messages > > > > > > > > > > # The authpriv file has restricted access. > > > > > authpriv.* > > > > /var/log/secure > > > > > > > > > > # Log all the mail messages in one place. > > > > > mail.* - > > > > /var/log/maillog > > > > > > > > > > > > > > > # Log cron stuff > > > > > cron.* - > > > > /var/log/cron > > > > > > > > > > # Everybody gets emergency messages > > > > > *.emerg * > > > > > > > > > > # Save news errors of level crit and higher in a special file. > > > > > uucp,news.crit - > > > > /var/log/spooler > > > > > > > > > > # Save boot messages also to boot.log > > > > > local7.* > > > > /var/log/boot.log > > > > > > > > > > # Remote Logging (we use TCP for reliable delivery) > > > > > # An on-disk queue is created for this action. If the remote host > > is > > > > > # down, messages are spooled to disk and sent when it is up > > again. > > > > > #$WorkDirectory /rsyslog/spool # where to place spool files > > > > > #$ActionQueueFileName uniqName # unique name prefix for spool > > files > > > > > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as > > > > possible) > > > > > #$ActionQueueSaveOnShutdown on # save messages to disk on > > shutdown > > > > > #$ActionQueueType LinkedList # run asynchronously > > > > > #$ActionResumeRetryCount -1 # infinite retries if host is down > > > > > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port > > optional > > > > > #*.* @@remote-host:514 > > > > > > > > > > > > > > > # ######### Receiving Messages from Remote Hosts ########## > > > > > # TCP Syslog Server: > > > > > # provides TCP syslog reception and GSS-API (if compiled to > > support > > > > it) > > > > > $ModLoad imtcp.so # load module > > > > > $InputTCPServerRun 1470 # start up TCP listener at port 514 > > > > > > > > > > # UDP Syslog Server: > > > > > $ModLoad imudp.so # provides UDP syslog reception > > > > > $UDPServerRun 514 # start a UDP syslog server at standard port > > 514 > > > > > > > > > > > > > > > SOME DEBUG OUTPUT: > > > > > > > > > > (see attached) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Steve Chupack | IT Systems Administrator > > > > > V: 877.327.8422 x 1242 > > > > > Steve.Chupack at dealer.com | www.dealer.com > > > > > > > > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Sep 12 17:24:16 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 12 Sep 2011 17:24:16 +0200 Subject: [rsyslog] Unable to implement on-disk queuing Message-ID: <002e01cc715f$e40487b3$100013ac@intern.adiscon.com> Can you mail me the complete debug log, please. Rainer Steve Chupack hat geschrieben:Thanks for taking the time to respond, I really appreciate it. OK, I think I can better describe this now. You are correct, rsyslog is not seeing a need to queue. My debug log is filled with the following entries, which clearly indicate it sees no need to queue or spool. So again, I think I am missing something very basic here... 7878.780426000:455e1940: main Q: entry added, size now 1 entries 7878.780439000:455e1940: wtpAdviseMaxWorkers signals busy 7878.780566000:455e1940: main Q: EnqueueMsg advised worker start 7878.780585000:423dc940: main Q: entry deleted, state 0, size now 0 entries I used the very basic config from your article on buffering. $MainMsgQueueSize 500 $WorkDirectory /var/log/rsyslogq # default location for work (spool) files $MainMsgQueueFileName mainq # set file name, also enables disk mode $ActionResumeRetryCount -1 # infinite retries on insert failure Mysql is definitely not running, as show by the following: 8220.440850000:41537940: Called LogError, msg: db error (2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) rsyslogd: db error (2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) 8220.440865000:41537940: logmsg: flags 1, from 'vt1hs1-netservices01', msg db error (2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) 8220.441040000:41537940: Called action, logging to ommysql 8220.573450000:41537940: Called action, logging to ommysql 8220.574801000:41537940: Called action, logging to ommysql 8220.602406000:41537940: Called action, logging to ommysql 8220.604330000:41537940: Called action, logging to ommysql On Mon, 12 Sep 2011 15:18:32 +0200 Rainer Gerhards wrote: > > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Steve Chupack > > Sent: Monday, September 12, 2011 3:02 PM > > To: rsyslog at lists.adiscon.com > > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > > > Hmmmm... the thing is, my $MainMsgQueueSize is only 5, and I had mysql > > turned off for 20 minutes... And my message flow rate is up around 1 > > per second. So I would have expected some queuing to happen. Is there > > anything else I should look at or try? > > I think the unusual extremely low queue sizes results in some watermarks set > to 0 and inhibiting DA queue startup. Set the main queue size to, say, 500 > or 1000 and try again. Note that you can use logger or the tcpflood tool to > injects lots of message in a short period. > > Rainer > > > > > On Mon, 12 Sep 2011 11:13:28 +0200 > > Rainer Gerhards wrote: > > > > > Queue files are only created if necessary. The debug log does not > > contain any > > > such situation. > > > > > > Rainer > > > > > > > -----Original Message----- > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > > bounces at lists.adiscon.com] On Behalf Of Steve Chupack > > > > Sent: Sunday, September 11, 2011 11:11 PM > > > > To: rsyslog at lists.adiscon.com > > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > > > > > > > Ooops... previous message's attachment was the config, not the > > debug > > > > output. Debug attached. > > > > > > > > > > > > > > > > On Sun, 11 Sep 2011 16:50:57 -0400 > > > > Steve Chupack wrote: > > > > > > > > > Hi, > > > > > > > > > > I have rsyslogd up and running and logging to mysql with > > loganalyzer > > > > as a front end. Very cool. > > > > > > > > > > However, I can't seem to get on-disk queuing working, which would > > be > > > > nice if mysql goes down or a table is locked because I'm purging > > > > records during a nightly maintenance cron. > > > > > > > > > > I never see any queue files created. I dropped the size of the > > memory > > > > queue to something ridiculously small, but still no queue files > > even > > > > after 20 minutes. > > > > > > > > > > I followed the instructions given here: > > > > http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along > > with a > > > > couple of other tutorials on rsyslogd and queuing. > > > > > > > > > > I'm quite sure I'm just completely missing something obvious. > > Details > > > > below... And thanks to anyone who might be able to point out what > > I'm > > > > doing wrong. > > > > > > > > > > -Steve > > > > > > > > > > > > > > > > > > > > MY CONFIG, please, no flames re the horrendously complex rules ;- > > ) > > > > > > > > > > # if you experience problems, check > > > > > # http://www.rsyslog.com/troubleshoot for assistance > > > > > > > > > > # rsyslog v3: load input modules > > > > > # If you do not load inputs, nothing happens! > > > > > # You may need to set the module load path if modules are not > > found. > > > > > > > > > > $ModLoad immark # provides --MARK-- message capability > > > > > $ModLoad imuxsock # provides support for local system logging > > (e.g. > > > > via logger command) > > > > > $ModLoad imklog # kernel logging (formerly provided by rklogd) > > > > > $ModLoad ommysql > > > > > > > > > > # SC 2010.11.11: configure disk caching in case mysql is > > unavailable > > > > > > > > > > $MainMsgQueueSize 5 > > > > > > > > > > $WorkDirectory /var/log/rsyslogq # default location for work > > (spool) > > > > files > > > > > > > > > > $ActionQueueType LinkedList # use asynchronous processing > > > > > $ActionQueueFileName dbq # set file name, also enables disk > > mode > > > > > $ActionResumeRetryCount -1 # infinite retries on insert failure > > > > > > > > > > # send snmpd INFO messages to the dustbin > > > > > > > > > > if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd' > > > > then ~ > > > > > > > > > > $template vtfw,"insert into vtfw (Message, Facility, FromHost, > > > > Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) > > values > > > > ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, > > > > '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', > > %iut%, > > > > '%syslogtag%')",SQL > > > > > > > > > > if ($fromhost-ip contains '10.128.255') then > > > > :ommysql:localhost,Syslog,root,mysqldb44;vtfw > > > > > & ~ > > > > > #if ($fromhost-ip contains '10.128.255') then ~ > > > > > > > > > > $template vt1hs1_switches,"insert into vt1hs1_switches (Message, > > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, > > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, > > '%HOSTNAME%', > > > > %syslogpriority%, '%timereported:::date-mysql%', > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > > > if $fromhost-ip contains '10.128.0' or $hostname contains_i > > 'core01' > > > > or $hostname contains_i 'core02' then > > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches > > > > > #if ($fromhost-ip contains '10.128.0' or $source=='vt1hs1-dc216- > > > > core01' or $source=='vt1hs1-dc216-core02') then /var/log/switches > > > > > & ~ > > > > > #if $fromhost-ip contains '10.128.0' or $hostname contains_i > > 'core01' > > > > or $hostname contains_i 'core02' then ~ > > > > > > > > > > $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message, > > Facility, > > > > FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, > > > > SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', > > > > %syslogpriority%, '%timereported:::date-mysql%', > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > > > if $fromhost-ip contains '10.128.244' then > > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi > > > > > & ~ > > > > > #if $fromhost-ip contains '10.128.244' then ~ > > > > > > > > > > $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message, > > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, > > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, > > '%HOSTNAME%', > > > > %syslogpriority%, '%timereported:::date-mysql%', > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == > > '10.128.24.44' > > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' > > or > > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or > > > > $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') > > and > > > > ($syslogseverity-text == 'warning' or $syslogseverity-text == > > 'warn' or > > > > $syslogseverity-text == 'err' or $syslogseverity-text == 'error' or > > > > $syslogseverity-text == 'crit' or $syslogseverity-text == 'alert' > > or > > > > $syslogseverity-text == 'emergency' or $syslogseverity-text == > > 'panic') > > > > then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere > > > > > & ~ > > > > > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == > > '10.128.24.44' > > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' > > or > > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or > > > > $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') > > and > > > > ($syslogseverity-text == 'notice') then ~ > > > > > > > > > > > > > > > #*.* :ommysql:localhost,Syslog,root,mysqldb44 > > > > > *.notice :ommysql:localhost,Syslog,root,mysqldb44 > > > > > > > > > > # Log all kernel messages to the console. > > > > > # Logging much else clutters up the screen. > > > > > #kern.* > > /dev/console > > > > > > > > > > # Log anything (except mail) of level info or higher. > > > > > # Don't log private authentication messages! > > > > > *.info;mail.none;authpriv.none;cron.none - > > > > /var/log/messages > > > > > > > > > > # The authpriv file has restricted access. > > > > > authpriv.* > > > > /var/log/secure > > > > > > > > > > # Log all the mail messages in one place. > > > > > mail.* - > > > > /var/log/maillog > > > > > > > > > > > > > > > # Log cron stuff > > > > > cron.* - > > > > /var/log/cron > > > > > > > > > > # Everybody gets emergency messages > > > > > *.emerg * > > > > > > > > > > # Save news errors of level crit and higher in a special file. > > > > > uucp,news.crit - > > > > /var/log/spooler > > > > > > > > > > # Save boot messages also to boot.log > > > > > local7.* > > > > /var/log/boot.log > > > > > > > > > > # Remote Logging (we use TCP for reliable delivery) > > > > > # An on-disk queue is created for this action. If the remote host > > is > > > > > # down, messages are spooled to disk and sent when it is up > > again. > > > > > #$WorkDirectory /rsyslog/spool # where to place spool files > > > > > #$ActionQueueFileName uniqName # unique name prefix for spool > > files > > > > > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as > > > > possible) > > > > > #$ActionQueueSaveOnShutdown on # save messages to disk on > > shutdown > > > > > #$ActionQueueType LinkedList # run asynchronously > > > > > #$ActionResumeRetryCount -1 # infinite retries if host is down > > > > > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port > > optional > > > > > #*.* @@remote-host:514 > > > > > > > > > > > > > > > # ######### Receiving Messages from Remote Hosts ########## > > > > > # TCP Syslog Server: > > > > > # provides TCP syslog reception and GSS-API (if compiled to > > support > > > > it) > > > > > $ModLoad imtcp.so # load module > > > > > $InputTCPServerRun 1470 # start up TCP listener at port 514 > > > > > > > > > > # UDP Syslog Server: > > > > > $ModLoad imudp.so # provides UDP syslog reception > > > > > $UDPServerRun 514 # start a UDP syslog server at standard port > > 514 > > > > > > > > > > > > > > > SOME DEBUG OUTPUT: > > > > > > > > > > (see attached) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Steve Chupack | IT Systems Administrator > > > > > V: 877.327.8422 x 1242 > > > > > Steve.Chupack at dealer.com | www.dealer.com > > > > > > > > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Sep 12 19:00:28 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 12 Sep 2011 19:00:28 +0200 Subject: [rsyslog] Unable to implement on-disk queuing In-Reply-To: <20110912103918.4a41fd18@stchupack-m4300> References: <20110911165057.77049b4d@stchupack-m4300><20110911171031.25bf90cc@stchupack-m4300><9B6E2A8877C38245BFB15CC491A11DA72811EE@GRFEXC.intern.adiscon.com><20110912090209.4d864ec6@stchupack-m4300><9B6E2A8877C38245BFB15CC491A11DA72811F1@GRFEXC.intern.adiscon.com> <20110912103918.4a41fd18@stchupack-m4300> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72811F4@GRFEXC.intern.adiscon.com> I think I saw the problem in the config as exposed by the debug log. If I saw correctly, you use $ActionResumeRetryCount -1 Some filter ~ Some filter :ommysql: The retry count is of type auto-reset. So the unlimited retries apply to the first action (the discard) and the sql writer uses the default value, which is to give up fairly quickly. It is important that you group these statements right in front of the actual action you want them to apply to -- actually, they *are* part of that action. I know its cumbersome, and things have and will continue to improve greatly in v6. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Steve Chupack > Sent: Monday, September 12, 2011 4:39 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > Thanks for taking the time to respond, I really appreciate it. > > OK, I think I can better describe this now. You are correct, rsyslog is not > seeing a need to queue. My debug log is filled with the following entries, > which clearly indicate it sees no need to queue or spool. So again, I think I am > missing something very basic here... > > > 7878.780426000:455e1940: main Q: entry added, size now 1 entries > 7878.780439000:455e1940: wtpAdviseMaxWorkers signals busy > 7878.780566000:455e1940: main Q: EnqueueMsg advised worker start > 7878.780585000:423dc940: main Q: entry deleted, state 0, size now 0 entries > > I used the very basic config from your article on buffering. > > $MainMsgQueueSize 500 > $WorkDirectory /var/log/rsyslogq # default location for work (spool) files > $MainMsgQueueFileName mainq # set file name, also enables disk mode > $ActionResumeRetryCount -1 # infinite retries on insert failure > > > Mysql is definitely not running, as show by the following: > > 8220.440850000:41537940: Called LogError, msg: db error (2002): Can't > connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) > rsyslogd: db error (2002): Can't connect to local MySQL server through socket > '/var/lib/mysql/mysql.sock' (2) > 8220.440865000:41537940: logmsg: flags 1, from 'vt1hs1-netservices01', msg > db error (2002): Can't connect to local MySQL server through socket > '/var/lib/mysql/mysql.sock' (2) > 8220.441040000:41537940: Called action, logging to ommysql > 8220.573450000:41537940: Called action, logging to ommysql > 8220.574801000:41537940: Called action, logging to ommysql > 8220.602406000:41537940: Called action, logging to ommysql > 8220.604330000:41537940: Called action, logging to ommysql > > > > On Mon, 12 Sep 2011 15:18:32 +0200 > Rainer Gerhards wrote: > > > > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Steve Chupack > > > Sent: Monday, September 12, 2011 3:02 PM > > > To: rsyslog at lists.adiscon.com > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > > > > > Hmmmm... the thing is, my $MainMsgQueueSize is only 5, and I had > > > mysql turned off for 20 minutes... And my message flow rate is up > > > around 1 per second. So I would have expected some queuing to > > > happen. Is there anything else I should look at or try? > > > > I think the unusual extremely low queue sizes results in some > > watermarks set to 0 and inhibiting DA queue startup. Set the main > > queue size to, say, 500 or 1000 and try again. Note that you can use > > logger or the tcpflood tool to injects lots of message in a short period. > > > > Rainer > > > > > > > > On Mon, 12 Sep 2011 11:13:28 +0200 > > > Rainer Gerhards wrote: > > > > > > > Queue files are only created if necessary. The debug log does not > > > contain any > > > > such situation. > > > > > > > > Rainer > > > > > > > > > -----Original Message----- > > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > > > bounces at lists.adiscon.com] On Behalf Of Steve Chupack > > > > > Sent: Sunday, September 11, 2011 11:11 PM > > > > > To: rsyslog at lists.adiscon.com > > > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > > > > > > > > > Ooops... previous message's attachment was the config, not the > > > debug > > > > > output. Debug attached. > > > > > > > > > > > > > > > > > > > > On Sun, 11 Sep 2011 16:50:57 -0400 Steve Chupack > > > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > I have rsyslogd up and running and logging to mysql with > > > loganalyzer > > > > > as a front end. Very cool. > > > > > > > > > > > > However, I can't seem to get on-disk queuing working, which > > > > > > would > > > be > > > > > nice if mysql goes down or a table is locked because I'm purging > > > > > records during a nightly maintenance cron. > > > > > > > > > > > > I never see any queue files created. I dropped the size of the > > > memory > > > > > queue to something ridiculously small, but still no queue files > > > even > > > > > after 20 minutes. > > > > > > > > > > > > I followed the instructions given here: > > > > > http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along > > > with a > > > > > couple of other tutorials on rsyslogd and queuing. > > > > > > > > > > > > I'm quite sure I'm just completely missing something obvious. > > > Details > > > > > below... And thanks to anyone who might be able to point out > > > > > what > > > I'm > > > > > doing wrong. > > > > > > > > > > > > -Steve > > > > > > > > > > > > > > > > > > > > > > > > MY CONFIG, please, no flames re the horrendously complex rules > > > > > > ;- > > > ) > > > > > > > > > > > > # if you experience problems, check # > > > > > > http://www.rsyslog.com/troubleshoot for assistance > > > > > > > > > > > > # rsyslog v3: load input modules # If you do not load inputs, > > > > > > nothing happens! > > > > > > # You may need to set the module load path if modules are not > > > found. > > > > > > > > > > > > $ModLoad immark # provides --MARK-- message capability > > > > > > $ModLoad imuxsock # provides support for local system logging > > > (e.g. > > > > > via logger command) > > > > > > $ModLoad imklog # kernel logging (formerly provided by rklogd) > > > > > > $ModLoad ommysql > > > > > > > > > > > > # SC 2010.11.11: configure disk caching in case mysql is > > > unavailable > > > > > > > > > > > > $MainMsgQueueSize 5 > > > > > > > > > > > > $WorkDirectory /var/log/rsyslogq # default location for work > > > (spool) > > > > > files > > > > > > > > > > > > $ActionQueueType LinkedList # use asynchronous processing > > > > > > $ActionQueueFileName dbq # set file name, also enables disk > > > mode > > > > > > $ActionResumeRetryCount -1 # infinite retries on insert > > > > > > failure > > > > > > > > > > > > # send snmpd INFO messages to the dustbin > > > > > > > > > > > > if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd' > > > > > then ~ > > > > > > > > > > > > $template vtfw,"insert into vtfw (Message, Facility, FromHost, > > > > > Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) > > > values > > > > > ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, > > > > > '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', > > > %iut%, > > > > > '%syslogtag%')",SQL > > > > > > > > > > > > if ($fromhost-ip contains '10.128.255') then > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vtfw > > > > > > & ~ > > > > > > #if ($fromhost-ip contains '10.128.255') then ~ > > > > > > > > > > > > $template vt1hs1_switches,"insert into vt1hs1_switches > > > > > > (Message, > > > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, > > > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, > > > '%HOSTNAME%', > > > > > %syslogpriority%, '%timereported:::date-mysql%', > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > > > > > if $fromhost-ip contains '10.128.0' or $hostname contains_i > > > 'core01' > > > > > or $hostname contains_i 'core02' then > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches > > > > > > #if ($fromhost-ip contains '10.128.0' or > > > > > > $source=='vt1hs1-dc216- > > > > > core01' or $source=='vt1hs1-dc216-core02') then > > > > > /var/log/switches > > > > > > & ~ > > > > > > #if $fromhost-ip contains '10.128.0' or $hostname contains_i > > > 'core01' > > > > > or $hostname contains_i 'core02' then ~ > > > > > > > > > > > > $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message, > > > Facility, > > > > > FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, > > > > > SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', > > > > > %syslogpriority%, '%timereported:::date-mysql%', > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > > > > > if $fromhost-ip contains '10.128.244' then > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi > > > > > > & ~ > > > > > > #if $fromhost-ip contains '10.128.244' then ~ > > > > > > > > > > > > $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message, > > > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, > > > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, > > > '%HOSTNAME%', > > > > > %syslogpriority%, '%timereported:::date-mysql%', > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == > > > '10.128.24.44' > > > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' > > > or > > > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' > > > > > or $fromhost-ip == '10.128.24.80' or $fromhost-ip == > > > > > '10.128.24.84') > > > and > > > > > ($syslogseverity-text == 'warning' or $syslogseverity-text == > > > 'warn' or > > > > > $syslogseverity-text == 'err' or $syslogseverity-text == 'error' > > > > > or $syslogseverity-text == 'crit' or $syslogseverity-text == 'alert' > > > or > > > > > $syslogseverity-text == 'emergency' or $syslogseverity-text == > > > 'panic') > > > > > then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere > > > > > > & ~ > > > > > > > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == > > > '10.128.24.44' > > > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' > > > or > > > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' > > > > > or $fromhost-ip == '10.128.24.80' or $fromhost-ip == > > > > > '10.128.24.84') > > > and > > > > > ($syslogseverity-text == 'notice') then ~ > > > > > > > > > > > > > > > > > > #*.* :ommysql:localhost,Syslog,root,mysqldb44 > > > > > > *.notice :ommysql:localhost,Syslog,root,mysqldb44 > > > > > > > > > > > > # Log all kernel messages to the console. > > > > > > # Logging much else clutters up the screen. > > > > > > #kern.* > > > /dev/console > > > > > > > > > > > > # Log anything (except mail) of level info or higher. > > > > > > # Don't log private authentication messages! > > > > > > *.info;mail.none;authpriv.none;cron.none - > > > > > /var/log/messages > > > > > > > > > > > > # The authpriv file has restricted access. > > > > > > authpriv.* > > > > > /var/log/secure > > > > > > > > > > > > # Log all the mail messages in one place. > > > > > > mail.* - > > > > > /var/log/maillog > > > > > > > > > > > > > > > > > > # Log cron stuff > > > > > > cron.* - > > > > > /var/log/cron > > > > > > > > > > > > # Everybody gets emergency messages > > > > > > *.emerg * > > > > > > > > > > > > # Save news errors of level crit and higher in a special file. > > > > > > uucp,news.crit - > > > > > /var/log/spooler > > > > > > > > > > > > # Save boot messages also to boot.log > > > > > > local7.* > > > > > /var/log/boot.log > > > > > > > > > > > > # Remote Logging (we use TCP for reliable delivery) # An > > > > > > on-disk queue is created for this action. If the remote host > > > is > > > > > > # down, messages are spooled to disk and sent when it is up > > > again. > > > > > > #$WorkDirectory /rsyslog/spool # where to place spool files > > > > > > #$ActionQueueFileName uniqName # unique name prefix for spool > > > files > > > > > > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as > > > > > possible) > > > > > > #$ActionQueueSaveOnShutdown on # save messages to disk on > > > shutdown > > > > > > #$ActionQueueType LinkedList # run asynchronously > > > > > > #$ActionResumeRetryCount -1 # infinite retries if host is down > > > > > > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port > > > optional > > > > > > #*.* @@remote-host:514 > > > > > > > > > > > > > > > > > > # ######### Receiving Messages from Remote Hosts ########## > # > > > > > > TCP Syslog Server: > > > > > > # provides TCP syslog reception and GSS-API (if compiled to > > > support > > > > > it) > > > > > > $ModLoad imtcp.so # load module $InputTCPServerRun 1470 # > > > > > > start up TCP listener at port 514 > > > > > > > > > > > > # UDP Syslog Server: > > > > > > $ModLoad imudp.so # provides UDP syslog reception > > > > > > $UDPServerRun 514 # start a UDP syslog server at standard port > > > 514 > > > > > > > > > > > > > > > > > > SOME DEBUG OUTPUT: > > > > > > > > > > > > (see attached) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Steve Chupack | IT Systems Administrator > > > > > > V: 877.327.8422 x 1242 > > > > > > Steve.Chupack at dealer.com | www.dealer.com > > > > > > > > > > > > > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From steve.chupack at dealer.com Mon Sep 12 22:06:08 2011 From: steve.chupack at dealer.com (Steve Chupack) Date: Mon, 12 Sep 2011 16:06:08 -0400 Subject: [rsyslog] Unable to implement on-disk queuing In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA72811F4@GRFEXC.intern.adiscon.com> References: <20110911165057.77049b4d@stchupack-m4300> <20110911171031.25bf90cc@stchupack-m4300> <9B6E2A8877C38245BFB15CC491A11DA72811EE@GRFEXC.intern.adiscon.com> <20110912090209.4d864ec6@stchupack-m4300> <9B6E2A8877C38245BFB15CC491A11DA72811F1@GRFEXC.intern.adiscon.com> <20110912103918.4a41fd18@stchupack-m4300> <9B6E2A8877C38245BFB15CC491A11DA72811F4@GRFEXC.intern.adiscon.com> Message-ID: <20110912160608.1290fa48@stchupack-m4300> Rainier, Thank you! I just tested and spool files are created and grow when mysql is off, then disappear when mysql comes back. The debug log also confirmed this behavior. Again, your help is hugely appreciated. Steve On Mon, 12 Sep 2011 19:00:28 +0200 Rainer Gerhards wrote: > I think I saw the problem in the config as exposed by the debug log. If I saw > correctly, you use > > $ActionResumeRetryCount -1 > Some filter ~ > Some filter :ommysql: > > The retry count is of type auto-reset. So the unlimited retries apply to the > first action (the discard) and the sql writer uses the default value, which > is to give up fairly quickly. It is important that you group these statements > right in front of the actual action you want them to apply to -- actually, > they *are* part of that action. I know its cumbersome, and things have and > will continue to improve greatly in v6. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Steve Chupack > > Sent: Monday, September 12, 2011 4:39 PM > > To: rsyslog at lists.adiscon.com > > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > > > Thanks for taking the time to respond, I really appreciate it. > > > > OK, I think I can better describe this now. You are correct, rsyslog is not > > seeing a need to queue. My debug log is filled with the following entries, > > which clearly indicate it sees no need to queue or spool. So again, I think > I am > > missing something very basic here... > > > > > > 7878.780426000:455e1940: main Q: entry added, size now 1 entries > > 7878.780439000:455e1940: wtpAdviseMaxWorkers signals busy > > 7878.780566000:455e1940: main Q: EnqueueMsg advised worker start > > 7878.780585000:423dc940: main Q: entry deleted, state 0, size now 0 entries > > > > I used the very basic config from your article on buffering. > > > > $MainMsgQueueSize 500 > > $WorkDirectory /var/log/rsyslogq # default location for work (spool) files > > $MainMsgQueueFileName mainq # set file name, also enables disk mode > > $ActionResumeRetryCount -1 # infinite retries on insert failure > > > > > > Mysql is definitely not running, as show by the following: > > > > 8220.440850000:41537940: Called LogError, msg: db error (2002): Can't > > connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' > (2) > > rsyslogd: db error (2002): Can't connect to local MySQL server through > socket > > '/var/lib/mysql/mysql.sock' (2) > > 8220.440865000:41537940: logmsg: flags 1, from 'vt1hs1-netservices01', msg > > db error (2002): Can't connect to local MySQL server through socket > > '/var/lib/mysql/mysql.sock' (2) > > 8220.441040000:41537940: Called action, logging to ommysql > > 8220.573450000:41537940: Called action, logging to ommysql > > 8220.574801000:41537940: Called action, logging to ommysql > > 8220.602406000:41537940: Called action, logging to ommysql > > 8220.604330000:41537940: Called action, logging to ommysql > > > > > > > > On Mon, 12 Sep 2011 15:18:32 +0200 > > Rainer Gerhards wrote: > > > > > > > > > > > > -----Original Message----- > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > > bounces at lists.adiscon.com] On Behalf Of Steve Chupack > > > > Sent: Monday, September 12, 2011 3:02 PM > > > > To: rsyslog at lists.adiscon.com > > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > > > > > > > Hmmmm... the thing is, my $MainMsgQueueSize is only 5, and I had > > > > mysql turned off for 20 minutes... And my message flow rate is up > > > > around 1 per second. So I would have expected some queuing to > > > > happen. Is there anything else I should look at or try? > > > > > > I think the unusual extremely low queue sizes results in some > > > watermarks set to 0 and inhibiting DA queue startup. Set the main > > > queue size to, say, 500 or 1000 and try again. Note that you can use > > > logger or the tcpflood tool to injects lots of message in a short period. > > > > > > Rainer > > > > > > > > > > > On Mon, 12 Sep 2011 11:13:28 +0200 > > > > Rainer Gerhards wrote: > > > > > > > > > Queue files are only created if necessary. The debug log does not > > > > contain any > > > > > such situation. > > > > > > > > > > Rainer > > > > > > > > > > > -----Original Message----- > > > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > > > > bounces at lists.adiscon.com] On Behalf Of Steve Chupack > > > > > > Sent: Sunday, September 11, 2011 11:11 PM > > > > > > To: rsyslog at lists.adiscon.com > > > > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > > > > > > > > > > > Ooops... previous message's attachment was the config, not the > > > > debug > > > > > > output. Debug attached. > > > > > > > > > > > > > > > > > > > > > > > > On Sun, 11 Sep 2011 16:50:57 -0400 Steve Chupack > > > > > > wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > I have rsyslogd up and running and logging to mysql with > > > > loganalyzer > > > > > > as a front end. Very cool. > > > > > > > > > > > > > > However, I can't seem to get on-disk queuing working, which > > > > > > > would > > > > be > > > > > > nice if mysql goes down or a table is locked because I'm purging > > > > > > records during a nightly maintenance cron. > > > > > > > > > > > > > > I never see any queue files created. I dropped the size of the > > > > memory > > > > > > queue to something ridiculously small, but still no queue files > > > > even > > > > > > after 20 minutes. > > > > > > > > > > > > > > I followed the instructions given here: > > > > > > http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along > > > > with a > > > > > > couple of other tutorials on rsyslogd and queuing. > > > > > > > > > > > > > > I'm quite sure I'm just completely missing something obvious. > > > > Details > > > > > > below... And thanks to anyone who might be able to point out > > > > > > what > > > > I'm > > > > > > doing wrong. > > > > > > > > > > > > > > -Steve > > > > > > > > > > > > > > > > > > > > > > > > > > > > MY CONFIG, please, no flames re the horrendously complex rules > > > > > > > ;- > > > > ) > > > > > > > > > > > > > > # if you experience problems, check # > > > > > > > http://www.rsyslog.com/troubleshoot for assistance > > > > > > > > > > > > > > # rsyslog v3: load input modules # If you do not load inputs, > > > > > > > nothing happens! > > > > > > > # You may need to set the module load path if modules are not > > > > found. > > > > > > > > > > > > > > $ModLoad immark # provides --MARK-- message capability > > > > > > > $ModLoad imuxsock # provides support for local system logging > > > > (e.g. > > > > > > via logger command) > > > > > > > $ModLoad imklog # kernel logging (formerly provided by rklogd) > > > > > > > $ModLoad ommysql > > > > > > > > > > > > > > # SC 2010.11.11: configure disk caching in case mysql is > > > > unavailable > > > > > > > > > > > > > > $MainMsgQueueSize 5 > > > > > > > > > > > > > > $WorkDirectory /var/log/rsyslogq # default location for work > > > > (spool) > > > > > > files > > > > > > > > > > > > > > $ActionQueueType LinkedList # use asynchronous processing > > > > > > > $ActionQueueFileName dbq # set file name, also enables disk > > > > mode > > > > > > > $ActionResumeRetryCount -1 # infinite retries on insert > > > > > > > failure > > > > > > > > > > > > > > # send snmpd INFO messages to the dustbin > > > > > > > > > > > > > > if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd' > > > > > > then ~ > > > > > > > > > > > > > > $template vtfw,"insert into vtfw (Message, Facility, FromHost, > > > > > > Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) > > > > values > > > > > > ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, > > > > > > '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', > > > > %iut%, > > > > > > '%syslogtag%')",SQL > > > > > > > > > > > > > > if ($fromhost-ip contains '10.128.255') then > > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vtfw > > > > > > > & ~ > > > > > > > #if ($fromhost-ip contains '10.128.255') then ~ > > > > > > > > > > > > > > $template vt1hs1_switches,"insert into vt1hs1_switches > > > > > > > (Message, > > > > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, > > > > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, > > > > '%HOSTNAME%', > > > > > > %syslogpriority%, '%timereported:::date-mysql%', > > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > > > > > > > if $fromhost-ip contains '10.128.0' or $hostname contains_i > > > > 'core01' > > > > > > or $hostname contains_i 'core02' then > > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches > > > > > > > #if ($fromhost-ip contains '10.128.0' or > > > > > > > $source=='vt1hs1-dc216- > > > > > > core01' or $source=='vt1hs1-dc216-core02') then > > > > > > /var/log/switches > > > > > > > & ~ > > > > > > > #if $fromhost-ip contains '10.128.0' or $hostname contains_i > > > > 'core01' > > > > > > or $hostname contains_i 'core02' then ~ > > > > > > > > > > > > > > $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message, > > > > Facility, > > > > > > FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, > > > > > > SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', > > > > > > %syslogpriority%, '%timereported:::date-mysql%', > > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > > > > > > > if $fromhost-ip contains '10.128.244' then > > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi > > > > > > > & ~ > > > > > > > #if $fromhost-ip contains '10.128.244' then ~ > > > > > > > > > > > > > > $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message, > > > > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, > > > > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, > > > > '%HOSTNAME%', > > > > > > %syslogpriority%, '%timereported:::date-mysql%', > > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == > > > > '10.128.24.44' > > > > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' > > > > or > > > > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' > > > > > > or $fromhost-ip == '10.128.24.80' or $fromhost-ip == > > > > > > '10.128.24.84') > > > > and > > > > > > ($syslogseverity-text == 'warning' or $syslogseverity-text == > > > > 'warn' or > > > > > > $syslogseverity-text == 'err' or $syslogseverity-text == 'error' > > > > > > or $syslogseverity-text == 'crit' or $syslogseverity-text == > 'alert' > > > > or > > > > > > $syslogseverity-text == 'emergency' or $syslogseverity-text == > > > > 'panic') > > > > > > then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere > > > > > > > & ~ > > > > > > > > > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == > > > > '10.128.24.44' > > > > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' > > > > or > > > > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' > > > > > > or $fromhost-ip == '10.128.24.80' or $fromhost-ip == > > > > > > '10.128.24.84') > > > > and > > > > > > ($syslogseverity-text == 'notice') then ~ > > > > > > > > > > > > > > > > > > > > > #*.* :ommysql:localhost,Syslog,root,mysqldb44 > > > > > > > *.notice :ommysql:localhost,Syslog,root,mysqldb44 > > > > > > > > > > > > > > # Log all kernel messages to the console. > > > > > > > # Logging much else clutters up the screen. > > > > > > > #kern.* > > > > /dev/console > > > > > > > > > > > > > > # Log anything (except mail) of level info or higher. > > > > > > > # Don't log private authentication messages! > > > > > > > *.info;mail.none;authpriv.none;cron.none - > > > > > > /var/log/messages > > > > > > > > > > > > > > # The authpriv file has restricted access. > > > > > > > authpriv.* > > > > > > /var/log/secure > > > > > > > > > > > > > > # Log all the mail messages in one place. > > > > > > > mail.* - > > > > > > /var/log/maillog > > > > > > > > > > > > > > > > > > > > > # Log cron stuff > > > > > > > cron.* - > > > > > > /var/log/cron > > > > > > > > > > > > > > # Everybody gets emergency messages > > > > > > > *.emerg * > > > > > > > > > > > > > > # Save news errors of level crit and higher in a special file. > > > > > > > uucp,news.crit - > > > > > > /var/log/spooler > > > > > > > > > > > > > > # Save boot messages also to boot.log > > > > > > > local7.* > > > > > > /var/log/boot.log > > > > > > > > > > > > > > # Remote Logging (we use TCP for reliable delivery) # An > > > > > > > on-disk queue is created for this action. If the remote host > > > > is > > > > > > > # down, messages are spooled to disk and sent when it is up > > > > again. > > > > > > > #$WorkDirectory /rsyslog/spool # where to place spool files > > > > > > > #$ActionQueueFileName uniqName # unique name prefix for spool > > > > files > > > > > > > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as > > > > > > possible) > > > > > > > #$ActionQueueSaveOnShutdown on # save messages to disk on > > > > shutdown > > > > > > > #$ActionQueueType LinkedList # run asynchronously > > > > > > > #$ActionResumeRetryCount -1 # infinite retries if host is down > > > > > > > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port > > > > optional > > > > > > > #*.* @@remote-host:514 > > > > > > > > > > > > > > > > > > > > > # ######### Receiving Messages from Remote Hosts ########## > > # > > > > > > > TCP Syslog Server: > > > > > > > # provides TCP syslog reception and GSS-API (if compiled to > > > > support > > > > > > it) > > > > > > > $ModLoad imtcp.so # load module $InputTCPServerRun 1470 # > > > > > > > start up TCP listener at port 514 > > > > > > > > > > > > > > # UDP Syslog Server: > > > > > > > $ModLoad imudp.so # provides UDP syslog reception > > > > > > > $UDPServerRun 514 # start a UDP syslog server at standard port > > > > 514 > > > > > > > > > > > > > > > > > > > > > SOME DEBUG OUTPUT: > > > > > > > > > > > > > > (see attached) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Steve Chupack | IT Systems Administrator > > > > > > > V: 877.327.8422 x 1242 > > > > > > > Steve.Chupack at dealer.com | www.dealer.com > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > rsyslog mailing list > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > > http://www.rsyslog.com > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From steve.chupack at dealer.com Mon Sep 12 22:06:27 2011 From: steve.chupack at dealer.com (Steve Chupack) Date: Mon, 12 Sep 2011 16:06:27 -0400 Subject: [rsyslog] Unable to implement on-disk queuing In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA72811F4@GRFEXC.intern.adiscon.com> References: <20110911165057.77049b4d@stchupack-m4300> <20110911171031.25bf90cc@stchupack-m4300> <9B6E2A8877C38245BFB15CC491A11DA72811EE@GRFEXC.intern.adiscon.com> <20110912090209.4d864ec6@stchupack-m4300> <9B6E2A8877C38245BFB15CC491A11DA72811F1@GRFEXC.intern.adiscon.com> <20110912103918.4a41fd18@stchupack-m4300> <9B6E2A8877C38245BFB15CC491A11DA72811F4@GRFEXC.intern.adiscon.com> Message-ID: <20110912160627.41bf0b5f@stchupack-m4300> Rainer, Thank you! I just tested and spool files are created and grow when mysql is off, then disappear when mysql comes back. The debug log also confirmed this behavior. Again, your help is hugely appreciated. Steve On Mon, 12 Sep 2011 19:00:28 +0200 Rainer Gerhards wrote: > I think I saw the problem in the config as exposed by the debug log. If I saw > correctly, you use > > $ActionResumeRetryCount -1 > Some filter ~ > Some filter :ommysql: > > The retry count is of type auto-reset. So the unlimited retries apply to the > first action (the discard) and the sql writer uses the default value, which > is to give up fairly quickly. It is important that you group these statements > right in front of the actual action you want them to apply to -- actually, > they *are* part of that action. I know its cumbersome, and things have and > will continue to improve greatly in v6. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Steve Chupack > > Sent: Monday, September 12, 2011 4:39 PM > > To: rsyslog at lists.adiscon.com > > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > > > Thanks for taking the time to respond, I really appreciate it. > > > > OK, I think I can better describe this now. You are correct, rsyslog is not > > seeing a need to queue. My debug log is filled with the following entries, > > which clearly indicate it sees no need to queue or spool. So again, I think > I am > > missing something very basic here... > > > > > > 7878.780426000:455e1940: main Q: entry added, size now 1 entries > > 7878.780439000:455e1940: wtpAdviseMaxWorkers signals busy > > 7878.780566000:455e1940: main Q: EnqueueMsg advised worker start > > 7878.780585000:423dc940: main Q: entry deleted, state 0, size now 0 entries > > > > I used the very basic config from your article on buffering. > > > > $MainMsgQueueSize 500 > > $WorkDirectory /var/log/rsyslogq # default location for work (spool) files > > $MainMsgQueueFileName mainq # set file name, also enables disk mode > > $ActionResumeRetryCount -1 # infinite retries on insert failure > > > > > > Mysql is definitely not running, as show by the following: > > > > 8220.440850000:41537940: Called LogError, msg: db error (2002): Can't > > connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' > (2) > > rsyslogd: db error (2002): Can't connect to local MySQL server through > socket > > '/var/lib/mysql/mysql.sock' (2) > > 8220.440865000:41537940: logmsg: flags 1, from 'vt1hs1-netservices01', msg > > db error (2002): Can't connect to local MySQL server through socket > > '/var/lib/mysql/mysql.sock' (2) > > 8220.441040000:41537940: Called action, logging to ommysql > > 8220.573450000:41537940: Called action, logging to ommysql > > 8220.574801000:41537940: Called action, logging to ommysql > > 8220.602406000:41537940: Called action, logging to ommysql > > 8220.604330000:41537940: Called action, logging to ommysql > > > > > > > > On Mon, 12 Sep 2011 15:18:32 +0200 > > Rainer Gerhards wrote: > > > > > > > > > > > > -----Original Message----- > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > > bounces at lists.adiscon.com] On Behalf Of Steve Chupack > > > > Sent: Monday, September 12, 2011 3:02 PM > > > > To: rsyslog at lists.adiscon.com > > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > > > > > > > Hmmmm... the thing is, my $MainMsgQueueSize is only 5, and I had > > > > mysql turned off for 20 minutes... And my message flow rate is up > > > > around 1 per second. So I would have expected some queuing to > > > > happen. Is there anything else I should look at or try? > > > > > > I think the unusual extremely low queue sizes results in some > > > watermarks set to 0 and inhibiting DA queue startup. Set the main > > > queue size to, say, 500 or 1000 and try again. Note that you can use > > > logger or the tcpflood tool to injects lots of message in a short period. > > > > > > Rainer > > > > > > > > > > > On Mon, 12 Sep 2011 11:13:28 +0200 > > > > Rainer Gerhards wrote: > > > > > > > > > Queue files are only created if necessary. The debug log does not > > > > contain any > > > > > such situation. > > > > > > > > > > Rainer > > > > > > > > > > > -----Original Message----- > > > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > > > > bounces at lists.adiscon.com] On Behalf Of Steve Chupack > > > > > > Sent: Sunday, September 11, 2011 11:11 PM > > > > > > To: rsyslog at lists.adiscon.com > > > > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > > > > > > > > > > > Ooops... previous message's attachment was the config, not the > > > > debug > > > > > > output. Debug attached. > > > > > > > > > > > > > > > > > > > > > > > > On Sun, 11 Sep 2011 16:50:57 -0400 Steve Chupack > > > > > > wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > I have rsyslogd up and running and logging to mysql with > > > > loganalyzer > > > > > > as a front end. Very cool. > > > > > > > > > > > > > > However, I can't seem to get on-disk queuing working, which > > > > > > > would > > > > be > > > > > > nice if mysql goes down or a table is locked because I'm purging > > > > > > records during a nightly maintenance cron. > > > > > > > > > > > > > > I never see any queue files created. I dropped the size of the > > > > memory > > > > > > queue to something ridiculously small, but still no queue files > > > > even > > > > > > after 20 minutes. > > > > > > > > > > > > > > I followed the instructions given here: > > > > > > http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along > > > > with a > > > > > > couple of other tutorials on rsyslogd and queuing. > > > > > > > > > > > > > > I'm quite sure I'm just completely missing something obvious. > > > > Details > > > > > > below... And thanks to anyone who might be able to point out > > > > > > what > > > > I'm > > > > > > doing wrong. > > > > > > > > > > > > > > -Steve > > > > > > > > > > > > > > > > > > > > > > > > > > > > MY CONFIG, please, no flames re the horrendously complex rules > > > > > > > ;- > > > > ) > > > > > > > > > > > > > > # if you experience problems, check # > > > > > > > http://www.rsyslog.com/troubleshoot for assistance > > > > > > > > > > > > > > # rsyslog v3: load input modules # If you do not load inputs, > > > > > > > nothing happens! > > > > > > > # You may need to set the module load path if modules are not > > > > found. > > > > > > > > > > > > > > $ModLoad immark # provides --MARK-- message capability > > > > > > > $ModLoad imuxsock # provides support for local system logging > > > > (e.g. > > > > > > via logger command) > > > > > > > $ModLoad imklog # kernel logging (formerly provided by rklogd) > > > > > > > $ModLoad ommysql > > > > > > > > > > > > > > # SC 2010.11.11: configure disk caching in case mysql is > > > > unavailable > > > > > > > > > > > > > > $MainMsgQueueSize 5 > > > > > > > > > > > > > > $WorkDirectory /var/log/rsyslogq # default location for work > > > > (spool) > > > > > > files > > > > > > > > > > > > > > $ActionQueueType LinkedList # use asynchronous processing > > > > > > > $ActionQueueFileName dbq # set file name, also enables disk > > > > mode > > > > > > > $ActionResumeRetryCount -1 # infinite retries on insert > > > > > > > failure > > > > > > > > > > > > > > # send snmpd INFO messages to the dustbin > > > > > > > > > > > > > > if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd' > > > > > > then ~ > > > > > > > > > > > > > > $template vtfw,"insert into vtfw (Message, Facility, FromHost, > > > > > > Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) > > > > values > > > > > > ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, > > > > > > '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', > > > > %iut%, > > > > > > '%syslogtag%')",SQL > > > > > > > > > > > > > > if ($fromhost-ip contains '10.128.255') then > > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vtfw > > > > > > > & ~ > > > > > > > #if ($fromhost-ip contains '10.128.255') then ~ > > > > > > > > > > > > > > $template vt1hs1_switches,"insert into vt1hs1_switches > > > > > > > (Message, > > > > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, > > > > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, > > > > '%HOSTNAME%', > > > > > > %syslogpriority%, '%timereported:::date-mysql%', > > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > > > > > > > if $fromhost-ip contains '10.128.0' or $hostname contains_i > > > > 'core01' > > > > > > or $hostname contains_i 'core02' then > > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches > > > > > > > #if ($fromhost-ip contains '10.128.0' or > > > > > > > $source=='vt1hs1-dc216- > > > > > > core01' or $source=='vt1hs1-dc216-core02') then > > > > > > /var/log/switches > > > > > > > & ~ > > > > > > > #if $fromhost-ip contains '10.128.0' or $hostname contains_i > > > > 'core01' > > > > > > or $hostname contains_i 'core02' then ~ > > > > > > > > > > > > > > $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message, > > > > Facility, > > > > > > FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, > > > > > > SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', > > > > > > %syslogpriority%, '%timereported:::date-mysql%', > > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > > > > > > > if $fromhost-ip contains '10.128.244' then > > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi > > > > > > > & ~ > > > > > > > #if $fromhost-ip contains '10.128.244' then ~ > > > > > > > > > > > > > > $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message, > > > > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, > > > > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, > > > > '%HOSTNAME%', > > > > > > %syslogpriority%, '%timereported:::date-mysql%', > > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == > > > > '10.128.24.44' > > > > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' > > > > or > > > > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' > > > > > > or $fromhost-ip == '10.128.24.80' or $fromhost-ip == > > > > > > '10.128.24.84') > > > > and > > > > > > ($syslogseverity-text == 'warning' or $syslogseverity-text == > > > > 'warn' or > > > > > > $syslogseverity-text == 'err' or $syslogseverity-text == 'error' > > > > > > or $syslogseverity-text == 'crit' or $syslogseverity-text == > 'alert' > > > > or > > > > > > $syslogseverity-text == 'emergency' or $syslogseverity-text == > > > > 'panic') > > > > > > then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere > > > > > > > & ~ > > > > > > > > > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == > > > > '10.128.24.44' > > > > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' > > > > or > > > > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' > > > > > > or $fromhost-ip == '10.128.24.80' or $fromhost-ip == > > > > > > '10.128.24.84') > > > > and > > > > > > ($syslogseverity-text == 'notice') then ~ > > > > > > > > > > > > > > > > > > > > > #*.* :ommysql:localhost,Syslog,root,mysqldb44 > > > > > > > *.notice :ommysql:localhost,Syslog,root,mysqldb44 > > > > > > > > > > > > > > # Log all kernel messages to the console. > > > > > > > # Logging much else clutters up the screen. > > > > > > > #kern.* > > > > /dev/console > > > > > > > > > > > > > > # Log anything (except mail) of level info or higher. > > > > > > > # Don't log private authentication messages! > > > > > > > *.info;mail.none;authpriv.none;cron.none - > > > > > > /var/log/messages > > > > > > > > > > > > > > # The authpriv file has restricted access. > > > > > > > authpriv.* > > > > > > /var/log/secure > > > > > > > > > > > > > > # Log all the mail messages in one place. > > > > > > > mail.* - > > > > > > /var/log/maillog > > > > > > > > > > > > > > > > > > > > > # Log cron stuff > > > > > > > cron.* - > > > > > > /var/log/cron > > > > > > > > > > > > > > # Everybody gets emergency messages > > > > > > > *.emerg * > > > > > > > > > > > > > > # Save news errors of level crit and higher in a special file. > > > > > > > uucp,news.crit - > > > > > > /var/log/spooler > > > > > > > > > > > > > > # Save boot messages also to boot.log > > > > > > > local7.* > > > > > > /var/log/boot.log > > > > > > > > > > > > > > # Remote Logging (we use TCP for reliable delivery) # An > > > > > > > on-disk queue is created for this action. If the remote host > > > > is > > > > > > > # down, messages are spooled to disk and sent when it is up > > > > again. > > > > > > > #$WorkDirectory /rsyslog/spool # where to place spool files > > > > > > > #$ActionQueueFileName uniqName # unique name prefix for spool > > > > files > > > > > > > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as > > > > > > possible) > > > > > > > #$ActionQueueSaveOnShutdown on # save messages to disk on > > > > shutdown > > > > > > > #$ActionQueueType LinkedList # run asynchronously > > > > > > > #$ActionResumeRetryCount -1 # infinite retries if host is down > > > > > > > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port > > > > optional > > > > > > > #*.* @@remote-host:514 > > > > > > > > > > > > > > > > > > > > > # ######### Receiving Messages from Remote Hosts ########## > > # > > > > > > > TCP Syslog Server: > > > > > > > # provides TCP syslog reception and GSS-API (if compiled to > > > > support > > > > > > it) > > > > > > > $ModLoad imtcp.so # load module $InputTCPServerRun 1470 # > > > > > > > start up TCP listener at port 514 > > > > > > > > > > > > > > # UDP Syslog Server: > > > > > > > $ModLoad imudp.so # provides UDP syslog reception > > > > > > > $UDPServerRun 514 # start a UDP syslog server at standard port > > > > 514 > > > > > > > > > > > > > > > > > > > > > SOME DEBUG OUTPUT: > > > > > > > > > > > > > > (see attached) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Steve Chupack | IT Systems Administrator > > > > > > > V: 877.327.8422 x 1242 > > > > > > > Steve.Chupack at dealer.com | www.dealer.com > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > rsyslog mailing list > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > > http://www.rsyslog.com > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From steve.chupack at dealer.com Mon Sep 12 22:09:48 2011 From: steve.chupack at dealer.com (Steve Chupack) Date: Mon, 12 Sep 2011 16:09:48 -0400 Subject: [rsyslog] Unable to implement on-disk queuing In-Reply-To: <20110912160627.41bf0b5f@stchupack-m4300> References: <20110911165057.77049b4d@stchupack-m4300> <20110911171031.25bf90cc@stchupack-m4300> <9B6E2A8877C38245BFB15CC491A11DA72811EE@GRFEXC.intern.adiscon.com> <20110912090209.4d864ec6@stchupack-m4300> <9B6E2A8877C38245BFB15CC491A11DA72811F1@GRFEXC.intern.adiscon.com> <20110912103918.4a41fd18@stchupack-m4300> <9B6E2A8877C38245BFB15CC491A11DA72811F4@GRFEXC.intern.adiscon.com> <20110912160627.41bf0b5f@stchupack-m4300> Message-ID: <20110912160948.06dbb9a5@stchupack-m4300> sorry, i incorrectly spelled your name the first time around on this. ;-) On Mon, 12 Sep 2011 16:06:27 -0400 Steve Chupack wrote: > Rainer, > > Thank you! I just tested and spool files are created and grow when mysql is off, then disappear when mysql comes back. The debug log also confirmed this behavior. > > Again, your help is hugely appreciated. > > Steve > > > > On Mon, 12 Sep 2011 19:00:28 +0200 > Rainer Gerhards wrote: > > > I think I saw the problem in the config as exposed by the debug log. If I saw > > correctly, you use > > > > $ActionResumeRetryCount -1 > > Some filter ~ > > Some filter :ommysql: > > > > The retry count is of type auto-reset. So the unlimited retries apply to the > > first action (the discard) and the sql writer uses the default value, which > > is to give up fairly quickly. It is important that you group these statements > > right in front of the actual action you want them to apply to -- actually, > > they *are* part of that action. I know its cumbersome, and things have and > > will continue to improve greatly in v6. > > > > Rainer > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Steve Chupack > > > Sent: Monday, September 12, 2011 4:39 PM > > > To: rsyslog at lists.adiscon.com > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > > > > > Thanks for taking the time to respond, I really appreciate it. > > > > > > OK, I think I can better describe this now. You are correct, rsyslog is not > > > seeing a need to queue. My debug log is filled with the following entries, > > > which clearly indicate it sees no need to queue or spool. So again, I think > > I am > > > missing something very basic here... > > > > > > > > > 7878.780426000:455e1940: main Q: entry added, size now 1 entries > > > 7878.780439000:455e1940: wtpAdviseMaxWorkers signals busy > > > 7878.780566000:455e1940: main Q: EnqueueMsg advised worker start > > > 7878.780585000:423dc940: main Q: entry deleted, state 0, size now 0 entries > > > > > > I used the very basic config from your article on buffering. > > > > > > $MainMsgQueueSize 500 > > > $WorkDirectory /var/log/rsyslogq # default location for work (spool) files > > > $MainMsgQueueFileName mainq # set file name, also enables disk mode > > > $ActionResumeRetryCount -1 # infinite retries on insert failure > > > > > > > > > Mysql is definitely not running, as show by the following: > > > > > > 8220.440850000:41537940: Called LogError, msg: db error (2002): Can't > > > connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' > > (2) > > > rsyslogd: db error (2002): Can't connect to local MySQL server through > > socket > > > '/var/lib/mysql/mysql.sock' (2) > > > 8220.440865000:41537940: logmsg: flags 1, from 'vt1hs1-netservices01', msg > > > db error (2002): Can't connect to local MySQL server through socket > > > '/var/lib/mysql/mysql.sock' (2) > > > 8220.441040000:41537940: Called action, logging to ommysql > > > 8220.573450000:41537940: Called action, logging to ommysql > > > 8220.574801000:41537940: Called action, logging to ommysql > > > 8220.602406000:41537940: Called action, logging to ommysql > > > 8220.604330000:41537940: Called action, logging to ommysql > > > > > > > > > > > > On Mon, 12 Sep 2011 15:18:32 +0200 > > > Rainer Gerhards wrote: > > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > > > bounces at lists.adiscon.com] On Behalf Of Steve Chupack > > > > > Sent: Monday, September 12, 2011 3:02 PM > > > > > To: rsyslog at lists.adiscon.com > > > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > > > > > > > > > Hmmmm... the thing is, my $MainMsgQueueSize is only 5, and I had > > > > > mysql turned off for 20 minutes... And my message flow rate is up > > > > > around 1 per second. So I would have expected some queuing to > > > > > happen. Is there anything else I should look at or try? > > > > > > > > I think the unusual extremely low queue sizes results in some > > > > watermarks set to 0 and inhibiting DA queue startup. Set the main > > > > queue size to, say, 500 or 1000 and try again. Note that you can use > > > > logger or the tcpflood tool to injects lots of message in a short period. > > > > > > > > Rainer > > > > > > > > > > > > > > On Mon, 12 Sep 2011 11:13:28 +0200 > > > > > Rainer Gerhards wrote: > > > > > > > > > > > Queue files are only created if necessary. The debug log does not > > > > > contain any > > > > > > such situation. > > > > > > > > > > > > Rainer > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > > > > > bounces at lists.adiscon.com] On Behalf Of Steve Chupack > > > > > > > Sent: Sunday, September 11, 2011 11:11 PM > > > > > > > To: rsyslog at lists.adiscon.com > > > > > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > > > > > > > > > > > > > Ooops... previous message's attachment was the config, not the > > > > > debug > > > > > > > output. Debug attached. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, 11 Sep 2011 16:50:57 -0400 Steve Chupack > > > > > > > wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > I have rsyslogd up and running and logging to mysql with > > > > > loganalyzer > > > > > > > as a front end. Very cool. > > > > > > > > > > > > > > > > However, I can't seem to get on-disk queuing working, which > > > > > > > > would > > > > > be > > > > > > > nice if mysql goes down or a table is locked because I'm purging > > > > > > > records during a nightly maintenance cron. > > > > > > > > > > > > > > > > I never see any queue files created. I dropped the size of the > > > > > memory > > > > > > > queue to something ridiculously small, but still no queue files > > > > > even > > > > > > > after 20 minutes. > > > > > > > > > > > > > > > > I followed the instructions given here: > > > > > > > http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along > > > > > with a > > > > > > > couple of other tutorials on rsyslogd and queuing. > > > > > > > > > > > > > > > > I'm quite sure I'm just completely missing something obvious. > > > > > Details > > > > > > > below... And thanks to anyone who might be able to point out > > > > > > > what > > > > > I'm > > > > > > > doing wrong. > > > > > > > > > > > > > > > > -Steve > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > MY CONFIG, please, no flames re the horrendously complex rules > > > > > > > > ;- > > > > > ) > > > > > > > > > > > > > > > > # if you experience problems, check # > > > > > > > > http://www.rsyslog.com/troubleshoot for assistance > > > > > > > > > > > > > > > > # rsyslog v3: load input modules # If you do not load inputs, > > > > > > > > nothing happens! > > > > > > > > # You may need to set the module load path if modules are not > > > > > found. > > > > > > > > > > > > > > > > $ModLoad immark # provides --MARK-- message capability > > > > > > > > $ModLoad imuxsock # provides support for local system logging > > > > > (e.g. > > > > > > > via logger command) > > > > > > > > $ModLoad imklog # kernel logging (formerly provided by rklogd) > > > > > > > > $ModLoad ommysql > > > > > > > > > > > > > > > > # SC 2010.11.11: configure disk caching in case mysql is > > > > > unavailable > > > > > > > > > > > > > > > > $MainMsgQueueSize 5 > > > > > > > > > > > > > > > > $WorkDirectory /var/log/rsyslogq # default location for work > > > > > (spool) > > > > > > > files > > > > > > > > > > > > > > > > $ActionQueueType LinkedList # use asynchronous processing > > > > > > > > $ActionQueueFileName dbq # set file name, also enables disk > > > > > mode > > > > > > > > $ActionResumeRetryCount -1 # infinite retries on insert > > > > > > > > failure > > > > > > > > > > > > > > > > # send snmpd INFO messages to the dustbin > > > > > > > > > > > > > > > > if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd' > > > > > > > then ~ > > > > > > > > > > > > > > > > $template vtfw,"insert into vtfw (Message, Facility, FromHost, > > > > > > > Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) > > > > > values > > > > > > > ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, > > > > > > > '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', > > > > > %iut%, > > > > > > > '%syslogtag%')",SQL > > > > > > > > > > > > > > > > if ($fromhost-ip contains '10.128.255') then > > > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vtfw > > > > > > > > & ~ > > > > > > > > #if ($fromhost-ip contains '10.128.255') then ~ > > > > > > > > > > > > > > > > $template vt1hs1_switches,"insert into vt1hs1_switches > > > > > > > > (Message, > > > > > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, > > > > > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, > > > > > '%HOSTNAME%', > > > > > > > %syslogpriority%, '%timereported:::date-mysql%', > > > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > > > > > > > > > if $fromhost-ip contains '10.128.0' or $hostname contains_i > > > > > 'core01' > > > > > > > or $hostname contains_i 'core02' then > > > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches > > > > > > > > #if ($fromhost-ip contains '10.128.0' or > > > > > > > > $source=='vt1hs1-dc216- > > > > > > > core01' or $source=='vt1hs1-dc216-core02') then > > > > > > > /var/log/switches > > > > > > > > & ~ > > > > > > > > #if $fromhost-ip contains '10.128.0' or $hostname contains_i > > > > > 'core01' > > > > > > > or $hostname contains_i 'core02' then ~ > > > > > > > > > > > > > > > > $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message, > > > > > Facility, > > > > > > > FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, > > > > > > > SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', > > > > > > > %syslogpriority%, '%timereported:::date-mysql%', > > > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > > > > > > > > > if $fromhost-ip contains '10.128.244' then > > > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi > > > > > > > > & ~ > > > > > > > > #if $fromhost-ip contains '10.128.244' then ~ > > > > > > > > > > > > > > > > $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message, > > > > > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, > > > > > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, > > > > > '%HOSTNAME%', > > > > > > > %syslogpriority%, '%timereported:::date-mysql%', > > > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > > > > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == > > > > > '10.128.24.44' > > > > > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' > > > > > or > > > > > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' > > > > > > > or $fromhost-ip == '10.128.24.80' or $fromhost-ip == > > > > > > > '10.128.24.84') > > > > > and > > > > > > > ($syslogseverity-text == 'warning' or $syslogseverity-text == > > > > > 'warn' or > > > > > > > $syslogseverity-text == 'err' or $syslogseverity-text == 'error' > > > > > > > or $syslogseverity-text == 'crit' or $syslogseverity-text == > > 'alert' > > > > > or > > > > > > > $syslogseverity-text == 'emergency' or $syslogseverity-text == > > > > > 'panic') > > > > > > > then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere > > > > > > > > & ~ > > > > > > > > > > > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == > > > > > '10.128.24.44' > > > > > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' > > > > > or > > > > > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' > > > > > > > or $fromhost-ip == '10.128.24.80' or $fromhost-ip == > > > > > > > '10.128.24.84') > > > > > and > > > > > > > ($syslogseverity-text == 'notice') then ~ > > > > > > > > > > > > > > > > > > > > > > > > #*.* :ommysql:localhost,Syslog,root,mysqldb44 > > > > > > > > *.notice :ommysql:localhost,Syslog,root,mysqldb44 > > > > > > > > > > > > > > > > # Log all kernel messages to the console. > > > > > > > > # Logging much else clutters up the screen. > > > > > > > > #kern.* > > > > > /dev/console > > > > > > > > > > > > > > > > # Log anything (except mail) of level info or higher. > > > > > > > > # Don't log private authentication messages! > > > > > > > > *.info;mail.none;authpriv.none;cron.none - > > > > > > > /var/log/messages > > > > > > > > > > > > > > > > # The authpriv file has restricted access. > > > > > > > > authpriv.* > > > > > > > /var/log/secure > > > > > > > > > > > > > > > > # Log all the mail messages in one place. > > > > > > > > mail.* - > > > > > > > /var/log/maillog > > > > > > > > > > > > > > > > > > > > > > > > # Log cron stuff > > > > > > > > cron.* - > > > > > > > /var/log/cron > > > > > > > > > > > > > > > > # Everybody gets emergency messages > > > > > > > > *.emerg * > > > > > > > > > > > > > > > > # Save news errors of level crit and higher in a special file. > > > > > > > > uucp,news.crit - > > > > > > > /var/log/spooler > > > > > > > > > > > > > > > > # Save boot messages also to boot.log > > > > > > > > local7.* > > > > > > > /var/log/boot.log > > > > > > > > > > > > > > > > # Remote Logging (we use TCP for reliable delivery) # An > > > > > > > > on-disk queue is created for this action. If the remote host > > > > > is > > > > > > > > # down, messages are spooled to disk and sent when it is up > > > > > again. > > > > > > > > #$WorkDirectory /rsyslog/spool # where to place spool files > > > > > > > > #$ActionQueueFileName uniqName # unique name prefix for spool > > > > > files > > > > > > > > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as > > > > > > > possible) > > > > > > > > #$ActionQueueSaveOnShutdown on # save messages to disk on > > > > > shutdown > > > > > > > > #$ActionQueueType LinkedList # run asynchronously > > > > > > > > #$ActionResumeRetryCount -1 # infinite retries if host is down > > > > > > > > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port > > > > > optional > > > > > > > > #*.* @@remote-host:514 > > > > > > > > > > > > > > > > > > > > > > > > # ######### Receiving Messages from Remote Hosts ########## > > > # > > > > > > > > TCP Syslog Server: > > > > > > > > # provides TCP syslog reception and GSS-API (if compiled to > > > > > support > > > > > > > it) > > > > > > > > $ModLoad imtcp.so # load module $InputTCPServerRun 1470 # > > > > > > > > start up TCP listener at port 514 > > > > > > > > > > > > > > > > # UDP Syslog Server: > > > > > > > > $ModLoad imudp.so # provides UDP syslog reception > > > > > > > > $UDPServerRun 514 # start a UDP syslog server at standard port > > > > > 514 > > > > > > > > > > > > > > > > > > > > > > > > SOME DEBUG OUTPUT: > > > > > > > > > > > > > > > > (see attached) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Steve Chupack | IT Systems Administrator > > > > > > > > V: 877.327.8422 x 1242 > > > > > > > > Steve.Chupack at dealer.com | www.dealer.com > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > rsyslog mailing list > > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > > > http://www.rsyslog.com > > > > > _______________________________________________ > > > > > rsyslog mailing list > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > > http://www.rsyslog.com > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From jpoling at moody.edu Wed Sep 14 22:39:18 2011 From: jpoling at moody.edu (Jeff Poling) Date: Wed, 14 Sep 2011 15:39:18 -0500 Subject: [rsyslog] Upgrade Message-ID: <9599A350A0A5884DB4E50D83F9287D0F01F315C3AB@exchmbx01.moody.edu> Does it matter in what order I do the upgrade? i.e., do I upgrade the collector system first and then all the other systems? Thanks, Jeff Jeffrey Poling System Administrator | Information Systems Moody Bible Institute 820 N. LaSalle Blvd., Chicago, IL 60610 312-329-8968 www.moodyministries.net >From the Word. To Life. From david at lang.hm Thu Sep 15 01:48:05 2011 From: david at lang.hm (david at lang.hm) Date: Wed, 14 Sep 2011 16:48:05 -0700 (PDT) Subject: [rsyslog] Upgrade In-Reply-To: <9599A350A0A5884DB4E50D83F9287D0F01F315C3AB@exchmbx01.moody.edu> References: <9599A350A0A5884DB4E50D83F9287D0F01F315C3AB@exchmbx01.moody.edu> Message-ID: unless you are also changing what protocol you use between machines, the order that you update does not matter. David Lang On Wed, 14 Sep 2011, Jeff Poling wrote: > Date: Wed, 14 Sep 2011 15:39:18 -0500 > From: Jeff Poling > Reply-To: rsyslog-users > To: "rsyslog at lists.adiscon.com" > Subject: [rsyslog] Upgrade > > Does it matter in what order I do the upgrade? i.e., do I upgrade the collector system first and then all the other systems? > > Thanks, > > Jeff > > Jeffrey Poling > System Administrator | Information Systems > Moody Bible Institute > 820 N. LaSalle Blvd., Chicago, IL 60610 > 312-329-8968 > www.moodyministries.net > From the Word. To Life. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From jpoling at moody.edu Thu Sep 15 05:37:55 2011 From: jpoling at moody.edu (Jeff Poling) Date: Wed, 14 Sep 2011 22:37:55 -0500 Subject: [rsyslog] Upgrade In-Reply-To: References: <9599A350A0A5884DB4E50D83F9287D0F01F315C3AB@exchmbx01.moody.edu> Message-ID: <9599A350A0A5884DB4E50D83F9287D0F01F315C410@exchmbx01.moody.edu> > unless you are also changing what protocol you use between machines, the > order that you update does not matter. Thank you! From yerrysherry at gmail.com Fri Sep 16 16:03:41 2011 From: yerrysherry at gmail.com (=?ISO-8859-1?Q?Gerrit_Ser=E9?=) Date: Fri, 16 Sep 2011 16:03:41 +0200 Subject: [rsyslog] installing,trying 6.3.4 => error 2207 Message-ID: Hey, I was looking at the version 6.3.4 and I did a standard installation: # lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 10.04.3 LTS Release: 10.04 Codename: lucid # ./configure --prefix=/opt/rsyslog/rsyslog-6.3.4 # /var/tmp/rsyslog-6.3.4# /opt/rsyslog/rsyslog-6.3.4/sbin/rsyslogd -v rsyslogd 6.3.4, compiled with: FEATURE_REGEXP: Yes FEATURE_LARGEFILE: No GSSAPI Kerberos 5 support: No FEATURE_DEBUG (debug build, slow code): No 32bit Atomic operations supported: Yes 64bit Atomic operations supported: No Runtime Instrumentation (slow code): No See http://www.rsyslog.com for more information. Then some extra config: # cat /etc/rsyslog.conf # /etc/rsyslog.conf Configuration file for rsyslog. # # For more information see # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html # # Default logging rules can be found in /etc/rsyslog.d/50-default.conf ################# #### MODULES #### ################# $ModLoad imuxsock # provides support for local system logging $ModLoad imklog # provides kernel logging support (previously done by rklogd) $ModLoad immark # provides --MARK-- message capability $KLogPath /proc/kmsg # provides UDP syslog reception $ModLoad imudp $UDPServerRun 514 # provides TCP syslog reception $ModLoad imtcp $InputTCPServerRun 514 ########################### #### GLOBAL DIRECTIVES #### ########################### # # Use traditional timestamp format. # To enable high precision timestamps, comment out the following line. # $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # Filter duplicated messages $RepeatedMsgReduction on # # Set the default permissions for all log files. # $FileOwner syslog $FileGroup adm $FileCreateMode 0640 $DirCreateMode 0755 $Umask 0022 $PrivDropToUser root $PrivDropToGroup root # # Include all config files in /etc/rsyslog.d/ # $IncludeConfig /etc/rsyslog.d/*.conf # cat /etc/rsyslog.d/10-local.conf $template DynaFile,"/var/log/servers/system-%HOSTNAME%.log" *.* -?DynaFile :fromhost-ip, !isequal, "127.0.0.1" ~ # /opt/rsyslog/rsyslog-6.3.4/sbin/rsyslogd rsyslogd: run failed with error -2207 (see rsyslog.h or try http://www.rsyslog.com/e/2207 to learn what that number means) I change the config: # cat /etc/rsyslog.d/10-local.conf $template DynaFile,"/var/log/servers/system-%HOSTNAME%.log" *.* -?DynaFile #:fromhost-ip, !isequal, "127.0.0.1" ~ Then, it will start the daemon. Is it a bug? Regards, Gerrit From rgerhards at hq.adiscon.com Sun Sep 18 11:19:26 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 18 Sep 2011 11:19:26 +0200 Subject: [rsyslog] installing,trying 6.3.4 => error 2207 In-Reply-To: References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA728124D@GRFEXC.intern.adiscon.com> This looks like a problem with the (new) grammar. I'll check and let you know. Most probably a bug fix is due ;) Thanks for reporting! Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Gerrit Ser? > Sent: Friday, September 16, 2011 4:04 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] installing,trying 6.3.4 => error 2207 > > Hey, > > I was looking at the version 6.3.4 and I did a standard installation: > > # lsb_release -a > No LSB modules are available. > Distributor ID: Ubuntu > Description: Ubuntu 10.04.3 LTS > Release: 10.04 > Codename: lucid > > # ./configure --prefix=/opt/rsyslog/rsyslog-6.3.4 > > # /var/tmp/rsyslog-6.3.4# /opt/rsyslog/rsyslog-6.3.4/sbin/rsyslogd -v > rsyslogd 6.3.4, compiled with: > FEATURE_REGEXP: Yes > FEATURE_LARGEFILE: No > GSSAPI Kerberos 5 support: No > FEATURE_DEBUG (debug build, slow code): No > 32bit Atomic operations supported: Yes > 64bit Atomic operations supported: No > Runtime Instrumentation (slow code): No > > See http://www.rsyslog.com for more information. > > Then some extra config: > > > # cat /etc/rsyslog.conf > # /etc/rsyslog.conf Configuration file for rsyslog. > # > # For more information see > # /usr/share/doc/rsyslog- > doc/html/rsyslog_conf.html > # > # Default logging rules can be found in /etc/rsyslog.d/50-default.conf > > > ################# > #### MODULES #### > ################# > > $ModLoad imuxsock # provides support for local system logging > $ModLoad imklog # provides kernel logging support (previously done by > rklogd) > $ModLoad immark # provides --MARK-- message capability > > $KLogPath /proc/kmsg > > # provides UDP syslog reception > $ModLoad imudp > $UDPServerRun 514 > > # provides TCP syslog reception > $ModLoad imtcp > $InputTCPServerRun 514 > > > ########################### > #### GLOBAL DIRECTIVES #### > ########################### > > # > # Use traditional timestamp format. > # To enable high precision timestamps, comment out the following line. > # > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > # Filter duplicated messages > $RepeatedMsgReduction on > > # > # Set the default permissions for all log files. > # > $FileOwner syslog > $FileGroup adm > $FileCreateMode 0640 > $DirCreateMode 0755 > $Umask 0022 > $PrivDropToUser root > $PrivDropToGroup root > > # > # Include all config files in /etc/rsyslog.d/ > # > $IncludeConfig /etc/rsyslog.d/*.conf > > # cat /etc/rsyslog.d/10-local.conf > $template DynaFile,"/var/log/servers/system-%HOSTNAME%.log" > *.* -?DynaFile > > :fromhost-ip, !isequal, "127.0.0.1" ~ > > # /opt/rsyslog/rsyslog-6.3.4/sbin/rsyslogd > rsyslogd: run failed with error -2207 (see rsyslog.h or try > http://www.rsyslog.com/e/2207 to learn what that number means) > > I change the config: > > # cat /etc/rsyslog.d/10-local.conf > $template DynaFile,"/var/log/servers/system-%HOSTNAME%.log" > *.* -?DynaFile > > #:fromhost-ip, !isequal, "127.0.0.1" ~ > > Then, it will start the daemon. Is it a bug? > > Regards, > Gerrit > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Sep 19 12:08:50 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 19 Sep 2011 12:08:50 +0200 Subject: [rsyslog] installing,trying 6.3.4 => error 2207 In-Reply-To: References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7281258@GRFEXC.intern.adiscon.com> Hi Gerrit, as I thought, a small (but somewhat hard to spot) problem with the grammar: dashes inside property names were not correctly processed. Patch is available here: http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=6ce87411a6c93b674a9c8139 28bc853620a4ae3b (the time was off on the system...) I'll probably do a new release today, even though not much has changed... Thanks again for reporting this problem! Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Gerrit Ser? > Sent: Friday, September 16, 2011 4:04 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] installing,trying 6.3.4 => error 2207 > > Hey, > > I was looking at the version 6.3.4 and I did a standard installation: > > # lsb_release -a > No LSB modules are available. > Distributor ID: Ubuntu > Description: Ubuntu 10.04.3 LTS > Release: 10.04 > Codename: lucid > > # ./configure --prefix=/opt/rsyslog/rsyslog-6.3.4 > > # /var/tmp/rsyslog-6.3.4# /opt/rsyslog/rsyslog-6.3.4/sbin/rsyslogd -v > rsyslogd 6.3.4, compiled with: > FEATURE_REGEXP: Yes > FEATURE_LARGEFILE: No > GSSAPI Kerberos 5 support: No > FEATURE_DEBUG (debug build, slow code): No > 32bit Atomic operations supported: Yes > 64bit Atomic operations supported: No > Runtime Instrumentation (slow code): No > > See http://www.rsyslog.com for more information. > > Then some extra config: > > > # cat /etc/rsyslog.conf > # /etc/rsyslog.conf Configuration file for rsyslog. > # > # For more information see > # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html > # > # Default logging rules can be found in /etc/rsyslog.d/50-default.conf > > > ################# > #### MODULES #### > ################# > > $ModLoad imuxsock # provides support for local system logging > $ModLoad imklog # provides kernel logging support (previously done by > rklogd) > $ModLoad immark # provides --MARK-- message capability > > $KLogPath /proc/kmsg > > # provides UDP syslog reception > $ModLoad imudp > $UDPServerRun 514 > > # provides TCP syslog reception > $ModLoad imtcp > $InputTCPServerRun 514 > > > ########################### > #### GLOBAL DIRECTIVES #### > ########################### > > # > # Use traditional timestamp format. > # To enable high precision timestamps, comment out the following line. > # > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > # Filter duplicated messages > $RepeatedMsgReduction on > > # > # Set the default permissions for all log files. > # > $FileOwner syslog > $FileGroup adm > $FileCreateMode 0640 > $DirCreateMode 0755 > $Umask 0022 > $PrivDropToUser root > $PrivDropToGroup root > > # > # Include all config files in /etc/rsyslog.d/ # $IncludeConfig > /etc/rsyslog.d/*.conf > > # cat /etc/rsyslog.d/10-local.conf > $template DynaFile,"/var/log/servers/system-%HOSTNAME%.log" > *.* -?DynaFile > > :fromhost-ip, !isequal, "127.0.0.1" ~ > > # /opt/rsyslog/rsyslog-6.3.4/sbin/rsyslogd > rsyslogd: run failed with error -2207 (see rsyslog.h or try > http://www.rsyslog.com/e/2207 to learn what that number means) > > I change the config: > > # cat /etc/rsyslog.d/10-local.conf > $template DynaFile,"/var/log/servers/system-%HOSTNAME%.log" > *.* -?DynaFile > > #:fromhost-ip, !isequal, "127.0.0.1" ~ > > Then, it will start the daemon. Is it a bug? > > Regards, > Gerrit > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From friedl at hq.adiscon.com Mon Sep 19 13:08:32 2011 From: friedl at hq.adiscon.com (Florian Riedl) Date: Mon, 19 Sep 2011 13:08:32 +0200 Subject: [rsyslog] rsyslog 6.3.6 (v6-devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA728125F@GRFEXC.intern.adiscon.com> This is primarily a maintenance release fixing a really annoying problem with reading the config file. ChangeLog: http://www.rsyslog.com/changelog-for-6-3-6-v6-devel/ Download: http://www.rsyslog.com/rsyslog-6-3-6-v6-devel/ As always, feedback is appreciated. Best regards, Florian Riedl From shell.heriyanto at gmail.com Thu Sep 22 13:07:45 2011 From: shell.heriyanto at gmail.com (heriyanto) Date: Thu, 22 Sep 2011 18:07:45 +0700 Subject: [rsyslog] [ASK]Need help rsyslog for JBOSS AS stacktrace and timestamp Message-ID: <4E7B1701.4020007@gmail.com> Dear All, rsyslog its very cool, i already using for all my server several country. Its very help to centralize our log files. But i still get stuck when try to make rsyslog log just like JBOSS AS output log(server.log,etc). 1. Can we make just send the file to another host using rsyslog? without any additional(date,time,host,progname,etc) i already imfile module but its still give timestamp :( double timestamp :( 2. Can we remove timestamp from rsyslog to output? 3. Can we make rsyslog just showing date and messages, without date, host and progname? template do like that? 4. Any template for JBOSS AS stacktrace? Great appreciate every reply. Thank yours for help. Best regards Heriyanto From rgerhards at hq.adiscon.com Thu Sep 22 17:10:34 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 22 Sep 2011 17:10:34 +0200 Subject: [rsyslog] [ASK]Need help rsyslog for JBOSS AS stacktrace andtimestamp In-Reply-To: <4E7B1701.4020007@gmail.com> References: <4E7B1701.4020007@gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA728129A@GRFEXC.intern.adiscon.com> Mhhh... rsyslog is not a generic file transfer tool. Given your description, it sounds like you really want to just have the files copied over to some other location. Isn't something like rsync more appropriate for the job? Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of heriyanto > Sent: Thursday, September 22, 2011 1:08 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] [ASK]Need help rsyslog for JBOSS AS stacktrace > andtimestamp > > Dear All, > > rsyslog its very cool, i already using for all my server several > country. Its very help to centralize our log files. > But i still get stuck when try to make rsyslog log just like JBOSS AS > output log(server.log,etc). > > 1. Can we make just send the file to another host using rsyslog? > without > any additional(date,time,host,progname,etc) i already imfile module but > its still give timestamp :( double timestamp :( > 2. Can we remove timestamp from rsyslog to output? > 3. Can we make rsyslog just showing date and messages, without date, > host and progname? template do like that? > 4. Any template for JBOSS AS stacktrace? > > Great appreciate every reply. Thank yours for help. > > Best regards > > Heriyanto > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Thu Sep 22 22:24:54 2011 From: david at lang.hm (david at lang.hm) Date: Thu, 22 Sep 2011 13:24:54 -0700 (PDT) Subject: [rsyslog] [ASK]Need help rsyslog for JBOSS AS stacktrace and timestamp In-Reply-To: <4E7B1701.4020007@gmail.com> References: <4E7B1701.4020007@gmail.com> Message-ID: On Thu, 22 Sep 2011, heriyanto wrote: > Dear All, > > rsyslog its very cool, i already using for all my server several country. Its > very help to centralize our log files. > But i still get stuck when try to make rsyslog log just like JBOSS AS output > log(server.log,etc). > > 1. Can we make just send the file to another host using rsyslog? without any > additional(date,time,host,progname,etc) i already imfile module but its still > give timestamp :( double timestamp :( > 2. Can we remove timestamp from rsyslog to output? > 3. Can we make rsyslog just showing date and messages, without date, host and > progname? template do like that? > 4. Any template for JBOSS AS stacktrace? > > Great appreciate every reply. Thank yours for help. you really don't want to try and mess with changing the over-the-wire protocol, but you can setup a template for the far side that ignores the timestamp that rsyslog puts in it (which I think is what you are saying for #2 and #3) I'm not sure what you are looking for for #4 one thing to remember, syslog is based on line-formatted messages. many Java error messages are significantly more complex (in my limited experience they are many lines, if not pages long, with all but the first line indented). To help with this I wrote a modification for the imfile module that could combine all these lines into one very long message. As this gets sent the newlines in the message will be replaced by and escaped version of it (#nnn), so it will be one long line on the far side. I don't know of any way to change them back in the output template, but it would be pretty trivial to send them to a program to do the conversion (it doesn't need to be some custom program, sed will do the job) David Lang From shell.heriyanto at gmail.com Fri Sep 23 08:54:04 2011 From: shell.heriyanto at gmail.com (heriyanto) Date: Fri, 23 Sep 2011 13:54:04 +0700 Subject: [rsyslog] [ASK]Need help rsyslog for JBOSS AS stacktrace andtimestamp In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA728129A@GRFEXC.intern.adiscon.com> References: <4E7B1701.4020007@gmail.com> <9B6E2A8877C38245BFB15CC491A11DA728129A@GRFEXC.intern.adiscon.com> Message-ID: <4E7C2D0C.4050307@gmail.com> Thank for your reply Rainer, yes rsync is appropriate for the job, but i love rsyslog :). On 09/22/2011 10:10 PM, Rainer Gerhards wrote: > Mhhh... rsyslog is not a generic file transfer tool. Given your description, > it sounds like you really want to just have the files copied over to some > other location. Isn't something like rsync more appropriate for the job? > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of heriyanto >> Sent: Thursday, September 22, 2011 1:08 PM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] [ASK]Need help rsyslog for JBOSS AS stacktrace >> andtimestamp >> >> Dear All, >> >> rsyslog its very cool, i already using for all my server several >> country. Its very help to centralize our log files. >> But i still get stuck when try to make rsyslog log just like JBOSS AS >> output log(server.log,etc). >> >> 1. Can we make just send the file to another host using rsyslog? >> without >> any additional(date,time,host,progname,etc) i already imfile module but >> its still give timestamp :( double timestamp :( >> 2. Can we remove timestamp from rsyslog to output? >> 3. Can we make rsyslog just showing date and messages, without date, >> host and progname? template do like that? >> 4. Any template for JBOSS AS stacktrace? >> >> Great appreciate every reply. Thank yours for help. >> >> Best regards >> >> Heriyanto >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Fri Sep 23 08:56:51 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 23 Sep 2011 08:56:51 +0200 Subject: [rsyslog] [ASK]Need help rsyslog for JBOSS AS stacktraceandtimestamp In-Reply-To: <4E7C2D0C.4050307@gmail.com> References: <4E7B1701.4020007@gmail.com><9B6E2A8877C38245BFB15CC491A11DA728129A@GRFEXC.intern.adiscon.com> <4E7C2D0C.4050307@gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72812A1@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of heriyanto > Sent: Friday, September 23, 2011 8:54 AM > To: rsyslog-users > Subject: Re: [rsyslog] [ASK]Need help rsyslog for JBOSS AS > stacktraceandtimestamp > > Thank for your reply Rainer, yes rsync is appropriate for the job, but > i > love rsyslog :). Well, while of course I appreciate your preference towards rsyslog, but experience shows that things begin to become really complicated and tend to break if you use some tool for a border case that it was not really designed for. So if your real need is to transfer those files unaltered, I'd suggest to use rsync or something along these lines. Of course, if you intend to do filtering, different processing etc, things are different. David has already provided very good advise in this case. Rainer > > On 09/22/2011 10:10 PM, Rainer Gerhards wrote: > > Mhhh... rsyslog is not a generic file transfer tool. Given your > description, > > it sounds like you really want to just have the files copied over to > some > > other location. Isn't something like rsync more appropriate for the > job? > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of heriyanto > >> Sent: Thursday, September 22, 2011 1:08 PM > >> To: rsyslog at lists.adiscon.com > >> Subject: [rsyslog] [ASK]Need help rsyslog for JBOSS AS stacktrace > >> andtimestamp > >> > >> Dear All, > >> > >> rsyslog its very cool, i already using for all my server several > >> country. Its very help to centralize our log files. > >> But i still get stuck when try to make rsyslog log just like JBOSS > AS > >> output log(server.log,etc). > >> > >> 1. Can we make just send the file to another host using rsyslog? > >> without > >> any additional(date,time,host,progname,etc) i already imfile module > but > >> its still give timestamp :( double timestamp :( > >> 2. Can we remove timestamp from rsyslog to output? > >> 3. Can we make rsyslog just showing date and messages, without date, > >> host and progname? template do like that? > >> 4. Any template for JBOSS AS stacktrace? > >> > >> Great appreciate every reply. Thank yours for help. > >> > >> Best regards > >> > >> Heriyanto > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From shell.heriyanto at gmail.com Fri Sep 23 10:01:26 2011 From: shell.heriyanto at gmail.com (heriyanto) Date: Fri, 23 Sep 2011 15:01:26 +0700 Subject: [rsyslog] [ASK]Need help rsyslog for JBOSS AS stacktrace and timestamp In-Reply-To: References: <4E7B1701.4020007@gmail.com> Message-ID: <4E7C3CD6.7060007@gmail.com> Thank very much for your reply David, its helpful. my goal is make jboss stacktrace as one event in splunk. If we can make it one long line, its great, its should be work in splunk report, even timestamp theres. what the rsyslog version that you modification for the imfile module that combine all that lines into one very long message? can you give me the configuration sample? because i already play with imfile. did you have any sample configuration to send them to a program to do conversion with sed? can we just show the time on rsyslog date format? or using 'if' if the first character is space we will make its one line. thank very much for any help, 4 days i already find out about this issue, with my bos always watching at me :D Best regards, Heriyanto On 09/23/2011 03:24 AM, david at lang.hm wrote: > On Thu, 22 Sep 2011, heriyanto wrote: > >> Dear All, >> >> rsyslog its very cool, i already using for all my server several >> country. Its very help to centralize our log files. >> But i still get stuck when try to make rsyslog log just like JBOSS AS >> output log(server.log,etc). >> >> 1. Can we make just send the file to another host using rsyslog? >> without any additional(date,time,host,progname,etc) i already imfile >> module but its still give timestamp :( double timestamp :( >> 2. Can we remove timestamp from rsyslog to output? >> 3. Can we make rsyslog just showing date and messages, without date, >> host and progname? template do like that? >> 4. Any template for JBOSS AS stacktrace? >> >> Great appreciate every reply. Thank yours for help. > > you really don't want to try and mess with changing the over-the-wire > protocol, but you can setup a template for the far side that ignores > the timestamp that rsyslog puts in it (which I think is what you are > saying for #2 and #3) > > I'm not sure what you are looking for for #4 > > one thing to remember, syslog is based on line-formatted messages. > many Java error messages are significantly more complex (in my limited > experience they are many lines, if not pages long, with all but the > first line indented). To help with this I wrote a modification for the > imfile module that could combine all these lines into one very long > message. As this gets sent the newlines in the message will be > replaced by and escaped version of it (#nnn), so it will be one long > line on the far side. I don't know of any way to change them back in > the output template, but it would be pretty trivial to send them to a > program to do the conversion (it doesn't need to be some custom > program, sed will do the job) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From jsabo at criminal.org Fri Sep 23 18:50:40 2011 From: jsabo at criminal.org (Jonathan Sabo) Date: Fri, 23 Sep 2011 12:50:40 -0400 Subject: [rsyslog] matching apache server status and error codes Message-ID: Does anyone have any rsyslog config to match apache server status and error codes? I'm trying to find a way to log error codes 500's and 400's to different logs. Would just like some advice on how to go about doing that... Thanks, Jonathan From oliver at obeattie.com Fri Sep 23 19:07:07 2011 From: oliver at obeattie.com (Oliver Beattie) Date: Fri, 23 Sep 2011 18:07:07 +0100 Subject: [rsyslog] matching apache server status and error codes In-Reply-To: References: Message-ID: <-1247782554610428040@unknownmsgid> Is this not something that would be achieved better by using Apache's error log directive to do the filtering? http://httpd.apache.org/docs/2.2/mod/core.html#errorlog On Sep 23, 2011, at 5:50 PM, Jonathan Sabo wrote: > Does anyone have any rsyslog config to match apache server status and > error codes? I'm trying to find a way to log error codes 500's and > 400's to different logs. Would just like some advice on how to go > about doing that... > > Thanks, > > Jonathan > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From jsabo at criminal.org Fri Sep 23 20:08:51 2011 From: jsabo at criminal.org (Jonathan Sabo) Date: Fri, 23 Sep 2011 14:08:51 -0400 Subject: [rsyslog] matching apache server status and error codes In-Reply-To: <-1247782554610428040@unknownmsgid> References: <-1247782554610428040@unknownmsgid> Message-ID: You can't log to different files based on error codes. So no this is not something you would achieve with the error log directives or the access log filtering based on the environment variables... I want to do it with rsyslog filters... "Although we have just shown that conditional logging is very powerful and flexible, it is not the only way to control the contents of the logs. Log files are more useful when they contain a complete record of server activity. It is often easier to simply post-process the log files to remove requests that you do not want to consider." Any one out there doing this already? Thanks, Jonathan On Fri, Sep 23, 2011 at 1:07 PM, Oliver Beattie wrote: > Is this not something that would be achieved better by using Apache's > error log directive to do the filtering? > http://httpd.apache.org/docs/2.2/mod/core.html#errorlog > > On Sep 23, 2011, at 5:50 PM, Jonathan Sabo wrote: > >> Does anyone have any rsyslog config to match apache server status and >> error codes? ?I'm trying to find a way to log error codes 500's and >> 400's to different logs. ?Would just like some advice on how to go >> about doing that... >> >> Thanks, >> >> Jonathan >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Fri Sep 23 21:27:19 2011 From: david at lang.hm (david at lang.hm) Date: Fri, 23 Sep 2011 12:27:19 -0700 (PDT) Subject: [rsyslog] [ASK]Need help rsyslog for JBOSS AS stacktrace and timestamp In-Reply-To: <4E7C3CD6.7060007@gmail.com> References: <4E7B1701.4020007@gmail.com> <4E7C3CD6.7060007@gmail.com> Message-ID: I don't remember the version that this got into, if you use the latest stable 5.x release you should be good. if you are wanting to get things into splunk, you have two options. 1. use rsyslog to combine everything into one line and send it in the one-line format to splunk. to do this 'right' you should teach splunk that #nnn is a control character and should be treated as a separator. 2. run a splunk agent against the native log file and let it put things into splunk. you are probably better off running the splunk agent where your log file is than you are to try to copy the logfile to a different machine and have it processed there. David Lang On Fri, 23 Sep 2011, heriyanto wrote: > Thank very much for your reply David, its helpful. my goal is make jboss > stacktrace as one event in splunk. > If we can make it one long line, its great, its should be work in splunk > report, even timestamp theres. > what the rsyslog version that you modification for the imfile module that > combine all that lines into one very long message? can you give me the > configuration sample? because i already play with imfile. > did you have any sample configuration to send them to a program to do > conversion with sed? > can we just show the time on rsyslog date format? or using 'if' if the first > character is space we will make its one line. > thank very much for any help, 4 days i already find out about this issue, > with my bos always watching at me :D > > Best regards, > > Heriyanto > > > On 09/23/2011 03:24 AM, david at lang.hm wrote: >> On Thu, 22 Sep 2011, heriyanto wrote: >> >>> Dear All, >>> >>> rsyslog its very cool, i already using for all my server several country. >>> Its very help to centralize our log files. >>> But i still get stuck when try to make rsyslog log just like JBOSS AS >>> output log(server.log,etc). >>> >>> 1. Can we make just send the file to another host using rsyslog? without >>> any additional(date,time,host,progname,etc) i already imfile module but >>> its still give timestamp :( double timestamp :( >>> 2. Can we remove timestamp from rsyslog to output? >>> 3. Can we make rsyslog just showing date and messages, without date, host >>> and progname? template do like that? >>> 4. Any template for JBOSS AS stacktrace? >>> >>> Great appreciate every reply. Thank yours for help. >> >> you really don't want to try and mess with changing the over-the-wire >> protocol, but you can setup a template for the far side that ignores the >> timestamp that rsyslog puts in it (which I think is what you are saying for >> #2 and #3) >> >> I'm not sure what you are looking for for #4 >> >> one thing to remember, syslog is based on line-formatted messages. many >> Java error messages are significantly more complex (in my limited >> experience they are many lines, if not pages long, with all but the first >> line indented). To help with this I wrote a modification for the imfile >> module that could combine all these lines into one very long message. As >> this gets sent the newlines in the message will be replaced by and escaped >> version of it (#nnn), so it will be one long line on the far side. I don't >> know of any way to change them back in the output template, but it would be >> pretty trivial to send them to a program to do the conversion (it doesn't >> need to be some custom program, sed will do the job) >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Fri Sep 23 21:28:45 2011 From: david at lang.hm (david at lang.hm) Date: Fri, 23 Sep 2011 12:28:45 -0700 (PDT) Subject: [rsyslog] [ASK]Need help rsyslog for JBOSS AS stacktraceandtimestamp In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA72812A1@GRFEXC.intern.adiscon.com> References: <4E7B1701.4020007@gmail.com><9B6E2A8877C38245BFB15CC491A11DA728129A@GRFEXC.intern.adiscon.com> <4E7C2D0C.4050307@gmail.com> <9B6E2A8877C38245BFB15CC491A11DA72812A1@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 23 Sep 2011, Rainer Gerhards wrote: > Well, while of course I appreciate your preference towards rsyslog, but > experience shows that things begin to become really complicated and tend to > break if you use some tool for a border case that it was not really designed > for. So if your real need is to transfer those files unaltered, I'd suggest > to use rsync or something along these lines. Of course, if you intend to do > filtering, different processing etc, things are different. David has already > provided very good advise in this case. I went looking in the documentation for the log separator control and couldn't find it (I was tryng to send a link to someone else). was I just missing it in some obvious place or is it missing from the page on imfile and the master config page? David Lang From rgerhards at hq.adiscon.com Fri Sep 23 21:31:16 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 23 Sep 2011 21:31:16 +0200 Subject: [rsyslog] [ASK]Need help rsyslog for JBOSS AS stacktraceandtimestamp In-Reply-To: References: <4E7B1701.4020007@gmail.com><9B6E2A8877C38245BFB15CC491A11DA728129A@GRFEXC.intern.adiscon.com><4E7C2D0C.4050307@gmail.com><9B6E2A8877C38245BFB15CC491A11DA72812A1@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72812A6@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, September 23, 2011 9:29 PM > To: rsyslog-users > Subject: Re: [rsyslog] [ASK]Need help rsyslog for JBOSS AS > stacktraceandtimestamp > > On Fri, 23 Sep 2011, Rainer Gerhards wrote: > > > Well, while of course I appreciate your preference towards rsyslog, > but > > experience shows that things begin to become really complicated and > tend to > > break if you use some tool for a border case that it was not really > designed > > for. So if your real need is to transfer those files unaltered, I'd > suggest > > to use rsync or something along these lines. Of course, if you intend > to do > > filtering, different processing etc, things are different. David has > already > > provided very good advise in this case. > > I went looking in the documentation for the log separator control and > couldn't find it (I was tryng to send a link to someone else). was I > just > missing it in some obvious place or is it missing from the page on > imfile > and the master config page? Quite honestly, I think we never had any doc on that... I know I merged your patch, but there was no doc accompanying it and I flagged it as experimental, so I never investigated it in depth. Rainer From david at lang.hm Fri Sep 23 21:33:01 2011 From: david at lang.hm (david at lang.hm) Date: Fri, 23 Sep 2011 12:33:01 -0700 (PDT) Subject: [rsyslog] matching apache server status and error codes In-Reply-To: References: Message-ID: On Fri, 23 Sep 2011, Jonathan Sabo wrote: > Does anyone have any rsyslog config to match apache server status and > error codes? I'm trying to find a way to log error codes 500's and > 400's to different logs. Would just like some advice on how to go > about doing that... it gets a little ugly to do this with the default log format because you don't have a really good way of matching only at a specific position (the fields are space separated and earlier fields can contain embedded spaces) if you can change your format to put the result code in a more predictable place it becomes much easier to match. for example, if you were to put the result code first, you could then match for %msg starting with '4' or '5'. without changing the format, you can do a regex match for " 4[0-9][0-9] " and probably get pretty good results (you may get some false positives, but by including the spaces before and after the value it hopefully won't be too bad) David Lang From sean at conman.org Fri Sep 23 22:05:59 2011 From: sean at conman.org (Sean Conner) Date: Fri, 23 Sep 2011 16:05:59 -0400 Subject: [rsyslog] matching apache server status and error codes In-Reply-To: References: Message-ID: <20110923200559.GA31754@brevard.conman.org> It was thus said that the Great david at lang.hm once stated: > On Fri, 23 Sep 2011, Jonathan Sabo wrote: > > without changing the format, you can do a regex match for " 4[0-9][0-9] " > and probably get pretty good results (you may get some false positives, > but by including the spaces before and after the value it hopefully won't > be too bad) A better regex would be \" 4[0-9][0-9] [0-9]+ \" Just doing a " 4[0-9][0-9] " could possibly a valid response that was in the 400-409 byte range. What I gave above at least does a better matching of that part of the line (if you are using one of the default formats from Apache). -spc From victor.lu at citi.com Mon Sep 26 18:18:35 2011 From: victor.lu at citi.com (Lu, Victor ) Date: Mon, 26 Sep 2011 11:18:35 -0500 Subject: [rsyslog] Duplicated messages on Solaris Message-ID: <35B12B7283BF44478AFA717323EE52951CA8AB3585@extxmb32.nam.nsroot.net> On Solaris, 1) If I use both $Modload ImkLog and $Modload imsolaris, A logger command will always generate message twice. 2011-09-26T11:08:46-04:00 i8-420-02 test: [ID 702911 user.notice] This is a test 2011-09-26T11:08:46.962612-04:00 i8-420-02 kernel: Sep 26 11:08:46 test: [ID 702911 user.notice] This is a test su command will return only one message. 2011-09-26T12:08:21.643321-04:00 i8-420-02 kernel: Sep 26 12:08:21 su: [ID 366847 auth.info] 'su root' succeeded for vl10243 on /dev/pts/4 2) If I use $Modload imklog only, the logger command will return only one message. 2011-09-26T12:02:20.667780-04:00 i8-420-02 kernel: Sep 26 12:02:20 test: [ID 702911 user.notice] this is a test su command will return only one message. 2011-09-26T12:02:47.700657-04:00 i8-420-02 kernel: Sep 26 12:02:47 su: [ID 366847 auth.info] 'su root' succeeded for vl10243 on /dev/pts/4 3) If I use $Modload imsolaris only The logger command will return the following message. 2011-09-26T12:06:01-04:00 i8-420-02 test: [ID 702911 user.notice] this is a test su command will not return any message. I only need one message to be generated in the system log (same on Linux), not duplicated. It looks like I can use imklog module alone to capture both kernel and logger command message. But I am not sure if I still could miss other type of system events without using imsolaris module. For the kernel message generated, I don't like duplicated time stamp For example, the following event, 2011-09-26T12:02:20.667780-04:00 i8-420-02 kernel: Sep 26 12:02:20 test: [ID 702911 user.notice] this is a test The timestamp after kernel: Sep 26 12:02:20 because I already have the event time 2011-09-26T12:02:20.667780-04:00. Any suggestions? Anybody have a sample rsyslog.conf on Solaris to share? Thanks Victor Lu From rgerhards at hq.adiscon.com Mon Sep 26 18:55:25 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 26 Sep 2011 18:55:25 +0200 Subject: [rsyslog] Duplicated messages on Solaris In-Reply-To: <35B12B7283BF44478AFA717323EE52951CA8AB3585@extxmb32.nam.nsroot.net> References: <35B12B7283BF44478AFA717323EE52951CA8AB3585@extxmb32.nam.nsroot.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72812A9@GRFEXC.intern.adiscon.com> Mhhh... I have no idea why Solaris' logger writes to both locations. But I also don't see how I should tell which one to drop... As of the timestamps: are you sure you use the newest version of the branch in question? I remember that I recently fixed something in that regard. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Lu, Victor > Sent: Monday, September 26, 2011 6:19 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Duplicated messages on Solaris > > On Solaris, > > > 1) If I use both $Modload ImkLog and $Modload imsolaris, > > A logger command will always generate message twice. > 2011-09-26T11:08:46-04:00 i8-420-02 test: [ID 702911 > user.notice] This is a test > 2011-09-26T11:08:46.962612-04:00 i8-420-02 kernel: Sep 26 > 11:08:46 test: [ID 702911 user.notice] This is a test > > su command will return only one message. > 2011-09-26T12:08:21.643321-04:00 i8-420-02 kernel: Sep 26 > 12:08:21 su: [ID 366847 auth.info] 'su root' succeeded for vl10243 on > /dev/pts/4 > > > 2) If I use $Modload imklog only, the logger command will return > only one message. > > 2011-09-26T12:02:20.667780-04:00 i8-420-02 kernel: Sep 26 > 12:02:20 test: [ID 702911 user.notice] this is a test > > su command will return only one message. > > 2011-09-26T12:02:47.700657-04:00 i8-420-02 kernel: Sep 26 > 12:02:47 su: [ID 366847 auth.info] 'su root' succeeded for vl10243 on > /dev/pts/4 > > > 3) If I use $Modload imsolaris only > > The logger command will return the following message. > > 2011-09-26T12:06:01-04:00 i8-420-02 test: [ID 702911 > user.notice] this is a test > > su command will not return any message. > > I only need one message to be generated in the system log (same on > Linux), not duplicated. > > It looks like I can use imklog module alone to capture both kernel and > logger command message. But I am not sure if I still could miss other > type of system events without using imsolaris module. > > For the kernel message generated, I don't like duplicated time stamp > > For example, the following event, > 2011-09-26T12:02:20.667780-04:00 i8-420-02 kernel: Sep 26 12:02:20 > test: [ID 702911 user.notice] this is a test > > The timestamp after kernel: Sep 26 12:02:20 because I already have > the event time 2011-09-26T12:02:20.667780-04:00. > > Any suggestions? Anybody have a sample rsyslog.conf on Solaris to > share? > > Thanks > > Victor Lu > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From victor.lu at citi.com Mon Sep 26 19:24:57 2011 From: victor.lu at citi.com (Lu, Victor ) Date: Mon, 26 Sep 2011 12:24:57 -0500 Subject: [rsyslog] Duplicated messages on Solaris In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA72812A9@GRFEXC.intern.adiscon.com> References: <35B12B7283BF44478AFA717323EE52951CA8AB3585@extxmb32.nam.nsroot.net> <9B6E2A8877C38245BFB15CC491A11DA72812A9@GRFEXC.intern.adiscon.com> Message-ID: <35B12B7283BF44478AFA717323EE52951CA8AB3731@extxmb32.nam.nsroot.net> Hi Rainer, Thanks for quick response. For product version, I am using the latest stable version 5.8.5. Could you let me know which version fixed timestamp issue and how the message look like after the fix. The following is what you posted on the web site. Is this because of special kernel input device that produced duplicated message? Any suggestions to have the same behavior like what we have on Linux? Website http://www.rsyslog.com/doc/imsolaris.html Solaris Input Module Module Name: imsolaris Author: Rainer Gerhards Description: Reads local Solaris log messages including the kernel log. This module is specifically tailored for Solaris. Under Solaris, there is no special kernel input device. Instead, both kernel messages as well as messages emitted via syslog() are received from a single source. This module obeys the Solaris door() mechanism to detect a running syslogd instance. As such, only one can be active at one time. If it detects another active intance at startup, the module disables itself, but rsyslog will continue to run. Configuration Directives: $IMSolarisLogSocketName This is the name of the log socket (stream) to read. If not given, /dev/log is read. -----Original Message----- From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards Sent: Monday, September 26, 2011 12:55 PM To: rsyslog-users Subject: Re: [rsyslog] Duplicated messages on Solaris Mhhh... I have no idea why Solaris' logger writes to both locations. But I also don't see how I should tell which one to drop... As of the timestamps: are you sure you use the newest version of the branch in question? I remember that I recently fixed something in that regard. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Lu, Victor > Sent: Monday, September 26, 2011 6:19 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Duplicated messages on Solaris > > On Solaris, > > > 1) If I use both $Modload ImkLog and $Modload imsolaris, > > A logger command will always generate message twice. > 2011-09-26T11:08:46-04:00 i8-420-02 test: [ID 702911 > user.notice] This is a test > 2011-09-26T11:08:46.962612-04:00 i8-420-02 kernel: Sep 26 > 11:08:46 test: [ID 702911 user.notice] This is a test > > su command will return only one message. > 2011-09-26T12:08:21.643321-04:00 i8-420-02 kernel: Sep 26 > 12:08:21 su: [ID 366847 auth.info] 'su root' succeeded for vl10243 on > /dev/pts/4 > > > 2) If I use $Modload imklog only, the logger command will return > only one message. > > 2011-09-26T12:02:20.667780-04:00 i8-420-02 kernel: Sep 26 > 12:02:20 test: [ID 702911 user.notice] this is a test > > su command will return only one message. > > 2011-09-26T12:02:47.700657-04:00 i8-420-02 kernel: Sep 26 > 12:02:47 su: [ID 366847 auth.info] 'su root' succeeded for vl10243 on > /dev/pts/4 > > > 3) If I use $Modload imsolaris only > > The logger command will return the following message. > > 2011-09-26T12:06:01-04:00 i8-420-02 test: [ID 702911 > user.notice] this is a test > > su command will not return any message. > > I only need one message to be generated in the system log (same on > Linux), not duplicated. > > It looks like I can use imklog module alone to capture both kernel and > logger command message. But I am not sure if I still could miss other > type of system events without using imsolaris module. > > For the kernel message generated, I don't like duplicated time stamp > > For example, the following event, > 2011-09-26T12:02:20.667780-04:00 i8-420-02 kernel: Sep 26 12:02:20 > test: [ID 702911 user.notice] this is a test > > The timestamp after kernel: Sep 26 12:02:20 because I already have > the event time 2011-09-26T12:02:20.667780-04:00. > > Any suggestions? Anybody have a sample rsyslog.conf on Solaris to > share? > > Thanks > > Victor Lu > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From jsabo at criminal.org Mon Sep 26 23:05:27 2011 From: jsabo at criminal.org (Jonathan Sabo) Date: Mon, 26 Sep 2011 17:05:27 -0400 Subject: [rsyslog] matching apache server status and error codes In-Reply-To: <20110923200559.GA31754@brevard.conman.org> References: <20110923200559.GA31754@brevard.conman.org> Message-ID: I'm having some trouble getting rsyslog to accept the syntax... if $programname == 'apache' and $msg regex \" 2[0-9][0-9] [0-9]+ \" then @loghost & ~ I was trying a bunch of different versions of above and wasn't able to get any of them to work correctly. 0273.869497216:main thread: skipped whitespace, stream now 'regex " 4[0-9][0-9] " then @loghost' 0273.869510366:main thread: skipped whitespace, stream now 'regex " 4[0-9][0-9] " then @loghost' 0273.869523524:main thread: parser has an invalid word (token) 'regex' Can you use regex with expression based filters? Is there a way to do what I'm trying to do? Thanks, Jonathan On Fri, Sep 23, 2011 at 4:05 PM, Sean Conner wrote: > It was thus said that the Great david at lang.hm once stated: >> On Fri, 23 Sep 2011, Jonathan Sabo wrote: >> >> without changing the format, you can do a regex match for " 4[0-9][0-9] " >> and probably get pretty good results (you may get some false positives, >> but by including the spaces before and after the value it hopefully won't >> be too bad) > > ?A better regex would be > > ? ? ? ?\" 4[0-9][0-9] [0-9]+ \" > > ?Just doing a " 4[0-9][0-9] " could possibly a valid response that was in > the 400-409 byte range. ?What I gave above at least does a better matching > of that part of the line (if you are using one of the default formats from > Apache). > > ?-spc > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From victor.lu at citi.com Wed Sep 28 19:08:45 2011 From: victor.lu at citi.com (Lu, Victor ) Date: Wed, 28 Sep 2011 12:08:45 -0500 Subject: [rsyslog] Kill -HUP does not work on Solaris version 5.8.5 Message-ID: <35B12B7283BF44478AFA717323EE52951CA8DAB17C@extxmb32.nam.nsroot.net> Hi There, I am testing to send kill -HUP command to rsyslogd process on Solaris, It is supposed to reread the /etc/rsyslog.conf file, but it did not. Anybody experience the same issue? It works on RHEL platform though. The rsyslogd version of the software is 5.8.5 for Solaris. Thanks Victor From abiacco at formatdynamics.com Wed Sep 28 19:33:25 2011 From: abiacco at formatdynamics.com (Anthony J. Biacco) Date: Wed, 28 Sep 2011 11:33:25 -0600 Subject: [rsyslog] inputfilemonitor with log truncate Message-ID: <40D5E3C3CBE03D45A776E7C358BDE908042AD36A@missoula.formatdynamics.com> Hello, I'm running 5.8.4 compiled on centos 5.6. I'm running rsyslog on a couple of my app servers and using the imfile module to read my tomcat logs, and then I forward them onto a central rsyslog server, also 5.8.4, with dated directories. Every week or so I want to zero out/truncate my tomcat logs on the app servers. So I've tried just doing a "echo > /path/to/log" and see how it went. Well, then I tried to append some text "echo BLAH >> /path/to/log" to the tomcat log and the file monitor didn't seem to pick it up and forward it on to central. So I HUPed rsyslogd. Retry append, still nothing. Set "$HUPisRestart On". Retry, still nothing. Restart rsyslogd service. Retry, still nothing. Remove rsyslog state file for the log file, Restart rsyslogd service. Retry, still nothing. Stop rsyslogd service. Remove rsyslog state file for the log file. Start rsyslogd service. Retry. WORKS. Is this really how it's supposed to work? Or am I missing something? Thanks in advance for any help. Command line switches are: -c4 -f /etc/rsyslog.conf -i /var/run/rsyslogd.pid Relevant config is: # Input config $UDPServerRun 514 $InputTCPServerRun 514 $MaxMessageSize 64k $ActionQueueType LinkedList $ActionQueueFileName disk-assisted-queue-file $ActionResumeRetryCount -1 $ActionQueueMaxDiskSpace 500M $ActionQueueSaveOnShutdown on $OMFileFlushOnTXEnd on $ActionFileDefaultTemplate TraditionalFormat $template RFC3164fmt,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%" # catalina.out $InputFileName /data/logs/tomcat/catalina.out $InputFileTag tomcat-out: $InputFileStateFile stat-catalina.out $InputFileSeverity debug $InputFileFacility local0 $InputRunFileMonitor $InputFilePollInterval 3 $InputFilePersistStateInterval 5000 # Output to corporate *.* @@x.x.x.x:xxx;RFC3164fmt $ActionExecOnlyWhenPreviousIsSuspended on & /var/log/localbuffer $ActionExecOnlyWhenPreviousIsSuspended off -Tony --------------------------- Manager, IT Operations Format Dynamics, Inc. P: 303-228-7327 F: 303-228-7305 abiacco at formatdynamics.com http://www.formatdynamics.com From mbiebl at gmail.com Wed Sep 28 23:10:36 2011 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 28 Sep 2011 23:10:36 +0200 Subject: [rsyslog] Kill -HUP does not work on Solaris version 5.8.5 In-Reply-To: <35B12B7283BF44478AFA717323EE52951CA8DAB17C@extxmb32.nam.nsroot.net> References: <35B12B7283BF44478AFA717323EE52951CA8DAB17C@extxmb32.nam.nsroot.net> Message-ID: 2011/9/28 Lu, Victor : > Hi There, > > I am testing to send kill -HUP command to rsyslogd process on Solaris, It is supposed to reread the /etc/rsyslog.conf file, but it did not. Anybody experience the same issue? It works on RHEL platform though. > > The rsyslogd version of the software is 5.8.5 for Solaris. http://www.rsyslog.com/doc/v4compatibility.html http://www.rsyslog.com/doc/v5compatibility.html -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Sep 29 07:43:07 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 29 Sep 2011 07:43:07 +0200 Subject: [rsyslog] inputfilemonitor with log truncate In-Reply-To: <40D5E3C3CBE03D45A776E7C358BDE908042AD36A@missoula.formatdynamics.com> References: <40D5E3C3CBE03D45A776E7C358BDE908042AD36A@missoula.formatdynamics.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72812C4@GRFEXC.intern.adiscon.com> File trunction is NOT handled by the current imfile. Previous versions tried to do that, but this resulted in a big nightmare, because there are various unavoidable races in the process that often resulted in a full re-send of the data. So it is important that a new file is created (aka the old one deleted). We could not see any problem with that approach, as a log file typically is appended but never truncated in some place. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Anthony J. Biacco > Sent: Wednesday, September 28, 2011 7:33 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] inputfilemonitor with log truncate > > Hello, > > I'm running 5.8.4 compiled on centos 5.6. > I'm running rsyslog on a couple of my app servers and using the imfile > module to read my tomcat logs, and then I forward them onto a central > rsyslog server, also 5.8.4, with dated directories. > Every week or so I want to zero out/truncate my tomcat logs on the app > servers. So I've tried just doing a "echo > /path/to/log" and see how > it > went. > Well, then I tried to append some text "echo BLAH >> /path/to/log" to > the tomcat log and the file monitor didn't seem to pick it up and > forward it on to central. > So I HUPed rsyslogd. Retry append, still nothing. > Set "$HUPisRestart On". Retry, still nothing. > Restart rsyslogd service. Retry, still nothing. > Remove rsyslog state file for the log file, Restart rsyslogd service. > Retry, still nothing. > Stop rsyslogd service. Remove rsyslog state file for the log file. > Start > rsyslogd service. Retry. WORKS. > > Is this really how it's supposed to work? Or am I missing something? > Thanks in advance for any help. > > Command line switches are: > > -c4 -f /etc/rsyslog.conf -i /var/run/rsyslogd.pid > > Relevant config is: > > # Input config > $UDPServerRun 514 > $InputTCPServerRun 514 > $MaxMessageSize 64k > > $ActionQueueType LinkedList > $ActionQueueFileName disk-assisted-queue-file > $ActionResumeRetryCount -1 > $ActionQueueMaxDiskSpace 500M > $ActionQueueSaveOnShutdown on > $OMFileFlushOnTXEnd on > > $ActionFileDefaultTemplate TraditionalFormat > $template RFC3164fmt,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%" > > # catalina.out > $InputFileName /data/logs/tomcat/catalina.out > $InputFileTag tomcat-out: > $InputFileStateFile stat-catalina.out > $InputFileSeverity debug > $InputFileFacility local0 > $InputRunFileMonitor > > $InputFilePollInterval 3 > $InputFilePersistStateInterval 5000 > > # Output to corporate > *.* @@x.x.x.x:xxx;RFC3164fmt > $ActionExecOnlyWhenPreviousIsSuspended on > & /var/log/localbuffer > $ActionExecOnlyWhenPreviousIsSuspended off > > -Tony > --------------------------- > Manager, IT Operations > Format Dynamics, Inc. > P: 303-228-7327 > F: 303-228-7305 > abiacco at formatdynamics.com > http://www.formatdynamics.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From bison at garbagebrain.org Thu Sep 29 18:36:39 2011 From: bison at garbagebrain.org (Brad Ison) Date: Thu, 29 Sep 2011 11:36:39 -0500 Subject: [rsyslog] Incorrect dynamic file names Message-ID: Hello, I'm trying to confirm if there's something wrong with my configuration, or if I may be running into a bug. I'm using rsyslog 5.8.1 and running into an issue with some log messages (seemingly randomly) being written to the wrong files. On my client machines I forward messages for a number of applications whose names all begin with "app_" to a remote server using TCP and TLS. Here's the relevant part of my client config: $template Custom_ForwardFormat,"<%PRI%>%TIMESTAMP:::date-rfc3339% %FROMHOST% %syslogtag%%msg:::sp-if-no-1st-sp%%msg%" $ActionQueueType LinkedList $ActionQueueFileName forward $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on if $programname startswith 'app_' then @@10.0.0.1:10514;Custom_ForwardFormat And on the server I send these to separate files based on the value of %PROGRAMNAME%: $template Custom_TraditionalFileFormat,"%TIMESTAMP% %FROMHOST% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" $template production-file,"/var/log/company/production/%PROGRAMNAME%.log" $template production-errors-file,"/var/log/company/production/%PROGRAMNAME%-errors.log" $RuleSet production $ActionQueueType LinkedList # use asynchronous processing $ActionQueueFileName production # set file name, also enables disk mode $ActionResumeRetryCount -1 # infinite retries on insert failure $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down $RulesetCreateMainQueue on # create ruleset-specific queue *.* ?production-file;Custom_TraditionalFileFormat *.err ?production-errors-file;Custom_TraditionalFileFormat $InputTCPServerBindRuleset production $InputTCPServerRun 10514 So, for the most part everything works normally, and I get files like this: /var/log/company/production/app_location_engine.log /var/log/company/production/app_location_webapp.log But, occasionaly, I'll get odd files like: /var/log/company/production/app_location_webine-errors.log /var/log/company/production/.log However, the app names appear correctly in the syslog tag logged to messages in those files. Any help is much appreciated! -- Brad From rgerhards at hq.adiscon.com Fri Sep 30 07:33:56 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 30 Sep 2011 07:33:56 +0200 Subject: [rsyslog] Incorrect dynamic file names In-Reply-To: References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72812CE@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Brad Ison > Sent: Thursday, September 29, 2011 6:37 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Incorrect dynamic file names > > Hello, > > I'm trying to confirm if there's something wrong with my > configuration, or if I may be running into a bug. I'm using rsyslog > 5.8.1 and running into an issue with some log messages (seemingly You should upgrade to the current version. 5.8.1 is missing many patches. I guess the problem goes away once you have done that... Rainer > randomly) being written to the wrong files. > > On my client machines I forward messages for a number of applications > whose names all begin with "app_" to a remote server using TCP and > TLS. Here's the relevant part of my client config: > > $template Custom_ForwardFormat,"<%PRI%>%TIMESTAMP:::date-rfc3339% > %FROMHOST% %syslogtag%%msg:::sp-if-no-1st-sp%%msg%" > > $ActionQueueType LinkedList > $ActionQueueFileName forward > $ActionResumeRetryCount -1 > $ActionQueueSaveOnShutdown on > > if $programname startswith 'app_' then > @@10.0.0.1:10514;Custom_ForwardFormat > > > And on the server I send these to separate files based on the value of > %PROGRAMNAME%: > > $template Custom_TraditionalFileFormat,"%TIMESTAMP% %FROMHOST% > %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" > > $template production- > file,"/var/log/company/production/%PROGRAMNAME%.log" > $template production-errors- > file,"/var/log/company/production/%PROGRAMNAME%-errors.log" > > $RuleSet production > $ActionQueueType LinkedList # use asynchronous processing > $ActionQueueFileName production # set file name, also enables disk > mode > $ActionResumeRetryCount -1 # infinite retries on insert failure > $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog > shuts down > $RulesetCreateMainQueue on # create ruleset-specific queue > *.* ?production-file;Custom_TraditionalFileFormat > *.err ?production-errors-file;Custom_TraditionalFileFormat > > $InputTCPServerBindRuleset production > $InputTCPServerRun 10514 > > > So, for the most part everything works normally, and I get files like > this: > > /var/log/company/production/app_location_engine.log > /var/log/company/production/app_location_webapp.log > > But, occasionaly, I'll get odd files like: > > /var/log/company/production/app_location_webine-errors.log > /var/log/company/production/.log > > However, the app names appear correctly in the syslog tag logged to > messages in those files. > > Any help is much appreciated! > > -- > Brad > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com