From bxu at italkbb.com  Mon May  9 17:48:50 2011
From: bxu at italkbb.com (bxu)
Date: Mon, 9 May 2011 10:48:50 -0500
Subject: [rsyslog] help : CPU high load of rsyslog writing to Oracle
Message-ID: <002101cc0e60$9aa7f6d0$cff7e470$@italkbb.com>

Hi All,


I have configured rsyslog writing to Oracle.The writing was successful.But
after several messages have been made into Oracle,the CPU use percentage of
rsyslogd goes up to 99%-100%.It is abnormally too much high.
There's no problem with the cpu usage when writing to files.

Here is my rsyslogd.conf:
#====================================================
#*.* >192.168.45.213,rsyslog

# Use traditional timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Provides kernel logging support (previously done by rklogd)
$ModLoad imklog
# Provides support for local system logging (e.g. via logger command)
$ModLoad imuxsock

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages

# The authpriv file has restricted access.
authpriv.* /var/log/secure

# Log all the mail messages in one place.
mail.* -/var/log/maillog

# Log cron stuff
cron.* /var/log/cron

# Everybody gets emergency messages
*.emerg *

# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log
local7.* /var/log/boot.log

#Load External Listender
$modload imudp


$UDPServerRun 514
#SPA Messages
#*.* /var/log/linksys.log
$template DynaFile,"/var/log/system-%HOSTNAME%.log"
*.* -?DynaFile


################
#### ORACLE ####
################

$ModLoad omoracle
$OmoracleDBUser syslog
$OmoracleDBPassword syslog
$OmoracleDB syslog
$OmoracleBatchSize 1
$OmoracleBatchItemSize 4096
$OmoracleStatementTemplate OmoracleStatement

$template OmoracleStatement,"INSERT INTO TEST(hostname,message)
VALUES(:hostname,:msg)"
$template TestStmt,"%hostname%%msg%"

*.*      :omoracle:;TestStmt
#====================================================

Really appreciate anyone's help!

Best 


From david at lang.hm  Mon May  9 19:56:30 2011
From: david at lang.hm (david at lang.hm)
Date: Mon, 9 May 2011 10:56:30 -0700 (PDT)
Subject: [rsyslog] help : CPU high load of rsyslog writing to Oracle
In-Reply-To: <002101cc0e60$9aa7f6d0$cff7e470$@italkbb.com>
References: <002101cc0e60$9aa7f6d0$cff7e470$@italkbb.com>
Message-ID: <alpine.DEB.2.00.1105091055030.25291@asgard.lang.hm>

given that it only happens after a few messages, I would suspect that 
there is something odd with the messages.

could you try writing the messages to a file using the Oracle template and 
then look and see if there is anything odd about the messages when it goes 
nuts?

the other option is to get a debug dump, but unless it happens right away, 
that is likely to be a large dump.

David Lang

  On Mon, 9 May 2011, bxu wrote:

> Date: Mon, 9 May 2011 10:48:50 -0500
> From: bxu <bxu at italkbb.com>
> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] help : CPU high load of rsyslog writing to Oracle
> 
> Hi All,
>
>
> I have configured rsyslog writing to Oracle.The writing was successful.But
> after several messages have been made into Oracle,the CPU use percentage of
> rsyslogd goes up to 99%-100%.It is abnormally too much high.
> There's no problem with the cpu usage when writing to files.
>
> Here is my rsyslogd.conf:
> #====================================================
> #*.* >192.168.45.213,rsyslog
>
> # Use traditional timestamp format
> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>
> # Provides kernel logging support (previously done by rklogd)
> $ModLoad imklog
> # Provides support for local system logging (e.g. via logger command)
> $ModLoad imuxsock
>
> # Log all kernel messages to the console.
> # Logging much else clutters up the screen.
> #kern.* /dev/console
>
> # Log anything (except mail) of level info or higher.
> # Don't log private authentication messages!
> *.info;mail.none;authpriv.none;cron.none /var/log/messages
>
> # The authpriv file has restricted access.
> authpriv.* /var/log/secure
>
> # Log all the mail messages in one place.
> mail.* -/var/log/maillog
>
> # Log cron stuff
> cron.* /var/log/cron
>
> # Everybody gets emergency messages
> *.emerg *
>
> # Save news errors of level crit and higher in a special file.
> uucp,news.crit /var/log/spooler
>
> # Save boot messages also to boot.log
> local7.* /var/log/boot.log
>
> #Load External Listender
> $modload imudp
>
>
> $UDPServerRun 514
> #SPA Messages
> #*.* /var/log/linksys.log
> $template DynaFile,"/var/log/system-%HOSTNAME%.log"
> *.* -?DynaFile
>
>
> ################
> #### ORACLE ####
> ################
>
> $ModLoad omoracle
> $OmoracleDBUser syslog
> $OmoracleDBPassword syslog
> $OmoracleDB syslog
> $OmoracleBatchSize 1
> $OmoracleBatchItemSize 4096
> $OmoracleStatementTemplate OmoracleStatement
>
> $template OmoracleStatement,"INSERT INTO TEST(hostname,message)
> VALUES(:hostname,:msg)"
> $template TestStmt,"%hostname%%msg%"
>
> *.*      :omoracle:;TestStmt
> #====================================================
>
> Really appreciate anyone's help!
>
> Best
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From bxu at italkbb.com  Tue May 10 07:25:14 2011
From: bxu at italkbb.com (bxu)
Date: Tue, 10 May 2011 00:25:14 -0500
Subject: [rsyslog] help : CPU high load of rsyslog writing to Oracle
In-Reply-To: <alpine.DEB.2.00.1105091055030.25291@asgard.lang.hm>
References: <002101cc0e60$9aa7f6d0$cff7e470$@italkbb.com>
	<alpine.DEB.2.00.1105091055030.25291@asgard.lang.hm>
Message-ID: <001801cc0ed2$a7081eb0$f5185c10$@italkbb.com>

Hi David,
	I tried as your recommendation.Seems there's no odd message when
writing to file using the Oracle template.
	I also tried that only write some part of the message(SIP
message,using 'contains' in rsyslog.conf) to Oracle and all others
message(iptables,etc.) were writing to files.The CPU is working at normal
usage.But once I put all the messages into both Oracle and files,the CPU
usage will be keeping at 99-100%. The concurrent messages per second is
about 4 or 5.
	What's wrong?
	Thanks very much.

Best,
Henry

-----Original Message-----
From: rsyslog-bounces at lists.adiscon.com
[mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
Sent: Monday, May 09, 2011 12:57 PM
To: rsyslog-users
Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to Oracle

given that it only happens after a few messages, I would suspect that there
is something odd with the messages.

could you try writing the messages to a file using the Oracle template and
then look and see if there is anything odd about the messages when it goes
nuts?

the other option is to get a debug dump, but unless it happens right away,
that is likely to be a large dump.

David Lang

  On Mon, 9 May 2011, bxu wrote:

> Date: Mon, 9 May 2011 10:48:50 -0500
> From: bxu <bxu at italkbb.com>
> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] help : CPU high load of rsyslog writing to Oracle
> 
> Hi All,
>
>
> I have configured rsyslog writing to Oracle.The writing was 
> successful.But after several messages have been made into Oracle,the 
> CPU use percentage of rsyslogd goes up to 99%-100%.It is abnormally too
much high.
> There's no problem with the cpu usage when writing to files.
>
> Here is my rsyslogd.conf:
> #====================================================
> #*.* >192.168.45.213,rsyslog
>
> # Use traditional timestamp format
> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>
> # Provides kernel logging support (previously done by rklogd) $ModLoad 
> imklog # Provides support for local system logging (e.g. via logger 
> command) $ModLoad imuxsock
>
> # Log all kernel messages to the console.
> # Logging much else clutters up the screen.
> #kern.* /dev/console
>
> # Log anything (except mail) of level info or higher.
> # Don't log private authentication messages!
> *.info;mail.none;authpriv.none;cron.none /var/log/messages
>
> # The authpriv file has restricted access.
> authpriv.* /var/log/secure
>
> # Log all the mail messages in one place.
> mail.* -/var/log/maillog
>
> # Log cron stuff
> cron.* /var/log/cron
>
> # Everybody gets emergency messages
> *.emerg *
>
> # Save news errors of level crit and higher in a special file.
> uucp,news.crit /var/log/spooler
>
> # Save boot messages also to boot.log
> local7.* /var/log/boot.log
>
> #Load External Listender
> $modload imudp
>
>
> $UDPServerRun 514
> #SPA Messages
> #*.* /var/log/linksys.log
> $template DynaFile,"/var/log/system-%HOSTNAME%.log"
> *.* -?DynaFile
>
>
> ################
> #### ORACLE ####
> ################
>
> $ModLoad omoracle
> $OmoracleDBUser syslog
> $OmoracleDBPassword syslog
> $OmoracleDB syslog
> $OmoracleBatchSize 1
> $OmoracleBatchItemSize 4096
> $OmoracleStatementTemplate OmoracleStatement
>
> $template OmoracleStatement,"INSERT INTO TEST(hostname,message) 
> VALUES(:hostname,:msg)"
> $template TestStmt,"%hostname%%msg%"
>
> *.*      :omoracle:;TestStmt
> #====================================================
>
> Really appreciate anyone's help!
>
> Best
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

--
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.


From david at lang.hm  Tue May 10 07:34:58 2011
From: david at lang.hm (david at lang.hm)
Date: Mon, 9 May 2011 22:34:58 -0700 (PDT)
Subject: [rsyslog] help : CPU high load of rsyslog writing to Oracle
In-Reply-To: <alpine.DEB.2.00.1105091055030.25291@asgard.lang.hm>
References: <002101cc0e60$9aa7f6d0$cff7e470$@italkbb.com>
	<alpine.DEB.2.00.1105091055030.25291@asgard.lang.hm>
Message-ID: <alpine.DEB.2.00.1105092231490.6302@asgard.lang.hm>

I accidently deleted your reply, so I am replying to my message

4-5 messages/secshould not be any problem at all.

since writing a copy of the sql commands to a file doesn't show anything 
strange, the next step will be to try and run it in debug mode to see if 
it logs anything strange as you start having problems.

the next step after that will probably be to do a strace of the thread 
that is using all the CPU

I don't see where you mention what version of rsyslog you are using.

David Lang

On Mon, 9 May 2011, david at lang.hm wrote:

> Date: Mon, 9 May 2011 10:56:30 -0700 (PDT)
> From: david at lang.hm
> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> To: rsyslog-users <rsyslog at lists.adiscon.com>
> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to Oracle
> 
> given that it only happens after a few messages, I would suspect that there 
> is something odd with the messages.
>
> could you try writing the messages to a file using the Oracle template and 
> then look and see if there is anything odd about the messages when it goes 
> nuts?
>
> the other option is to get a debug dump, but unless it happens right away, 
> that is likely to be a large dump.
>
> David Lang
>
> On Mon, 9 May 2011, bxu wrote:
>
>> Date: Mon, 9 May 2011 10:48:50 -0500
>> From: bxu <bxu at italkbb.com>
>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>> To: rsyslog at lists.adiscon.com
>> Subject: [rsyslog] help : CPU high load of rsyslog writing to Oracle
>> 
>> Hi All,
>> 
>> 
>> I have configured rsyslog writing to Oracle.The writing was successful.But
>> after several messages have been made into Oracle,the CPU use percentage of
>> rsyslogd goes up to 99%-100%.It is abnormally too much high.
>> There's no problem with the cpu usage when writing to files.
>> 
>> Here is my rsyslogd.conf:
>> #====================================================
>> #*.* >192.168.45.213,rsyslog
>> 
>> # Use traditional timestamp format
>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>> 
>> # Provides kernel logging support (previously done by rklogd)
>> $ModLoad imklog
>> # Provides support for local system logging (e.g. via logger command)
>> $ModLoad imuxsock
>> 
>> # Log all kernel messages to the console.
>> # Logging much else clutters up the screen.
>> #kern.* /dev/console
>> 
>> # Log anything (except mail) of level info or higher.
>> # Don't log private authentication messages!
>> *.info;mail.none;authpriv.none;cron.none /var/log/messages
>> 
>> # The authpriv file has restricted access.
>> authpriv.* /var/log/secure
>> 
>> # Log all the mail messages in one place.
>> mail.* -/var/log/maillog
>> 
>> # Log cron stuff
>> cron.* /var/log/cron
>> 
>> # Everybody gets emergency messages
>> *.emerg *
>> 
>> # Save news errors of level crit and higher in a special file.
>> uucp,news.crit /var/log/spooler
>> 
>> # Save boot messages also to boot.log
>> local7.* /var/log/boot.log
>> 
>> #Load External Listender
>> $modload imudp
>> 
>> 
>> $UDPServerRun 514
>> #SPA Messages
>> #*.* /var/log/linksys.log
>> $template DynaFile,"/var/log/system-%HOSTNAME%.log"
>> *.* -?DynaFile
>> 
>> 
>> ################
>> #### ORACLE ####
>> ################
>> 
>> $ModLoad omoracle
>> $OmoracleDBUser syslog
>> $OmoracleDBPassword syslog
>> $OmoracleDB syslog
>> $OmoracleBatchSize 1
>> $OmoracleBatchItemSize 4096
>> $OmoracleStatementTemplate OmoracleStatement
>> 
>> $template OmoracleStatement,"INSERT INTO TEST(hostname,message)
>> VALUES(:hostname,:msg)"
>> $template TestStmt,"%hostname%%msg%"
>> 
>> *.*      :omoracle:;TestStmt
>> #====================================================
>> 
>> Really appreciate anyone's help!
>> 
>> Best
>> 
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From bxu at italkbb.com  Tue May 10 08:06:36 2011
From: bxu at italkbb.com (bxu)
Date: Tue, 10 May 2011 01:06:36 -0500
Subject: [rsyslog] help : CPU high load of rsyslog writing to Oracle
In-Reply-To: <alpine.DEB.2.00.1105092231490.6302@asgard.lang.hm>
References: <002101cc0e60$9aa7f6d0$cff7e470$@italkbb.com>	<alpine.DEB.2.00.1105091055030.25291@asgard.lang.hm>
	<alpine.DEB.2.00.1105092231490.6302@asgard.lang.hm>
Message-ID: <001c01cc0ed8$6ea9ac40$4bfd04c0$@italkbb.com>

Hi David,
	The rsyslogd version I tried are 5.8.0(stable) and 6.1.7(devel).
	I straced rsyslogd when CPU usage is keeping at 100%.

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 43.47    0.066967        2309        29           read
 38.28    0.058979         855        69           epoll_wait
 18.17    0.027994        9331         3           select
  0.05    0.000081           1       114           gettimeofday
  0.03    0.000041           0       178        69 recvfrom
  0.00    0.000000           0         3           recvmsg
------ ----------- ----------- --------- --------- ----------------
100.00    0.154062                   396        69 total


Best,
Henry

-----Original Message-----
From: rsyslog-bounces at lists.adiscon.com
[mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
Sent: Tuesday, May 10, 2011 12:35 AM
To: rsyslog-users
Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to Oracle

I accidently deleted your reply, so I am replying to my message

4-5 messages/secshould not be any problem at all.

since writing a copy of the sql commands to a file doesn't show anything
strange, the next step will be to try and run it in debug mode to see if it
logs anything strange as you start having problems.

the next step after that will probably be to do a strace of the thread that
is using all the CPU

I don't see where you mention what version of rsyslog you are using.

David Lang

On Mon, 9 May 2011, david at lang.hm wrote:

> Date: Mon, 9 May 2011 10:56:30 -0700 (PDT)
> From: david at lang.hm
> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> To: rsyslog-users <rsyslog at lists.adiscon.com>
> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to 
> Oracle
> 
> given that it only happens after a few messages, I would suspect that 
> there is something odd with the messages.
>
> could you try writing the messages to a file using the Oracle template 
> and then look and see if there is anything odd about the messages when 
> it goes nuts?
>
> the other option is to get a debug dump, but unless it happens right 
> away, that is likely to be a large dump.
>
> David Lang
>
> On Mon, 9 May 2011, bxu wrote:
>
>> Date: Mon, 9 May 2011 10:48:50 -0500
>> From: bxu <bxu at italkbb.com>
>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>> To: rsyslog at lists.adiscon.com
>> Subject: [rsyslog] help : CPU high load of rsyslog writing to Oracle
>> 
>> Hi All,
>> 
>> 
>> I have configured rsyslog writing to Oracle.The writing was 
>> successful.But after several messages have been made into Oracle,the 
>> CPU use percentage of rsyslogd goes up to 99%-100%.It is abnormally too
much high.
>> There's no problem with the cpu usage when writing to files.
>> 
>> Here is my rsyslogd.conf:
>> #====================================================
>> #*.* >192.168.45.213,rsyslog
>> 
>> # Use traditional timestamp format
>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>> 
>> # Provides kernel logging support (previously done by rklogd) 
>> $ModLoad imklog # Provides support for local system logging (e.g. via 
>> logger command) $ModLoad imuxsock
>> 
>> # Log all kernel messages to the console.
>> # Logging much else clutters up the screen.
>> #kern.* /dev/console
>> 
>> # Log anything (except mail) of level info or higher.
>> # Don't log private authentication messages!
>> *.info;mail.none;authpriv.none;cron.none /var/log/messages
>> 
>> # The authpriv file has restricted access.
>> authpriv.* /var/log/secure
>> 
>> # Log all the mail messages in one place.
>> mail.* -/var/log/maillog
>> 
>> # Log cron stuff
>> cron.* /var/log/cron
>> 
>> # Everybody gets emergency messages
>> *.emerg *
>> 
>> # Save news errors of level crit and higher in a special file.
>> uucp,news.crit /var/log/spooler
>> 
>> # Save boot messages also to boot.log
>> local7.* /var/log/boot.log
>> 
>> #Load External Listender
>> $modload imudp
>> 
>> 
>> $UDPServerRun 514
>> #SPA Messages
>> #*.* /var/log/linksys.log
>> $template DynaFile,"/var/log/system-%HOSTNAME%.log"
>> *.* -?DynaFile
>> 
>> 
>> ################
>> #### ORACLE ####
>> ################
>> 
>> $ModLoad omoracle
>> $OmoracleDBUser syslog
>> $OmoracleDBPassword syslog
>> $OmoracleDB syslog
>> $OmoracleBatchSize 1
>> $OmoracleBatchItemSize 4096
>> $OmoracleStatementTemplate OmoracleStatement
>> 
>> $template OmoracleStatement,"INSERT INTO TEST(hostname,message) 
>> VALUES(:hostname,:msg)"
>> $template TestStmt,"%hostname%%msg%"
>> 
>> *.*      :omoracle:;TestStmt
>> #====================================================
>> 
>> Really appreciate anyone's help!
>> 
>> Best
>> 
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

--
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.


From david at lang.hm  Tue May 10 08:19:17 2011
From: david at lang.hm (david at lang.hm)
Date: Mon, 9 May 2011 23:19:17 -0700 (PDT)
Subject: [rsyslog] help : CPU high load of rsyslog writing to Oracle
In-Reply-To: <001c01cc0ed8$6ea9ac40$4bfd04c0$@italkbb.com>
References: <002101cc0e60$9aa7f6d0$cff7e470$@italkbb.com>
	<alpine.DEB.2.00.1105091055030.25291@asgard.lang.hm>
	<alpine.DEB.2.00.1105092231490.6302@asgard.lang.hm>
	<001c01cc0ed8$6ea9ac40$4bfd04c0$@italkbb.com>
Message-ID: <alpine.DEB.2.00.1105092313190.6302@asgard.lang.hm>

rsyslog uses multiple threads, if you can tell which thread is using all 
the CPU and then do a strace of it and dump the sequence of calls that it 
makes to process one of the messages it may help

I don't have any idea what's going on here, so I'm just working through 
the steps that I would be doing if I was running into the problem

try ps -eLf to list the different threads

in top, hit 'H' to run on per-thread reporting (at which point you may get 
some additional info on the right about what the thread is doing)

from the stats you are showing here, it looks like the thread you got with 
your strace is the parent that's receiving messages (lots of gettimeofday 
and receive/read type calls) the correct one should have write calls in it 
(among others)

the versions you are running are new enough that we shouldn't have any 
old, solved problems hitting us.

  On Tue, 10 May 2011, bxu 
wrote:

> Hi David,
> 	The rsyslogd version I tried are 5.8.0(stable) and 6.1.7(devel).
> 	I straced rsyslogd when CPU usage is keeping at 100%.
>
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 43.47    0.066967        2309        29           read
> 38.28    0.058979         855        69           epoll_wait
> 18.17    0.027994        9331         3           select
>  0.05    0.000081           1       114           gettimeofday
>  0.03    0.000041           0       178        69 recvfrom
>  0.00    0.000000           0         3           recvmsg
> ------ ----------- ----------- --------- --------- ----------------
> 100.00    0.154062                   396        69 total
>
>
> Best,
> Henry
>
> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, May 10, 2011 12:35 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to Oracle
>
> I accidently deleted your reply, so I am replying to my message
>
> 4-5 messages/secshould not be any problem at all.
>
> since writing a copy of the sql commands to a file doesn't show anything
> strange, the next step will be to try and run it in debug mode to see if it
> logs anything strange as you start having problems.
>
> the next step after that will probably be to do a strace of the thread that
> is using all the CPU
>
> I don't see where you mention what version of rsyslog you are using.
>
> David Lang
>
> On Mon, 9 May 2011, david at lang.hm wrote:
>
>> Date: Mon, 9 May 2011 10:56:30 -0700 (PDT)
>> From: david at lang.hm
>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>> To: rsyslog-users <rsyslog at lists.adiscon.com>
>> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to
>> Oracle
>>
>> given that it only happens after a few messages, I would suspect that
>> there is something odd with the messages.
>>
>> could you try writing the messages to a file using the Oracle template
>> and then look and see if there is anything odd about the messages when
>> it goes nuts?
>>
>> the other option is to get a debug dump, but unless it happens right
>> away, that is likely to be a large dump.
>>
>> David Lang
>>
>> On Mon, 9 May 2011, bxu wrote:
>>
>>> Date: Mon, 9 May 2011 10:48:50 -0500
>>> From: bxu <bxu at italkbb.com>
>>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>>> To: rsyslog at lists.adiscon.com
>>> Subject: [rsyslog] help : CPU high load of rsyslog writing to Oracle
>>>
>>> Hi All,
>>>
>>>
>>> I have configured rsyslog writing to Oracle.The writing was
>>> successful.But after several messages have been made into Oracle,the
>>> CPU use percentage of rsyslogd goes up to 99%-100%.It is abnormally too
> much high.
>>> There's no problem with the cpu usage when writing to files.
>>>
>>> Here is my rsyslogd.conf:
>>> #====================================================
>>> #*.* >192.168.45.213,rsyslog
>>>
>>> # Use traditional timestamp format
>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>
>>> # Provides kernel logging support (previously done by rklogd)
>>> $ModLoad imklog # Provides support for local system logging (e.g. via
>>> logger command) $ModLoad imuxsock
>>>
>>> # Log all kernel messages to the console.
>>> # Logging much else clutters up the screen.
>>> #kern.* /dev/console
>>>
>>> # Log anything (except mail) of level info or higher.
>>> # Don't log private authentication messages!
>>> *.info;mail.none;authpriv.none;cron.none /var/log/messages
>>>
>>> # The authpriv file has restricted access.
>>> authpriv.* /var/log/secure
>>>
>>> # Log all the mail messages in one place.
>>> mail.* -/var/log/maillog
>>>
>>> # Log cron stuff
>>> cron.* /var/log/cron
>>>
>>> # Everybody gets emergency messages
>>> *.emerg *
>>>
>>> # Save news errors of level crit and higher in a special file.
>>> uucp,news.crit /var/log/spooler
>>>
>>> # Save boot messages also to boot.log
>>> local7.* /var/log/boot.log
>>>
>>> #Load External Listender
>>> $modload imudp
>>>
>>>
>>> $UDPServerRun 514
>>> #SPA Messages
>>> #*.* /var/log/linksys.log
>>> $template DynaFile,"/var/log/system-%HOSTNAME%.log"
>>> *.* -?DynaFile
>>>
>>>
>>> ################
>>> #### ORACLE ####
>>> ################
>>>
>>> $ModLoad omoracle
>>> $OmoracleDBUser syslog
>>> $OmoracleDBPassword syslog
>>> $OmoracleDB syslog
>>> $OmoracleBatchSize 1
>>> $OmoracleBatchItemSize 4096
>>> $OmoracleStatementTemplate OmoracleStatement
>>>
>>> $template OmoracleStatement,"INSERT INTO TEST(hostname,message)
>>> VALUES(:hostname,:msg)"
>>> $template TestStmt,"%hostname%%msg%"
>>>
>>> *.*      :omoracle:;TestStmt
>>> #====================================================
>>>
>>> Really appreciate anyone's help!
>>>
>>> Best
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>
> --
> This message has been scanned for viruses and dangerous content by
> MailScanner, and is believed to be clean.
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From bxu at italkbb.com  Tue May 10 19:12:27 2011
From: bxu at italkbb.com (bxu)
Date: Tue, 10 May 2011 12:12:27 -0500
Subject: [rsyslog] help : CPU high load of rsyslog writing to Oracle
References: <002101cc0e60$9aa7f6d0$cff7e470$@italkbb.com>	<alpine.DEB.2.00.1105091055030.25291@asgard.lang.hm>	<alpine.DEB.2.00.1105092231490.6302@asgard.lang.hm>	<001c01cc0ed8$6ea9ac40$4bfd04c0$@italkbb.com>
	<alpine.DEB.2.00.1105092313190.6302@asgard.lang.hm> 
Message-ID: <004801cc0f35$731e9850$595bc8f0$@italkbb.com>

Hi David,
	Here are what I'm doing:
	1. oracle configuration of rsyslogd.conf

################
#### ORACLE ####
################

$ModLoad omoracle
$OmoracleDBUser syslog
$OmoracleDBPassword syslog
$OmoracleDB syslog
$OmoracleBatchSize 1
$OmoracleBatchItemSize 4096
$OmoracleStatementTemplate OmoracleStatement

$template OmoracleStatement,"INSERT INTO TEST(Hostname,Message)
VALUES(:hostname,:msg)"
$template TestStmt,"%hostname%%msg%"
$template TestStmtFile,"INSERT INTO TEST(Hostname,Message)
VALUES('%hostname%','%msg%')\n"

#$WorkDirectory /rsyslog/work # default location for work (spool) files

#$ActionQueueType LinkedList # use asynchronous processing
#$ActionQueueFileName dbq    # set file name, also enables disk mode
#$ActionResumeRetryCount -1  # infinite retries on insert failure

:msg,contains,"SIP"             :omoracle:;TestStmt

& ~

*.*                             /var/log/syslog-test;TestStmtFile

	2. running command :  rsyslogd -c5 -nd > tt &

	3, TOP information

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

13470 root      25   0  244m  11m 7636 R 99.9  0.1   1:06.39 rs:main Q:Reg

11410 root      15   0 12768 1144  832 R  0.3  0.0   0:13.52 top

    1 root      15   0 10372  692  580 S  0.0  0.0   0:00.51 init

	4. when CPU usage is keeping at 100%,the summary of strace by thread
13470:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 51.26    0.019996        9998         2           epoll_wait
 46.12    0.017990        1799        10           read
  2.56    0.000999         999         1           select
  0.06    0.000022           2        12         2 recvfrom
  0.00    0.000000           0        64           write
  0.00    0.000000           0         1           recvmsg
  0.00    0.000000           0        17           gettimeofday
  0.00    0.000000           0        62           clock_gettime
------ ----------- ----------- --------- --------- ----------------
100.00    0.039007                   169         2 total

	5. when CPU usage is keeping at 100%,some of the call sequence of
strace by thread 13470:

[pid 13469]      0.000000 select(1, NULL, NULL, NULL, {86097, 801000}
<unfinished ...>
[pid 13473]      0.000089 clock_gettime(CLOCK_REALTIME,  <unfinished ...>
[pid 13472]      0.000054 select(9, [8], NULL, NULL, NULL <unfinished ...>
[pid 13471]      0.000066 read(3,  <unfinished ...>
[pid 13473]      0.000048 <... clock_gettime resumed> {1305046294,
274947000}) = 0 <0.000138>
[pid 13473]      0.000067 write(1, "6294.274947000:43899940: imudp: "...,
66) = 66 <0.000042>
[pid 13473]      0.000137 epoll_wait(12,  <unfinished ...>
[pid 13471]      1.421497 <... read resumed> "<6>iptables: log and drop
IN=eth"..., 4095) = 200 <1.421706>
[pid 13471]      0.000095 gettimeofday({1305046295, 696914}, NULL) = 0
<0.000022>
[pid 13471]      0.000130 clock_gettime(CLOCK_REALTIME, {1305046295,
697046000}) = 0 <0.000022>
[pid 13471]      0.000104 write(1, "6295.697046000:42497940: main Q:"...,
81) = 81 <0.000037>
[pid 13471]      0.000127 clock_gettime(CLOCK_REALTIME, {1305046295,
697276000}) = 0 <0.000020>
[pid 13471]      0.000102 write(1, "6295.697276000:42497940: main Q:"...,
65) = 65 <0.000031>
[pid 13471]      0.000120 read(3, "<6>iptables: log and drop IN=eth"...,
4095) = 200 <4.250732>
[pid 13471]      4.250863 gettimeofday({1305046299, 948362}, NULL) = 0
<0.000021>
[pid 13471]      0.000129 clock_gettime(CLOCK_REALTIME, {1305046299,
948491000}) = 0 <0.000021>
[pid 13471]      0.000106 write(1, "6299.948491000:42497940: main Q:"...,
81) = 81 <0.000051>
[pid 13471]      0.000140 clock_gettime(CLOCK_REALTIME, {1305046299,
948735000}) = 0 <0.000021>
[pid 13471]      0.000101 write(1, "6299.948735000:42497940: main Q:"...,
65) = 65 <0.000031>
[pid 13471]      0.000123 read(3, "<6>iptables: log and drop IN=eth"...,
4095) = 200 <2.178833>
[pid 13471]      2.178931 gettimeofday({1305046302, 127891}, NULL) = 0
<0.000022>
[pid 13471]      0.000112 clock_gettime(CLOCK_REALTIME, {1305046302,
128003000}) = 0 <0.000021>
[pid 13471]      0.000101 write(1, "6302.128003000:42497940: main Q:"...,
81) = 81 <0.000036>
[pid 13471]      0.000121 clock_gettime(CLOCK_REALTIME, {1305046302,
128225000}) = 0 <0.000021>
[pid 13471]      0.000102 write(1, "6302.128225000:42497940: main Q:"...,
65) = 65 <0.000031>
[pid 13471]      0.000115 read(3, "<6>iptables: log and drop IN=eth"...,
4095) = 200 <2.431092>
[pid 13471]      2.431228 gettimeofday({1305046304, 559672}, NULL) = 0
<0.000022>
[pid 13471]      0.000132 clock_gettime(CLOCK_REALTIME, {1305046304,
559803000}) = 0 <0.000021>
[pid 13471]      0.000110 write(1, "6304.559803000:42497940: main Q:"...,
81) = 81 <0.000052>
[pid 13471]      0.000140 clock_gettime(CLOCK_REALTIME, {1305046304,
560052000}) = 0 <0.000021>
[pid 13471]      0.000102 write(1, "6304.560052000:42497940: main Q:"...,
65) = 65 <0.000031>
[pid 13471]      0.000119 read(3, "<6>iptables: log and drop IN=eth"...,
4095) = 197 <0.678610>
[pid 13471]      0.678704 gettimeofday({1305046305, 238976}, NULL) = 0
<0.000021>
[pid 13471]      0.000110 clock_gettime(CLOCK_REALTIME, {1305046305,
239087000}) = 0 <0.000024>
[pid 13471]      0.000105 write(1, "6305.239087000:42497940: main Q:"...,
81) = 81 <0.000035>
[pid 13471]      0.000120 clock_gettime(CLOCK_REALTIME, {1305046305,
239312000}) = 0 <0.000020>
[pid 13471]      0.000102 write(1, "6305.239312000:42497940: main Q:"...,
65) = 65 <0.000030>
[pid 13471]      0.000114 read(3, "<6>iptables: log and drop IN=eth"...,
4095) = 200 <3.058797>
[pid 13471]      3.058929 gettimeofday({1305046308, 298458}, NULL) = 0
<0.000022>
[pid 13471]      0.000131 clock_gettime(CLOCK_REALTIME, {1305046308,
298589000}) = 0 <0.000021>
[pid 13471]      0.000106 write(1, "6308.298589000:42497940: main Q:"...,
81) = 81 <0.000058>
[pid 13471]      0.000148 clock_gettime(CLOCK_REALTIME, {1305046308,
298841000}) = 0 <0.000021>
[pid 13471]      0.000101 write(1, "6308.298841000:42497940: main Q:"...,
65) = 65 <0.000093>
[pid 13471]      0.000181 read(3, "<6>iptables: log and drop IN=eth"...,
4095) = 197 <0.213955>
[pid 13471]      0.214082 gettimeofday({1305046308, 513206}, NULL) = 0
<0.000021>
[pid 13471]      0.000128 clock_gettime(CLOCK_REALTIME, {1305046308,
513335000}) = 0 <0.000021>
[pid 13471]      0.000106 write(1, "6308.513335000:42497940: main Q:"...,
81) = 81 <0.000042>
[pid 13471]      0.000130 clock_gettime(CLOCK_REALTIME, {1305046308,
513570000}) = 0 <0.000021>
[pid 13471]      0.000102 write(1, "6308.513570000:42497940: main Q:"...,
65) = 65 <0.000030>
[pid 13471]      0.000122 read(3, "<6>iptables: log and drop IN=eth"...,
4095) = 196 <0.835091>
[pid 13471]      0.835187 gettimeofday({1305046309, 348980}, NULL) = 0
<0.000021>
[pid 13471]      0.000110 clock_gettime(CLOCK_REALTIME, {1305046309,
349091000}) = 0 <0.000021>
[pid 13471]      0.000102 write(1, "6309.349091000:42497940: main Q:"...,
81) = 81 <0.000035>
[pid 13471]      0.000121 clock_gettime(CLOCK_REALTIME, {1305046309,
349313000}) = 0 <0.000020>
[pid 13471]      0.000102 write(1, "6309.349313000:42497940: main Q:"...,
65) = 65 <0.000030>
[pid 13471]      0.000114 read(3,  <unfinished ...>
[pid 13473]      2.018865 <... epoll_wait resumed> {{EPOLLIN, {u32=2,
u64=2}}}, 10, 4294967295) = 1 <17.093033>
[pid 13473]      0.000086 clock_gettime(CLOCK_REALTIME, {1305046311,
368481000}) = 0 <0.000020>
[pid 13473]      0.000098 write(1, "6311.368481000:43899940: imudp: "...,
65) = 65 <0.000036>	 


	6. Attached is the log file while rsyslogd is running in debug mode
	    In the log,after "omoracle insertion to DB succeeded", there's
no more oracle log information related.

	
	Thank you very much.

Best,
Henry

-----Original Message-----
From: rsyslog-bounces at lists.adiscon.com
[mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
Sent: Tuesday, May 10, 2011 1:19 AM
To: rsyslog-users
Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to Oracle

rsyslog uses multiple threads, if you can tell which thread is using all the
CPU and then do a strace of it and dump the sequence of calls that it makes
to process one of the messages it may help

I don't have any idea what's going on here, so I'm just working through the
steps that I would be doing if I was running into the problem

try ps -eLf to list the different threads

in top, hit 'H' to run on per-thread reporting (at which point you may get
some additional info on the right about what the thread is doing)

from the stats you are showing here, it looks like the thread you got with
your strace is the parent that's receiving messages (lots of gettimeofday
and receive/read type calls) the correct one should have write calls in it
(among others)

the versions you are running are new enough that we shouldn't have any old,
solved problems hitting us.

  On Tue, 10 May 2011, bxu
wrote:

> Hi David,
> 	The rsyslogd version I tried are 5.8.0(stable) and 6.1.7(devel).
> 	I straced rsyslogd when CPU usage is keeping at 100%.
>
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 43.47    0.066967        2309        29           read
> 38.28    0.058979         855        69           epoll_wait
> 18.17    0.027994        9331         3           select
>  0.05    0.000081           1       114           gettimeofday
>  0.03    0.000041           0       178        69 recvfrom
>  0.00    0.000000           0         3           recvmsg
> ------ ----------- ----------- --------- --------- ----------------
> 100.00    0.154062                   396        69 total
>
>
> Best,
> Henry
>
> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, May 10, 2011 12:35 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to 
> Oracle
>
> I accidently deleted your reply, so I am replying to my message
>
> 4-5 messages/secshould not be any problem at all.
>
> since writing a copy of the sql commands to a file doesn't show 
> anything strange, the next step will be to try and run it in debug 
> mode to see if it logs anything strange as you start having problems.
>
> the next step after that will probably be to do a strace of the thread 
> that is using all the CPU
>
> I don't see where you mention what version of rsyslog you are using.
>
> David Lang
>
> On Mon, 9 May 2011, david at lang.hm wrote:
>
>> Date: Mon, 9 May 2011 10:56:30 -0700 (PDT)
>> From: david at lang.hm
>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>> To: rsyslog-users <rsyslog at lists.adiscon.com>
>> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to 
>> Oracle
>>
>> given that it only happens after a few messages, I would suspect that 
>> there is something odd with the messages.
>>
>> could you try writing the messages to a file using the Oracle 
>> template and then look and see if there is anything odd about the 
>> messages when it goes nuts?
>>
>> the other option is to get a debug dump, but unless it happens right 
>> away, that is likely to be a large dump.
>>
>> David Lang
>>
>> On Mon, 9 May 2011, bxu wrote:
>>
>>> Date: Mon, 9 May 2011 10:48:50 -0500
>>> From: bxu <bxu at italkbb.com>
>>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>>> To: rsyslog at lists.adiscon.com
>>> Subject: [rsyslog] help : CPU high load of rsyslog writing to Oracle
>>>
>>> Hi All,
>>>
>>>
>>> I have configured rsyslog writing to Oracle.The writing was 
>>> successful.But after several messages have been made into Oracle,the 
>>> CPU use percentage of rsyslogd goes up to 99%-100%.It is abnormally 
>>> too
> much high.
>>> There's no problem with the cpu usage when writing to files.
>>>
>>> Here is my rsyslogd.conf:
>>> #====================================================
>>> #*.* >192.168.45.213,rsyslog
>>>
>>> # Use traditional timestamp format
>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>
>>> # Provides kernel logging support (previously done by rklogd) 
>>> $ModLoad imklog # Provides support for local system logging (e.g.
>>> via logger command) $ModLoad imuxsock
>>>
>>> # Log all kernel messages to the console.
>>> # Logging much else clutters up the screen.
>>> #kern.* /dev/console
>>>
>>> # Log anything (except mail) of level info or higher.
>>> # Don't log private authentication messages!
>>> *.info;mail.none;authpriv.none;cron.none /var/log/messages
>>>
>>> # The authpriv file has restricted access.
>>> authpriv.* /var/log/secure
>>>
>>> # Log all the mail messages in one place.
>>> mail.* -/var/log/maillog
>>>
>>> # Log cron stuff
>>> cron.* /var/log/cron
>>>
>>> # Everybody gets emergency messages
>>> *.emerg *
>>>
>>> # Save news errors of level crit and higher in a special file.
>>> uucp,news.crit /var/log/spooler
>>>
>>> # Save boot messages also to boot.log
>>> local7.* /var/log/boot.log
>>>
>>> #Load External Listender
>>> $modload imudp
>>>
>>>
>>> $UDPServerRun 514
>>> #SPA Messages
>>> #*.* /var/log/linksys.log
>>> $template DynaFile,"/var/log/system-%HOSTNAME%.log"
>>> *.* -?DynaFile
>>>
>>>
>>> ################
>>> #### ORACLE ####
>>> ################
>>>
>>> $ModLoad omoracle
>>> $OmoracleDBUser syslog
>>> $OmoracleDBPassword syslog
>>> $OmoracleDB syslog
>>> $OmoracleBatchSize 1
>>> $OmoracleBatchItemSize 4096
>>> $OmoracleStatementTemplate OmoracleStatement
>>>
>>> $template OmoracleStatement,"INSERT INTO TEST(hostname,message) 
>>> VALUES(:hostname,:msg)"
>>> $template TestStmt,"%hostname%%msg%"
>>>
>>> *.*      :omoracle:;TestStmt
>>> #====================================================
>>>
>>> Really appreciate anyone's help!
>>>
>>> Best
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>
> --
> This message has been scanned for viruses and dangerous content by 
> MailScanner, and is believed to be clean.
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

--
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tt
Type: application/octet-stream
Size: 464915 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20110510/31ea033e/attachment-0001.obj>

From david at lang.hm  Wed May 11 03:28:54 2011
From: david at lang.hm (david at lang.hm)
Date: Tue, 10 May 2011 18:28:54 -0700 (PDT)
Subject: [rsyslog] help : CPU high load of rsyslog writing to Oracle
In-Reply-To: <004801cc0f35$731e9850$595bc8f0$@italkbb.com>
References: <002101cc0e60$9aa7f6d0$cff7e470$@italkbb.com>
	<alpine.DEB.2.00.1105091055030.25291@asgard.lang.hm>
	<alpine.DEB.2.00.1105092231490.6302@asgard.lang.hm>
	<001c01cc0ed8$6ea9ac40$4bfd04c0$@italkbb.com>
	<alpine.DEB.2.00.1105092313190.6302@asgard.lang.hm>
	<004801cc0f35$731e9850$595bc8f0$@italkbb.com>
Message-ID: <alpine.DEB.2.00.1105101820220.25291@asgard.lang.hm>

thanks for the info.

when you printed the TOP info, was that with 'H' (thread details) turned 
on?

I'm puzzled by the fact that you are doing a strace of the thread 13470 
and it's showing actions by many different pids (it looks like everything 
except 13470

can you try doing

strace -ff -o rsyslog.strace. -C  -ttt -T -s 100  rsyslogd -c5 -nd >rsyslog.debug

this will create a bunch of strace files. we shouldn't need to look at all 
of them, just the one using all the CPU (and possibly only the end of that 
one)

while this is running, do a ps -eLf

also, run top -H and note which thread is using all the CPU (you may want 
to try toggling H while it's running to make sure you are getting the 
thread info not the consolodated process info)

I'm about to head home and when I get home I'll look at the debug output 
that you send.

one thing I'm not remembering. are the logs getting into the database?

David Lang

On Tue, 10 May 2011, bxu wrote:

> Hi David,
> 	Here are what I'm doing:
> 	1. oracle configuration of rsyslogd.conf
>
> ################
> #### ORACLE ####
> ################
>
> $ModLoad omoracle
> $OmoracleDBUser syslog
> $OmoracleDBPassword syslog
> $OmoracleDB syslog
> $OmoracleBatchSize 1
> $OmoracleBatchItemSize 4096
> $OmoracleStatementTemplate OmoracleStatement
>
> $template OmoracleStatement,"INSERT INTO TEST(Hostname,Message)
> VALUES(:hostname,:msg)"
> $template TestStmt,"%hostname%%msg%"
> $template TestStmtFile,"INSERT INTO TEST(Hostname,Message)
> VALUES('%hostname%','%msg%')\n"
>
> #$WorkDirectory /rsyslog/work # default location for work (spool) files
>
> #$ActionQueueType LinkedList # use asynchronous processing
> #$ActionQueueFileName dbq    # set file name, also enables disk mode
> #$ActionResumeRetryCount -1  # infinite retries on insert failure
>
> :msg,contains,"SIP"             :omoracle:;TestStmt
>
> & ~
>
> *.*                             /var/log/syslog-test;TestStmtFile
>
> 	2. running command :  rsyslogd -c5 -nd > tt &
>
> 	3, TOP information
>
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>
> 13470 root      25   0  244m  11m 7636 R 99.9  0.1   1:06.39 rs:main Q:Reg
>
> 11410 root      15   0 12768 1144  832 R  0.3  0.0   0:13.52 top
>
>    1 root      15   0 10372  692  580 S  0.0  0.0   0:00.51 init
>
> 	4. when CPU usage is keeping at 100%,the summary of strace by thread
> 13470:
>
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 51.26    0.019996        9998         2           epoll_wait
> 46.12    0.017990        1799        10           read
>  2.56    0.000999         999         1           select
>  0.06    0.000022           2        12         2 recvfrom
>  0.00    0.000000           0        64           write
>  0.00    0.000000           0         1           recvmsg
>  0.00    0.000000           0        17           gettimeofday
>  0.00    0.000000           0        62           clock_gettime
> ------ ----------- ----------- --------- --------- ----------------
> 100.00    0.039007                   169         2 total
>
> 	5. when CPU usage is keeping at 100%,some of the call sequence of
> strace by thread 13470:
>
> [pid 13469]      0.000000 select(1, NULL, NULL, NULL, {86097, 801000}
> <unfinished ...>
> [pid 13473]      0.000089 clock_gettime(CLOCK_REALTIME,  <unfinished ...>
> [pid 13472]      0.000054 select(9, [8], NULL, NULL, NULL <unfinished ...>
> [pid 13471]      0.000066 read(3,  <unfinished ...>
> [pid 13473]      0.000048 <... clock_gettime resumed> {1305046294,
> 274947000}) = 0 <0.000138>
> [pid 13473]      0.000067 write(1, "6294.274947000:43899940: imudp: "...,
> 66) = 66 <0.000042>
> [pid 13473]      0.000137 epoll_wait(12,  <unfinished ...>
> [pid 13471]      1.421497 <... read resumed> "<6>iptables: log and drop
> IN=eth"..., 4095) = 200 <1.421706>
> [pid 13471]      0.000095 gettimeofday({1305046295, 696914}, NULL) = 0
> <0.000022>
> [pid 13471]      0.000130 clock_gettime(CLOCK_REALTIME, {1305046295,
> 697046000}) = 0 <0.000022>
> [pid 13471]      0.000104 write(1, "6295.697046000:42497940: main Q:"...,
> 81) = 81 <0.000037>
> [pid 13471]      0.000127 clock_gettime(CLOCK_REALTIME, {1305046295,
> 697276000}) = 0 <0.000020>
> [pid 13471]      0.000102 write(1, "6295.697276000:42497940: main Q:"...,
> 65) = 65 <0.000031>
> [pid 13471]      0.000120 read(3, "<6>iptables: log and drop IN=eth"...,
> 4095) = 200 <4.250732>
> [pid 13471]      4.250863 gettimeofday({1305046299, 948362}, NULL) = 0
> <0.000021>
> [pid 13471]      0.000129 clock_gettime(CLOCK_REALTIME, {1305046299,
> 948491000}) = 0 <0.000021>
> [pid 13471]      0.000106 write(1, "6299.948491000:42497940: main Q:"...,
> 81) = 81 <0.000051>
> [pid 13471]      0.000140 clock_gettime(CLOCK_REALTIME, {1305046299,
> 948735000}) = 0 <0.000021>
> [pid 13471]      0.000101 write(1, "6299.948735000:42497940: main Q:"...,
> 65) = 65 <0.000031>
> [pid 13471]      0.000123 read(3, "<6>iptables: log and drop IN=eth"...,
> 4095) = 200 <2.178833>
> [pid 13471]      2.178931 gettimeofday({1305046302, 127891}, NULL) = 0
> <0.000022>
> [pid 13471]      0.000112 clock_gettime(CLOCK_REALTIME, {1305046302,
> 128003000}) = 0 <0.000021>
> [pid 13471]      0.000101 write(1, "6302.128003000:42497940: main Q:"...,
> 81) = 81 <0.000036>
> [pid 13471]      0.000121 clock_gettime(CLOCK_REALTIME, {1305046302,
> 128225000}) = 0 <0.000021>
> [pid 13471]      0.000102 write(1, "6302.128225000:42497940: main Q:"...,
> 65) = 65 <0.000031>
> [pid 13471]      0.000115 read(3, "<6>iptables: log and drop IN=eth"...,
> 4095) = 200 <2.431092>
> [pid 13471]      2.431228 gettimeofday({1305046304, 559672}, NULL) = 0
> <0.000022>
> [pid 13471]      0.000132 clock_gettime(CLOCK_REALTIME, {1305046304,
> 559803000}) = 0 <0.000021>
> [pid 13471]      0.000110 write(1, "6304.559803000:42497940: main Q:"...,
> 81) = 81 <0.000052>
> [pid 13471]      0.000140 clock_gettime(CLOCK_REALTIME, {1305046304,
> 560052000}) = 0 <0.000021>
> [pid 13471]      0.000102 write(1, "6304.560052000:42497940: main Q:"...,
> 65) = 65 <0.000031>
> [pid 13471]      0.000119 read(3, "<6>iptables: log and drop IN=eth"...,
> 4095) = 197 <0.678610>
> [pid 13471]      0.678704 gettimeofday({1305046305, 238976}, NULL) = 0
> <0.000021>
> [pid 13471]      0.000110 clock_gettime(CLOCK_REALTIME, {1305046305,
> 239087000}) = 0 <0.000024>
> [pid 13471]      0.000105 write(1, "6305.239087000:42497940: main Q:"...,
> 81) = 81 <0.000035>
> [pid 13471]      0.000120 clock_gettime(CLOCK_REALTIME, {1305046305,
> 239312000}) = 0 <0.000020>
> [pid 13471]      0.000102 write(1, "6305.239312000:42497940: main Q:"...,
> 65) = 65 <0.000030>
> [pid 13471]      0.000114 read(3, "<6>iptables: log and drop IN=eth"...,
> 4095) = 200 <3.058797>
> [pid 13471]      3.058929 gettimeofday({1305046308, 298458}, NULL) = 0
> <0.000022>
> [pid 13471]      0.000131 clock_gettime(CLOCK_REALTIME, {1305046308,
> 298589000}) = 0 <0.000021>
> [pid 13471]      0.000106 write(1, "6308.298589000:42497940: main Q:"...,
> 81) = 81 <0.000058>
> [pid 13471]      0.000148 clock_gettime(CLOCK_REALTIME, {1305046308,
> 298841000}) = 0 <0.000021>
> [pid 13471]      0.000101 write(1, "6308.298841000:42497940: main Q:"...,
> 65) = 65 <0.000093>
> [pid 13471]      0.000181 read(3, "<6>iptables: log and drop IN=eth"...,
> 4095) = 197 <0.213955>
> [pid 13471]      0.214082 gettimeofday({1305046308, 513206}, NULL) = 0
> <0.000021>
> [pid 13471]      0.000128 clock_gettime(CLOCK_REALTIME, {1305046308,
> 513335000}) = 0 <0.000021>
> [pid 13471]      0.000106 write(1, "6308.513335000:42497940: main Q:"...,
> 81) = 81 <0.000042>
> [pid 13471]      0.000130 clock_gettime(CLOCK_REALTIME, {1305046308,
> 513570000}) = 0 <0.000021>
> [pid 13471]      0.000102 write(1, "6308.513570000:42497940: main Q:"...,
> 65) = 65 <0.000030>
> [pid 13471]      0.000122 read(3, "<6>iptables: log and drop IN=eth"...,
> 4095) = 196 <0.835091>
> [pid 13471]      0.835187 gettimeofday({1305046309, 348980}, NULL) = 0
> <0.000021>
> [pid 13471]      0.000110 clock_gettime(CLOCK_REALTIME, {1305046309,
> 349091000}) = 0 <0.000021>
> [pid 13471]      0.000102 write(1, "6309.349091000:42497940: main Q:"...,
> 81) = 81 <0.000035>
> [pid 13471]      0.000121 clock_gettime(CLOCK_REALTIME, {1305046309,
> 349313000}) = 0 <0.000020>
> [pid 13471]      0.000102 write(1, "6309.349313000:42497940: main Q:"...,
> 65) = 65 <0.000030>
> [pid 13471]      0.000114 read(3,  <unfinished ...>
> [pid 13473]      2.018865 <... epoll_wait resumed> {{EPOLLIN, {u32=2,
> u64=2}}}, 10, 4294967295) = 1 <17.093033>
> [pid 13473]      0.000086 clock_gettime(CLOCK_REALTIME, {1305046311,
> 368481000}) = 0 <0.000020>
> [pid 13473]      0.000098 write(1, "6311.368481000:43899940: imudp: "...,
> 65) = 65 <0.000036>
>
>
> 	6. Attached is the log file while rsyslogd is running in debug mode
> 	    In the log,after "omoracle insertion to DB succeeded", there's
> no more oracle log information related.
>
>
> 	Thank you very much.
>
> Best,
> Henry
>
> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, May 10, 2011 1:19 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to Oracle
>
> rsyslog uses multiple threads, if you can tell which thread is using all the
> CPU and then do a strace of it and dump the sequence of calls that it makes
> to process one of the messages it may help
>
> I don't have any idea what's going on here, so I'm just working through the
> steps that I would be doing if I was running into the problem
>
> try ps -eLf to list the different threads
>
> in top, hit 'H' to run on per-thread reporting (at which point you may get
> some additional info on the right about what the thread is doing)
>
> from the stats you are showing here, it looks like the thread you got with
> your strace is the parent that's receiving messages (lots of gettimeofday
> and receive/read type calls) the correct one should have write calls in it
> (among others)
>
> the versions you are running are new enough that we shouldn't have any old,
> solved problems hitting us.
>
>  On Tue, 10 May 2011, bxu
> wrote:
>
>> Hi David,
>> 	The rsyslogd version I tried are 5.8.0(stable) and 6.1.7(devel).
>> 	I straced rsyslogd when CPU usage is keeping at 100%.
>>
>> % time     seconds  usecs/call     calls    errors syscall
>> ------ ----------- ----------- --------- --------- ----------------
>> 43.47    0.066967        2309        29           read
>> 38.28    0.058979         855        69           epoll_wait
>> 18.17    0.027994        9331         3           select
>>  0.05    0.000081           1       114           gettimeofday
>>  0.03    0.000041           0       178        69 recvfrom
>>  0.00    0.000000           0         3           recvmsg
>> ------ ----------- ----------- --------- --------- ----------------
>> 100.00    0.154062                   396        69 total
>>
>>
>> Best,
>> Henry
>>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Tuesday, May 10, 2011 12:35 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to
>> Oracle
>>
>> I accidently deleted your reply, so I am replying to my message
>>
>> 4-5 messages/secshould not be any problem at all.
>>
>> since writing a copy of the sql commands to a file doesn't show
>> anything strange, the next step will be to try and run it in debug
>> mode to see if it logs anything strange as you start having problems.
>>
>> the next step after that will probably be to do a strace of the thread
>> that is using all the CPU
>>
>> I don't see where you mention what version of rsyslog you are using.
>>
>> David Lang
>>
>> On Mon, 9 May 2011, david at lang.hm wrote:
>>
>>> Date: Mon, 9 May 2011 10:56:30 -0700 (PDT)
>>> From: david at lang.hm
>>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>>> To: rsyslog-users <rsyslog at lists.adiscon.com>
>>> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to
>>> Oracle
>>>
>>> given that it only happens after a few messages, I would suspect that
>>> there is something odd with the messages.
>>>
>>> could you try writing the messages to a file using the Oracle
>>> template and then look and see if there is anything odd about the
>>> messages when it goes nuts?
>>>
>>> the other option is to get a debug dump, but unless it happens right
>>> away, that is likely to be a large dump.
>>>
>>> David Lang
>>>
>>> On Mon, 9 May 2011, bxu wrote:
>>>
>>>> Date: Mon, 9 May 2011 10:48:50 -0500
>>>> From: bxu <bxu at italkbb.com>
>>>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>>>> To: rsyslog at lists.adiscon.com
>>>> Subject: [rsyslog] help : CPU high load of rsyslog writing to Oracle
>>>>
>>>> Hi All,
>>>>
>>>>
>>>> I have configured rsyslog writing to Oracle.The writing was
>>>> successful.But after several messages have been made into Oracle,the
>>>> CPU use percentage of rsyslogd goes up to 99%-100%.It is abnormally
>>>> too
>> much high.
>>>> There's no problem with the cpu usage when writing to files.
>>>>
>>>> Here is my rsyslogd.conf:
>>>> #====================================================
>>>> #*.* >192.168.45.213,rsyslog
>>>>
>>>> # Use traditional timestamp format
>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>>
>>>> # Provides kernel logging support (previously done by rklogd)
>>>> $ModLoad imklog # Provides support for local system logging (e.g.
>>>> via logger command) $ModLoad imuxsock
>>>>
>>>> # Log all kernel messages to the console.
>>>> # Logging much else clutters up the screen.
>>>> #kern.* /dev/console
>>>>
>>>> # Log anything (except mail) of level info or higher.
>>>> # Don't log private authentication messages!
>>>> *.info;mail.none;authpriv.none;cron.none /var/log/messages
>>>>
>>>> # The authpriv file has restricted access.
>>>> authpriv.* /var/log/secure
>>>>
>>>> # Log all the mail messages in one place.
>>>> mail.* -/var/log/maillog
>>>>
>>>> # Log cron stuff
>>>> cron.* /var/log/cron
>>>>
>>>> # Everybody gets emergency messages
>>>> *.emerg *
>>>>
>>>> # Save news errors of level crit and higher in a special file.
>>>> uucp,news.crit /var/log/spooler
>>>>
>>>> # Save boot messages also to boot.log
>>>> local7.* /var/log/boot.log
>>>>
>>>> #Load External Listender
>>>> $modload imudp
>>>>
>>>>
>>>> $UDPServerRun 514
>>>> #SPA Messages
>>>> #*.* /var/log/linksys.log
>>>> $template DynaFile,"/var/log/system-%HOSTNAME%.log"
>>>> *.* -?DynaFile
>>>>
>>>>
>>>> ################
>>>> #### ORACLE ####
>>>> ################
>>>>
>>>> $ModLoad omoracle
>>>> $OmoracleDBUser syslog
>>>> $OmoracleDBPassword syslog
>>>> $OmoracleDB syslog
>>>> $OmoracleBatchSize 1
>>>> $OmoracleBatchItemSize 4096
>>>> $OmoracleStatementTemplate OmoracleStatement
>>>>
>>>> $template OmoracleStatement,"INSERT INTO TEST(hostname,message)
>>>> VALUES(:hostname,:msg)"
>>>> $template TestStmt,"%hostname%%msg%"
>>>>
>>>> *.*      :omoracle:;TestStmt
>>>> #====================================================
>>>>
>>>> Really appreciate anyone's help!
>>>>
>>>> Best
>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
>> --
>> This message has been scanned for viruses and dangerous content by
>> MailScanner, and is believed to be clean.
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>
> --
> This message has been scanned for viruses and dangerous content by
> MailScanner, and is believed to be clean.
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tt
Type: application/octet-stream
Size: 464915 bytes
Desc: 
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20110510/c96c3ac5/attachment-0001.obj>
-------------- next part --------------
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Wed May 11 07:30:45 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 11 May 2011 07:30:45 +0200
Subject: [rsyslog] help : CPU high load of rsyslog writing to Oracle
In-Reply-To: <004801cc0f35$731e9850$595bc8f0$@italkbb.com>
References: <002101cc0e60$9aa7f6d0$cff7e470$@italkbb.com>	<alpine.DEB.2.00.1105091055030.25291@asgard.lang.hm>	<alpine.DEB.2.00.1105092231490.6302@asgard.lang.hm>	<001c01cc0ed8$6ea9ac40$4bfd04c0$@italkbb.com><alpine.DEB.2.00.1105092313190.6302@asgard.lang.hm>
	<004801cc0f35$731e9850$595bc8f0$@italkbb.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DE132@GRFEXC.intern.adiscon.com>

I also had a look at the debug log, but it does look very normal, I do not
see any sign of a loop. Was that really created while rsyslog was looping?
All I see is that omoracle receives and stores data and then some new data
comes in via UDP ... and then the log ends. 

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of bxu
> Sent: Tuesday, May 10, 2011 7:12 PM
> To: 'rsyslog-users'
> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to
> Oracle
> 
> Hi David,
> 	Here are what I'm doing:
> 	1. oracle configuration of rsyslogd.conf
> 
> ################
> #### ORACLE ####
> ################
> 
> $ModLoad omoracle
> $OmoracleDBUser syslog
> $OmoracleDBPassword syslog
> $OmoracleDB syslog
> $OmoracleBatchSize 1
> $OmoracleBatchItemSize 4096
> $OmoracleStatementTemplate OmoracleStatement
> 
> $template OmoracleStatement,"INSERT INTO TEST(Hostname,Message)
> VALUES(:hostname,:msg)"
> $template TestStmt,"%hostname%%msg%"
> $template TestStmtFile,"INSERT INTO TEST(Hostname,Message)
> VALUES('%hostname%','%msg%')\n"
> 
> #$WorkDirectory /rsyslog/work # default location for work (spool) files
> 
> #$ActionQueueType LinkedList # use asynchronous processing
> #$ActionQueueFileName dbq    # set file name, also enables disk mode
> #$ActionResumeRetryCount -1  # infinite retries on insert failure
> 
> :msg,contains,"SIP"             :omoracle:;TestStmt
> 
> & ~
> 
> *.*                             /var/log/syslog-test;TestStmtFile
> 
> 	2. running command :  rsyslogd -c5 -nd > tt &
> 
> 	3, TOP information
> 
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 
> 13470 root      25   0  244m  11m 7636 R 99.9  0.1   1:06.39 rs:main
> Q:Reg
> 
> 11410 root      15   0 12768 1144  832 R  0.3  0.0   0:13.52 top
> 
>     1 root      15   0 10372  692  580 S  0.0  0.0   0:00.51 init
> 
> 	4. when CPU usage is keeping at 100%,the summary of strace by
> thread
> 13470:
> 
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
>  51.26    0.019996        9998         2           epoll_wait
>  46.12    0.017990        1799        10           read
>   2.56    0.000999         999         1           select
>   0.06    0.000022           2        12         2 recvfrom
>   0.00    0.000000           0        64           write
>   0.00    0.000000           0         1           recvmsg
>   0.00    0.000000           0        17           gettimeofday
>   0.00    0.000000           0        62           clock_gettime
> ------ ----------- ----------- --------- --------- ----------------
> 100.00    0.039007                   169         2 total
> 
> 	5. when CPU usage is keeping at 100%,some of the call sequence of
> strace by thread 13470:
> 
> [pid 13469]      0.000000 select(1, NULL, NULL, NULL, {86097, 801000}
> <unfinished ...>
> [pid 13473]      0.000089 clock_gettime(CLOCK_REALTIME,  <unfinished
> ...>
> [pid 13472]      0.000054 select(9, [8], NULL, NULL, NULL <unfinished
> ...>
> [pid 13471]      0.000066 read(3,  <unfinished ...>
> [pid 13473]      0.000048 <... clock_gettime resumed> {1305046294,
> 274947000}) = 0 <0.000138>
> [pid 13473]      0.000067 write(1, "6294.274947000:43899940: imudp:
> "...,
> 66) = 66 <0.000042>
> [pid 13473]      0.000137 epoll_wait(12,  <unfinished ...>
> [pid 13471]      1.421497 <... read resumed> "<6>iptables: log and drop
> IN=eth"..., 4095) = 200 <1.421706>
> [pid 13471]      0.000095 gettimeofday({1305046295, 696914}, NULL) = 0
> <0.000022>
> [pid 13471]      0.000130 clock_gettime(CLOCK_REALTIME, {1305046295,
> 697046000}) = 0 <0.000022>
> [pid 13471]      0.000104 write(1, "6295.697046000:42497940: main
> Q:"...,
> 81) = 81 <0.000037>
> [pid 13471]      0.000127 clock_gettime(CLOCK_REALTIME, {1305046295,
> 697276000}) = 0 <0.000020>
> [pid 13471]      0.000102 write(1, "6295.697276000:42497940: main
> Q:"...,
> 65) = 65 <0.000031>
> [pid 13471]      0.000120 read(3, "<6>iptables: log and drop
> IN=eth"...,
> 4095) = 200 <4.250732>
> [pid 13471]      4.250863 gettimeofday({1305046299, 948362}, NULL) = 0
> <0.000021>
> [pid 13471]      0.000129 clock_gettime(CLOCK_REALTIME, {1305046299,
> 948491000}) = 0 <0.000021>
> [pid 13471]      0.000106 write(1, "6299.948491000:42497940: main
> Q:"...,
> 81) = 81 <0.000051>
> [pid 13471]      0.000140 clock_gettime(CLOCK_REALTIME, {1305046299,
> 948735000}) = 0 <0.000021>
> [pid 13471]      0.000101 write(1, "6299.948735000:42497940: main
> Q:"...,
> 65) = 65 <0.000031>
> [pid 13471]      0.000123 read(3, "<6>iptables: log and drop
> IN=eth"...,
> 4095) = 200 <2.178833>
> [pid 13471]      2.178931 gettimeofday({1305046302, 127891}, NULL) = 0
> <0.000022>
> [pid 13471]      0.000112 clock_gettime(CLOCK_REALTIME, {1305046302,
> 128003000}) = 0 <0.000021>
> [pid 13471]      0.000101 write(1, "6302.128003000:42497940: main
> Q:"...,
> 81) = 81 <0.000036>
> [pid 13471]      0.000121 clock_gettime(CLOCK_REALTIME, {1305046302,
> 128225000}) = 0 <0.000021>
> [pid 13471]      0.000102 write(1, "6302.128225000:42497940: main
> Q:"...,
> 65) = 65 <0.000031>
> [pid 13471]      0.000115 read(3, "<6>iptables: log and drop
> IN=eth"...,
> 4095) = 200 <2.431092>
> [pid 13471]      2.431228 gettimeofday({1305046304, 559672}, NULL) = 0
> <0.000022>
> [pid 13471]      0.000132 clock_gettime(CLOCK_REALTIME, {1305046304,
> 559803000}) = 0 <0.000021>
> [pid 13471]      0.000110 write(1, "6304.559803000:42497940: main
> Q:"...,
> 81) = 81 <0.000052>
> [pid 13471]      0.000140 clock_gettime(CLOCK_REALTIME, {1305046304,
> 560052000}) = 0 <0.000021>
> [pid 13471]      0.000102 write(1, "6304.560052000:42497940: main
> Q:"...,
> 65) = 65 <0.000031>
> [pid 13471]      0.000119 read(3, "<6>iptables: log and drop
> IN=eth"...,
> 4095) = 197 <0.678610>
> [pid 13471]      0.678704 gettimeofday({1305046305, 238976}, NULL) = 0
> <0.000021>
> [pid 13471]      0.000110 clock_gettime(CLOCK_REALTIME, {1305046305,
> 239087000}) = 0 <0.000024>
> [pid 13471]      0.000105 write(1, "6305.239087000:42497940: main
> Q:"...,
> 81) = 81 <0.000035>
> [pid 13471]      0.000120 clock_gettime(CLOCK_REALTIME, {1305046305,
> 239312000}) = 0 <0.000020>
> [pid 13471]      0.000102 write(1, "6305.239312000:42497940: main
> Q:"...,
> 65) = 65 <0.000030>
> [pid 13471]      0.000114 read(3, "<6>iptables: log and drop
> IN=eth"...,
> 4095) = 200 <3.058797>
> [pid 13471]      3.058929 gettimeofday({1305046308, 298458}, NULL) = 0
> <0.000022>
> [pid 13471]      0.000131 clock_gettime(CLOCK_REALTIME, {1305046308,
> 298589000}) = 0 <0.000021>
> [pid 13471]      0.000106 write(1, "6308.298589000:42497940: main
> Q:"...,
> 81) = 81 <0.000058>
> [pid 13471]      0.000148 clock_gettime(CLOCK_REALTIME, {1305046308,
> 298841000}) = 0 <0.000021>
> [pid 13471]      0.000101 write(1, "6308.298841000:42497940: main
> Q:"...,
> 65) = 65 <0.000093>
> [pid 13471]      0.000181 read(3, "<6>iptables: log and drop
> IN=eth"...,
> 4095) = 197 <0.213955>
> [pid 13471]      0.214082 gettimeofday({1305046308, 513206}, NULL) = 0
> <0.000021>
> [pid 13471]      0.000128 clock_gettime(CLOCK_REALTIME, {1305046308,
> 513335000}) = 0 <0.000021>
> [pid 13471]      0.000106 write(1, "6308.513335000:42497940: main
> Q:"...,
> 81) = 81 <0.000042>
> [pid 13471]      0.000130 clock_gettime(CLOCK_REALTIME, {1305046308,
> 513570000}) = 0 <0.000021>
> [pid 13471]      0.000102 write(1, "6308.513570000:42497940: main
> Q:"...,
> 65) = 65 <0.000030>
> [pid 13471]      0.000122 read(3, "<6>iptables: log and drop
> IN=eth"...,
> 4095) = 196 <0.835091>
> [pid 13471]      0.835187 gettimeofday({1305046309, 348980}, NULL) = 0
> <0.000021>
> [pid 13471]      0.000110 clock_gettime(CLOCK_REALTIME, {1305046309,
> 349091000}) = 0 <0.000021>
> [pid 13471]      0.000102 write(1, "6309.349091000:42497940: main
> Q:"...,
> 81) = 81 <0.000035>
> [pid 13471]      0.000121 clock_gettime(CLOCK_REALTIME, {1305046309,
> 349313000}) = 0 <0.000020>
> [pid 13471]      0.000102 write(1, "6309.349313000:42497940: main
> Q:"...,
> 65) = 65 <0.000030>
> [pid 13471]      0.000114 read(3,  <unfinished ...>
> [pid 13473]      2.018865 <... epoll_wait resumed> {{EPOLLIN, {u32=2,
> u64=2}}}, 10, 4294967295) = 1 <17.093033>
> [pid 13473]      0.000086 clock_gettime(CLOCK_REALTIME, {1305046311,
> 368481000}) = 0 <0.000020>
> [pid 13473]      0.000098 write(1, "6311.368481000:43899940: imudp:
> "...,
> 65) = 65 <0.000036>
> 
> 
> 	6. Attached is the log file while rsyslogd is running in debug
> mode
> 	    In the log,after "omoracle insertion to DB succeeded",
> there's
> no more oracle log information related.
> 
> 
> 	Thank you very much.
> 
> Best,
> Henry
> 
> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, May 10, 2011 1:19 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to
> Oracle
> 
> rsyslog uses multiple threads, if you can tell which thread is using
> all the
> CPU and then do a strace of it and dump the sequence of calls that it
> makes
> to process one of the messages it may help
> 
> I don't have any idea what's going on here, so I'm just working through
> the
> steps that I would be doing if I was running into the problem
> 
> try ps -eLf to list the different threads
> 
> in top, hit 'H' to run on per-thread reporting (at which point you may
> get
> some additional info on the right about what the thread is doing)
> 
> from the stats you are showing here, it looks like the thread you got
> with
> your strace is the parent that's receiving messages (lots of
> gettimeofday
> and receive/read type calls) the correct one should have write calls in
> it
> (among others)
> 
> the versions you are running are new enough that we shouldn't have any
> old,
> solved problems hitting us.
> 
>   On Tue, 10 May 2011, bxu
> wrote:
> 
> > Hi David,
> > 	The rsyslogd version I tried are 5.8.0(stable) and 6.1.7(devel).
> > 	I straced rsyslogd when CPU usage is keeping at 100%.
> >
> > % time     seconds  usecs/call     calls    errors syscall
> > ------ ----------- ----------- --------- --------- ----------------
> > 43.47    0.066967        2309        29           read
> > 38.28    0.058979         855        69           epoll_wait
> > 18.17    0.027994        9331         3           select
> >  0.05    0.000081           1       114           gettimeofday
> >  0.03    0.000041           0       178        69 recvfrom
> >  0.00    0.000000           0         3           recvmsg
> > ------ ----------- ----------- --------- --------- ----------------
> > 100.00    0.154062                   396        69 total
> >
> >
> > Best,
> > Henry
> >
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com
> > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> > Sent: Tuesday, May 10, 2011 12:35 AM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to
> > Oracle
> >
> > I accidently deleted your reply, so I am replying to my message
> >
> > 4-5 messages/secshould not be any problem at all.
> >
> > since writing a copy of the sql commands to a file doesn't show
> > anything strange, the next step will be to try and run it in debug
> > mode to see if it logs anything strange as you start having problems.
> >
> > the next step after that will probably be to do a strace of the
> thread
> > that is using all the CPU
> >
> > I don't see where you mention what version of rsyslog you are using.
> >
> > David Lang
> >
> > On Mon, 9 May 2011, david at lang.hm wrote:
> >
> >> Date: Mon, 9 May 2011 10:56:30 -0700 (PDT)
> >> From: david at lang.hm
> >> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> >> To: rsyslog-users <rsyslog at lists.adiscon.com>
> >> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to
> >> Oracle
> >>
> >> given that it only happens after a few messages, I would suspect
> that
> >> there is something odd with the messages.
> >>
> >> could you try writing the messages to a file using the Oracle
> >> template and then look and see if there is anything odd about the
> >> messages when it goes nuts?
> >>
> >> the other option is to get a debug dump, but unless it happens right
> >> away, that is likely to be a large dump.
> >>
> >> David Lang
> >>
> >> On Mon, 9 May 2011, bxu wrote:
> >>
> >>> Date: Mon, 9 May 2011 10:48:50 -0500
> >>> From: bxu <bxu at italkbb.com>
> >>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> >>> To: rsyslog at lists.adiscon.com
> >>> Subject: [rsyslog] help : CPU high load of rsyslog writing to
> Oracle
> >>>
> >>> Hi All,
> >>>
> >>>
> >>> I have configured rsyslog writing to Oracle.The writing was
> >>> successful.But after several messages have been made into
> Oracle,the
> >>> CPU use percentage of rsyslogd goes up to 99%-100%.It is abnormally
> >>> too
> > much high.
> >>> There's no problem with the cpu usage when writing to files.
> >>>
> >>> Here is my rsyslogd.conf:
> >>> #====================================================
> >>> #*.* >192.168.45.213,rsyslog
> >>>
> >>> # Use traditional timestamp format
> >>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
> >>>
> >>> # Provides kernel logging support (previously done by rklogd)
> >>> $ModLoad imklog # Provides support for local system logging (e.g.
> >>> via logger command) $ModLoad imuxsock
> >>>
> >>> # Log all kernel messages to the console.
> >>> # Logging much else clutters up the screen.
> >>> #kern.* /dev/console
> >>>
> >>> # Log anything (except mail) of level info or higher.
> >>> # Don't log private authentication messages!
> >>> *.info;mail.none;authpriv.none;cron.none /var/log/messages
> >>>
> >>> # The authpriv file has restricted access.
> >>> authpriv.* /var/log/secure
> >>>
> >>> # Log all the mail messages in one place.
> >>> mail.* -/var/log/maillog
> >>>
> >>> # Log cron stuff
> >>> cron.* /var/log/cron
> >>>
> >>> # Everybody gets emergency messages
> >>> *.emerg *
> >>>
> >>> # Save news errors of level crit and higher in a special file.
> >>> uucp,news.crit /var/log/spooler
> >>>
> >>> # Save boot messages also to boot.log
> >>> local7.* /var/log/boot.log
> >>>
> >>> #Load External Listender
> >>> $modload imudp
> >>>
> >>>
> >>> $UDPServerRun 514
> >>> #SPA Messages
> >>> #*.* /var/log/linksys.log
> >>> $template DynaFile,"/var/log/system-%HOSTNAME%.log"
> >>> *.* -?DynaFile
> >>>
> >>>
> >>> ################
> >>> #### ORACLE ####
> >>> ################
> >>>
> >>> $ModLoad omoracle
> >>> $OmoracleDBUser syslog
> >>> $OmoracleDBPassword syslog
> >>> $OmoracleDB syslog
> >>> $OmoracleBatchSize 1
> >>> $OmoracleBatchItemSize 4096
> >>> $OmoracleStatementTemplate OmoracleStatement
> >>>
> >>> $template OmoracleStatement,"INSERT INTO TEST(hostname,message)
> >>> VALUES(:hostname,:msg)"
> >>> $template TestStmt,"%hostname%%msg%"
> >>>
> >>> *.*      :omoracle:;TestStmt
> >>> #====================================================
> >>>
> >>> Really appreciate anyone's help!
> >>>
> >>> Best
> >>>
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >>>
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >>
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> > --
> > This message has been scanned for viruses and dangerous content by
> > MailScanner, and is believed to be clean.
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 
> --
> This message has been scanned for viruses and dangerous content by
> MailScanner, and is believed to be clean.

From bxu at italkbb.com  Wed May 11 08:15:03 2011
From: bxu at italkbb.com (bxu)
Date: Wed, 11 May 2011 01:15:03 -0500
Subject: [rsyslog] help : CPU high load of rsyslog writing to Oracle
In-Reply-To: <alpine.DEB.2.00.1105101820220.25291@asgard.lang.hm>
References: <002101cc0e60$9aa7f6d0$cff7e470$@italkbb.com>	<alpine.DEB.2.00.1105091055030.25291@asgard.lang.hm>	<alpine.DEB.2.00.1105092231490.6302@asgard.lang.hm>	<001c01cc0ed8$6ea9ac40$4bfd04c0$@italkbb.com>	<alpine.DEB.2.00.1105092313190.6302@asgard.lang.hm>	<004801cc0f35$731e9850$595bc8f0$@italkbb.com>
	<alpine.DEB.2.00.1105101820220.25291@asgard.lang.hm>
Message-ID: <002701cc0fa2$c849cc60$58dd6520$@italkbb.com>

Hi David,
	Yes. The 'H' was turned on for the TOP info.

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 49.99    0.008990           5      1858       929 futex
 33.34    0.005996         857         7           select
 16.67    0.002997          20       151           read
  0.00    0.000000           0       542           recvfrom
  0.00    0.000000           0         7           recvmsg
  0.00    0.000000           0       468           gettimeofday
  0.00    0.000000           0      1860           clock_gettime
------ ----------- ----------- --------- --------- ----------------
100.00    0.017983                  4893       929 total

	As seen from the above TOP info,there are 929 errors for futex
call.Sometimes there were errors for recvmsg call where I did more
straces.Does this relate to the CPU high load?

	I'll do more tests as your suggestion.

Thanks,
Henry

-----Original Message-----
From: rsyslog-bounces at lists.adiscon.com
[mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
Sent: Tuesday, May 10, 2011 8:29 PM
To: rsyslog-users
Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to Oracle

thanks for the info.

when you printed the TOP info, was that with 'H' (thread details) turned on?

I'm puzzled by the fact that you are doing a strace of the thread 13470 and
it's showing actions by many different pids (it looks like everything except
13470

can you try doing

strace -ff -o rsyslog.strace. -C  -ttt -T -s 100  rsyslogd -c5 -nd
>rsyslog.debug

this will create a bunch of strace files. we shouldn't need to look at all
of them, just the one using all the CPU (and possibly only the end of that
one)

while this is running, do a ps -eLf

also, run top -H and note which thread is using all the CPU (you may want to
try toggling H while it's running to make sure you are getting the thread
info not the consolodated process info)

I'm about to head home and when I get home I'll look at the debug output
that you send.

one thing I'm not remembering. are the logs getting into the database?

David Lang

On Tue, 10 May 2011, bxu wrote:

> Hi David,
> 	Here are what I'm doing:
> 	1. oracle configuration of rsyslogd.conf
>
> ################
> #### ORACLE ####
> ################
>
> $ModLoad omoracle
> $OmoracleDBUser syslog
> $OmoracleDBPassword syslog
> $OmoracleDB syslog
> $OmoracleBatchSize 1
> $OmoracleBatchItemSize 4096
> $OmoracleStatementTemplate OmoracleStatement
>
> $template OmoracleStatement,"INSERT INTO TEST(Hostname,Message) 
> VALUES(:hostname,:msg)"
> $template TestStmt,"%hostname%%msg%"
> $template TestStmtFile,"INSERT INTO TEST(Hostname,Message) 
> VALUES('%hostname%','%msg%')\n"
>
> #$WorkDirectory /rsyslog/work # default location for work (spool) 
> files
>
> #$ActionQueueType LinkedList # use asynchronous processing
> #$ActionQueueFileName dbq    # set file name, also enables disk mode
> #$ActionResumeRetryCount -1  # infinite retries on insert failure
>
> :msg,contains,"SIP"             :omoracle:;TestStmt
>
> & ~
>
> *.*                             /var/log/syslog-test;TestStmtFile
>
> 	2. running command :  rsyslogd -c5 -nd > tt &
>
> 	3, TOP information
>
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>
> 13470 root      25   0  244m  11m 7636 R 99.9  0.1   1:06.39 rs:main Q:Reg
>
> 11410 root      15   0 12768 1144  832 R  0.3  0.0   0:13.52 top
>
>    1 root      15   0 10372  692  580 S  0.0  0.0   0:00.51 init
>
> 	4. when CPU usage is keeping at 100%,the summary of strace by thread
> 13470:
>
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 51.26    0.019996        9998         2           epoll_wait
> 46.12    0.017990        1799        10           read
>  2.56    0.000999         999         1           select
>  0.06    0.000022           2        12         2 recvfrom
>  0.00    0.000000           0        64           write
>  0.00    0.000000           0         1           recvmsg
>  0.00    0.000000           0        17           gettimeofday
>  0.00    0.000000           0        62           clock_gettime
> ------ ----------- ----------- --------- --------- ----------------
> 100.00    0.039007                   169         2 total
>
> 	5. when CPU usage is keeping at 100%,some of the call sequence of 
> strace by thread 13470:
>
> [pid 13469]      0.000000 select(1, NULL, NULL, NULL, {86097, 801000}
> <unfinished ...>
> [pid 13473]      0.000089 clock_gettime(CLOCK_REALTIME,  <unfinished ...>
> [pid 13472]      0.000054 select(9, [8], NULL, NULL, NULL <unfinished ...>
> [pid 13471]      0.000066 read(3,  <unfinished ...>
> [pid 13473]      0.000048 <... clock_gettime resumed> {1305046294,
> 274947000}) = 0 <0.000138>
> [pid 13473]      0.000067 write(1, "6294.274947000:43899940: imudp: "...,
> 66) = 66 <0.000042>
> [pid 13473]      0.000137 epoll_wait(12,  <unfinished ...>
> [pid 13471]      1.421497 <... read resumed> "<6>iptables: log and drop
> IN=eth"..., 4095) = 200 <1.421706>
> [pid 13471]      0.000095 gettimeofday({1305046295, 696914}, NULL) = 0
> <0.000022>
> [pid 13471]      0.000130 clock_gettime(CLOCK_REALTIME, {1305046295,
> 697046000}) = 0 <0.000022>
> [pid 13471]      0.000104 write(1, "6295.697046000:42497940: main Q:"...,
> 81) = 81 <0.000037>
> [pid 13471]      0.000127 clock_gettime(CLOCK_REALTIME, {1305046295,
> 697276000}) = 0 <0.000020>
> [pid 13471]      0.000102 write(1, "6295.697276000:42497940: main Q:"...,
> 65) = 65 <0.000031>
> [pid 13471]      0.000120 read(3, "<6>iptables: log and drop IN=eth"...,
> 4095) = 200 <4.250732>
> [pid 13471]      4.250863 gettimeofday({1305046299, 948362}, NULL) = 0
> <0.000021>
> [pid 13471]      0.000129 clock_gettime(CLOCK_REALTIME, {1305046299,
> 948491000}) = 0 <0.000021>
> [pid 13471]      0.000106 write(1, "6299.948491000:42497940: main Q:"...,
> 81) = 81 <0.000051>
> [pid 13471]      0.000140 clock_gettime(CLOCK_REALTIME, {1305046299,
> 948735000}) = 0 <0.000021>
> [pid 13471]      0.000101 write(1, "6299.948735000:42497940: main Q:"...,
> 65) = 65 <0.000031>
> [pid 13471]      0.000123 read(3, "<6>iptables: log and drop IN=eth"...,
> 4095) = 200 <2.178833>
> [pid 13471]      2.178931 gettimeofday({1305046302, 127891}, NULL) = 0
> <0.000022>
> [pid 13471]      0.000112 clock_gettime(CLOCK_REALTIME, {1305046302,
> 128003000}) = 0 <0.000021>
> [pid 13471]      0.000101 write(1, "6302.128003000:42497940: main Q:"...,
> 81) = 81 <0.000036>
> [pid 13471]      0.000121 clock_gettime(CLOCK_REALTIME, {1305046302,
> 128225000}) = 0 <0.000021>
> [pid 13471]      0.000102 write(1, "6302.128225000:42497940: main Q:"...,
> 65) = 65 <0.000031>
> [pid 13471]      0.000115 read(3, "<6>iptables: log and drop IN=eth"...,
> 4095) = 200 <2.431092>
> [pid 13471]      2.431228 gettimeofday({1305046304, 559672}, NULL) = 0
> <0.000022>
> [pid 13471]      0.000132 clock_gettime(CLOCK_REALTIME, {1305046304,
> 559803000}) = 0 <0.000021>
> [pid 13471]      0.000110 write(1, "6304.559803000:42497940: main Q:"...,
> 81) = 81 <0.000052>
> [pid 13471]      0.000140 clock_gettime(CLOCK_REALTIME, {1305046304,
> 560052000}) = 0 <0.000021>
> [pid 13471]      0.000102 write(1, "6304.560052000:42497940: main Q:"...,
> 65) = 65 <0.000031>
> [pid 13471]      0.000119 read(3, "<6>iptables: log and drop IN=eth"...,
> 4095) = 197 <0.678610>
> [pid 13471]      0.678704 gettimeofday({1305046305, 238976}, NULL) = 0
> <0.000021>
> [pid 13471]      0.000110 clock_gettime(CLOCK_REALTIME, {1305046305,
> 239087000}) = 0 <0.000024>
> [pid 13471]      0.000105 write(1, "6305.239087000:42497940: main Q:"...,
> 81) = 81 <0.000035>
> [pid 13471]      0.000120 clock_gettime(CLOCK_REALTIME, {1305046305,
> 239312000}) = 0 <0.000020>
> [pid 13471]      0.000102 write(1, "6305.239312000:42497940: main Q:"...,
> 65) = 65 <0.000030>
> [pid 13471]      0.000114 read(3, "<6>iptables: log and drop IN=eth"...,
> 4095) = 200 <3.058797>
> [pid 13471]      3.058929 gettimeofday({1305046308, 298458}, NULL) = 0
> <0.000022>
> [pid 13471]      0.000131 clock_gettime(CLOCK_REALTIME, {1305046308,
> 298589000}) = 0 <0.000021>
> [pid 13471]      0.000106 write(1, "6308.298589000:42497940: main Q:"...,
> 81) = 81 <0.000058>
> [pid 13471]      0.000148 clock_gettime(CLOCK_REALTIME, {1305046308,
> 298841000}) = 0 <0.000021>
> [pid 13471]      0.000101 write(1, "6308.298841000:42497940: main Q:"...,
> 65) = 65 <0.000093>
> [pid 13471]      0.000181 read(3, "<6>iptables: log and drop IN=eth"...,
> 4095) = 197 <0.213955>
> [pid 13471]      0.214082 gettimeofday({1305046308, 513206}, NULL) = 0
> <0.000021>
> [pid 13471]      0.000128 clock_gettime(CLOCK_REALTIME, {1305046308,
> 513335000}) = 0 <0.000021>
> [pid 13471]      0.000106 write(1, "6308.513335000:42497940: main Q:"...,
> 81) = 81 <0.000042>
> [pid 13471]      0.000130 clock_gettime(CLOCK_REALTIME, {1305046308,
> 513570000}) = 0 <0.000021>
> [pid 13471]      0.000102 write(1, "6308.513570000:42497940: main Q:"...,
> 65) = 65 <0.000030>
> [pid 13471]      0.000122 read(3, "<6>iptables: log and drop IN=eth"...,
> 4095) = 196 <0.835091>
> [pid 13471]      0.835187 gettimeofday({1305046309, 348980}, NULL) = 0
> <0.000021>
> [pid 13471]      0.000110 clock_gettime(CLOCK_REALTIME, {1305046309,
> 349091000}) = 0 <0.000021>
> [pid 13471]      0.000102 write(1, "6309.349091000:42497940: main Q:"...,
> 81) = 81 <0.000035>
> [pid 13471]      0.000121 clock_gettime(CLOCK_REALTIME, {1305046309,
> 349313000}) = 0 <0.000020>
> [pid 13471]      0.000102 write(1, "6309.349313000:42497940: main Q:"...,
> 65) = 65 <0.000030>
> [pid 13471]      0.000114 read(3,  <unfinished ...>
> [pid 13473]      2.018865 <... epoll_wait resumed> {{EPOLLIN, {u32=2,
> u64=2}}}, 10, 4294967295) = 1 <17.093033>
> [pid 13473]      0.000086 clock_gettime(CLOCK_REALTIME, {1305046311,
> 368481000}) = 0 <0.000020>
> [pid 13473]      0.000098 write(1, "6311.368481000:43899940: imudp: "...,
> 65) = 65 <0.000036>
>
>
> 	6. Attached is the log file while rsyslogd is running in debug mode
> 	    In the log,after "omoracle insertion to DB succeeded", there's
no 
> more oracle log information related.
>
>
> 	Thank you very much.
>
> Best,
> Henry
>
> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, May 10, 2011 1:19 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to 
> Oracle
>
> rsyslog uses multiple threads, if you can tell which thread is using 
> all the CPU and then do a strace of it and dump the sequence of calls 
> that it makes to process one of the messages it may help
>
> I don't have any idea what's going on here, so I'm just working 
> through the steps that I would be doing if I was running into the 
> problem
>
> try ps -eLf to list the different threads
>
> in top, hit 'H' to run on per-thread reporting (at which point you may 
> get some additional info on the right about what the thread is doing)
>
> from the stats you are showing here, it looks like the thread you got 
> with your strace is the parent that's receiving messages (lots of 
> gettimeofday and receive/read type calls) the correct one should have 
> write calls in it (among others)
>
> the versions you are running are new enough that we shouldn't have any 
> old, solved problems hitting us.
>
>  On Tue, 10 May 2011, bxu
> wrote:
>
>> Hi David,
>> 	The rsyslogd version I tried are 5.8.0(stable) and 6.1.7(devel).
>> 	I straced rsyslogd when CPU usage is keeping at 100%.
>>
>> % time     seconds  usecs/call     calls    errors syscall
>> ------ ----------- ----------- --------- --------- ----------------
>> 43.47    0.066967        2309        29           read
>> 38.28    0.058979         855        69           epoll_wait
>> 18.17    0.027994        9331         3           select
>>  0.05    0.000081           1       114           gettimeofday
>>  0.03    0.000041           0       178        69 recvfrom
>>  0.00    0.000000           0         3           recvmsg
>> ------ ----------- ----------- --------- --------- ----------------
>> 100.00    0.154062                   396        69 total
>>
>>
>> Best,
>> Henry
>>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com 
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Tuesday, May 10, 2011 12:35 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to 
>> Oracle
>>
>> I accidently deleted your reply, so I am replying to my message
>>
>> 4-5 messages/secshould not be any problem at all.
>>
>> since writing a copy of the sql commands to a file doesn't show 
>> anything strange, the next step will be to try and run it in debug 
>> mode to see if it logs anything strange as you start having problems.
>>
>> the next step after that will probably be to do a strace of the 
>> thread that is using all the CPU
>>
>> I don't see where you mention what version of rsyslog you are using.
>>
>> David Lang
>>
>> On Mon, 9 May 2011, david at lang.hm wrote:
>>
>>> Date: Mon, 9 May 2011 10:56:30 -0700 (PDT)
>>> From: david at lang.hm
>>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>>> To: rsyslog-users <rsyslog at lists.adiscon.com>
>>> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to 
>>> Oracle
>>>
>>> given that it only happens after a few messages, I would suspect 
>>> that there is something odd with the messages.
>>>
>>> could you try writing the messages to a file using the Oracle 
>>> template and then look and see if there is anything odd about the 
>>> messages when it goes nuts?
>>>
>>> the other option is to get a debug dump, but unless it happens right 
>>> away, that is likely to be a large dump.
>>>
>>> David Lang
>>>
>>> On Mon, 9 May 2011, bxu wrote:
>>>
>>>> Date: Mon, 9 May 2011 10:48:50 -0500
>>>> From: bxu <bxu at italkbb.com>
>>>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>>>> To: rsyslog at lists.adiscon.com
>>>> Subject: [rsyslog] help : CPU high load of rsyslog writing to 
>>>> Oracle
>>>>
>>>> Hi All,
>>>>
>>>>
>>>> I have configured rsyslog writing to Oracle.The writing was 
>>>> successful.But after several messages have been made into 
>>>> Oracle,the CPU use percentage of rsyslogd goes up to 99%-100%.It is 
>>>> abnormally too
>> much high.
>>>> There's no problem with the cpu usage when writing to files.
>>>>
>>>> Here is my rsyslogd.conf:
>>>> #====================================================
>>>> #*.* >192.168.45.213,rsyslog
>>>>
>>>> # Use traditional timestamp format
>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>>
>>>> # Provides kernel logging support (previously done by rklogd) 
>>>> $ModLoad imklog # Provides support for local system logging (e.g.
>>>> via logger command) $ModLoad imuxsock
>>>>
>>>> # Log all kernel messages to the console.
>>>> # Logging much else clutters up the screen.
>>>> #kern.* /dev/console
>>>>
>>>> # Log anything (except mail) of level info or higher.
>>>> # Don't log private authentication messages!
>>>> *.info;mail.none;authpriv.none;cron.none /var/log/messages
>>>>
>>>> # The authpriv file has restricted access.
>>>> authpriv.* /var/log/secure
>>>>
>>>> # Log all the mail messages in one place.
>>>> mail.* -/var/log/maillog
>>>>
>>>> # Log cron stuff
>>>> cron.* /var/log/cron
>>>>
>>>> # Everybody gets emergency messages *.emerg *
>>>>
>>>> # Save news errors of level crit and higher in a special file.
>>>> uucp,news.crit /var/log/spooler
>>>>
>>>> # Save boot messages also to boot.log
>>>> local7.* /var/log/boot.log
>>>>
>>>> #Load External Listender
>>>> $modload imudp
>>>>
>>>>
>>>> $UDPServerRun 514
>>>> #SPA Messages
>>>> #*.* /var/log/linksys.log
>>>> $template DynaFile,"/var/log/system-%HOSTNAME%.log"
>>>> *.* -?DynaFile
>>>>
>>>>
>>>> ################
>>>> #### ORACLE ####
>>>> ################
>>>>
>>>> $ModLoad omoracle
>>>> $OmoracleDBUser syslog
>>>> $OmoracleDBPassword syslog
>>>> $OmoracleDB syslog
>>>> $OmoracleBatchSize 1
>>>> $OmoracleBatchItemSize 4096
>>>> $OmoracleStatementTemplate OmoracleStatement
>>>>
>>>> $template OmoracleStatement,"INSERT INTO TEST(hostname,message) 
>>>> VALUES(:hostname,:msg)"
>>>> $template TestStmt,"%hostname%%msg%"
>>>>
>>>> *.*      :omoracle:;TestStmt
>>>> #====================================================
>>>>
>>>> Really appreciate anyone's help!
>>>>
>>>> Best
>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
>> --
>> This message has been scanned for viruses and dangerous content by 
>> MailScanner, and is believed to be clean.
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>
> --
> This message has been scanned for viruses and dangerous content by 
> MailScanner, and is believed to be clean.
>
--
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.



From rgerhards at hq.adiscon.com  Wed May 11 12:22:20 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 11 May 2011 12:22:20 +0200
Subject: [rsyslog] small client soap library
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DE138@GRFEXC.intern.adiscon.com>

Hi all,

I may be in the position that I may need to talk to a soap server from a
plugin. Can anyone recommend a small library for doing so (preferably built
on top of libxml2)? So far I have found csoap[1], but this project looks
abandoned.

Thanks,
Rainer

[1] http://csoap.sourceforge.net/

From bxu at italkbb.com  Wed May 11 21:47:55 2011
From: bxu at italkbb.com (bxu)
Date: Wed, 11 May 2011 14:47:55 -0500
Subject: [rsyslog] help : CPU high load of rsyslog writing to Oracle
References: <002101cc0e60$9aa7f6d0$cff7e470$@italkbb.com>	<alpine.DEB.2.00.1105091055030.25291@asgard.lang.hm>	<alpine.DEB.2.00.1105092231490.6302@asgard.lang.hm>	<001c01cc0ed8$6ea9ac40$4bfd04c0$@italkbb.com>	<alpine.DEB.2.00.1105092313190.6302@asgard.lang.hm>	<004801cc0f35$731e9850$595bc8f0$@italkbb.com>
	<alpine.DEB.2.00.1105101820220.25291@asgard.lang.hm> 
Message-ID: <001a01cc1014$556f7560$004e6020$@italkbb.com>

Attached is the debug file when use " strace -ff -o rsyslog.strace. -C  -ttt
-T -s 100  rsyslogd -c5 -nd >rsyslog.debug".
Thread ID 23298 consumes almost all the CPU time,which is used to write to
Oracle.
In the file 'rsyslog.strace..23298',there's no more log after "
1305142850.448202 write(1, "action call returned 0\n", 23) = 23
<0.000027>",seems this thread was blocked.
I think there is a loop after some wrong.

-----Original Message-----
From: bxu [mailto:bxu at italkbb.com] 
Sent: Wednesday, May 11, 2011 1:15 AM
To: 'rsyslog-users'
Subject: RE: [rsyslog] help : CPU high load of rsyslog writing to Oracle

Hi David,
	Yes. The 'H' was turned on for the TOP info.

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 49.99    0.008990           5      1858       929 futex
 33.34    0.005996         857         7           select
 16.67    0.002997          20       151           read
  0.00    0.000000           0       542           recvfrom
  0.00    0.000000           0         7           recvmsg
  0.00    0.000000           0       468           gettimeofday
  0.00    0.000000           0      1860           clock_gettime
------ ----------- ----------- --------- --------- ----------------
100.00    0.017983                  4893       929 total

	As seen from the above TOP info,there are 929 errors for futex
call.Sometimes there were errors for recvmsg call where I did more
straces.Does this relate to the CPU high load?

	I'll do more tests as your suggestion.

Thanks,
Henry

-----Original Message-----
From: rsyslog-bounces at lists.adiscon.com
[mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
Sent: Tuesday, May 10, 2011 8:29 PM
To: rsyslog-users
Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to Oracle

thanks for the info.

when you printed the TOP info, was that with 'H' (thread details) turned on?

I'm puzzled by the fact that you are doing a strace of the thread 13470 and
it's showing actions by many different pids (it looks like everything except
13470

can you try doing

strace -ff -o rsyslog.strace. -C  -ttt -T -s 100  rsyslogd -c5 -nd
>rsyslog.debug

this will create a bunch of strace files. we shouldn't need to look at all
of them, just the one using all the CPU (and possibly only the end of that
one)

while this is running, do a ps -eLf

also, run top -H and note which thread is using all the CPU (you may want to
try toggling H while it's running to make sure you are getting the thread
info not the consolodated process info)

I'm about to head home and when I get home I'll look at the debug output
that you send.

one thing I'm not remembering. are the logs getting into the database?

David Lang

On Tue, 10 May 2011, bxu wrote:

> Hi David,
> 	Here are what I'm doing:
> 	1. oracle configuration of rsyslogd.conf
>
> ################
> #### ORACLE ####
> ################
>
> $ModLoad omoracle
> $OmoracleDBUser syslog
> $OmoracleDBPassword syslog
> $OmoracleDB syslog
> $OmoracleBatchSize 1
> $OmoracleBatchItemSize 4096
> $OmoracleStatementTemplate OmoracleStatement
>
> $template OmoracleStatement,"INSERT INTO TEST(Hostname,Message) 
> VALUES(:hostname,:msg)"
> $template TestStmt,"%hostname%%msg%"
> $template TestStmtFile,"INSERT INTO TEST(Hostname,Message) 
> VALUES('%hostname%','%msg%')\n"
>
> #$WorkDirectory /rsyslog/work # default location for work (spool) 
> files
>
> #$ActionQueueType LinkedList # use asynchronous processing
> #$ActionQueueFileName dbq    # set file name, also enables disk mode
> #$ActionResumeRetryCount -1  # infinite retries on insert failure
>
> :msg,contains,"SIP"             :omoracle:;TestStmt
>
> & ~
>
> *.*                             /var/log/syslog-test;TestStmtFile
>
> 	2. running command :  rsyslogd -c5 -nd > tt &
>
> 	3, TOP information
>
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>
> 13470 root      25   0  244m  11m 7636 R 99.9  0.1   1:06.39 rs:main Q:Reg
>
> 11410 root      15   0 12768 1144  832 R  0.3  0.0   0:13.52 top
>
>    1 root      15   0 10372  692  580 S  0.0  0.0   0:00.51 init
>
> 	4. when CPU usage is keeping at 100%,the summary of strace by thread
> 13470:
>
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 51.26    0.019996        9998         2           epoll_wait
> 46.12    0.017990        1799        10           read
>  2.56    0.000999         999         1           select
>  0.06    0.000022           2        12         2 recvfrom
>  0.00    0.000000           0        64           write
>  0.00    0.000000           0         1           recvmsg
>  0.00    0.000000           0        17           gettimeofday
>  0.00    0.000000           0        62           clock_gettime
> ------ ----------- ----------- --------- --------- ----------------
> 100.00    0.039007                   169         2 total
>
> 	5. when CPU usage is keeping at 100%,some of the call sequence of 
> strace by thread 13470:
>
> [pid 13469]      0.000000 select(1, NULL, NULL, NULL, {86097, 801000}
> <unfinished ...>
> [pid 13473]      0.000089 clock_gettime(CLOCK_REALTIME,  <unfinished ...>
> [pid 13472]      0.000054 select(9, [8], NULL, NULL, NULL <unfinished ...>
> [pid 13471]      0.000066 read(3,  <unfinished ...>
> [pid 13473]      0.000048 <... clock_gettime resumed> {1305046294,
> 274947000}) = 0 <0.000138>
> [pid 13473]      0.000067 write(1, "6294.274947000:43899940: imudp: "...,
> 66) = 66 <0.000042>
> [pid 13473]      0.000137 epoll_wait(12,  <unfinished ...>
> [pid 13471]      1.421497 <... read resumed> "<6>iptables: log and drop
> IN=eth"..., 4095) = 200 <1.421706>
> [pid 13471]      0.000095 gettimeofday({1305046295, 696914}, NULL) = 0
> <0.000022>
> [pid 13471]      0.000130 clock_gettime(CLOCK_REALTIME, {1305046295,
> 697046000}) = 0 <0.000022>
> [pid 13471]      0.000104 write(1, "6295.697046000:42497940: main Q:"...,
> 81) = 81 <0.000037>
> [pid 13471]      0.000127 clock_gettime(CLOCK_REALTIME, {1305046295,
> 697276000}) = 0 <0.000020>
> [pid 13471]      0.000102 write(1, "6295.697276000:42497940: main Q:"...,
> 65) = 65 <0.000031>
> [pid 13471]      0.000120 read(3, "<6>iptables: log and drop IN=eth"...,
> 4095) = 200 <4.250732>
> [pid 13471]      4.250863 gettimeofday({1305046299, 948362}, NULL) = 0
> <0.000021>
> [pid 13471]      0.000129 clock_gettime(CLOCK_REALTIME, {1305046299,
> 948491000}) = 0 <0.000021>
> [pid 13471]      0.000106 write(1, "6299.948491000:42497940: main Q:"...,
> 81) = 81 <0.000051>
> [pid 13471]      0.000140 clock_gettime(CLOCK_REALTIME, {1305046299,
> 948735000}) = 0 <0.000021>
> [pid 13471]      0.000101 write(1, "6299.948735000:42497940: main Q:"...,
> 65) = 65 <0.000031>
> [pid 13471]      0.000123 read(3, "<6>iptables: log and drop IN=eth"...,
> 4095) = 200 <2.178833>
> [pid 13471]      2.178931 gettimeofday({1305046302, 127891}, NULL) = 0
> <0.000022>
> [pid 13471]      0.000112 clock_gettime(CLOCK_REALTIME, {1305046302,
> 128003000}) = 0 <0.000021>
> [pid 13471]      0.000101 write(1, "6302.128003000:42497940: main Q:"...,
> 81) = 81 <0.000036>
> [pid 13471]      0.000121 clock_gettime(CLOCK_REALTIME, {1305046302,
> 128225000}) = 0 <0.000021>
> [pid 13471]      0.000102 write(1, "6302.128225000:42497940: main Q:"...,
> 65) = 65 <0.000031>
> [pid 13471]      0.000115 read(3, "<6>iptables: log and drop IN=eth"...,
> 4095) = 200 <2.431092>
> [pid 13471]      2.431228 gettimeofday({1305046304, 559672}, NULL) = 0
> <0.000022>
> [pid 13471]      0.000132 clock_gettime(CLOCK_REALTIME, {1305046304,
> 559803000}) = 0 <0.000021>
> [pid 13471]      0.000110 write(1, "6304.559803000:42497940: main Q:"...,
> 81) = 81 <0.000052>
> [pid 13471]      0.000140 clock_gettime(CLOCK_REALTIME, {1305046304,
> 560052000}) = 0 <0.000021>
> [pid 13471]      0.000102 write(1, "6304.560052000:42497940: main Q:"...,
> 65) = 65 <0.000031>
> [pid 13471]      0.000119 read(3, "<6>iptables: log and drop IN=eth"...,
> 4095) = 197 <0.678610>
> [pid 13471]      0.678704 gettimeofday({1305046305, 238976}, NULL) = 0
> <0.000021>
> [pid 13471]      0.000110 clock_gettime(CLOCK_REALTIME, {1305046305,
> 239087000}) = 0 <0.000024>
> [pid 13471]      0.000105 write(1, "6305.239087000:42497940: main Q:"...,
> 81) = 81 <0.000035>
> [pid 13471]      0.000120 clock_gettime(CLOCK_REALTIME, {1305046305,
> 239312000}) = 0 <0.000020>
> [pid 13471]      0.000102 write(1, "6305.239312000:42497940: main Q:"...,
> 65) = 65 <0.000030>
> [pid 13471]      0.000114 read(3, "<6>iptables: log and drop IN=eth"...,
> 4095) = 200 <3.058797>
> [pid 13471]      3.058929 gettimeofday({1305046308, 298458}, NULL) = 0
> <0.000022>
> [pid 13471]      0.000131 clock_gettime(CLOCK_REALTIME, {1305046308,
> 298589000}) = 0 <0.000021>
> [pid 13471]      0.000106 write(1, "6308.298589000:42497940: main Q:"...,
> 81) = 81 <0.000058>
> [pid 13471]      0.000148 clock_gettime(CLOCK_REALTIME, {1305046308,
> 298841000}) = 0 <0.000021>
> [pid 13471]      0.000101 write(1, "6308.298841000:42497940: main Q:"...,
> 65) = 65 <0.000093>
> [pid 13471]      0.000181 read(3, "<6>iptables: log and drop IN=eth"...,
> 4095) = 197 <0.213955>
> [pid 13471]      0.214082 gettimeofday({1305046308, 513206}, NULL) = 0
> <0.000021>
> [pid 13471]      0.000128 clock_gettime(CLOCK_REALTIME, {1305046308,
> 513335000}) = 0 <0.000021>
> [pid 13471]      0.000106 write(1, "6308.513335000:42497940: main Q:"...,
> 81) = 81 <0.000042>
> [pid 13471]      0.000130 clock_gettime(CLOCK_REALTIME, {1305046308,
> 513570000}) = 0 <0.000021>
> [pid 13471]      0.000102 write(1, "6308.513570000:42497940: main Q:"...,
> 65) = 65 <0.000030>
> [pid 13471]      0.000122 read(3, "<6>iptables: log and drop IN=eth"...,
> 4095) = 196 <0.835091>
> [pid 13471]      0.835187 gettimeofday({1305046309, 348980}, NULL) = 0
> <0.000021>
> [pid 13471]      0.000110 clock_gettime(CLOCK_REALTIME, {1305046309,
> 349091000}) = 0 <0.000021>
> [pid 13471]      0.000102 write(1, "6309.349091000:42497940: main Q:"...,
> 81) = 81 <0.000035>
> [pid 13471]      0.000121 clock_gettime(CLOCK_REALTIME, {1305046309,
> 349313000}) = 0 <0.000020>
> [pid 13471]      0.000102 write(1, "6309.349313000:42497940: main Q:"...,
> 65) = 65 <0.000030>
> [pid 13471]      0.000114 read(3,  <unfinished ...>
> [pid 13473]      2.018865 <... epoll_wait resumed> {{EPOLLIN, {u32=2,
> u64=2}}}, 10, 4294967295) = 1 <17.093033>
> [pid 13473]      0.000086 clock_gettime(CLOCK_REALTIME, {1305046311,
> 368481000}) = 0 <0.000020>
> [pid 13473]      0.000098 write(1, "6311.368481000:43899940: imudp: "...,
> 65) = 65 <0.000036>
>
>
> 	6. Attached is the log file while rsyslogd is running in debug mode
> 	    In the log,after "omoracle insertion to DB succeeded", there's
no 
> more oracle log information related.
>
>
> 	Thank you very much.
>
> Best,
> Henry
>
> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, May 10, 2011 1:19 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to 
> Oracle
>
> rsyslog uses multiple threads, if you can tell which thread is using 
> all the CPU and then do a strace of it and dump the sequence of calls 
> that it makes to process one of the messages it may help
>
> I don't have any idea what's going on here, so I'm just working 
> through the steps that I would be doing if I was running into the 
> problem
>
> try ps -eLf to list the different threads
>
> in top, hit 'H' to run on per-thread reporting (at which point you may 
> get some additional info on the right about what the thread is doing)
>
> from the stats you are showing here, it looks like the thread you got 
> with your strace is the parent that's receiving messages (lots of 
> gettimeofday and receive/read type calls) the correct one should have 
> write calls in it (among others)
>
> the versions you are running are new enough that we shouldn't have any 
> old, solved problems hitting us.
>
>  On Tue, 10 May 2011, bxu
> wrote:
>
>> Hi David,
>> 	The rsyslogd version I tried are 5.8.0(stable) and 6.1.7(devel).
>> 	I straced rsyslogd when CPU usage is keeping at 100%.
>>
>> % time     seconds  usecs/call     calls    errors syscall
>> ------ ----------- ----------- --------- --------- ----------------
>> 43.47    0.066967        2309        29           read
>> 38.28    0.058979         855        69           epoll_wait
>> 18.17    0.027994        9331         3           select
>>  0.05    0.000081           1       114           gettimeofday
>>  0.03    0.000041           0       178        69 recvfrom
>>  0.00    0.000000           0         3           recvmsg
>> ------ ----------- ----------- --------- --------- ----------------
>> 100.00    0.154062                   396        69 total
>>
>>
>> Best,
>> Henry
>>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com 
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Tuesday, May 10, 2011 12:35 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to 
>> Oracle
>>
>> I accidently deleted your reply, so I am replying to my message
>>
>> 4-5 messages/secshould not be any problem at all.
>>
>> since writing a copy of the sql commands to a file doesn't show 
>> anything strange, the next step will be to try and run it in debug 
>> mode to see if it logs anything strange as you start having problems.
>>
>> the next step after that will probably be to do a strace of the 
>> thread that is using all the CPU
>>
>> I don't see where you mention what version of rsyslog you are using.
>>
>> David Lang
>>
>> On Mon, 9 May 2011, david at lang.hm wrote:
>>
>>> Date: Mon, 9 May 2011 10:56:30 -0700 (PDT)
>>> From: david at lang.hm
>>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>>> To: rsyslog-users <rsyslog at lists.adiscon.com>
>>> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to 
>>> Oracle
>>>
>>> given that it only happens after a few messages, I would suspect 
>>> that there is something odd with the messages.
>>>
>>> could you try writing the messages to a file using the Oracle 
>>> template and then look and see if there is anything odd about the 
>>> messages when it goes nuts?
>>>
>>> the other option is to get a debug dump, but unless it happens right 
>>> away, that is likely to be a large dump.
>>>
>>> David Lang
>>>
>>> On Mon, 9 May 2011, bxu wrote:
>>>
>>>> Date: Mon, 9 May 2011 10:48:50 -0500
>>>> From: bxu <bxu at italkbb.com>
>>>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>>>> To: rsyslog at lists.adiscon.com
>>>> Subject: [rsyslog] help : CPU high load of rsyslog writing to 
>>>> Oracle
>>>>
>>>> Hi All,
>>>>
>>>>
>>>> I have configured rsyslog writing to Oracle.The writing was 
>>>> successful.But after several messages have been made into 
>>>> Oracle,the CPU use percentage of rsyslogd goes up to 99%-100%.It is 
>>>> abnormally too
>> much high.
>>>> There's no problem with the cpu usage when writing to files.
>>>>
>>>> Here is my rsyslogd.conf:
>>>> #====================================================
>>>> #*.* >192.168.45.213,rsyslog
>>>>
>>>> # Use traditional timestamp format
>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>>
>>>> # Provides kernel logging support (previously done by rklogd) 
>>>> $ModLoad imklog # Provides support for local system logging (e.g.
>>>> via logger command) $ModLoad imuxsock
>>>>
>>>> # Log all kernel messages to the console.
>>>> # Logging much else clutters up the screen.
>>>> #kern.* /dev/console
>>>>
>>>> # Log anything (except mail) of level info or higher.
>>>> # Don't log private authentication messages!
>>>> *.info;mail.none;authpriv.none;cron.none /var/log/messages
>>>>
>>>> # The authpriv file has restricted access.
>>>> authpriv.* /var/log/secure
>>>>
>>>> # Log all the mail messages in one place.
>>>> mail.* -/var/log/maillog
>>>>
>>>> # Log cron stuff
>>>> cron.* /var/log/cron
>>>>
>>>> # Everybody gets emergency messages *.emerg *
>>>>
>>>> # Save news errors of level crit and higher in a special file.
>>>> uucp,news.crit /var/log/spooler
>>>>
>>>> # Save boot messages also to boot.log
>>>> local7.* /var/log/boot.log
>>>>
>>>> #Load External Listender
>>>> $modload imudp
>>>>
>>>>
>>>> $UDPServerRun 514
>>>> #SPA Messages
>>>> #*.* /var/log/linksys.log
>>>> $template DynaFile,"/var/log/system-%HOSTNAME%.log"
>>>> *.* -?DynaFile
>>>>
>>>>
>>>> ################
>>>> #### ORACLE ####
>>>> ################
>>>>
>>>> $ModLoad omoracle
>>>> $OmoracleDBUser syslog
>>>> $OmoracleDBPassword syslog
>>>> $OmoracleDB syslog
>>>> $OmoracleBatchSize 1
>>>> $OmoracleBatchItemSize 4096
>>>> $OmoracleStatementTemplate OmoracleStatement
>>>>
>>>> $template OmoracleStatement,"INSERT INTO TEST(hostname,message) 
>>>> VALUES(:hostname,:msg)"
>>>> $template TestStmt,"%hostname%%msg%"
>>>>
>>>> *.*      :omoracle:;TestStmt
>>>> #====================================================
>>>>
>>>> Really appreciate anyone's help!
>>>>
>>>> Best
>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
>> --
>> This message has been scanned for viruses and dangerous content by 
>> MailScanner, and is believed to be clean.
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>
> --
> This message has been scanned for viruses and dangerous content by 
> MailScanner, and is believed to be clean.
>
--
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: rsyslog.debug.rar
Type: application/octet-stream
Size: 242872 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20110511/9b264f29/attachment-0001.obj>

From david at lang.hm  Thu May 12 06:22:34 2011
From: david at lang.hm (david at lang.hm)
Date: Wed, 11 May 2011 21:22:34 -0700 (PDT)
Subject: [rsyslog] help : CPU high load of rsyslog writing to Oracle
In-Reply-To: <001a01cc1014$556f7560$004e6020$@italkbb.com>
References: <002101cc0e60$9aa7f6d0$cff7e470$@italkbb.com>
	<alpine.DEB.2.00.1105091055030.25291@asgard.lang.hm>
	<alpine.DEB.2.00.1105092231490.6302@asgard.lang.hm>
	<001c01cc0ed8$6ea9ac40$4bfd04c0$@italkbb.com>
	<alpine.DEB.2.00.1105092313190.6302@asgard.lang.hm>
	<004801cc0f35$731e9850$595bc8f0$@italkbb.com>
	<alpine.DEB.2.00.1105101820220.25291@asgard.lang.hm>
	<001a01cc1014$556f7560$004e6020$@italkbb.com>
Message-ID: <alpine.DEB.2.00.1105112119200.16882@asgard.lang.hm>

this should point us at the problem (although I think Rainer will need to 
look at this).

one minor correction, if the thread was blocked it would be eating zero 
CPU, not 100%, so it probably went into some loop that is completly in 
userspace after getting that response.

it may (or may not) help to get more of the rsyslog.strace..23298 file, 
how large is it?

David Lang

On Wed, 11 May 2011, bxu wrote:

> Attached is the debug file when use " strace -ff -o rsyslog.strace. -C  -ttt
> -T -s 100  rsyslogd -c5 -nd >rsyslog.debug".
> Thread ID 23298 consumes almost all the CPU time,which is used to write to
> Oracle.
> In the file 'rsyslog.strace..23298',there's no more log after "
> 1305142850.448202 write(1, "action call returned 0\n", 23) = 23
> <0.000027>",seems this thread was blocked.
> I think there is a loop after some wrong.
>
> -----Original Message-----
> From: bxu [mailto:bxu at italkbb.com]
> Sent: Wednesday, May 11, 2011 1:15 AM
> To: 'rsyslog-users'
> Subject: RE: [rsyslog] help : CPU high load of rsyslog writing to Oracle
>
> Hi David,
> 	Yes. The 'H' was turned on for the TOP info.
>
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 49.99    0.008990           5      1858       929 futex
> 33.34    0.005996         857         7           select
> 16.67    0.002997          20       151           read
>  0.00    0.000000           0       542           recvfrom
>  0.00    0.000000           0         7           recvmsg
>  0.00    0.000000           0       468           gettimeofday
>  0.00    0.000000           0      1860           clock_gettime
> ------ ----------- ----------- --------- --------- ----------------
> 100.00    0.017983                  4893       929 total
>
> 	As seen from the above TOP info,there are 929 errors for futex
> call.Sometimes there were errors for recvmsg call where I did more
> straces.Does this relate to the CPU high load?
>
> 	I'll do more tests as your suggestion.
>
> Thanks,
> Henry
>
> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, May 10, 2011 8:29 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to Oracle
>
> thanks for the info.
>
> when you printed the TOP info, was that with 'H' (thread details) turned on?
>
> I'm puzzled by the fact that you are doing a strace of the thread 13470 and
> it's showing actions by many different pids (it looks like everything except
> 13470
>
> can you try doing
>
> strace -ff -o rsyslog.strace. -C  -ttt -T -s 100  rsyslogd -c5 -nd
>> rsyslog.debug
>
> this will create a bunch of strace files. we shouldn't need to look at all
> of them, just the one using all the CPU (and possibly only the end of that
> one)
>
> while this is running, do a ps -eLf
>
> also, run top -H and note which thread is using all the CPU (you may want to
> try toggling H while it's running to make sure you are getting the thread
> info not the consolodated process info)
>
> I'm about to head home and when I get home I'll look at the debug output
> that you send.
>
> one thing I'm not remembering. are the logs getting into the database?
>
> David Lang
>
> On Tue, 10 May 2011, bxu wrote:
>
>> Hi David,
>> 	Here are what I'm doing:
>> 	1. oracle configuration of rsyslogd.conf
>>
>> ################
>> #### ORACLE ####
>> ################
>>
>> $ModLoad omoracle
>> $OmoracleDBUser syslog
>> $OmoracleDBPassword syslog
>> $OmoracleDB syslog
>> $OmoracleBatchSize 1
>> $OmoracleBatchItemSize 4096
>> $OmoracleStatementTemplate OmoracleStatement
>>
>> $template OmoracleStatement,"INSERT INTO TEST(Hostname,Message)
>> VALUES(:hostname,:msg)"
>> $template TestStmt,"%hostname%%msg%"
>> $template TestStmtFile,"INSERT INTO TEST(Hostname,Message)
>> VALUES('%hostname%','%msg%')\n"
>>
>> #$WorkDirectory /rsyslog/work # default location for work (spool)
>> files
>>
>> #$ActionQueueType LinkedList # use asynchronous processing
>> #$ActionQueueFileName dbq    # set file name, also enables disk mode
>> #$ActionResumeRetryCount -1  # infinite retries on insert failure
>>
>> :msg,contains,"SIP"             :omoracle:;TestStmt
>>
>> & ~
>>
>> *.*                             /var/log/syslog-test;TestStmtFile
>>
>> 	2. running command :  rsyslogd -c5 -nd > tt &
>>
>> 	3, TOP information
>>
>>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>
>> 13470 root      25   0  244m  11m 7636 R 99.9  0.1   1:06.39 rs:main Q:Reg
>>
>> 11410 root      15   0 12768 1144  832 R  0.3  0.0   0:13.52 top
>>
>>    1 root      15   0 10372  692  580 S  0.0  0.0   0:00.51 init
>>
>> 	4. when CPU usage is keeping at 100%,the summary of strace by thread
>> 13470:
>>
>> % time     seconds  usecs/call     calls    errors syscall
>> ------ ----------- ----------- --------- --------- ----------------
>> 51.26    0.019996        9998         2           epoll_wait
>> 46.12    0.017990        1799        10           read
>>  2.56    0.000999         999         1           select
>>  0.06    0.000022           2        12         2 recvfrom
>>  0.00    0.000000           0        64           write
>>  0.00    0.000000           0         1           recvmsg
>>  0.00    0.000000           0        17           gettimeofday
>>  0.00    0.000000           0        62           clock_gettime
>> ------ ----------- ----------- --------- --------- ----------------
>> 100.00    0.039007                   169         2 total
>>
>> 	5. when CPU usage is keeping at 100%,some of the call sequence of
>> strace by thread 13470:
>>
>> [pid 13469]      0.000000 select(1, NULL, NULL, NULL, {86097, 801000}
>> <unfinished ...>
>> [pid 13473]      0.000089 clock_gettime(CLOCK_REALTIME,  <unfinished ...>
>> [pid 13472]      0.000054 select(9, [8], NULL, NULL, NULL <unfinished ...>
>> [pid 13471]      0.000066 read(3,  <unfinished ...>
>> [pid 13473]      0.000048 <... clock_gettime resumed> {1305046294,
>> 274947000}) = 0 <0.000138>
>> [pid 13473]      0.000067 write(1, "6294.274947000:43899940: imudp: "...,
>> 66) = 66 <0.000042>
>> [pid 13473]      0.000137 epoll_wait(12,  <unfinished ...>
>> [pid 13471]      1.421497 <... read resumed> "<6>iptables: log and drop
>> IN=eth"..., 4095) = 200 <1.421706>
>> [pid 13471]      0.000095 gettimeofday({1305046295, 696914}, NULL) = 0
>> <0.000022>
>> [pid 13471]      0.000130 clock_gettime(CLOCK_REALTIME, {1305046295,
>> 697046000}) = 0 <0.000022>
>> [pid 13471]      0.000104 write(1, "6295.697046000:42497940: main Q:"...,
>> 81) = 81 <0.000037>
>> [pid 13471]      0.000127 clock_gettime(CLOCK_REALTIME, {1305046295,
>> 697276000}) = 0 <0.000020>
>> [pid 13471]      0.000102 write(1, "6295.697276000:42497940: main Q:"...,
>> 65) = 65 <0.000031>
>> [pid 13471]      0.000120 read(3, "<6>iptables: log and drop IN=eth"...,
>> 4095) = 200 <4.250732>
>> [pid 13471]      4.250863 gettimeofday({1305046299, 948362}, NULL) = 0
>> <0.000021>
>> [pid 13471]      0.000129 clock_gettime(CLOCK_REALTIME, {1305046299,
>> 948491000}) = 0 <0.000021>
>> [pid 13471]      0.000106 write(1, "6299.948491000:42497940: main Q:"...,
>> 81) = 81 <0.000051>
>> [pid 13471]      0.000140 clock_gettime(CLOCK_REALTIME, {1305046299,
>> 948735000}) = 0 <0.000021>
>> [pid 13471]      0.000101 write(1, "6299.948735000:42497940: main Q:"...,
>> 65) = 65 <0.000031>
>> [pid 13471]      0.000123 read(3, "<6>iptables: log and drop IN=eth"...,
>> 4095) = 200 <2.178833>
>> [pid 13471]      2.178931 gettimeofday({1305046302, 127891}, NULL) = 0
>> <0.000022>
>> [pid 13471]      0.000112 clock_gettime(CLOCK_REALTIME, {1305046302,
>> 128003000}) = 0 <0.000021>
>> [pid 13471]      0.000101 write(1, "6302.128003000:42497940: main Q:"...,
>> 81) = 81 <0.000036>
>> [pid 13471]      0.000121 clock_gettime(CLOCK_REALTIME, {1305046302,
>> 128225000}) = 0 <0.000021>
>> [pid 13471]      0.000102 write(1, "6302.128225000:42497940: main Q:"...,
>> 65) = 65 <0.000031>
>> [pid 13471]      0.000115 read(3, "<6>iptables: log and drop IN=eth"...,
>> 4095) = 200 <2.431092>
>> [pid 13471]      2.431228 gettimeofday({1305046304, 559672}, NULL) = 0
>> <0.000022>
>> [pid 13471]      0.000132 clock_gettime(CLOCK_REALTIME, {1305046304,
>> 559803000}) = 0 <0.000021>
>> [pid 13471]      0.000110 write(1, "6304.559803000:42497940: main Q:"...,
>> 81) = 81 <0.000052>
>> [pid 13471]      0.000140 clock_gettime(CLOCK_REALTIME, {1305046304,
>> 560052000}) = 0 <0.000021>
>> [pid 13471]      0.000102 write(1, "6304.560052000:42497940: main Q:"...,
>> 65) = 65 <0.000031>
>> [pid 13471]      0.000119 read(3, "<6>iptables: log and drop IN=eth"...,
>> 4095) = 197 <0.678610>
>> [pid 13471]      0.678704 gettimeofday({1305046305, 238976}, NULL) = 0
>> <0.000021>
>> [pid 13471]      0.000110 clock_gettime(CLOCK_REALTIME, {1305046305,
>> 239087000}) = 0 <0.000024>
>> [pid 13471]      0.000105 write(1, "6305.239087000:42497940: main Q:"...,
>> 81) = 81 <0.000035>
>> [pid 13471]      0.000120 clock_gettime(CLOCK_REALTIME, {1305046305,
>> 239312000}) = 0 <0.000020>
>> [pid 13471]      0.000102 write(1, "6305.239312000:42497940: main Q:"...,
>> 65) = 65 <0.000030>
>> [pid 13471]      0.000114 read(3, "<6>iptables: log and drop IN=eth"...,
>> 4095) = 200 <3.058797>
>> [pid 13471]      3.058929 gettimeofday({1305046308, 298458}, NULL) = 0
>> <0.000022>
>> [pid 13471]      0.000131 clock_gettime(CLOCK_REALTIME, {1305046308,
>> 298589000}) = 0 <0.000021>
>> [pid 13471]      0.000106 write(1, "6308.298589000:42497940: main Q:"...,
>> 81) = 81 <0.000058>
>> [pid 13471]      0.000148 clock_gettime(CLOCK_REALTIME, {1305046308,
>> 298841000}) = 0 <0.000021>
>> [pid 13471]      0.000101 write(1, "6308.298841000:42497940: main Q:"...,
>> 65) = 65 <0.000093>
>> [pid 13471]      0.000181 read(3, "<6>iptables: log and drop IN=eth"...,
>> 4095) = 197 <0.213955>
>> [pid 13471]      0.214082 gettimeofday({1305046308, 513206}, NULL) = 0
>> <0.000021>
>> [pid 13471]      0.000128 clock_gettime(CLOCK_REALTIME, {1305046308,
>> 513335000}) = 0 <0.000021>
>> [pid 13471]      0.000106 write(1, "6308.513335000:42497940: main Q:"...,
>> 81) = 81 <0.000042>
>> [pid 13471]      0.000130 clock_gettime(CLOCK_REALTIME, {1305046308,
>> 513570000}) = 0 <0.000021>
>> [pid 13471]      0.000102 write(1, "6308.513570000:42497940: main Q:"...,
>> 65) = 65 <0.000030>
>> [pid 13471]      0.000122 read(3, "<6>iptables: log and drop IN=eth"...,
>> 4095) = 196 <0.835091>
>> [pid 13471]      0.835187 gettimeofday({1305046309, 348980}, NULL) = 0
>> <0.000021>
>> [pid 13471]      0.000110 clock_gettime(CLOCK_REALTIME, {1305046309,
>> 349091000}) = 0 <0.000021>
>> [pid 13471]      0.000102 write(1, "6309.349091000:42497940: main Q:"...,
>> 81) = 81 <0.000035>
>> [pid 13471]      0.000121 clock_gettime(CLOCK_REALTIME, {1305046309,
>> 349313000}) = 0 <0.000020>
>> [pid 13471]      0.000102 write(1, "6309.349313000:42497940: main Q:"...,
>> 65) = 65 <0.000030>
>> [pid 13471]      0.000114 read(3,  <unfinished ...>
>> [pid 13473]      2.018865 <... epoll_wait resumed> {{EPOLLIN, {u32=2,
>> u64=2}}}, 10, 4294967295) = 1 <17.093033>
>> [pid 13473]      0.000086 clock_gettime(CLOCK_REALTIME, {1305046311,
>> 368481000}) = 0 <0.000020>
>> [pid 13473]      0.000098 write(1, "6311.368481000:43899940: imudp: "...,
>> 65) = 65 <0.000036>
>>
>>
>> 	6. Attached is the log file while rsyslogd is running in debug mode
>> 	    In the log,after "omoracle insertion to DB succeeded", there's
> no
>> more oracle log information related.
>>
>>
>> 	Thank you very much.
>>
>> Best,
>> Henry
>>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Tuesday, May 10, 2011 1:19 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to
>> Oracle
>>
>> rsyslog uses multiple threads, if you can tell which thread is using
>> all the CPU and then do a strace of it and dump the sequence of calls
>> that it makes to process one of the messages it may help
>>
>> I don't have any idea what's going on here, so I'm just working
>> through the steps that I would be doing if I was running into the
>> problem
>>
>> try ps -eLf to list the different threads
>>
>> in top, hit 'H' to run on per-thread reporting (at which point you may
>> get some additional info on the right about what the thread is doing)
>>
>> from the stats you are showing here, it looks like the thread you got
>> with your strace is the parent that's receiving messages (lots of
>> gettimeofday and receive/read type calls) the correct one should have
>> write calls in it (among others)
>>
>> the versions you are running are new enough that we shouldn't have any
>> old, solved problems hitting us.
>>
>>  On Tue, 10 May 2011, bxu
>> wrote:
>>
>>> Hi David,
>>> 	The rsyslogd version I tried are 5.8.0(stable) and 6.1.7(devel).
>>> 	I straced rsyslogd when CPU usage is keeping at 100%.
>>>
>>> % time     seconds  usecs/call     calls    errors syscall
>>> ------ ----------- ----------- --------- --------- ----------------
>>> 43.47    0.066967        2309        29           read
>>> 38.28    0.058979         855        69           epoll_wait
>>> 18.17    0.027994        9331         3           select
>>>  0.05    0.000081           1       114           gettimeofday
>>>  0.03    0.000041           0       178        69 recvfrom
>>>  0.00    0.000000           0         3           recvmsg
>>> ------ ----------- ----------- --------- --------- ----------------
>>> 100.00    0.154062                   396        69 total
>>>
>>>
>>> Best,
>>> Henry
>>>
>>> -----Original Message-----
>>> From: rsyslog-bounces at lists.adiscon.com
>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>> Sent: Tuesday, May 10, 2011 12:35 AM
>>> To: rsyslog-users
>>> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to
>>> Oracle
>>>
>>> I accidently deleted your reply, so I am replying to my message
>>>
>>> 4-5 messages/secshould not be any problem at all.
>>>
>>> since writing a copy of the sql commands to a file doesn't show
>>> anything strange, the next step will be to try and run it in debug
>>> mode to see if it logs anything strange as you start having problems.
>>>
>>> the next step after that will probably be to do a strace of the
>>> thread that is using all the CPU
>>>
>>> I don't see where you mention what version of rsyslog you are using.
>>>
>>> David Lang
>>>
>>> On Mon, 9 May 2011, david at lang.hm wrote:
>>>
>>>> Date: Mon, 9 May 2011 10:56:30 -0700 (PDT)
>>>> From: david at lang.hm
>>>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>>>> To: rsyslog-users <rsyslog at lists.adiscon.com>
>>>> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to
>>>> Oracle
>>>>
>>>> given that it only happens after a few messages, I would suspect
>>>> that there is something odd with the messages.
>>>>
>>>> could you try writing the messages to a file using the Oracle
>>>> template and then look and see if there is anything odd about the
>>>> messages when it goes nuts?
>>>>
>>>> the other option is to get a debug dump, but unless it happens right
>>>> away, that is likely to be a large dump.
>>>>
>>>> David Lang
>>>>
>>>> On Mon, 9 May 2011, bxu wrote:
>>>>
>>>>> Date: Mon, 9 May 2011 10:48:50 -0500
>>>>> From: bxu <bxu at italkbb.com>
>>>>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>>>>> To: rsyslog at lists.adiscon.com
>>>>> Subject: [rsyslog] help : CPU high load of rsyslog writing to
>>>>> Oracle
>>>>>
>>>>> Hi All,
>>>>>
>>>>>
>>>>> I have configured rsyslog writing to Oracle.The writing was
>>>>> successful.But after several messages have been made into
>>>>> Oracle,the CPU use percentage of rsyslogd goes up to 99%-100%.It is
>>>>> abnormally too
>>> much high.
>>>>> There's no problem with the cpu usage when writing to files.
>>>>>
>>>>> Here is my rsyslogd.conf:
>>>>> #====================================================
>>>>> #*.* >192.168.45.213,rsyslog
>>>>>
>>>>> # Use traditional timestamp format
>>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>>>
>>>>> # Provides kernel logging support (previously done by rklogd)
>>>>> $ModLoad imklog # Provides support for local system logging (e.g.
>>>>> via logger command) $ModLoad imuxsock
>>>>>
>>>>> # Log all kernel messages to the console.
>>>>> # Logging much else clutters up the screen.
>>>>> #kern.* /dev/console
>>>>>
>>>>> # Log anything (except mail) of level info or higher.
>>>>> # Don't log private authentication messages!
>>>>> *.info;mail.none;authpriv.none;cron.none /var/log/messages
>>>>>
>>>>> # The authpriv file has restricted access.
>>>>> authpriv.* /var/log/secure
>>>>>
>>>>> # Log all the mail messages in one place.
>>>>> mail.* -/var/log/maillog
>>>>>
>>>>> # Log cron stuff
>>>>> cron.* /var/log/cron
>>>>>
>>>>> # Everybody gets emergency messages *.emerg *
>>>>>
>>>>> # Save news errors of level crit and higher in a special file.
>>>>> uucp,news.crit /var/log/spooler
>>>>>
>>>>> # Save boot messages also to boot.log
>>>>> local7.* /var/log/boot.log
>>>>>
>>>>> #Load External Listender
>>>>> $modload imudp
>>>>>
>>>>>
>>>>> $UDPServerRun 514
>>>>> #SPA Messages
>>>>> #*.* /var/log/linksys.log
>>>>> $template DynaFile,"/var/log/system-%HOSTNAME%.log"
>>>>> *.* -?DynaFile
>>>>>
>>>>>
>>>>> ################
>>>>> #### ORACLE ####
>>>>> ################
>>>>>
>>>>> $ModLoad omoracle
>>>>> $OmoracleDBUser syslog
>>>>> $OmoracleDBPassword syslog
>>>>> $OmoracleDB syslog
>>>>> $OmoracleBatchSize 1
>>>>> $OmoracleBatchItemSize 4096
>>>>> $OmoracleStatementTemplate OmoracleStatement
>>>>>
>>>>> $template OmoracleStatement,"INSERT INTO TEST(hostname,message)
>>>>> VALUES(:hostname,:msg)"
>>>>> $template TestStmt,"%hostname%%msg%"
>>>>>
>>>>> *.*      :omoracle:;TestStmt
>>>>> #====================================================
>>>>>
>>>>> Really appreciate anyone's help!
>>>>>
>>>>> Best
>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com
>>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>>> --
>>> This message has been scanned for viruses and dangerous content by
>>> MailScanner, and is believed to be clean.
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
>> --
>> This message has been scanned for viruses and dangerous content by
>> MailScanner, and is believed to be clean.
>>
> --
> This message has been scanned for viruses and dangerous content by
> MailScanner, and is believed to be clean.
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rsyslog.debug.rar
Type: application/octet-stream
Size: 242872 bytes
Desc: 
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20110511/67f774b0/attachment-0001.obj>
-------------- next part --------------
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

From bxu at italkbb.com  Thu May 12 07:02:50 2011
From: bxu at italkbb.com (Henry Xu)
Date: Thu, 12 May 2011 00:02:50 -0500
Subject: [rsyslog] help : CPU high load of rsyslog writing to Oracle
In-Reply-To: <alpine.DEB.2.00.1105112119200.16882@asgard.lang.hm>
References: <002101cc0e60$9aa7f6d0$cff7e470$@italkbb.com>	<alpine.DEB.2.00.1105091055030.25291@asgard.lang.hm>	<alpine.DEB.2.00.1105092231490.6302@asgard.lang.hm>	<001c01cc0ed8$6ea9ac40$4bfd04c0$@italkbb.com>	<alpine.DEB.2.00.1105092313190.6302@asgard.lang.hm>	<004801cc0f35$731e9850$595bc8f0$@italkbb.com>	<alpine.DEB.2.00.1105101820220.25291@asgard.lang.hm>	<001a01cc1014$556f7560$004e6020$@italkbb.com>
	<alpine.DEB.2.00.1105112119200.16882@asgard.lang.hm>
Message-ID: <003601cc1061$dac8b300$905a1900$@italkbb.com>

Attached is another trace of rsyslogd with thread ID 2665 which hosted all
the CPU.
That's all the log info.

-----Original Message-----
From: rsyslog-bounces at lists.adiscon.com
[mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
Sent: Wednesday, May 11, 2011 11:23 PM
To: rsyslog-users
Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to Oracle

this should point us at the problem (although I think Rainer will need to
look at this).

one minor correction, if the thread was blocked it would be eating zero CPU,
not 100%, so it probably went into some loop that is completly in userspace
after getting that response.

it may (or may not) help to get more of the rsyslog.strace..23298 file, how
large is it?

David Lang

On Wed, 11 May 2011, bxu wrote:

> Attached is the debug file when use " strace -ff -o rsyslog.strace. -C  
> -ttt -T -s 100  rsyslogd -c5 -nd >rsyslog.debug".
> Thread ID 23298 consumes almost all the CPU time,which is used to 
> write to Oracle.
> In the file 'rsyslog.strace..23298',there's no more log after "
> 1305142850.448202 write(1, "action call returned 0\n", 23) = 23 
> <0.000027>",seems this thread was blocked.
> I think there is a loop after some wrong.
>
> -----Original Message-----
> From: bxu [mailto:bxu at italkbb.com]
> Sent: Wednesday, May 11, 2011 1:15 AM
> To: 'rsyslog-users'
> Subject: RE: [rsyslog] help : CPU high load of rsyslog writing to 
> Oracle
>
> Hi David,
> 	Yes. The 'H' was turned on for the TOP info.
>
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 49.99    0.008990           5      1858       929 futex
> 33.34    0.005996         857         7           select
> 16.67    0.002997          20       151           read
>  0.00    0.000000           0       542           recvfrom
>  0.00    0.000000           0         7           recvmsg
>  0.00    0.000000           0       468           gettimeofday
>  0.00    0.000000           0      1860           clock_gettime
> ------ ----------- ----------- --------- --------- ----------------
> 100.00    0.017983                  4893       929 total
>
> 	As seen from the above TOP info,there are 929 errors for futex 
> call.Sometimes there were errors for recvmsg call where I did more 
> straces.Does this relate to the CPU high load?
>
> 	I'll do more tests as your suggestion.
>
> Thanks,
> Henry
>
> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, May 10, 2011 8:29 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to 
> Oracle
>
> thanks for the info.
>
> when you printed the TOP info, was that with 'H' (thread details) turned
on?
>
> I'm puzzled by the fact that you are doing a strace of the thread 
> 13470 and it's showing actions by many different pids (it looks like 
> everything except
> 13470
>
> can you try doing
>
> strace -ff -o rsyslog.strace. -C  -ttt -T -s 100  rsyslogd -c5 -nd
>> rsyslog.debug
>
> this will create a bunch of strace files. we shouldn't need to look at 
> all of them, just the one using all the CPU (and possibly only the end 
> of that
> one)
>
> while this is running, do a ps -eLf
>
> also, run top -H and note which thread is using all the CPU (you may 
> want to try toggling H while it's running to make sure you are getting 
> the thread info not the consolodated process info)
>
> I'm about to head home and when I get home I'll look at the debug 
> output that you send.
>
> one thing I'm not remembering. are the logs getting into the database?
>
> David Lang
>
> On Tue, 10 May 2011, bxu wrote:
>
>> Hi David,
>> 	Here are what I'm doing:
>> 	1. oracle configuration of rsyslogd.conf
>>
>> ################
>> #### ORACLE ####
>> ################
>>
>> $ModLoad omoracle
>> $OmoracleDBUser syslog
>> $OmoracleDBPassword syslog
>> $OmoracleDB syslog
>> $OmoracleBatchSize 1
>> $OmoracleBatchItemSize 4096
>> $OmoracleStatementTemplate OmoracleStatement
>>
>> $template OmoracleStatement,"INSERT INTO TEST(Hostname,Message) 
>> VALUES(:hostname,:msg)"
>> $template TestStmt,"%hostname%%msg%"
>> $template TestStmtFile,"INSERT INTO TEST(Hostname,Message) 
>> VALUES('%hostname%','%msg%')\n"
>>
>> #$WorkDirectory /rsyslog/work # default location for work (spool) 
>> files
>>
>> #$ActionQueueType LinkedList # use asynchronous processing
>> #$ActionQueueFileName dbq    # set file name, also enables disk mode
>> #$ActionResumeRetryCount -1  # infinite retries on insert failure
>>
>> :msg,contains,"SIP"             :omoracle:;TestStmt
>>
>> & ~
>>
>> *.*                             /var/log/syslog-test;TestStmtFile
>>
>> 	2. running command :  rsyslogd -c5 -nd > tt &
>>
>> 	3, TOP information
>>
>>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>
>> 13470 root      25   0  244m  11m 7636 R 99.9  0.1   1:06.39 rs:main
Q:Reg
>>
>> 11410 root      15   0 12768 1144  832 R  0.3  0.0   0:13.52 top
>>
>>    1 root      15   0 10372  692  580 S  0.0  0.0   0:00.51 init
>>
>> 	4. when CPU usage is keeping at 100%,the summary of strace by thread
>> 13470:
>>
>> % time     seconds  usecs/call     calls    errors syscall
>> ------ ----------- ----------- --------- --------- ----------------
>> 51.26    0.019996        9998         2           epoll_wait
>> 46.12    0.017990        1799        10           read
>>  2.56    0.000999         999         1           select
>>  0.06    0.000022           2        12         2 recvfrom
>>  0.00    0.000000           0        64           write
>>  0.00    0.000000           0         1           recvmsg
>>  0.00    0.000000           0        17           gettimeofday
>>  0.00    0.000000           0        62           clock_gettime
>> ------ ----------- ----------- --------- --------- ----------------
>> 100.00    0.039007                   169         2 total
>>
>> 	5. when CPU usage is keeping at 100%,some of the call sequence of 
>> strace by thread 13470:
>>
>> [pid 13469]      0.000000 select(1, NULL, NULL, NULL, {86097, 801000}
>> <unfinished ...>
>> [pid 13473]      0.000089 clock_gettime(CLOCK_REALTIME,  <unfinished ...>
>> [pid 13472]      0.000054 select(9, [8], NULL, NULL, NULL <unfinished
...>
>> [pid 13471]      0.000066 read(3,  <unfinished ...>
>> [pid 13473]      0.000048 <... clock_gettime resumed> {1305046294,
>> 274947000}) = 0 <0.000138>
>> [pid 13473]      0.000067 write(1, "6294.274947000:43899940: imudp: "...,
>> 66) = 66 <0.000042>
>> [pid 13473]      0.000137 epoll_wait(12,  <unfinished ...>
>> [pid 13471]      1.421497 <... read resumed> "<6>iptables: log and drop
>> IN=eth"..., 4095) = 200 <1.421706>
>> [pid 13471]      0.000095 gettimeofday({1305046295, 696914}, NULL) = 0
>> <0.000022>
>> [pid 13471]      0.000130 clock_gettime(CLOCK_REALTIME, {1305046295,
>> 697046000}) = 0 <0.000022>
>> [pid 13471]      0.000104 write(1, "6295.697046000:42497940: main Q:"...,
>> 81) = 81 <0.000037>
>> [pid 13471]      0.000127 clock_gettime(CLOCK_REALTIME, {1305046295,
>> 697276000}) = 0 <0.000020>
>> [pid 13471]      0.000102 write(1, "6295.697276000:42497940: main Q:"...,
>> 65) = 65 <0.000031>
>> [pid 13471]      0.000120 read(3, "<6>iptables: log and drop IN=eth"...,
>> 4095) = 200 <4.250732>
>> [pid 13471]      4.250863 gettimeofday({1305046299, 948362}, NULL) = 0
>> <0.000021>
>> [pid 13471]      0.000129 clock_gettime(CLOCK_REALTIME, {1305046299,
>> 948491000}) = 0 <0.000021>
>> [pid 13471]      0.000106 write(1, "6299.948491000:42497940: main Q:"...,
>> 81) = 81 <0.000051>
>> [pid 13471]      0.000140 clock_gettime(CLOCK_REALTIME, {1305046299,
>> 948735000}) = 0 <0.000021>
>> [pid 13471]      0.000101 write(1, "6299.948735000:42497940: main Q:"...,
>> 65) = 65 <0.000031>
>> [pid 13471]      0.000123 read(3, "<6>iptables: log and drop IN=eth"...,
>> 4095) = 200 <2.178833>
>> [pid 13471]      2.178931 gettimeofday({1305046302, 127891}, NULL) = 0
>> <0.000022>
>> [pid 13471]      0.000112 clock_gettime(CLOCK_REALTIME, {1305046302,
>> 128003000}) = 0 <0.000021>
>> [pid 13471]      0.000101 write(1, "6302.128003000:42497940: main Q:"...,
>> 81) = 81 <0.000036>
>> [pid 13471]      0.000121 clock_gettime(CLOCK_REALTIME, {1305046302,
>> 128225000}) = 0 <0.000021>
>> [pid 13471]      0.000102 write(1, "6302.128225000:42497940: main Q:"...,
>> 65) = 65 <0.000031>
>> [pid 13471]      0.000115 read(3, "<6>iptables: log and drop IN=eth"...,
>> 4095) = 200 <2.431092>
>> [pid 13471]      2.431228 gettimeofday({1305046304, 559672}, NULL) = 0
>> <0.000022>
>> [pid 13471]      0.000132 clock_gettime(CLOCK_REALTIME, {1305046304,
>> 559803000}) = 0 <0.000021>
>> [pid 13471]      0.000110 write(1, "6304.559803000:42497940: main Q:"...,
>> 81) = 81 <0.000052>
>> [pid 13471]      0.000140 clock_gettime(CLOCK_REALTIME, {1305046304,
>> 560052000}) = 0 <0.000021>
>> [pid 13471]      0.000102 write(1, "6304.560052000:42497940: main Q:"...,
>> 65) = 65 <0.000031>
>> [pid 13471]      0.000119 read(3, "<6>iptables: log and drop IN=eth"...,
>> 4095) = 197 <0.678610>
>> [pid 13471]      0.678704 gettimeofday({1305046305, 238976}, NULL) = 0
>> <0.000021>
>> [pid 13471]      0.000110 clock_gettime(CLOCK_REALTIME, {1305046305,
>> 239087000}) = 0 <0.000024>
>> [pid 13471]      0.000105 write(1, "6305.239087000:42497940: main Q:"...,
>> 81) = 81 <0.000035>
>> [pid 13471]      0.000120 clock_gettime(CLOCK_REALTIME, {1305046305,
>> 239312000}) = 0 <0.000020>
>> [pid 13471]      0.000102 write(1, "6305.239312000:42497940: main Q:"...,
>> 65) = 65 <0.000030>
>> [pid 13471]      0.000114 read(3, "<6>iptables: log and drop IN=eth"...,
>> 4095) = 200 <3.058797>
>> [pid 13471]      3.058929 gettimeofday({1305046308, 298458}, NULL) = 0
>> <0.000022>
>> [pid 13471]      0.000131 clock_gettime(CLOCK_REALTIME, {1305046308,
>> 298589000}) = 0 <0.000021>
>> [pid 13471]      0.000106 write(1, "6308.298589000:42497940: main Q:"...,
>> 81) = 81 <0.000058>
>> [pid 13471]      0.000148 clock_gettime(CLOCK_REALTIME, {1305046308,
>> 298841000}) = 0 <0.000021>
>> [pid 13471]      0.000101 write(1, "6308.298841000:42497940: main Q:"...,
>> 65) = 65 <0.000093>
>> [pid 13471]      0.000181 read(3, "<6>iptables: log and drop IN=eth"...,
>> 4095) = 197 <0.213955>
>> [pid 13471]      0.214082 gettimeofday({1305046308, 513206}, NULL) = 0
>> <0.000021>
>> [pid 13471]      0.000128 clock_gettime(CLOCK_REALTIME, {1305046308,
>> 513335000}) = 0 <0.000021>
>> [pid 13471]      0.000106 write(1, "6308.513335000:42497940: main Q:"...,
>> 81) = 81 <0.000042>
>> [pid 13471]      0.000130 clock_gettime(CLOCK_REALTIME, {1305046308,
>> 513570000}) = 0 <0.000021>
>> [pid 13471]      0.000102 write(1, "6308.513570000:42497940: main Q:"...,
>> 65) = 65 <0.000030>
>> [pid 13471]      0.000122 read(3, "<6>iptables: log and drop IN=eth"...,
>> 4095) = 196 <0.835091>
>> [pid 13471]      0.835187 gettimeofday({1305046309, 348980}, NULL) = 0
>> <0.000021>
>> [pid 13471]      0.000110 clock_gettime(CLOCK_REALTIME, {1305046309,
>> 349091000}) = 0 <0.000021>
>> [pid 13471]      0.000102 write(1, "6309.349091000:42497940: main Q:"...,
>> 81) = 81 <0.000035>
>> [pid 13471]      0.000121 clock_gettime(CLOCK_REALTIME, {1305046309,
>> 349313000}) = 0 <0.000020>
>> [pid 13471]      0.000102 write(1, "6309.349313000:42497940: main Q:"...,
>> 65) = 65 <0.000030>
>> [pid 13471]      0.000114 read(3,  <unfinished ...>
>> [pid 13473]      2.018865 <... epoll_wait resumed> {{EPOLLIN, {u32=2,
>> u64=2}}}, 10, 4294967295) = 1 <17.093033>
>> [pid 13473]      0.000086 clock_gettime(CLOCK_REALTIME, {1305046311,
>> 368481000}) = 0 <0.000020>
>> [pid 13473]      0.000098 write(1, "6311.368481000:43899940: imudp: "...,
>> 65) = 65 <0.000036>
>>
>>
>> 	6. Attached is the log file while rsyslogd is running in debug mode
>> 	    In the log,after "omoracle insertion to DB succeeded", there's
> no
>> more oracle log information related.
>>
>>
>> 	Thank you very much.
>>
>> Best,
>> Henry
>>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com 
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Tuesday, May 10, 2011 1:19 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to 
>> Oracle
>>
>> rsyslog uses multiple threads, if you can tell which thread is using 
>> all the CPU and then do a strace of it and dump the sequence of calls 
>> that it makes to process one of the messages it may help
>>
>> I don't have any idea what's going on here, so I'm just working 
>> through the steps that I would be doing if I was running into the 
>> problem
>>
>> try ps -eLf to list the different threads
>>
>> in top, hit 'H' to run on per-thread reporting (at which point you 
>> may get some additional info on the right about what the thread is 
>> doing)
>>
>> from the stats you are showing here, it looks like the thread you got 
>> with your strace is the parent that's receiving messages (lots of 
>> gettimeofday and receive/read type calls) the correct one should have 
>> write calls in it (among others)
>>
>> the versions you are running are new enough that we shouldn't have 
>> any old, solved problems hitting us.
>>
>>  On Tue, 10 May 2011, bxu
>> wrote:
>>
>>> Hi David,
>>> 	The rsyslogd version I tried are 5.8.0(stable) and 6.1.7(devel).
>>> 	I straced rsyslogd when CPU usage is keeping at 100%.
>>>
>>> % time     seconds  usecs/call     calls    errors syscall
>>> ------ ----------- ----------- --------- --------- ----------------
>>> 43.47    0.066967        2309        29           read
>>> 38.28    0.058979         855        69           epoll_wait
>>> 18.17    0.027994        9331         3           select
>>>  0.05    0.000081           1       114           gettimeofday
>>>  0.03    0.000041           0       178        69 recvfrom
>>>  0.00    0.000000           0         3           recvmsg
>>> ------ ----------- ----------- --------- --------- ----------------
>>> 100.00    0.154062                   396        69 total
>>>
>>>
>>> Best,
>>> Henry
>>>
>>> -----Original Message-----
>>> From: rsyslog-bounces at lists.adiscon.com 
>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of 
>>> david at lang.hm
>>> Sent: Tuesday, May 10, 2011 12:35 AM
>>> To: rsyslog-users
>>> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to 
>>> Oracle
>>>
>>> I accidently deleted your reply, so I am replying to my message
>>>
>>> 4-5 messages/secshould not be any problem at all.
>>>
>>> since writing a copy of the sql commands to a file doesn't show 
>>> anything strange, the next step will be to try and run it in debug 
>>> mode to see if it logs anything strange as you start having problems.
>>>
>>> the next step after that will probably be to do a strace of the 
>>> thread that is using all the CPU
>>>
>>> I don't see where you mention what version of rsyslog you are using.
>>>
>>> David Lang
>>>
>>> On Mon, 9 May 2011, david at lang.hm wrote:
>>>
>>>> Date: Mon, 9 May 2011 10:56:30 -0700 (PDT)
>>>> From: david at lang.hm
>>>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>>>> To: rsyslog-users <rsyslog at lists.adiscon.com>
>>>> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to 
>>>> Oracle
>>>>
>>>> given that it only happens after a few messages, I would suspect 
>>>> that there is something odd with the messages.
>>>>
>>>> could you try writing the messages to a file using the Oracle 
>>>> template and then look and see if there is anything odd about the 
>>>> messages when it goes nuts?
>>>>
>>>> the other option is to get a debug dump, but unless it happens 
>>>> right away, that is likely to be a large dump.
>>>>
>>>> David Lang
>>>>
>>>> On Mon, 9 May 2011, bxu wrote:
>>>>
>>>>> Date: Mon, 9 May 2011 10:48:50 -0500
>>>>> From: bxu <bxu at italkbb.com>
>>>>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>>>>> To: rsyslog at lists.adiscon.com
>>>>> Subject: [rsyslog] help : CPU high load of rsyslog writing to 
>>>>> Oracle
>>>>>
>>>>> Hi All,
>>>>>
>>>>>
>>>>> I have configured rsyslog writing to Oracle.The writing was 
>>>>> successful.But after several messages have been made into 
>>>>> Oracle,the CPU use percentage of rsyslogd goes up to 99%-100%.It 
>>>>> is abnormally too
>>> much high.
>>>>> There's no problem with the cpu usage when writing to files.
>>>>>
>>>>> Here is my rsyslogd.conf:
>>>>> #====================================================
>>>>> #*.* >192.168.45.213,rsyslog
>>>>>
>>>>> # Use traditional timestamp format $ActionFileDefaultTemplate 
>>>>> RSYSLOG_TraditionalFileFormat
>>>>>
>>>>> # Provides kernel logging support (previously done by rklogd) 
>>>>> $ModLoad imklog # Provides support for local system logging (e.g.
>>>>> via logger command) $ModLoad imuxsock
>>>>>
>>>>> # Log all kernel messages to the console.
>>>>> # Logging much else clutters up the screen.
>>>>> #kern.* /dev/console
>>>>>
>>>>> # Log anything (except mail) of level info or higher.
>>>>> # Don't log private authentication messages!
>>>>> *.info;mail.none;authpriv.none;cron.none /var/log/messages
>>>>>
>>>>> # The authpriv file has restricted access.
>>>>> authpriv.* /var/log/secure
>>>>>
>>>>> # Log all the mail messages in one place.
>>>>> mail.* -/var/log/maillog
>>>>>
>>>>> # Log cron stuff
>>>>> cron.* /var/log/cron
>>>>>
>>>>> # Everybody gets emergency messages *.emerg *
>>>>>
>>>>> # Save news errors of level crit and higher in a special file.
>>>>> uucp,news.crit /var/log/spooler
>>>>>
>>>>> # Save boot messages also to boot.log
>>>>> local7.* /var/log/boot.log
>>>>>
>>>>> #Load External Listender
>>>>> $modload imudp
>>>>>
>>>>>
>>>>> $UDPServerRun 514
>>>>> #SPA Messages
>>>>> #*.* /var/log/linksys.log
>>>>> $template DynaFile,"/var/log/system-%HOSTNAME%.log"
>>>>> *.* -?DynaFile
>>>>>
>>>>>
>>>>> ################
>>>>> #### ORACLE ####
>>>>> ################
>>>>>
>>>>> $ModLoad omoracle
>>>>> $OmoracleDBUser syslog
>>>>> $OmoracleDBPassword syslog
>>>>> $OmoracleDB syslog
>>>>> $OmoracleBatchSize 1
>>>>> $OmoracleBatchItemSize 4096
>>>>> $OmoracleStatementTemplate OmoracleStatement
>>>>>
>>>>> $template OmoracleStatement,"INSERT INTO TEST(hostname,message) 
>>>>> VALUES(:hostname,:msg)"
>>>>> $template TestStmt,"%hostname%%msg%"
>>>>>
>>>>> *.*      :omoracle:;TestStmt
>>>>> #====================================================
>>>>>
>>>>> Really appreciate anyone's help!
>>>>>
>>>>> Best
>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com
>>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>>> --
>>> This message has been scanned for viruses and dangerous content by 
>>> MailScanner, and is believed to be clean.
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
>> --
>> This message has been scanned for viruses and dangerous content by 
>> MailScanner, and is believed to be clean.
>>
> --
> This message has been scanned for viruses and dangerous content by 
> MailScanner, and is believed to be clean.
>
>
--
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: rsyslog.debug.rar
Type: application/octet-stream
Size: 147499 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20110512/3bd9e5be/attachment.obj>

From rgerhards at hq.adiscon.com  Thu May 12 07:38:32 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 12 May 2011 07:38:32 +0200
Subject: [rsyslog] help : CPU high load of rsyslog writing to Oracle
In-Reply-To: <alpine.DEB.2.00.1105112119200.16882@asgard.lang.hm>
References: <002101cc0e60$9aa7f6d0$cff7e470$@italkbb.com><alpine.DEB.2.00.1105091055030.25291@asgard.lang.hm><alpine.DEB.2.00.1105092231490.6302@asgard.lang.hm><001c01cc0ed8$6ea9ac40$4bfd04c0$@italkbb.com><alpine.DEB.2.00.1105092313190.6302@asgard.lang.hm><004801cc0f35$731e9850$595bc8f0$@italkbb.com><alpine.DEB.2.00.1105101820220.25291@asgard.lang.hm><001a01cc1014$556f7560$004e6020$@italkbb.com>
	<alpine.DEB.2.00.1105112119200.16882@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DE142@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Thursday, May 12, 2011 6:23 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to
> Oracle
> 
> this should point us at the problem (although I think Rainer will need
> to
> look at this).

I did, but it looks like the current instrumentation is not sufficient to see
where exactly it is looping. It definitely has something to do with the call
to omoracle, but it looks like the engine enters a loop, and not omoracle.

One way to get more information is to turn on the debug option "logfuncflow".
This will require a version that is compiled with configure options
--enable-debug --enable-rtinst given. It will create a *very* verbose debug
log. 

The other (probably better) alternative is to attach gdb to the looping
thread and make it outout a couple of stack traces. 

What I would like to see is which area of code the problem is in, and then I
will probably add some more instrumentation there (if the problem isn't
immediately visible...).

> 
> one minor correction, if the thread was blocked it would be eating zero
> CPU, not 100%, so it probably went into some loop that is completly in
> userspace after getting that response.
> 
> it may (or may not) help to get more of the rsyslog.strace..23298 file,
> how large is it?

Please let me propose that the attachments be placed on some web site (e.g.
filebin if nothing else is available). While we recently raised the
attachment size limit to 512K, I am  still a bit concerned that many mails
with big attachments will offset some list members.

Rainer
> 
> David Lang
> 
> On Wed, 11 May 2011, bxu wrote:
> 
> > Attached is the debug file when use " strace -ff -o rsyslog.strace. -
> C  -ttt
> > -T -s 100  rsyslogd -c5 -nd >rsyslog.debug".
> > Thread ID 23298 consumes almost all the CPU time,which is used to
> write to
> > Oracle.
> > In the file 'rsyslog.strace..23298',there's no more log after "
> > 1305142850.448202 write(1, "action call returned 0\n", 23) = 23
> > <0.000027>",seems this thread was blocked.
> > I think there is a loop after some wrong.
> >
> > -----Original Message-----
> > From: bxu [mailto:bxu at italkbb.com]
> > Sent: Wednesday, May 11, 2011 1:15 AM
> > To: 'rsyslog-users'
> > Subject: RE: [rsyslog] help : CPU high load of rsyslog writing to
> Oracle
> >
> > Hi David,
> > 	Yes. The 'H' was turned on for the TOP info.
> >
> > % time     seconds  usecs/call     calls    errors syscall
> > ------ ----------- ----------- --------- --------- ----------------
> > 49.99    0.008990           5      1858       929 futex
> > 33.34    0.005996         857         7           select
> > 16.67    0.002997          20       151           read
> >  0.00    0.000000           0       542           recvfrom
> >  0.00    0.000000           0         7           recvmsg
> >  0.00    0.000000           0       468           gettimeofday
> >  0.00    0.000000           0      1860           clock_gettime
> > ------ ----------- ----------- --------- --------- ----------------
> > 100.00    0.017983                  4893       929 total
> >
> > 	As seen from the above TOP info,there are 929 errors for futex
> > call.Sometimes there were errors for recvmsg call where I did more
> > straces.Does this relate to the CPU high load?
> >
> > 	I'll do more tests as your suggestion.
> >
> > Thanks,
> > Henry
> >
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com
> > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> > Sent: Tuesday, May 10, 2011 8:29 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to
> Oracle
> >
> > thanks for the info.
> >
> > when you printed the TOP info, was that with 'H' (thread details)
> turned on?
> >
> > I'm puzzled by the fact that you are doing a strace of the thread
> 13470 and
> > it's showing actions by many different pids (it looks like everything
> except
> > 13470
> >
> > can you try doing
> >
> > strace -ff -o rsyslog.strace. -C  -ttt -T -s 100  rsyslogd -c5 -nd
> >> rsyslog.debug
> >
> > this will create a bunch of strace files. we shouldn't need to look
> at all
> > of them, just the one using all the CPU (and possibly only the end of
> that
> > one)
> >
> > while this is running, do a ps -eLf
> >
> > also, run top -H and note which thread is using all the CPU (you may
> want to
> > try toggling H while it's running to make sure you are getting the
> thread
> > info not the consolodated process info)
> >
> > I'm about to head home and when I get home I'll look at the debug
> output
> > that you send.
> >
> > one thing I'm not remembering. are the logs getting into the
> database?
> >
> > David Lang
> >
> > On Tue, 10 May 2011, bxu wrote:
> >
> >> Hi David,
> >> 	Here are what I'm doing:
> >> 	1. oracle configuration of rsyslogd.conf
> >>
> >> ################
> >> #### ORACLE ####
> >> ################
> >>
> >> $ModLoad omoracle
> >> $OmoracleDBUser syslog
> >> $OmoracleDBPassword syslog
> >> $OmoracleDB syslog
> >> $OmoracleBatchSize 1
> >> $OmoracleBatchItemSize 4096
> >> $OmoracleStatementTemplate OmoracleStatement
> >>
> >> $template OmoracleStatement,"INSERT INTO TEST(Hostname,Message)
> >> VALUES(:hostname,:msg)"
> >> $template TestStmt,"%hostname%%msg%"
> >> $template TestStmtFile,"INSERT INTO TEST(Hostname,Message)
> >> VALUES('%hostname%','%msg%')\n"
> >>
> >> #$WorkDirectory /rsyslog/work # default location for work (spool)
> >> files
> >>
> >> #$ActionQueueType LinkedList # use asynchronous processing
> >> #$ActionQueueFileName dbq    # set file name, also enables disk mode
> >> #$ActionResumeRetryCount -1  # infinite retries on insert failure
> >>
> >> :msg,contains,"SIP"             :omoracle:;TestStmt
> >>
> >> & ~
> >>
> >> *.*                             /var/log/syslog-test;TestStmtFile
> >>
> >> 	2. running command :  rsyslogd -c5 -nd > tt &
> >>
> >> 	3, TOP information
> >>
> >>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> >>
> >> 13470 root      25   0  244m  11m 7636 R 99.9  0.1   1:06.39 rs:main
> Q:Reg
> >>
> >> 11410 root      15   0 12768 1144  832 R  0.3  0.0   0:13.52 top
> >>
> >>    1 root      15   0 10372  692  580 S  0.0  0.0   0:00.51 init
> >>
> >> 	4. when CPU usage is keeping at 100%,the summary of strace by
> thread
> >> 13470:
> >>
> >> % time     seconds  usecs/call     calls    errors syscall
> >> ------ ----------- ----------- --------- --------- ----------------
> >> 51.26    0.019996        9998         2           epoll_wait
> >> 46.12    0.017990        1799        10           read
> >>  2.56    0.000999         999         1           select
> >>  0.06    0.000022           2        12         2 recvfrom
> >>  0.00    0.000000           0        64           write
> >>  0.00    0.000000           0         1           recvmsg
> >>  0.00    0.000000           0        17           gettimeofday
> >>  0.00    0.000000           0        62           clock_gettime
> >> ------ ----------- ----------- --------- --------- ----------------
> >> 100.00    0.039007                   169         2 total
> >>
> >> 	5. when CPU usage is keeping at 100%,some of the call sequence of
> >> strace by thread 13470:
> >>
> >> [pid 13469]      0.000000 select(1, NULL, NULL, NULL, {86097,
> 801000}
> >> <unfinished ...>
> >> [pid 13473]      0.000089 clock_gettime(CLOCK_REALTIME,  <unfinished
> ...>
> >> [pid 13472]      0.000054 select(9, [8], NULL, NULL, NULL
> <unfinished ...>
> >> [pid 13471]      0.000066 read(3,  <unfinished ...>
> >> [pid 13473]      0.000048 <... clock_gettime resumed> {1305046294,
> >> 274947000}) = 0 <0.000138>
> >> [pid 13473]      0.000067 write(1, "6294.274947000:43899940: imudp:
> "...,
> >> 66) = 66 <0.000042>
> >> [pid 13473]      0.000137 epoll_wait(12,  <unfinished ...>
> >> [pid 13471]      1.421497 <... read resumed> "<6>iptables: log and
> drop
> >> IN=eth"..., 4095) = 200 <1.421706>
> >> [pid 13471]      0.000095 gettimeofday({1305046295, 696914}, NULL) =
> 0
> >> <0.000022>
> >> [pid 13471]      0.000130 clock_gettime(CLOCK_REALTIME, {1305046295,
> >> 697046000}) = 0 <0.000022>
> >> [pid 13471]      0.000104 write(1, "6295.697046000:42497940: main
> Q:"...,
> >> 81) = 81 <0.000037>
> >> [pid 13471]      0.000127 clock_gettime(CLOCK_REALTIME, {1305046295,
> >> 697276000}) = 0 <0.000020>
> >> [pid 13471]      0.000102 write(1, "6295.697276000:42497940: main
> Q:"...,
> >> 65) = 65 <0.000031>
> >> [pid 13471]      0.000120 read(3, "<6>iptables: log and drop
> IN=eth"...,
> >> 4095) = 200 <4.250732>
> >> [pid 13471]      4.250863 gettimeofday({1305046299, 948362}, NULL) =
> 0
> >> <0.000021>
> >> [pid 13471]      0.000129 clock_gettime(CLOCK_REALTIME, {1305046299,
> >> 948491000}) = 0 <0.000021>
> >> [pid 13471]      0.000106 write(1, "6299.948491000:42497940: main
> Q:"...,
> >> 81) = 81 <0.000051>
> >> [pid 13471]      0.000140 clock_gettime(CLOCK_REALTIME, {1305046299,
> >> 948735000}) = 0 <0.000021>
> >> [pid 13471]      0.000101 write(1, "6299.948735000:42497940: main
> Q:"...,
> >> 65) = 65 <0.000031>
> >> [pid 13471]      0.000123 read(3, "<6>iptables: log and drop
> IN=eth"...,
> >> 4095) = 200 <2.178833>
> >> [pid 13471]      2.178931 gettimeofday({1305046302, 127891}, NULL) =
> 0
> >> <0.000022>
> >> [pid 13471]      0.000112 clock_gettime(CLOCK_REALTIME, {1305046302,
> >> 128003000}) = 0 <0.000021>
> >> [pid 13471]      0.000101 write(1, "6302.128003000:42497940: main
> Q:"...,
> >> 81) = 81 <0.000036>
> >> [pid 13471]      0.000121 clock_gettime(CLOCK_REALTIME, {1305046302,
> >> 128225000}) = 0 <0.000021>
> >> [pid 13471]      0.000102 write(1, "6302.128225000:42497940: main
> Q:"...,
> >> 65) = 65 <0.000031>
> >> [pid 13471]      0.000115 read(3, "<6>iptables: log and drop
> IN=eth"...,
> >> 4095) = 200 <2.431092>
> >> [pid 13471]      2.431228 gettimeofday({1305046304, 559672}, NULL) =
> 0
> >> <0.000022>
> >> [pid 13471]      0.000132 clock_gettime(CLOCK_REALTIME, {1305046304,
> >> 559803000}) = 0 <0.000021>
> >> [pid 13471]      0.000110 write(1, "6304.559803000:42497940: main
> Q:"...,
> >> 81) = 81 <0.000052>
> >> [pid 13471]      0.000140 clock_gettime(CLOCK_REALTIME, {1305046304,
> >> 560052000}) = 0 <0.000021>
> >> [pid 13471]      0.000102 write(1, "6304.560052000:42497940: main
> Q:"...,
> >> 65) = 65 <0.000031>
> >> [pid 13471]      0.000119 read(3, "<6>iptables: log and drop
> IN=eth"...,
> >> 4095) = 197 <0.678610>
> >> [pid 13471]      0.678704 gettimeofday({1305046305, 238976}, NULL) =
> 0
> >> <0.000021>
> >> [pid 13471]      0.000110 clock_gettime(CLOCK_REALTIME, {1305046305,
> >> 239087000}) = 0 <0.000024>
> >> [pid 13471]      0.000105 write(1, "6305.239087000:42497940: main
> Q:"...,
> >> 81) = 81 <0.000035>
> >> [pid 13471]      0.000120 clock_gettime(CLOCK_REALTIME, {1305046305,
> >> 239312000}) = 0 <0.000020>
> >> [pid 13471]      0.000102 write(1, "6305.239312000:42497940: main
> Q:"...,
> >> 65) = 65 <0.000030>
> >> [pid 13471]      0.000114 read(3, "<6>iptables: log and drop
> IN=eth"...,
> >> 4095) = 200 <3.058797>
> >> [pid 13471]      3.058929 gettimeofday({1305046308, 298458}, NULL) =
> 0
> >> <0.000022>
> >> [pid 13471]      0.000131 clock_gettime(CLOCK_REALTIME, {1305046308,
> >> 298589000}) = 0 <0.000021>
> >> [pid 13471]      0.000106 write(1, "6308.298589000:42497940: main
> Q:"...,
> >> 81) = 81 <0.000058>
> >> [pid 13471]      0.000148 clock_gettime(CLOCK_REALTIME, {1305046308,
> >> 298841000}) = 0 <0.000021>
> >> [pid 13471]      0.000101 write(1, "6308.298841000:42497940: main
> Q:"...,
> >> 65) = 65 <0.000093>
> >> [pid 13471]      0.000181 read(3, "<6>iptables: log and drop
> IN=eth"...,
> >> 4095) = 197 <0.213955>
> >> [pid 13471]      0.214082 gettimeofday({1305046308, 513206}, NULL) =
> 0
> >> <0.000021>
> >> [pid 13471]      0.000128 clock_gettime(CLOCK_REALTIME, {1305046308,
> >> 513335000}) = 0 <0.000021>
> >> [pid 13471]      0.000106 write(1, "6308.513335000:42497940: main
> Q:"...,
> >> 81) = 81 <0.000042>
> >> [pid 13471]      0.000130 clock_gettime(CLOCK_REALTIME, {1305046308,
> >> 513570000}) = 0 <0.000021>
> >> [pid 13471]      0.000102 write(1, "6308.513570000:42497940: main
> Q:"...,
> >> 65) = 65 <0.000030>
> >> [pid 13471]      0.000122 read(3, "<6>iptables: log and drop
> IN=eth"...,
> >> 4095) = 196 <0.835091>
> >> [pid 13471]      0.835187 gettimeofday({1305046309, 348980}, NULL) =
> 0
> >> <0.000021>
> >> [pid 13471]      0.000110 clock_gettime(CLOCK_REALTIME, {1305046309,
> >> 349091000}) = 0 <0.000021>
> >> [pid 13471]      0.000102 write(1, "6309.349091000:42497940: main
> Q:"...,
> >> 81) = 81 <0.000035>
> >> [pid 13471]      0.000121 clock_gettime(CLOCK_REALTIME, {1305046309,
> >> 349313000}) = 0 <0.000020>
> >> [pid 13471]      0.000102 write(1, "6309.349313000:42497940: main
> Q:"...,
> >> 65) = 65 <0.000030>
> >> [pid 13471]      0.000114 read(3,  <unfinished ...>
> >> [pid 13473]      2.018865 <... epoll_wait resumed> {{EPOLLIN,
> {u32=2,
> >> u64=2}}}, 10, 4294967295) = 1 <17.093033>
> >> [pid 13473]      0.000086 clock_gettime(CLOCK_REALTIME, {1305046311,
> >> 368481000}) = 0 <0.000020>
> >> [pid 13473]      0.000098 write(1, "6311.368481000:43899940: imudp:
> "...,
> >> 65) = 65 <0.000036>
> >>
> >>
> >> 	6. Attached is the log file while rsyslogd is running in debug
> mode
> >> 	    In the log,after "omoracle insertion to DB succeeded",
> there's
> > no
> >> more oracle log information related.
> >>
> >>
> >> 	Thank you very much.
> >>
> >> Best,
> >> Henry
> >>
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com
> >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of
> david at lang.hm
> >> Sent: Tuesday, May 10, 2011 1:19 AM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to
> >> Oracle
> >>
> >> rsyslog uses multiple threads, if you can tell which thread is using
> >> all the CPU and then do a strace of it and dump the sequence of
> calls
> >> that it makes to process one of the messages it may help
> >>
> >> I don't have any idea what's going on here, so I'm just working
> >> through the steps that I would be doing if I was running into the
> >> problem
> >>
> >> try ps -eLf to list the different threads
> >>
> >> in top, hit 'H' to run on per-thread reporting (at which point you
> may
> >> get some additional info on the right about what the thread is
> doing)
> >>
> >> from the stats you are showing here, it looks like the thread you
> got
> >> with your strace is the parent that's receiving messages (lots of
> >> gettimeofday and receive/read type calls) the correct one should
> have
> >> write calls in it (among others)
> >>
> >> the versions you are running are new enough that we shouldn't have
> any
> >> old, solved problems hitting us.
> >>
> >>  On Tue, 10 May 2011, bxu
> >> wrote:
> >>
> >>> Hi David,
> >>> 	The rsyslogd version I tried are 5.8.0(stable) and 6.1.7(devel).
> >>> 	I straced rsyslogd when CPU usage is keeping at 100%.
> >>>
> >>> % time     seconds  usecs/call     calls    errors syscall
> >>> ------ ----------- ----------- --------- --------- ----------------
> >>> 43.47    0.066967        2309        29           read
> >>> 38.28    0.058979         855        69           epoll_wait
> >>> 18.17    0.027994        9331         3           select
> >>>  0.05    0.000081           1       114           gettimeofday
> >>>  0.03    0.000041           0       178        69 recvfrom
> >>>  0.00    0.000000           0         3           recvmsg
> >>> ------ ----------- ----------- --------- --------- ----------------
> >>> 100.00    0.154062                   396        69 total
> >>>
> >>>
> >>> Best,
> >>> Henry
> >>>
> >>> -----Original Message-----
> >>> From: rsyslog-bounces at lists.adiscon.com
> >>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of
> david at lang.hm
> >>> Sent: Tuesday, May 10, 2011 12:35 AM
> >>> To: rsyslog-users
> >>> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to
> >>> Oracle
> >>>
> >>> I accidently deleted your reply, so I am replying to my message
> >>>
> >>> 4-5 messages/secshould not be any problem at all.
> >>>
> >>> since writing a copy of the sql commands to a file doesn't show
> >>> anything strange, the next step will be to try and run it in debug
> >>> mode to see if it logs anything strange as you start having
> problems.
> >>>
> >>> the next step after that will probably be to do a strace of the
> >>> thread that is using all the CPU
> >>>
> >>> I don't see where you mention what version of rsyslog you are
> using.
> >>>
> >>> David Lang
> >>>
> >>> On Mon, 9 May 2011, david at lang.hm wrote:
> >>>
> >>>> Date: Mon, 9 May 2011 10:56:30 -0700 (PDT)
> >>>> From: david at lang.hm
> >>>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> >>>> To: rsyslog-users <rsyslog at lists.adiscon.com>
> >>>> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to
> >>>> Oracle
> >>>>
> >>>> given that it only happens after a few messages, I would suspect
> >>>> that there is something odd with the messages.
> >>>>
> >>>> could you try writing the messages to a file using the Oracle
> >>>> template and then look and see if there is anything odd about the
> >>>> messages when it goes nuts?
> >>>>
> >>>> the other option is to get a debug dump, but unless it happens
> right
> >>>> away, that is likely to be a large dump.
> >>>>
> >>>> David Lang
> >>>>
> >>>> On Mon, 9 May 2011, bxu wrote:
> >>>>
> >>>>> Date: Mon, 9 May 2011 10:48:50 -0500
> >>>>> From: bxu <bxu at italkbb.com>
> >>>>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> >>>>> To: rsyslog at lists.adiscon.com
> >>>>> Subject: [rsyslog] help : CPU high load of rsyslog writing to
> >>>>> Oracle
> >>>>>
> >>>>> Hi All,
> >>>>>
> >>>>>
> >>>>> I have configured rsyslog writing to Oracle.The writing was
> >>>>> successful.But after several messages have been made into
> >>>>> Oracle,the CPU use percentage of rsyslogd goes up to 99%-100%.It
> is
> >>>>> abnormally too
> >>> much high.
> >>>>> There's no problem with the cpu usage when writing to files.
> >>>>>
> >>>>> Here is my rsyslogd.conf:
> >>>>> #====================================================
> >>>>> #*.* >192.168.45.213,rsyslog
> >>>>>
> >>>>> # Use traditional timestamp format
> >>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
> >>>>>
> >>>>> # Provides kernel logging support (previously done by rklogd)
> >>>>> $ModLoad imklog # Provides support for local system logging (e.g.
> >>>>> via logger command) $ModLoad imuxsock
> >>>>>
> >>>>> # Log all kernel messages to the console.
> >>>>> # Logging much else clutters up the screen.
> >>>>> #kern.* /dev/console
> >>>>>
> >>>>> # Log anything (except mail) of level info or higher.
> >>>>> # Don't log private authentication messages!
> >>>>> *.info;mail.none;authpriv.none;cron.none /var/log/messages
> >>>>>
> >>>>> # The authpriv file has restricted access.
> >>>>> authpriv.* /var/log/secure
> >>>>>
> >>>>> # Log all the mail messages in one place.
> >>>>> mail.* -/var/log/maillog
> >>>>>
> >>>>> # Log cron stuff
> >>>>> cron.* /var/log/cron
> >>>>>
> >>>>> # Everybody gets emergency messages *.emerg *
> >>>>>
> >>>>> # Save news errors of level crit and higher in a special file.
> >>>>> uucp,news.crit /var/log/spooler
> >>>>>
> >>>>> # Save boot messages also to boot.log
> >>>>> local7.* /var/log/boot.log
> >>>>>
> >>>>> #Load External Listender
> >>>>> $modload imudp
> >>>>>
> >>>>>
> >>>>> $UDPServerRun 514
> >>>>> #SPA Messages
> >>>>> #*.* /var/log/linksys.log
> >>>>> $template DynaFile,"/var/log/system-%HOSTNAME%.log"
> >>>>> *.* -?DynaFile
> >>>>>
> >>>>>
> >>>>> ################
> >>>>> #### ORACLE ####
> >>>>> ################
> >>>>>
> >>>>> $ModLoad omoracle
> >>>>> $OmoracleDBUser syslog
> >>>>> $OmoracleDBPassword syslog
> >>>>> $OmoracleDB syslog
> >>>>> $OmoracleBatchSize 1
> >>>>> $OmoracleBatchItemSize 4096
> >>>>> $OmoracleStatementTemplate OmoracleStatement
> >>>>>
> >>>>> $template OmoracleStatement,"INSERT INTO TEST(hostname,message)
> >>>>> VALUES(:hostname,:msg)"
> >>>>> $template TestStmt,"%hostname%%msg%"
> >>>>>
> >>>>> *.*      :omoracle:;TestStmt
> >>>>> #====================================================
> >>>>>
> >>>>> Really appreciate anyone's help!
> >>>>>
> >>>>> Best
> >>>>>
> >>>>> _______________________________________________
> >>>>> rsyslog mailing list
> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>> http://www.rsyslog.com
> >>>>>
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.com
> >>>>
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >>>
> >>> --
> >>> This message has been scanned for viruses and dangerous content by
> >>> MailScanner, and is believed to be clean.
> >>>
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >>>
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >>
> >> --
> >> This message has been scanned for viruses and dangerous content by
> >> MailScanner, and is believed to be clean.
> >>
> > --
> > This message has been scanned for viruses and dangerous content by
> > MailScanner, and is believed to be clean.
> >
> >

From rgerhards at hq.adiscon.com  Thu May 12 07:42:12 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 12 May 2011 07:42:12 +0200
Subject: [rsyslog] help : CPU high load of rsyslog writing to Oracle
In-Reply-To: <003601cc1061$dac8b300$905a1900$@italkbb.com>
References: <002101cc0e60$9aa7f6d0$cff7e470$@italkbb.com>	<alpine.DEB.2.00.1105091055030.25291@asgard.lang.hm>	<alpine.DEB.2.00.1105092231490.6302@asgard.lang.hm>	<001c01cc0ed8$6ea9ac40$4bfd04c0$@italkbb.com>	<alpine.DEB.2.00.1105092313190.6302@asgard.lang.hm>	<004801cc0f35$731e9850$595bc8f0$@italkbb.com>	<alpine.DEB.2.00.1105101820220.25291@asgard.lang.hm>	<001a01cc1014$556f7560$004e6020$@italkbb.com><alpine.DEB.2.00.1105112119200.16882@asgard.lang.hm>
	<003601cc1061$dac8b300$905a1900$@italkbb.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DE143@GRFEXC.intern.adiscon.com>

It would possibly be useful if you gave rsyslog another 5 seconds between the
first ctl-c and the successive ones. I then could see how the thread term is
handled, which gives some further clues. However, this is by far not as
informative as the other things I suggested, so please combine both of it.

Thanks,
Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Henry Xu
> Sent: Thursday, May 12, 2011 7:03 AM
> To: 'rsyslog-users'
> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to
> Oracle
> 
> Attached is another trace of rsyslogd with thread ID 2665 which hosted
> all
> the CPU.
> That's all the log info.
> 
> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Wednesday, May 11, 2011 11:23 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to
> Oracle
> 
> this should point us at the problem (although I think Rainer will need
> to
> look at this).
> 
> one minor correction, if the thread was blocked it would be eating zero
> CPU,
> not 100%, so it probably went into some loop that is completly in
> userspace
> after getting that response.
> 
> it may (or may not) help to get more of the rsyslog.strace..23298 file,
> how
> large is it?
> 
> David Lang
> 
> On Wed, 11 May 2011, bxu wrote:
> 
> > Attached is the debug file when use " strace -ff -o rsyslog.strace. -
> C
> > -ttt -T -s 100  rsyslogd -c5 -nd >rsyslog.debug".
> > Thread ID 23298 consumes almost all the CPU time,which is used to
> > write to Oracle.
> > In the file 'rsyslog.strace..23298',there's no more log after "
> > 1305142850.448202 write(1, "action call returned 0\n", 23) = 23
> > <0.000027>",seems this thread was blocked.
> > I think there is a loop after some wrong.
> >
> > -----Original Message-----
> > From: bxu [mailto:bxu at italkbb.com]
> > Sent: Wednesday, May 11, 2011 1:15 AM
> > To: 'rsyslog-users'
> > Subject: RE: [rsyslog] help : CPU high load of rsyslog writing to
> > Oracle
> >
> > Hi David,
> > 	Yes. The 'H' was turned on for the TOP info.
> >
> > % time     seconds  usecs/call     calls    errors syscall
> > ------ ----------- ----------- --------- --------- ----------------
> > 49.99    0.008990           5      1858       929 futex
> > 33.34    0.005996         857         7           select
> > 16.67    0.002997          20       151           read
> >  0.00    0.000000           0       542           recvfrom
> >  0.00    0.000000           0         7           recvmsg
> >  0.00    0.000000           0       468           gettimeofday
> >  0.00    0.000000           0      1860           clock_gettime
> > ------ ----------- ----------- --------- --------- ----------------
> > 100.00    0.017983                  4893       929 total
> >
> > 	As seen from the above TOP info,there are 929 errors for futex
> > call.Sometimes there were errors for recvmsg call where I did more
> > straces.Does this relate to the CPU high load?
> >
> > 	I'll do more tests as your suggestion.
> >
> > Thanks,
> > Henry
> >
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com
> > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> > Sent: Tuesday, May 10, 2011 8:29 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to
> > Oracle
> >
> > thanks for the info.
> >
> > when you printed the TOP info, was that with 'H' (thread details)
> turned
> on?
> >
> > I'm puzzled by the fact that you are doing a strace of the thread
> > 13470 and it's showing actions by many different pids (it looks like
> > everything except
> > 13470
> >
> > can you try doing
> >
> > strace -ff -o rsyslog.strace. -C  -ttt -T -s 100  rsyslogd -c5 -nd
> >> rsyslog.debug
> >
> > this will create a bunch of strace files. we shouldn't need to look
> at
> > all of them, just the one using all the CPU (and possibly only the
> end
> > of that
> > one)
> >
> > while this is running, do a ps -eLf
> >
> > also, run top -H and note which thread is using all the CPU (you may
> > want to try toggling H while it's running to make sure you are
> getting
> > the thread info not the consolodated process info)
> >
> > I'm about to head home and when I get home I'll look at the debug
> > output that you send.
> >
> > one thing I'm not remembering. are the logs getting into the
> database?
> >
> > David Lang
> >
> > On Tue, 10 May 2011, bxu wrote:
> >
> >> Hi David,
> >> 	Here are what I'm doing:
> >> 	1. oracle configuration of rsyslogd.conf
> >>
> >> ################
> >> #### ORACLE ####
> >> ################
> >>
> >> $ModLoad omoracle
> >> $OmoracleDBUser syslog
> >> $OmoracleDBPassword syslog
> >> $OmoracleDB syslog
> >> $OmoracleBatchSize 1
> >> $OmoracleBatchItemSize 4096
> >> $OmoracleStatementTemplate OmoracleStatement
> >>
> >> $template OmoracleStatement,"INSERT INTO TEST(Hostname,Message)
> >> VALUES(:hostname,:msg)"
> >> $template TestStmt,"%hostname%%msg%"
> >> $template TestStmtFile,"INSERT INTO TEST(Hostname,Message)
> >> VALUES('%hostname%','%msg%')\n"
> >>
> >> #$WorkDirectory /rsyslog/work # default location for work (spool)
> >> files
> >>
> >> #$ActionQueueType LinkedList # use asynchronous processing
> >> #$ActionQueueFileName dbq    # set file name, also enables disk mode
> >> #$ActionResumeRetryCount -1  # infinite retries on insert failure
> >>
> >> :msg,contains,"SIP"             :omoracle:;TestStmt
> >>
> >> & ~
> >>
> >> *.*                             /var/log/syslog-test;TestStmtFile
> >>
> >> 	2. running command :  rsyslogd -c5 -nd > tt &
> >>
> >> 	3, TOP information
> >>
> >>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> >>
> >> 13470 root      25   0  244m  11m 7636 R 99.9  0.1   1:06.39 rs:main
> Q:Reg
> >>
> >> 11410 root      15   0 12768 1144  832 R  0.3  0.0   0:13.52 top
> >>
> >>    1 root      15   0 10372  692  580 S  0.0  0.0   0:00.51 init
> >>
> >> 	4. when CPU usage is keeping at 100%,the summary of strace by
> thread
> >> 13470:
> >>
> >> % time     seconds  usecs/call     calls    errors syscall
> >> ------ ----------- ----------- --------- --------- ----------------
> >> 51.26    0.019996        9998         2           epoll_wait
> >> 46.12    0.017990        1799        10           read
> >>  2.56    0.000999         999         1           select
> >>  0.06    0.000022           2        12         2 recvfrom
> >>  0.00    0.000000           0        64           write
> >>  0.00    0.000000           0         1           recvmsg
> >>  0.00    0.000000           0        17           gettimeofday
> >>  0.00    0.000000           0        62           clock_gettime
> >> ------ ----------- ----------- --------- --------- ----------------
> >> 100.00    0.039007                   169         2 total
> >>
> >> 	5. when CPU usage is keeping at 100%,some of the call sequence of
> >> strace by thread 13470:
> >>
> >> [pid 13469]      0.000000 select(1, NULL, NULL, NULL, {86097,
> 801000}
> >> <unfinished ...>
> >> [pid 13473]      0.000089 clock_gettime(CLOCK_REALTIME,  <unfinished
> ...>
> >> [pid 13472]      0.000054 select(9, [8], NULL, NULL, NULL
> <unfinished
> ...>
> >> [pid 13471]      0.000066 read(3,  <unfinished ...>
> >> [pid 13473]      0.000048 <... clock_gettime resumed> {1305046294,
> >> 274947000}) = 0 <0.000138>
> >> [pid 13473]      0.000067 write(1, "6294.274947000:43899940: imudp:
> "...,
> >> 66) = 66 <0.000042>
> >> [pid 13473]      0.000137 epoll_wait(12,  <unfinished ...>
> >> [pid 13471]      1.421497 <... read resumed> "<6>iptables: log and
> drop
> >> IN=eth"..., 4095) = 200 <1.421706>
> >> [pid 13471]      0.000095 gettimeofday({1305046295, 696914}, NULL) =
> 0
> >> <0.000022>
> >> [pid 13471]      0.000130 clock_gettime(CLOCK_REALTIME, {1305046295,
> >> 697046000}) = 0 <0.000022>
> >> [pid 13471]      0.000104 write(1, "6295.697046000:42497940: main
> Q:"...,
> >> 81) = 81 <0.000037>
> >> [pid 13471]      0.000127 clock_gettime(CLOCK_REALTIME, {1305046295,
> >> 697276000}) = 0 <0.000020>
> >> [pid 13471]      0.000102 write(1, "6295.697276000:42497940: main
> Q:"...,
> >> 65) = 65 <0.000031>
> >> [pid 13471]      0.000120 read(3, "<6>iptables: log and drop
> IN=eth"...,
> >> 4095) = 200 <4.250732>
> >> [pid 13471]      4.250863 gettimeofday({1305046299, 948362}, NULL) =
> 0
> >> <0.000021>
> >> [pid 13471]      0.000129 clock_gettime(CLOCK_REALTIME, {1305046299,
> >> 948491000}) = 0 <0.000021>
> >> [pid 13471]      0.000106 write(1, "6299.948491000:42497940: main
> Q:"...,
> >> 81) = 81 <0.000051>
> >> [pid 13471]      0.000140 clock_gettime(CLOCK_REALTIME, {1305046299,
> >> 948735000}) = 0 <0.000021>
> >> [pid 13471]      0.000101 write(1, "6299.948735000:42497940: main
> Q:"...,
> >> 65) = 65 <0.000031>
> >> [pid 13471]      0.000123 read(3, "<6>iptables: log and drop
> IN=eth"...,
> >> 4095) = 200 <2.178833>
> >> [pid 13471]      2.178931 gettimeofday({1305046302, 127891}, NULL) =
> 0
> >> <0.000022>
> >> [pid 13471]      0.000112 clock_gettime(CLOCK_REALTIME, {1305046302,
> >> 128003000}) = 0 <0.000021>
> >> [pid 13471]      0.000101 write(1, "6302.128003000:42497940: main
> Q:"...,
> >> 81) = 81 <0.000036>
> >> [pid 13471]      0.000121 clock_gettime(CLOCK_REALTIME, {1305046302,
> >> 128225000}) = 0 <0.000021>
> >> [pid 13471]      0.000102 write(1, "6302.128225000:42497940: main
> Q:"...,
> >> 65) = 65 <0.000031>
> >> [pid 13471]      0.000115 read(3, "<6>iptables: log and drop
> IN=eth"...,
> >> 4095) = 200 <2.431092>
> >> [pid 13471]      2.431228 gettimeofday({1305046304, 559672}, NULL) =
> 0
> >> <0.000022>
> >> [pid 13471]      0.000132 clock_gettime(CLOCK_REALTIME, {1305046304,
> >> 559803000}) = 0 <0.000021>
> >> [pid 13471]      0.000110 write(1, "6304.559803000:42497940: main
> Q:"...,
> >> 81) = 81 <0.000052>
> >> [pid 13471]      0.000140 clock_gettime(CLOCK_REALTIME, {1305046304,
> >> 560052000}) = 0 <0.000021>
> >> [pid 13471]      0.000102 write(1, "6304.560052000:42497940: main
> Q:"...,
> >> 65) = 65 <0.000031>
> >> [pid 13471]      0.000119 read(3, "<6>iptables: log and drop
> IN=eth"...,
> >> 4095) = 197 <0.678610>
> >> [pid 13471]      0.678704 gettimeofday({1305046305, 238976}, NULL) =
> 0
> >> <0.000021>
> >> [pid 13471]      0.000110 clock_gettime(CLOCK_REALTIME, {1305046305,
> >> 239087000}) = 0 <0.000024>
> >> [pid 13471]      0.000105 write(1, "6305.239087000:42497940: main
> Q:"...,
> >> 81) = 81 <0.000035>
> >> [pid 13471]      0.000120 clock_gettime(CLOCK_REALTIME, {1305046305,
> >> 239312000}) = 0 <0.000020>
> >> [pid 13471]      0.000102 write(1, "6305.239312000:42497940: main
> Q:"...,
> >> 65) = 65 <0.000030>
> >> [pid 13471]      0.000114 read(3, "<6>iptables: log and drop
> IN=eth"...,
> >> 4095) = 200 <3.058797>
> >> [pid 13471]      3.058929 gettimeofday({1305046308, 298458}, NULL) =
> 0
> >> <0.000022>
> >> [pid 13471]      0.000131 clock_gettime(CLOCK_REALTIME, {1305046308,
> >> 298589000}) = 0 <0.000021>
> >> [pid 13471]      0.000106 write(1, "6308.298589000:42497940: main
> Q:"...,
> >> 81) = 81 <0.000058>
> >> [pid 13471]      0.000148 clock_gettime(CLOCK_REALTIME, {1305046308,
> >> 298841000}) = 0 <0.000021>
> >> [pid 13471]      0.000101 write(1, "6308.298841000:42497940: main
> Q:"...,
> >> 65) = 65 <0.000093>
> >> [pid 13471]      0.000181 read(3, "<6>iptables: log and drop
> IN=eth"...,
> >> 4095) = 197 <0.213955>
> >> [pid 13471]      0.214082 gettimeofday({1305046308, 513206}, NULL) =
> 0
> >> <0.000021>
> >> [pid 13471]      0.000128 clock_gettime(CLOCK_REALTIME, {1305046308,
> >> 513335000}) = 0 <0.000021>
> >> [pid 13471]      0.000106 write(1, "6308.513335000:42497940: main
> Q:"...,
> >> 81) = 81 <0.000042>
> >> [pid 13471]      0.000130 clock_gettime(CLOCK_REALTIME, {1305046308,
> >> 513570000}) = 0 <0.000021>
> >> [pid 13471]      0.000102 write(1, "6308.513570000:42497940: main
> Q:"...,
> >> 65) = 65 <0.000030>
> >> [pid 13471]      0.000122 read(3, "<6>iptables: log and drop
> IN=eth"...,
> >> 4095) = 196 <0.835091>
> >> [pid 13471]      0.835187 gettimeofday({1305046309, 348980}, NULL) =
> 0
> >> <0.000021>
> >> [pid 13471]      0.000110 clock_gettime(CLOCK_REALTIME, {1305046309,
> >> 349091000}) = 0 <0.000021>
> >> [pid 13471]      0.000102 write(1, "6309.349091000:42497940: main
> Q:"...,
> >> 81) = 81 <0.000035>
> >> [pid 13471]      0.000121 clock_gettime(CLOCK_REALTIME, {1305046309,
> >> 349313000}) = 0 <0.000020>
> >> [pid 13471]      0.000102 write(1, "6309.349313000:42497940: main
> Q:"...,
> >> 65) = 65 <0.000030>
> >> [pid 13471]      0.000114 read(3,  <unfinished ...>
> >> [pid 13473]      2.018865 <... epoll_wait resumed> {{EPOLLIN,
> {u32=2,
> >> u64=2}}}, 10, 4294967295) = 1 <17.093033>
> >> [pid 13473]      0.000086 clock_gettime(CLOCK_REALTIME, {1305046311,
> >> 368481000}) = 0 <0.000020>
> >> [pid 13473]      0.000098 write(1, "6311.368481000:43899940: imudp:
> "...,
> >> 65) = 65 <0.000036>
> >>
> >>
> >> 	6. Attached is the log file while rsyslogd is running in debug
> mode
> >> 	    In the log,after "omoracle insertion to DB succeeded",
> there's
> > no
> >> more oracle log information related.
> >>
> >>
> >> 	Thank you very much.
> >>
> >> Best,
> >> Henry
> >>
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com
> >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of
> david at lang.hm
> >> Sent: Tuesday, May 10, 2011 1:19 AM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to
> >> Oracle
> >>
> >> rsyslog uses multiple threads, if you can tell which thread is using
> >> all the CPU and then do a strace of it and dump the sequence of
> calls
> >> that it makes to process one of the messages it may help
> >>
> >> I don't have any idea what's going on here, so I'm just working
> >> through the steps that I would be doing if I was running into the
> >> problem
> >>
> >> try ps -eLf to list the different threads
> >>
> >> in top, hit 'H' to run on per-thread reporting (at which point you
> >> may get some additional info on the right about what the thread is
> >> doing)
> >>
> >> from the stats you are showing here, it looks like the thread you
> got
> >> with your strace is the parent that's receiving messages (lots of
> >> gettimeofday and receive/read type calls) the correct one should
> have
> >> write calls in it (among others)
> >>
> >> the versions you are running are new enough that we shouldn't have
> >> any old, solved problems hitting us.
> >>
> >>  On Tue, 10 May 2011, bxu
> >> wrote:
> >>
> >>> Hi David,
> >>> 	The rsyslogd version I tried are 5.8.0(stable) and 6.1.7(devel).
> >>> 	I straced rsyslogd when CPU usage is keeping at 100%.
> >>>
> >>> % time     seconds  usecs/call     calls    errors syscall
> >>> ------ ----------- ----------- --------- --------- ----------------
> >>> 43.47    0.066967        2309        29           read
> >>> 38.28    0.058979         855        69           epoll_wait
> >>> 18.17    0.027994        9331         3           select
> >>>  0.05    0.000081           1       114           gettimeofday
> >>>  0.03    0.000041           0       178        69 recvfrom
> >>>  0.00    0.000000           0         3           recvmsg
> >>> ------ ----------- ----------- --------- --------- ----------------
> >>> 100.00    0.154062                   396        69 total
> >>>
> >>>
> >>> Best,
> >>> Henry
> >>>
> >>> -----Original Message-----
> >>> From: rsyslog-bounces at lists.adiscon.com
> >>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of
> >>> david at lang.hm
> >>> Sent: Tuesday, May 10, 2011 12:35 AM
> >>> To: rsyslog-users
> >>> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to
> >>> Oracle
> >>>
> >>> I accidently deleted your reply, so I am replying to my message
> >>>
> >>> 4-5 messages/secshould not be any problem at all.
> >>>
> >>> since writing a copy of the sql commands to a file doesn't show
> >>> anything strange, the next step will be to try and run it in debug
> >>> mode to see if it logs anything strange as you start having
> problems.
> >>>
> >>> the next step after that will probably be to do a strace of the
> >>> thread that is using all the CPU
> >>>
> >>> I don't see where you mention what version of rsyslog you are
> using.
> >>>
> >>> David Lang
> >>>
> >>> On Mon, 9 May 2011, david at lang.hm wrote:
> >>>
> >>>> Date: Mon, 9 May 2011 10:56:30 -0700 (PDT)
> >>>> From: david at lang.hm
> >>>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> >>>> To: rsyslog-users <rsyslog at lists.adiscon.com>
> >>>> Subject: Re: [rsyslog] help : CPU high load of rsyslog writing to
> >>>> Oracle
> >>>>
> >>>> given that it only happens after a few messages, I would suspect
> >>>> that there is something odd with the messages.
> >>>>
> >>>> could you try writing the messages to a file using the Oracle
> >>>> template and then look and see if there is anything odd about the
> >>>> messages when it goes nuts?
> >>>>
> >>>> the other option is to get a debug dump, but unless it happens
> >>>> right away, that is likely to be a large dump.
> >>>>
> >>>> David Lang
> >>>>
> >>>> On Mon, 9 May 2011, bxu wrote:
> >>>>
> >>>>> Date: Mon, 9 May 2011 10:48:50 -0500
> >>>>> From: bxu <bxu at italkbb.com>
> >>>>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> >>>>> To: rsyslog at lists.adiscon.com
> >>>>> Subject: [rsyslog] help : CPU high load of rsyslog writing to
> >>>>> Oracle
> >>>>>
> >>>>> Hi All,
> >>>>>
> >>>>>
> >>>>> I have configured rsyslog writing to Oracle.The writing was
> >>>>> successful.But after several messages have been made into
> >>>>> Oracle,the CPU use percentage of rsyslogd goes up to 99%-100%.It
> >>>>> is abnormally too
> >>> much high.
> >>>>> There's no problem with the cpu usage when writing to files.
> >>>>>
> >>>>> Here is my rsyslogd.conf:
> >>>>> #====================================================
> >>>>> #*.* >192.168.45.213,rsyslog
> >>>>>
> >>>>> # Use traditional timestamp format $ActionFileDefaultTemplate
> >>>>> RSYSLOG_TraditionalFileFormat
> >>>>>
> >>>>> # Provides kernel logging support (previously done by rklogd)
> >>>>> $ModLoad imklog # Provides support for local system logging (e.g.
> >>>>> via logger command) $ModLoad imuxsock
> >>>>>
> >>>>> # Log all kernel messages to the console.
> >>>>> # Logging much else clutters up the screen.
> >>>>> #kern.* /dev/console
> >>>>>
> >>>>> # Log anything (except mail) of level info or higher.
> >>>>> # Don't log private authentication messages!
> >>>>> *.info;mail.none;authpriv.none;cron.none /var/log/messages
> >>>>>
> >>>>> # The authpriv file has restricted access.
> >>>>> authpriv.* /var/log/secure
> >>>>>
> >>>>> # Log all the mail messages in one place.
> >>>>> mail.* -/var/log/maillog
> >>>>>
> >>>>> # Log cron stuff
> >>>>> cron.* /var/log/cron
> >>>>>
> >>>>> # Everybody gets emergency messages *.emerg *
> >>>>>
> >>>>> # Save news errors of level crit and higher in a special file.
> >>>>> uucp,news.crit /var/log/spooler
> >>>>>
> >>>>> # Save boot messages also to boot.log
> >>>>> local7.* /var/log/boot.log
> >>>>>
> >>>>> #Load External Listender
> >>>>> $modload imudp
> >>>>>
> >>>>>
> >>>>> $UDPServerRun 514
> >>>>> #SPA Messages
> >>>>> #*.* /var/log/linksys.log
> >>>>> $template DynaFile,"/var/log/system-%HOSTNAME%.log"
> >>>>> *.* -?DynaFile
> >>>>>
> >>>>>
> >>>>> ################
> >>>>> #### ORACLE ####
> >>>>> ################
> >>>>>
> >>>>> $ModLoad omoracle
> >>>>> $OmoracleDBUser syslog
> >>>>> $OmoracleDBPassword syslog
> >>>>> $OmoracleDB syslog
> >>>>> $OmoracleBatchSize 1
> >>>>> $OmoracleBatchItemSize 4096
> >>>>> $OmoracleStatementTemplate OmoracleStatement
> >>>>>
> >>>>> $template OmoracleStatement,"INSERT INTO TEST(hostname,message)
> >>>>> VALUES(:hostname,:msg)"
> >>>>> $template TestStmt,"%hostname%%msg%"
> >>>>>
> >>>>> *.*      :omoracle:;TestStmt
> >>>>> #====================================================
> >>>>>
> >>>>> Really appreciate anyone's help!
> >>>>>
> >>>>> Best
> >>>>>
> >>>>> _______________________________________________
> >>>>> rsyslog mailing list
> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>> http://www.rsyslog.com
> >>>>>
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.com
> >>>>
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >>>
> >>> --
> >>> This message has been scanned for viruses and dangerous content by
> >>> MailScanner, and is believed to be clean.
> >>>
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >>>
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >>
> >> --
> >> This message has been scanned for viruses and dangerous content by
> >> MailScanner, and is believed to be clean.
> >>
> > --
> > This message has been scanned for viruses and dangerous content by
> > MailScanner, and is believed to be clean.
> >
> >
> --
> This message has been scanned for viruses and dangerous content by
> MailScanner, and is believed to be clean.


From tobias.dinse at stegbauer.info  Thu May 12 17:52:58 2011
From: tobias.dinse at stegbauer.info (Tobias Dinse)
Date: Thu, 12 May 2011 17:52:58 +0200
Subject: [rsyslog] missing hostnames
Message-ID: <4DCC025A.8060007@stegbauer.info>

Hi,

i tried to send the Syslog of our Netgear Switch to a Rsyslog Server but 
the hostname in /var/log/syslog of the Switch is missing. I configured 
the name on the Webinterface. All other Computers hostnames are shown. 
Any idea?

thanks and

regards

Tobias

-- 
# Stegbauer Datawork
# Tobias Dinse
# Oberjulbachring 9, 84387 Julbach


From Ole.Rahn at t-systems.com  Wed May 18 12:55:56 2011
From: Ole.Rahn at t-systems.com (Ole.Rahn at t-systems.com)
Date: Wed, 18 May 2011 12:55:56 +0200
Subject: [rsyslog] Heavy stability problems when using TLS
Message-ID: <50CCFFD1B9C6424389383F7A13A03446013CFA30CFE3@HE101451.emea1.cds.t-internal.com>

Hi,

I guess, it is not a new topic, but I could not find any solution, yet. As always, deadlines are near and things seems to get worse and worse, so I thought I give the list a try...

We aim to use rsyslog for a centralized log server and log transmission should be as safe and secure as possible. In our case, that means we want to use syslog via TCP/TLS wherever possible and fall back to TCP or UDP for those nodes that don't support TLS (or even TCP).

I sent a post in April, already, in order to find out, if it is possible to run plain TCP and TCP/TLS on the same server instance (seems it does not work, so far). At this time, we managed to connect a few nodes via TCP/TLS, but the server and apparently even the clients keep crashing several times a day, causing a loss messages.

For obvious reasons, this is not what we want, at all.

I already found a bug report, that seems to point in the right direction, but I don't see a solution even though the status is "resolved":

*         http://bugzilla.adiscon.com/show_bug.cgi?id=194

*         and its spin off http://bugzilla.adiscon.com/show_bug.cgi?id=222

What was the solution?
What can be done to prevent the crashes?

I can provide whatever information you want (at least in regard to the server), but I would rather not post our config to the public.

The server is using rsyslog 5.8.0 on Solaris and the clients that are currently connected run version 4.4.2 on Red Hat.

Best regards
Ole




From kaiwang.chen at gmail.com  Wed May 18 14:01:49 2011
From: kaiwang.chen at gmail.com (Kaiwang Chen)
Date: Wed, 18 May 2011 20:01:49 +0800
Subject: [rsyslog] HOSTNAME and programname extraction failure,
 when rsyslog as receiver, stock syslog as sender
Message-ID: <BANLkTimy_GcSTjVu+DUZKO6oZhuGiOy+UQ@mail.gmail.com>

Hello,

I was trying to configure rsyslog(rsyslog-3.22.1-3.el5_5.1) as
receiver, stock syslog(sysklogd-1.4.1-46.el5) as sender.

The rsyslogd listened on udp/514, and used dynamic filenames with
protocol23  message formatting:
$ModLoad imudp
$UDPServerRun 514
$ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
$template DynFile,"/var/log/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/rsyslog.log"
*.*                                                  ?DynFile

The sender generated several entries(3rd, 4th) violating the
"syslogtag: message" convention
May 18 19:40:17 dns1 kernel: Kernel logging (proc) stopped.
May 18 19:40:17 dns1 kernel: Kernel log daemon terminating.
May 18 19:40:18 dns1 exiting on signal 15
May 18 19:40:18 dns1 syslogd 1.4.1: restart.
May 18 19:40:18 dns1 kernel: klogd 1.4.1, log source = /proc/kmsg started.

Take 3rd entry for example, the receiver mistook 'exiting' and 'on' to
be %HOSTNAME% and %programname%:
<6>1 2011-05-18T19:40:12.592370+08:00 bogon kernel - - -  Kernel
logging (proc) stopped.
<6>1 2011-05-18T19:40:12.592693+08:00 bogon kernel - - -  Kernel log
daemon terminating.
<46>1 2011-05-18T19:40:13.697115+08:00 exiting on - - -  signal 15
<46>1 2011-05-18T19:40:13.806302+08:00 syslogd 1.4.1 - - -  restart.
<6>1 2011-05-18T19:40:13.811331+08:00 bogon kernel - - -  klogd 1.4.1,
log source = /proc/kmsg started.
So, it went to /var/log/hosts/exiting/2011/05/18/rsyslog.log, and
that's definitely the wrong place.

How to deal with this case?


Thanks,
Kaiwang

From david at lang.hm  Wed May 18 18:59:57 2011
From: david at lang.hm (david at lang.hm)
Date: Wed, 18 May 2011 09:59:57 -0700 (PDT)
Subject: [rsyslog] HOSTNAME and programname extraction failure,
 when rsyslog as receiver, stock syslog as sender
In-Reply-To: <BANLkTimy_GcSTjVu+DUZKO6oZhuGiOy+UQ@mail.gmail.com>
References: <BANLkTimy_GcSTjVu+DUZKO6oZhuGiOy+UQ@mail.gmail.com>
Message-ID: <alpine.DEB.2.02.1105180955320.19924@asgard.lang.hm>

On Wed, 18 May 2011, Kaiwang Chen wrote:

> Hello,
>
> I was trying to configure rsyslog(rsyslog-3.22.1-3.el5_5.1) as
> receiver, stock syslog(sysklogd-1.4.1-46.el5) as sender.
>
> The rsyslogd listened on udp/514, and used dynamic filenames with
> protocol23  message formatting:
> $ModLoad imudp
> $UDPServerRun 514
> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
> $template DynFile,"/var/log/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/rsyslog.log"
> *.*                                                  ?DynFile
>
> The sender generated several entries(3rd, 4th) violating the
> "syslogtag: message" convention
> May 18 19:40:17 dns1 kernel: Kernel logging (proc) stopped.
> May 18 19:40:17 dns1 kernel: Kernel log daemon terminating.
> May 18 19:40:18 dns1 exiting on signal 15
> May 18 19:40:18 dns1 syslogd 1.4.1: restart.
> May 18 19:40:18 dns1 kernel: klogd 1.4.1, log source = /proc/kmsg started.
>
> Take 3rd entry for example, the receiver mistook 'exiting' and 'on' to
> be %HOSTNAME% and %programname%:

my guess is that what was sent was actually

<46>May 18 19:40:18 exiting on signal 15

unfortunantly, without logging %raw% somewhere or sniffing the traffic 
it's impossible to tell

if you can catch this (which shouldn't be too hard) we can look at the 
traffic and tell if there is any way to identify this.

If I am right about what was sent, there is no way for a listener to tell 
that 'exiting' is not a hostname.

the best that would be possible would be to make a blacklist of things 
that show up that are not hostnames and make something to notice this and 
put in the fromhost-ip instead.

David Lang

> <6>1 2011-05-18T19:40:12.592370+08:00 bogon kernel - - -  Kernel
> logging (proc) stopped.
> <6>1 2011-05-18T19:40:12.592693+08:00 bogon kernel - - -  Kernel log
> daemon terminating.
> <46>1 2011-05-18T19:40:13.697115+08:00 exiting on - - -  signal 15
> <46>1 2011-05-18T19:40:13.806302+08:00 syslogd 1.4.1 - - -  restart.
> <6>1 2011-05-18T19:40:13.811331+08:00 bogon kernel - - -  klogd 1.4.1,
> log source = /proc/kmsg started.
> So, it went to /var/log/hosts/exiting/2011/05/18/rsyslog.log, and
> that's definitely the wrong place.
>
> How to deal with this case?
>
>
> Thanks,
> Kaiwang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From kaiwang.chen at gmail.com  Wed May 18 19:59:53 2011
From: kaiwang.chen at gmail.com (Kaiwang Chen)
Date: Thu, 19 May 2011 01:59:53 +0800
Subject: [rsyslog] HOSTNAME and programname extraction failure,
 when rsyslog as receiver, stock syslog as sender
In-Reply-To: <alpine.DEB.2.02.1105180955320.19924@asgard.lang.hm>
References: <BANLkTimy_GcSTjVu+DUZKO6oZhuGiOy+UQ@mail.gmail.com>
	<alpine.DEB.2.02.1105180955320.19924@asgard.lang.hm>
Message-ID: <BANLkTimkQ6fnjQ5Qu653TLi-_M2fvvpZbQ@mail.gmail.com>

With the following configuration snippet:

$template rawfmt,"%rawmsg%\n"
*.*    /var/log/rawmessages;rawfmt

And the following command to capture packets:

tcpdump -i eth1 udp port 514 -nnn -s0 -vvv -w syslog.pcap

It turned out that the raw messages and captured UDP payload(one line
per packet) were the same:

<6>kernel: Kernel logging (proc) stopped.
<6>kernel: Kernel log daemon terminating.
<46>exiting on signal 15
<46>syslogd 1.4.1: restart.
<6>kernel: klogd 1.4.1, log source = /proc/kmsg started.

In the case of 3rd entry, the payload in hex string is:

3c 34 36 3e 65 78 69 74  69 6e 67 20 6f 6e 20 73 69 67 6e 61 6c 20 31 35  0a
 <  4   6  >   e  x   i    t    i    n   g   _  o   n  _   s   i   g
n   a  l    _   1   5   \n

where spaces are represented by underscores.


Thanks,
Kaiwang

2011/5/19  <david at lang.hm>:
> On Wed, 18 May 2011, Kaiwang Chen wrote:
>
>> Hello,
>>
>> I was trying to configure rsyslog(rsyslog-3.22.1-3.el5_5.1) as
>> receiver, stock syslog(sysklogd-1.4.1-46.el5) as sender.
>>
>> The rsyslogd listened on udp/514, and used dynamic filenames with
>> protocol23 ?message formatting:
>> $ModLoad imudp
>> $UDPServerRun 514
>> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
>> $template
>> DynFile,"/var/log/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/rsyslog.log"
>> *.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??DynFile
>>
>> The sender generated several entries(3rd, 4th) violating the
>> "syslogtag: message" convention
>> May 18 19:40:17 dns1 kernel: Kernel logging (proc) stopped.
>> May 18 19:40:17 dns1 kernel: Kernel log daemon terminating.
>> May 18 19:40:18 dns1 exiting on signal 15
>> May 18 19:40:18 dns1 syslogd 1.4.1: restart.
>> May 18 19:40:18 dns1 kernel: klogd 1.4.1, log source = /proc/kmsg started.
>>
>> Take 3rd entry for example, the receiver mistook 'exiting' and 'on' to
>> be %HOSTNAME% and %programname%:
>
> my guess is that what was sent was actually
>
> <46>May 18 19:40:18 exiting on signal 15
>
> unfortunantly, without logging %raw% somewhere or sniffing the traffic it's
> impossible to tell
>
> if you can catch this (which shouldn't be too hard) we can look at the
> traffic and tell if there is any way to identify this.
>
> If I am right about what was sent, there is no way for a listener to tell
> that 'exiting' is not a hostname.
>
> the best that would be possible would be to make a blacklist of things that
> show up that are not hostnames and make something to notice this and put in
> the fromhost-ip instead.
>
> David Lang
>
>> <6>1 2011-05-18T19:40:12.592370+08:00 bogon kernel - - - ?Kernel
>> logging (proc) stopped.
>> <6>1 2011-05-18T19:40:12.592693+08:00 bogon kernel - - - ?Kernel log
>> daemon terminating.
>> <46>1 2011-05-18T19:40:13.697115+08:00 exiting on - - - ?signal 15
>> <46>1 2011-05-18T19:40:13.806302+08:00 syslogd 1.4.1 - - - ?restart.
>> <6>1 2011-05-18T19:40:13.811331+08:00 bogon kernel - - - ?klogd 1.4.1,
>> log source = /proc/kmsg started.
>> So, it went to /var/log/hosts/exiting/2011/05/18/rsyslog.log, and
>> that's definitely the wrong place.
>>
>> How to deal with this case?
>>
>>
>> Thanks,
>> Kaiwang
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From rory at ooma.com  Thu May 19 01:44:03 2011
From: rory at ooma.com (Rory Toma)
Date: Wed, 18 May 2011 16:44:03 -0700
Subject: [rsyslog] Question on host failover
Message-ID: <4DD459C3.9020301@ooma.com>

I have the following for my clients' rsyslog.conf files:

$DefaultNetStreamDriverCAFile /etc/ca.pem

$DefaultNetStreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode anon

$ActionResumeInterval 29

$WorkDirectory /var/log

$ModLoad imuxsock
$SystemLogSocketName /var/log/log
$OptimizeForUniprocessor on

*.* @@<machine>:110
$ActionExecOnlyWhenPreviousIsSuspended on
*.* @@<machine>:143
$ActionExecOnlyWhenPreviousIsSuspended off



If I block port 110 from the client, I would expect that it would 
failover to port 143. I am not seeing this. Do I have to do something 
different here? It doesn't work if I remove the ActionExec directive, 
either. In either case, it seems to keep sending to port 110, no matter 
what. In this case, <machine> is the same physical machine, with the 
same IP address, it's just a different syslog receiver process.

This is rsyslog-5.6.2

thx

From rgerhards at hq.adiscon.com  Thu May 19 07:31:40 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 19 May 2011 07:31:40 +0200
Subject: [rsyslog] Question on host failover
In-Reply-To: <4DD459C3.9020301@ooma.com>
References: <4DD459C3.9020301@ooma.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DE198@GRFEXC.intern.adiscon.com>

This is a bug in older versions prior to 5.8.0.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rory Toma
> Sent: Thursday, May 19, 2011 1:44 AM
> To: rsyslog-users
> Subject: [rsyslog] Question on host failover
> 
> I have the following for my clients' rsyslog.conf files:
> 
> $DefaultNetStreamDriverCAFile /etc/ca.pem
> 
> $DefaultNetStreamDriver gtls
> $ActionSendStreamDriverMode 1
> $ActionSendStreamDriverAuthMode anon
> 
> $ActionResumeInterval 29
> 
> $WorkDirectory /var/log
> 
> $ModLoad imuxsock
> $SystemLogSocketName /var/log/log
> $OptimizeForUniprocessor on
> 
> *.* @@<machine>:110
> $ActionExecOnlyWhenPreviousIsSuspended on
> *.* @@<machine>:143
> $ActionExecOnlyWhenPreviousIsSuspended off
> 
> 
> 
> If I block port 110 from the client, I would expect that it would
> failover to port 143. I am not seeing this. Do I have to do something
> different here? It doesn't work if I remove the ActionExec directive,
> either. In either case, it seems to keep sending to port 110, no matter
> what. In this case, <machine> is the same physical machine, with the
> same IP address, it's just a different syslog receiver process.
> 
> This is rsyslog-5.6.2
> 
> thx
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Thu May 19 07:50:07 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 19 May 2011 07:50:07 +0200
Subject: [rsyslog] HOSTNAME and programname extraction failure,
	when rsyslog as receiver, stock syslog as sender
In-Reply-To: <BANLkTimy_GcSTjVu+DUZKO6oZhuGiOy+UQ@mail.gmail.com>
References: <BANLkTimy_GcSTjVu+DUZKO6oZhuGiOy+UQ@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DE199@GRFEXC.intern.adiscon.com>

I addition to what David already said:

http://www.rsyslog.com/doc/syslog_parsing.html

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
> Sent: Wednesday, May 18, 2011 2:02 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] HOSTNAME and programname extraction failure, when
> rsyslog as receiver, stock syslog as sender
> 
> Hello,
> 
> I was trying to configure rsyslog(rsyslog-3.22.1-3.el5_5.1) as
> receiver, stock syslog(sysklogd-1.4.1-46.el5) as sender.
> 
> The rsyslogd listened on udp/514, and used dynamic filenames with
> protocol23  message formatting:
> $ModLoad imudp
> $UDPServerRun 514
> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
> $template
> DynFile,"/var/log/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/rsyslog.log"
> *.*                                                  ?DynFile
> 
> The sender generated several entries(3rd, 4th) violating the
> "syslogtag: message" convention
> May 18 19:40:17 dns1 kernel: Kernel logging (proc) stopped.
> May 18 19:40:17 dns1 kernel: Kernel log daemon terminating.
> May 18 19:40:18 dns1 exiting on signal 15
> May 18 19:40:18 dns1 syslogd 1.4.1: restart.
> May 18 19:40:18 dns1 kernel: klogd 1.4.1, log source = /proc/kmsg
> started.
> 
> Take 3rd entry for example, the receiver mistook 'exiting' and 'on' to
> be %HOSTNAME% and %programname%:
> <6>1 2011-05-18T19:40:12.592370+08:00 bogon kernel - - -  Kernel
> logging (proc) stopped.
> <6>1 2011-05-18T19:40:12.592693+08:00 bogon kernel - - -  Kernel log
> daemon terminating.
> <46>1 2011-05-18T19:40:13.697115+08:00 exiting on - - -  signal 15
> <46>1 2011-05-18T19:40:13.806302+08:00 syslogd 1.4.1 - - -  restart.
> <6>1 2011-05-18T19:40:13.811331+08:00 bogon kernel - - -  klogd 1.4.1,
> log source = /proc/kmsg started.
> So, it went to /var/log/hosts/exiting/2011/05/18/rsyslog.log, and
> that's definitely the wrong place.
> 
> How to deal with this case?
> 
> 
> Thanks,
> Kaiwang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From tbergfeld at hq.adiscon.com  Thu May 19 13:27:58 2011
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Thu, 19 May 2011 13:27:58 +0200
Subject: [rsyslog] rsyslog 5.8.1 (v5-stable) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DE1A0@GRFEXC.intern.adiscon.com>

This is a maintenance release offering a set of bug fixes. It is suggested
that users of the v5-stable branch update to this release.

ChangeLog:

http://www.rsyslog.com/changelog-for-5-8-1-v5-stable/

Download:

http://www.rsyslog.com/rsyslog-5-8-1-v5-stable/

As always, feedback is appreciated.

Best regards,
Tom Bergfeld
--

Support
=======
Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html.

From kaiwang.chen at gmail.com  Thu May 19 18:10:27 2011
From: kaiwang.chen at gmail.com (Kaiwang Chen)
Date: Fri, 20 May 2011 00:10:27 +0800
Subject: [rsyslog] HOSTNAME and programname extraction failure,
 when rsyslog as receiver, stock syslog as sender
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DE199@GRFEXC.intern.adiscon.com>
References: <BANLkTimy_GcSTjVu+DUZKO6oZhuGiOy+UQ@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA71DE199@GRFEXC.intern.adiscon.com>
Message-ID: <BANLkTinttx=2TEhW=KsczJhwrzczCxEPUg@mail.gmail.com>

Looks like it's the stock sysklog in CentOS5.6 that generated bad
formatted logs(without timestamp and syslogtag), and with a rsyslog
3.x installation I have no choice but not use stock sysklog.

Thanks,
Kaiwang

2011/5/19 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> I addition to what David already said:
>
> http://www.rsyslog.com/doc/syslog_parsing.html
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
>> Sent: Wednesday, May 18, 2011 2:02 PM
>> To: rsyslog at lists.adiscon.com
>> Subject: [rsyslog] HOSTNAME and programname extraction failure, when
>> rsyslog as receiver, stock syslog as sender
>>
>> Hello,
>>
>> I was trying to configure rsyslog(rsyslog-3.22.1-3.el5_5.1) as
>> receiver, stock syslog(sysklogd-1.4.1-46.el5) as sender.
>>
>> The rsyslogd listened on udp/514, and used dynamic filenames with
>> protocol23 ?message formatting:
>> $ModLoad imudp
>> $UDPServerRun 514
>> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
>> $template
>> DynFile,"/var/log/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/rsyslog.log"
>> *.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??DynFile
>>
>> The sender generated several entries(3rd, 4th) violating the
>> "syslogtag: message" convention
>> May 18 19:40:17 dns1 kernel: Kernel logging (proc) stopped.
>> May 18 19:40:17 dns1 kernel: Kernel log daemon terminating.
>> May 18 19:40:18 dns1 exiting on signal 15
>> May 18 19:40:18 dns1 syslogd 1.4.1: restart.
>> May 18 19:40:18 dns1 kernel: klogd 1.4.1, log source = /proc/kmsg
>> started.
>>
>> Take 3rd entry for example, the receiver mistook 'exiting' and 'on' to
>> be %HOSTNAME% and %programname%:
>> <6>1 2011-05-18T19:40:12.592370+08:00 bogon kernel - - - ?Kernel
>> logging (proc) stopped.
>> <6>1 2011-05-18T19:40:12.592693+08:00 bogon kernel - - - ?Kernel log
>> daemon terminating.
>> <46>1 2011-05-18T19:40:13.697115+08:00 exiting on - - - ?signal 15
>> <46>1 2011-05-18T19:40:13.806302+08:00 syslogd 1.4.1 - - - ?restart.
>> <6>1 2011-05-18T19:40:13.811331+08:00 bogon kernel - - - ?klogd 1.4.1,
>> log source = /proc/kmsg started.
>> So, it went to /var/log/hosts/exiting/2011/05/18/rsyslog.log, and
>> that's definitely the wrong place.
>>
>> How to deal with this case?
>>
>>
>> Thanks,
>> Kaiwang
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From bxu at italkbb.com  Thu May 19 18:15:09 2011
From: bxu at italkbb.com (Henry Xu)
Date: Thu, 19 May 2011 11:15:09 -0500
Subject: [rsyslog] rsyslog 5.8.1 (v5-stable) released
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DE1A0@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA71DE1A0@GRFEXC.intern.adiscon.com>
Message-ID: <000601cc163f$f0a5cc10$d1f16430$@italkbb.com>

Still not fix the CPU 100%'s problem when writing to Oracle.

-----Original Message-----
From: rsyslog-bounces at lists.adiscon.com
[mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Tom Bergfeld
Sent: Thursday, May 19, 2011 06:28
To: rsyslog-users
Subject: [rsyslog] rsyslog 5.8.1 (v5-stable) released

This is a maintenance release offering a set of bug fixes. It is suggested
that users of the v5-stable branch update to this release.

ChangeLog:

http://www.rsyslog.com/changelog-for-5-8-1-v5-stable/

Download:

http://www.rsyslog.com/rsyslog-5-8-1-v5-stable/

As always, feedback is appreciated.

Best regards,
Tom Bergfeld
--

Support
=======
Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help
finance continued maintenance.  Adiscon GmbH, a privately held German
company, is currently funding rsyslog development. We are always looking for
interesting development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

--
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.


From rgerhards at hq.adiscon.com  Thu May 19 18:49:14 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 19 May 2011 18:49:14 +0200
Subject: [rsyslog] rsyslog 5.8.1 (v5-stable) released
In-Reply-To: <000601cc163f$f0a5cc10$d1f16430$@italkbb.com>
References: <9B6E2A8877C38245BFB15CC491A11DA71DE1A0@GRFEXC.intern.adiscon.com>
	<000601cc163f$f0a5cc10$d1f16430$@italkbb.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DE1A4@GRFEXC.intern.adiscon.com>

As I said, it's a contributed module I have absolutely no ability to modify
that source... (neither from a knowledge nor from a equipment PoV).

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Henry Xu
> Sent: Thursday, May 19, 2011 6:15 PM
> To: 'rsyslog-users'
> Subject: Re: [rsyslog] rsyslog 5.8.1 (v5-stable) released
> 
> Still not fix the CPU 100%'s problem when writing to Oracle.
> 
> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Tom Bergfeld
> Sent: Thursday, May 19, 2011 06:28
> To: rsyslog-users
> Subject: [rsyslog] rsyslog 5.8.1 (v5-stable) released
> 
> This is a maintenance release offering a set of bug fixes. It is
> suggested
> that users of the v5-stable branch update to this release.
> 
> ChangeLog:
> 
> http://www.rsyslog.com/changelog-for-5-8-1-v5-stable/
> 
> Download:
> 
> http://www.rsyslog.com/rsyslog-5-8-1-v5-stable/
> 
> As always, feedback is appreciated.
> 
> Best regards,
> Tom Bergfeld
> --
> 
> Support
> =======
> Improving rsyslog is costly, but you can help!  We are looking for
> organizations that find rsyslog useful and wish to contribute back.
> You can
> contribute by reporting bugs, improve the software, or donate money or
> equipment.
> 
> Commercial support contracts for rsyslog are available, and they help
> finance continued maintenance.  Adiscon GmbH, a privately held German
> company, is currently funding rsyslog development. We are always
> looking for
> interesting development projects. For details on how to help, please
> see
> http://www.rsyslog.com/doc-how2help.html.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 
> --
> This message has been scanned for viruses and dangerous content by
> MailScanner, and is believed to be clean.
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Thu May 19 18:49:51 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 19 May 2011 18:49:51 +0200
Subject: [rsyslog] rsyslog 5.8.1 (v5-stable) released
In-Reply-To: <000601cc163f$f0a5cc10$d1f16430$@italkbb.com>
References: <9B6E2A8877C38245BFB15CC491A11DA71DE1A0@GRFEXC.intern.adiscon.com>
	<000601cc163f$f0a5cc10$d1f16430$@italkbb.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DE1A5@GRFEXC.intern.adiscon.com>

Oh, and to provide some better advise. Have you tried to connect to Oracle
via omlibdbi?
Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Henry Xu
> Sent: Thursday, May 19, 2011 6:15 PM
> To: 'rsyslog-users'
> Subject: Re: [rsyslog] rsyslog 5.8.1 (v5-stable) released
> 
> Still not fix the CPU 100%'s problem when writing to Oracle.
> 
> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Tom Bergfeld
> Sent: Thursday, May 19, 2011 06:28
> To: rsyslog-users
> Subject: [rsyslog] rsyslog 5.8.1 (v5-stable) released
> 
> This is a maintenance release offering a set of bug fixes. It is
> suggested
> that users of the v5-stable branch update to this release.
> 
> ChangeLog:
> 
> http://www.rsyslog.com/changelog-for-5-8-1-v5-stable/
> 
> Download:
> 
> http://www.rsyslog.com/rsyslog-5-8-1-v5-stable/
> 
> As always, feedback is appreciated.
> 
> Best regards,
> Tom Bergfeld
> --
> 
> Support
> =======
> Improving rsyslog is costly, but you can help!  We are looking for
> organizations that find rsyslog useful and wish to contribute back.
> You can
> contribute by reporting bugs, improve the software, or donate money or
> equipment.
> 
> Commercial support contracts for rsyslog are available, and they help
> finance continued maintenance.  Adiscon GmbH, a privately held German
> company, is currently funding rsyslog development. We are always
> looking for
> interesting development projects. For details on how to help, please
> see
> http://www.rsyslog.com/doc-how2help.html.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 
> --
> This message has been scanned for viruses and dangerous content by
> MailScanner, and is believed to be clean.
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Thu May 19 18:53:41 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 19 May 2011 18:53:41 +0200
Subject: [rsyslog] rsyslog 5.8.1 (v5-stable) released
In-Reply-To: <000601cc163f$f0a5cc10$d1f16430$@italkbb.com>
References: <9B6E2A8877C38245BFB15CC491A11DA71DE1A0@GRFEXC.intern.adiscon.com>
	<000601cc163f$f0a5cc10$d1f16430$@italkbb.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DE1A6@GRFEXC.intern.adiscon.com>

Wait... Now my memory comes back. This is probably a problem in the core. I
asked you for some debugging information on May,12th, which I never received.
Once I get it, I can see if it is rsyslog core related. However, this may
require some further cooperation on your part.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Henry Xu
> Sent: Thursday, May 19, 2011 6:15 PM
> To: 'rsyslog-users'
> Subject: Re: [rsyslog] rsyslog 5.8.1 (v5-stable) released
> 
> Still not fix the CPU 100%'s problem when writing to Oracle.
> 
> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Tom Bergfeld
> Sent: Thursday, May 19, 2011 06:28
> To: rsyslog-users
> Subject: [rsyslog] rsyslog 5.8.1 (v5-stable) released
> 
> This is a maintenance release offering a set of bug fixes. It is
> suggested
> that users of the v5-stable branch update to this release.
> 
> ChangeLog:
> 
> http://www.rsyslog.com/changelog-for-5-8-1-v5-stable/
> 
> Download:
> 
> http://www.rsyslog.com/rsyslog-5-8-1-v5-stable/
> 
> As always, feedback is appreciated.
> 
> Best regards,
> Tom Bergfeld
> --
> 
> Support
> =======
> Improving rsyslog is costly, but you can help!  We are looking for
> organizations that find rsyslog useful and wish to contribute back.
> You can
> contribute by reporting bugs, improve the software, or donate money or
> equipment.
> 
> Commercial support contracts for rsyslog are available, and they help
> finance continued maintenance.  Adiscon GmbH, a privately held German
> company, is currently funding rsyslog development. We are always
> looking for
> interesting development projects. For details on how to help, please
> see
> http://www.rsyslog.com/doc-how2help.html.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 
> --
> This message has been scanned for viruses and dangerous content by
> MailScanner, and is believed to be clean.
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From bxu at italkbb.com  Thu May 19 18:57:30 2011
From: bxu at italkbb.com (Henry Xu)
Date: Thu, 19 May 2011 11:57:30 -0500
Subject: [rsyslog] rsyslog 5.8.1 (v5-stable) released
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DE1A6@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA71DE1A0@GRFEXC.intern.adiscon.com>	<000601cc163f$f0a5cc10$d1f16430$@italkbb.com>
	<9B6E2A8877C38245BFB15CC491A11DA71DE1A6@GRFEXC.intern.adiscon.com>
Message-ID: <000701cc1645$da679e00$8f36da00$@italkbb.com>

Hi Rainer,
	I haven't tried libdbi. I'm busy with other stuff so far. Maybe
sometime I will try the omlibdbi and do the debug.

	Thanks a lot for your advice.

Best,
Henry

-----Original Message-----
From: rsyslog-bounces at lists.adiscon.com
[mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
Sent: Thursday, May 19, 2011 11:54
To: rsyslog-users
Subject: Re: [rsyslog] rsyslog 5.8.1 (v5-stable) released

Wait... Now my memory comes back. This is probably a problem in the core. I
asked you for some debugging information on May,12th, which I never
received.
Once I get it, I can see if it is rsyslog core related. However, this may
require some further cooperation on your part.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- 
> bounces at lists.adiscon.com] On Behalf Of Henry Xu
> Sent: Thursday, May 19, 2011 6:15 PM
> To: 'rsyslog-users'
> Subject: Re: [rsyslog] rsyslog 5.8.1 (v5-stable) released
> 
> Still not fix the CPU 100%'s problem when writing to Oracle.
> 
> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Tom Bergfeld
> Sent: Thursday, May 19, 2011 06:28
> To: rsyslog-users
> Subject: [rsyslog] rsyslog 5.8.1 (v5-stable) released
> 
> This is a maintenance release offering a set of bug fixes. It is 
> suggested that users of the v5-stable branch update to this release.
> 
> ChangeLog:
> 
> http://www.rsyslog.com/changelog-for-5-8-1-v5-stable/
> 
> Download:
> 
> http://www.rsyslog.com/rsyslog-5-8-1-v5-stable/
> 
> As always, feedback is appreciated.
> 
> Best regards,
> Tom Bergfeld
> --
> 
> Support
> =======
> Improving rsyslog is costly, but you can help!  We are looking for 
> organizations that find rsyslog useful and wish to contribute back.
> You can
> contribute by reporting bugs, improve the software, or donate money or 
> equipment.
> 
> Commercial support contracts for rsyslog are available, and they help 
> finance continued maintenance.  Adiscon GmbH, a privately held German 
> company, is currently funding rsyslog development. We are always 
> looking for interesting development projects. For details on how to 
> help, please see http://www.rsyslog.com/doc-how2help.html.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 
> --
> This message has been scanned for viruses and dangerous content by 
> MailScanner, and is believed to be clean.
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

--
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.


From rgerhards at hq.adiscon.com  Thu May 19 18:58:58 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 19 May 2011 18:58:58 +0200
Subject: [rsyslog] rsyslog 5.8.1 (v5-stable) released
In-Reply-To: <000701cc1645$da679e00$8f36da00$@italkbb.com>
References: <9B6E2A8877C38245BFB15CC491A11DA71DE1A0@GRFEXC.intern.adiscon.com>	<000601cc163f$f0a5cc10$d1f16430$@italkbb.com><9B6E2A8877C38245BFB15CC491A11DA71DE1A6@GRFEXC.intern.adiscon.com>
	<000701cc1645$da679e00$8f36da00$@italkbb.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DE1A7@GRFEXC.intern.adiscon.com>



> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Henry Xu
> Sent: Thursday, May 19, 2011 6:58 PM
> To: 'rsyslog-users'
> Subject: Re: [rsyslog] rsyslog 5.8.1 (v5-stable) released
> 
> Hi Rainer,
> 	I haven't tried libdbi. I'm busy with other stuff so far.

*This* problem I know very well. So if the debug is not important enough for
you at this time, you can for sure bet that it is not important for me either
;)

Rainer
 Maybe
> sometime I will try the omlibdbi and do the debug.
> 
> 	Thanks a lot for your advice.
> 
> Best,
> Henry
> 
> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Thursday, May 19, 2011 11:54
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 5.8.1 (v5-stable) released
> 
> Wait... Now my memory comes back. This is probably a problem in the
> core. I
> asked you for some debugging information on May,12th, which I never
> received.
> Once I get it, I can see if it is rsyslog core related. However, this
> may
> require some further cooperation on your part.
> 
> Rainer
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Henry Xu
> > Sent: Thursday, May 19, 2011 6:15 PM
> > To: 'rsyslog-users'
> > Subject: Re: [rsyslog] rsyslog 5.8.1 (v5-stable) released
> >
> > Still not fix the CPU 100%'s problem when writing to Oracle.
> >
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com
> > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Tom Bergfeld
> > Sent: Thursday, May 19, 2011 06:28
> > To: rsyslog-users
> > Subject: [rsyslog] rsyslog 5.8.1 (v5-stable) released
> >
> > This is a maintenance release offering a set of bug fixes. It is
> > suggested that users of the v5-stable branch update to this release.
> >
> > ChangeLog:
> >
> > http://www.rsyslog.com/changelog-for-5-8-1-v5-stable/
> >
> > Download:
> >
> > http://www.rsyslog.com/rsyslog-5-8-1-v5-stable/
> >
> > As always, feedback is appreciated.
> >
> > Best regards,
> > Tom Bergfeld
> > --
> >
> > Support
> > =======
> > Improving rsyslog is costly, but you can help!  We are looking for
> > organizations that find rsyslog useful and wish to contribute back.
> > You can
> > contribute by reporting bugs, improve the software, or donate money
> or
> > equipment.
> >
> > Commercial support contracts for rsyslog are available, and they help
> > finance continued maintenance.  Adiscon GmbH, a privately held German
> > company, is currently funding rsyslog development. We are always
> > looking for interesting development projects. For details on how to
> > help, please see http://www.rsyslog.com/doc-how2help.html.
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> > --
> > This message has been scanned for viruses and dangerous content by
> > MailScanner, and is believed to be clean.
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 
> --
> This message has been scanned for viruses and dangerous content by
> MailScanner, and is believed to be clean.
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From david at lang.hm  Fri May 20 03:25:21 2011
From: david at lang.hm (david at lang.hm)
Date: Thu, 19 May 2011 18:25:21 -0700 (PDT)
Subject: [rsyslog] HOSTNAME and programname extraction failure,
 when rsyslog as receiver, stock syslog as sender
In-Reply-To: <BANLkTinttx=2TEhW=KsczJhwrzczCxEPUg@mail.gmail.com>
References: <BANLkTimy_GcSTjVu+DUZKO6oZhuGiOy+UQ@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA71DE199@GRFEXC.intern.adiscon.com>
	<BANLkTinttx=2TEhW=KsczJhwrzczCxEPUg@mail.gmail.com>
Message-ID: <alpine.DEB.2.02.1105191824300.30095@asgard.lang.hm>

if it's being generated without a timestamp, rsyslog should be able to 
detect that. can you get a raw log and verify that?

David Lang

On Fri, 20 May 2011, Kaiwang Chen wrote:

> Date: Fri, 20 May 2011 00:10:27 +0800
> From: Kaiwang Chen <kaiwang.chen at gmail.com>
> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> To: rsyslog-users <rsyslog at lists.adiscon.com>
> Subject: Re: [rsyslog] HOSTNAME and programname extraction failure,
>     when rsyslog as receiver, stock syslog as sender
> 
> Looks like it's the stock sysklog in CentOS5.6 that generated bad
> formatted logs(without timestamp and syslogtag), and with a rsyslog
> 3.x installation I have no choice but not use stock sysklog.
>
> Thanks,
> Kaiwang
>
> 2011/5/19 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>> I addition to what David already said:
>>
>> http://www.rsyslog.com/doc/syslog_parsing.html
>>
>> Rainer
>>
>>> -----Original Message-----
>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
>>> Sent: Wednesday, May 18, 2011 2:02 PM
>>> To: rsyslog at lists.adiscon.com
>>> Subject: [rsyslog] HOSTNAME and programname extraction failure, when
>>> rsyslog as receiver, stock syslog as sender
>>>
>>> Hello,
>>>
>>> I was trying to configure rsyslog(rsyslog-3.22.1-3.el5_5.1) as
>>> receiver, stock syslog(sysklogd-1.4.1-46.el5) as sender.
>>>
>>> The rsyslogd listened on udp/514, and used dynamic filenames with
>>> protocol23 ?message formatting:
>>> $ModLoad imudp
>>> $UDPServerRun 514
>>> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
>>> $template
>>> DynFile,"/var/log/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/rsyslog.log"
>>> *.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??DynFile
>>>
>>> The sender generated several entries(3rd, 4th) violating the
>>> "syslogtag: message" convention
>>> May 18 19:40:17 dns1 kernel: Kernel logging (proc) stopped.
>>> May 18 19:40:17 dns1 kernel: Kernel log daemon terminating.
>>> May 18 19:40:18 dns1 exiting on signal 15
>>> May 18 19:40:18 dns1 syslogd 1.4.1: restart.
>>> May 18 19:40:18 dns1 kernel: klogd 1.4.1, log source = /proc/kmsg
>>> started.
>>>
>>> Take 3rd entry for example, the receiver mistook 'exiting' and 'on' to
>>> be %HOSTNAME% and %programname%:
>>> <6>1 2011-05-18T19:40:12.592370+08:00 bogon kernel - - - ?Kernel
>>> logging (proc) stopped.
>>> <6>1 2011-05-18T19:40:12.592693+08:00 bogon kernel - - - ?Kernel log
>>> daemon terminating.
>>> <46>1 2011-05-18T19:40:13.697115+08:00 exiting on - - - ?signal 15
>>> <46>1 2011-05-18T19:40:13.806302+08:00 syslogd 1.4.1 - - - ?restart.
>>> <6>1 2011-05-18T19:40:13.811331+08:00 bogon kernel - - - ?klogd 1.4.1,
>>> log source = /proc/kmsg started.
>>> So, it went to /var/log/hosts/exiting/2011/05/18/rsyslog.log, and
>>> that's definitely the wrong place.
>>>
>>> How to deal with this case?
>>>
>>>
>>> Thanks,
>>> Kaiwang
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From kaiwang.chen at gmail.com  Fri May 20 03:35:15 2011
From: kaiwang.chen at gmail.com (Kaiwang Chen)
Date: Fri, 20 May 2011 09:35:15 +0800
Subject: [rsyslog] HOSTNAME and programname extraction failure,
 when rsyslog as receiver, stock syslog as sender
In-Reply-To: <alpine.DEB.2.02.1105191824300.30095@asgard.lang.hm>
References: <BANLkTimy_GcSTjVu+DUZKO6oZhuGiOy+UQ@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA71DE199@GRFEXC.intern.adiscon.com>
	<BANLkTinttx=2TEhW=KsczJhwrzczCxEPUg@mail.gmail.com>
	<alpine.DEB.2.02.1105191824300.30095@asgard.lang.hm>
Message-ID: <BANLkTikeV5MFt1LEdgG5vPOTS=daXYYEnA@mail.gmail.com>

Yes, it's
3c 34 36 3e 65 78 69 74  69 6e 67 20 6f 6e 20 73 69 67 6e 61 6c 20 31 35  0a
 <  4   6  >   e  x   i    t    i    n   g   _  o   n  _   s   i   g
n   a  l    _   1   5   \n
where spaces are represented by underscores.

Thanks,
Kaiwang

2011/5/20  <david at lang.hm>:
> if it's being generated without a timestamp, rsyslog should be able to
> detect that. can you get a raw log and verify that?
>
> David Lang
>
> On Fri, 20 May 2011, Kaiwang Chen wrote:
>
>> Date: Fri, 20 May 2011 00:10:27 +0800
>> From: Kaiwang Chen <kaiwang.chen at gmail.com>
>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>> To: rsyslog-users <rsyslog at lists.adiscon.com>
>> Subject: Re: [rsyslog] HOSTNAME and programname extraction failure,
>> ? ?when rsyslog as receiver, stock syslog as sender
>>
>> Looks like it's the stock sysklog in CentOS5.6 that generated bad
>> formatted logs(without timestamp and syslogtag), and with a rsyslog
>> 3.x installation I have no choice but not use stock sysklog.
>>
>> Thanks,
>> Kaiwang
>>
>> 2011/5/19 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>>>
>>> I addition to what David already said:
>>>
>>> http://www.rsyslog.com/doc/syslog_parsing.html
>>>
>>> Rainer
>>>
>>>> -----Original Message-----
>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
>>>> Sent: Wednesday, May 18, 2011 2:02 PM
>>>> To: rsyslog at lists.adiscon.com
>>>> Subject: [rsyslog] HOSTNAME and programname extraction failure, when
>>>> rsyslog as receiver, stock syslog as sender
>>>>
>>>> Hello,
>>>>
>>>> I was trying to configure rsyslog(rsyslog-3.22.1-3.el5_5.1) as
>>>> receiver, stock syslog(sysklogd-1.4.1-46.el5) as sender.
>>>>
>>>> The rsyslogd listened on udp/514, and used dynamic filenames with
>>>> protocol23 ?message formatting:
>>>> $ModLoad imudp
>>>> $UDPServerRun 514
>>>> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
>>>> $template
>>>> DynFile,"/var/log/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/rsyslog.log"
>>>> *.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??DynFile
>>>>
>>>> The sender generated several entries(3rd, 4th) violating the
>>>> "syslogtag: message" convention
>>>> May 18 19:40:17 dns1 kernel: Kernel logging (proc) stopped.
>>>> May 18 19:40:17 dns1 kernel: Kernel log daemon terminating.
>>>> May 18 19:40:18 dns1 exiting on signal 15
>>>> May 18 19:40:18 dns1 syslogd 1.4.1: restart.
>>>> May 18 19:40:18 dns1 kernel: klogd 1.4.1, log source = /proc/kmsg
>>>> started.
>>>>
>>>> Take 3rd entry for example, the receiver mistook 'exiting' and 'on' to
>>>> be %HOSTNAME% and %programname%:
>>>> <6>1 2011-05-18T19:40:12.592370+08:00 bogon kernel - - - ?Kernel
>>>> logging (proc) stopped.
>>>> <6>1 2011-05-18T19:40:12.592693+08:00 bogon kernel - - - ?Kernel log
>>>> daemon terminating.
>>>> <46>1 2011-05-18T19:40:13.697115+08:00 exiting on - - - ?signal 15
>>>> <46>1 2011-05-18T19:40:13.806302+08:00 syslogd 1.4.1 - - - ?restart.
>>>> <6>1 2011-05-18T19:40:13.811331+08:00 bogon kernel - - - ?klogd 1.4.1,
>>>> log source = /proc/kmsg started.
>>>> So, it went to /var/log/hosts/exiting/2011/05/18/rsyslog.log, and
>>>> that's definitely the wrong place.
>>>>
>>>> How to deal with this case?
>>>>
>>>>
>>>> Thanks,
>>>> Kaiwang
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>
>

From Ole.Rahn at t-systems.com  Fri May 20 09:12:04 2011
From: Ole.Rahn at t-systems.com (Ole.Rahn at t-systems.com)
Date: Fri, 20 May 2011 09:12:04 +0200
Subject: [rsyslog] Heavy stability problems when using TLS
In-Reply-To: <50CCFFD1B9C6424389383F7A13A03446013CFA30CFE3@HE101451.emea1.cds.t-internal.com>
References: <50CCFFD1B9C6424389383F7A13A03446013CFA30CFE3@HE101451.emea1.cds.t-internal.com>
Message-ID: <50CCFFD1B9C6424389383F7A13A03446013CFA4397DF@HE101451.emea1.cds.t-internal.com>

... still crashes with 5.8.1 ...

In the core dumps (as far as I can read something in there - had to switch them to minimal content, otherwise they rapidly fill up my HD, unfortunately that makes them unreadable to pstack), I see messages like:
* errno=50: Cannot assign requested address. socket: 3
* Urign requested address. socket (...binary rubbish...) Bind error, errno=50: Cannot assign requested address. socket: 3


No luck in capturing a crash, when running rsyslog in debug mode, so far, will keep on trying...

Any advice?

Ole

-----Urspr?ngliche Nachricht-----
Von: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-bounces at lists.adiscon.com] Im Auftrag von Rahn, Ole Johann
Gesendet: Mittwoch, 18. Mai 2011 12:56
An: rsyslog at lists.adiscon.com
Betreff: [rsyslog] Heavy stability problems when using TLS

Hi,

I guess, it is not a new topic, but I could not find any solution, yet. As always, deadlines are near and things seems to get worse and worse, so I thought I give the list a try...

We aim to use rsyslog for a centralized log server and log transmission should be as safe and secure as possible. In our case, that means we want to use syslog via TCP/TLS wherever possible and fall back to TCP or UDP for those nodes that don't support TLS (or even TCP).

I sent a post in April, already, in order to find out, if it is possible to run plain TCP and TCP/TLS on the same server instance (seems it does not work, so far). At this time, we managed to connect a few nodes via TCP/TLS, but the server and apparently even the clients keep crashing several times a day, causing a loss messages.

For obvious reasons, this is not what we want, at all.

I already found a bug report, that seems to point in the right direction, but I don't see a solution even though the status is "resolved":

*         http://bugzilla.adiscon.com/show_bug.cgi?id=194

*         and its spin off http://bugzilla.adiscon.com/show_bug.cgi?id=222

What was the solution?
What can be done to prevent the crashes?

I can provide whatever information you want (at least in regard to the server), but I would rather not post our config to the public.

The server is using rsyslog 5.8.0 on Solaris and the clients that are currently connected run version 4.4.2 on Red Hat.

Best regards
Ole



_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Fri May 20 09:30:26 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 20 May 2011 09:30:26 +0200
Subject: [rsyslog] Heavy stability problems when using TLS
In-Reply-To: <50CCFFD1B9C6424389383F7A13A03446013CFA4397DF@HE101451.emea1.cds.t-internal.com>
References: <50CCFFD1B9C6424389383F7A13A03446013CFA30CFE3@HE101451.emea1.cds.t-internal.com>
	<50CCFFD1B9C6424389383F7A13A03446013CFA4397DF@HE101451.emea1.cds.t-internal.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DE1AA@GRFEXC.intern.adiscon.com>

Consolidated from both messages (so read in full ;))

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Ole.Rahn at t-systems.com
> Sent: Friday, May 20, 2011 9:12 AM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] Heavy stability problems when using TLS
> 
> ... still crashes with 5.8.1 ...
> 
> In the core dumps (as far as I can read something in there - had to
> switch them to minimal content, otherwise they rapidly fill up my HD,
> unfortunately that makes them unreadable to pstack), I see messages
> like:
> * errno=50: Cannot assign requested address. socket: 3
> * Urign requested address. socket (...binary rubbish...) Bind error,
> errno=50: Cannot assign requested address. socket: 3
> 
> 
> No luck in capturing a crash, when running rsyslog in debug mode, so
> far, will keep on trying...
> 
> Any advice?
> 
> We aim to use rsyslog for a centralized log server and log transmission
> should be as safe and secure as possible. In our case, that means we
> want to use syslog via TCP/TLS wherever possible and fall back to TCP
> or UDP for those nodes that don't support TLS (or even TCP).
> 
> I sent a post in April, already, in order to find out, if it is
> possible to run plain TCP and TCP/TLS on the same server instance
> (seems it does not work, so far).

Can you elaborate on what you exactly desire. I think it is possible even
without modification.

> At this time, we managed to connect a
> few nodes via TCP/TLS, but the server and apparently even the clients
> keep crashing several times a day, causing a loss messages.
> 
> For obvious reasons, this is not what we want, at all.
> 
> I already found a bug report, that seems to point in the right
> direction, but I don't see a solution even though the status is
> "resolved":
> 
> *         http://bugzilla.adiscon.com/show_bug.cgi?id=194
> 
> *         and its spin off
> http://bugzilla.adiscon.com/show_bug.cgi?id=222
> 
> What was the solution?
You need to read the tracker in full. Unfortunately, some new bug was mangled
into it when the real one was fixed. Bug 222 handles this new problem. 222
has solution information for it towards the end. 

>From what I read, I think your problem is not related to these two trackers
(except that it seems to occur with TLS as well, which is not really a
relationship ;)).

> What can be done to prevent the crashes?
> 
> I can provide whatever information you want (at least in regard to the
> server), but I would rather not post our config to the public.
> 
> The server is using rsyslog 5.8.0 on Solaris and the clients that are
> currently connected run version 4.4.2 on Red Hat.

I'd suggest to upgrade the clients first, and see if they continue crashing.
Based on the results, let's look further. Alternatively, move the server to a
Linux box and check if it keep crashing. Solaris is very different in regard
to threading, so this may give us a clue.

HTH
Rainer
> 
> Best regards
> Ole
> 
> 
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From Ole.Rahn at t-systems.com  Fri May 20 11:49:10 2011
From: Ole.Rahn at t-systems.com (Ole.Rahn at t-systems.com)
Date: Fri, 20 May 2011 11:49:10 +0200
Subject: [rsyslog]  Heavy stability problems when using TLS
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DE1AA@GRFEXC.intern.adiscon.com>
References: <50CCFFD1B9C6424389383F7A13A03446013CFA30CFE3@HE101451.emea1.cds.t-internal.com>
	<50CCFFD1B9C6424389383F7A13A03446013CFA4397DF@HE101451.emea1.cds.t-internal.com>
	<9B6E2A8877C38245BFB15CC491A11DA71DE1AA@GRFEXC.intern.adiscon.com>
Message-ID: <50CCFFD1B9C6424389383F7A13A03446013CFA439B8D@HE101451.emea1.cds.t-internal.com>

Hi Rainer,

thank you for your reply, unfortunately my options are quite limited, here...

But, first things first:
What I am trying to achieve is this: 
> ... use rsyslog for a centralized log server and log 
> transmission should be as safe and secure as possible. In our case, 
> that means we want to use syslog via TCP/TLS wherever possible and 
> fall back to TCP or UDP for those nodes that don't support TLS (or even TCP).

In more detail, our plan is to have rsyslog listening on 3 ports:
* 514 for syslog/UDP
* 10514 for syslog/plain TCP
* 10515 for syslog/TLS/TCP

UDP seems fine (in order to debug the crashes, I disabled it recently, but until then it worked well).
For the TCP stuff, I tried to first start a listener on port 10514 (plain TCP), set all the TLS parameters and then start the TLS listener on port 10515. That resulted in port 10515 accepting plain TCP messages, but no TLS. Not starting the listener on port 10514 in the first place made port 10515 run as expected (TLS). Since, all nodes that were configured to forward logs to the central server so far support TLS, I decided to "postpone" this problem for now - I guess it is some misconfiguration. To be more specific the relevant part of the configuration was:

$ModLoad immark # provides --MARK-- message capability
#$ModLoad imudp # provides UDP syslog reception
$ModLoad imtcp # provides TCP syslog reception and GSS-API (if compiled to support it)
$ModLoad imsolaris # Reads local Solaris log messages including the kernel log.

#############################
# plain TCP                 #
#############################

#$InputTCPServerRun 10514

#############################
# plain UDP                 #
#############################

#$UDPServerRun 514

#############################
# TCP/TLS                   #
#############################

# make gtls driver the default
$DefaultNetstreamDriver gtls

# certificate files
$DefaultNetstreamDriverCAFile /etc/rsyslog/CA/ca-cert.pem
$DefaultNetstreamDriverCertFile /etc/rsyslog/machine_certificates/machine-cert.pem
$DefaultNetstreamDriverKeyFile /etc/rsyslog/machine_certificates/machine-key.pem

# Hosts
$InputTCPServerStreamDriverPermittedPeer host1
$InputTCPServerStreamDriverPermittedPeer host2
(...)

$InputTCPServerStreamDriverAuthMode x509/name
$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
$InputTCPServerRun 10515 # start up listener at port 10515

If I can not get it working like this, I could still imagine to start more instances of rsyslog, where each run another port, or something like that - I am not really concerned about this topic, so far.


About the crashes:
This problem cannot be postponed, as it has the potential to become a show-stopper for rsyslog in our environment.
I am working in a quite big environment and while I have the server in my hands (however, can't change it to another OS due to company policies - solaris would not have been my first choice!), the clients are managed by a vendor. Since logging is not their main business, they don't support any version changes or alike.

At the bottom line:
* I can change the rsyslog installation/configuration on the server.
* I can collect any logs etc. on the server.
* I can set up a Linux box, but can not put it in the real network.
* I can not change anything on the clients, but I can request configuration changes
* I might be able to have the vendor collect logs on the client(s).

I know (very well!!), that this makes it quite difficult to identify the problem, but I can't help it... :-(

What would you suggest me to do in order to find the problem under this conditions?

Best regards
Ole


-----Urspr?ngliche Nachricht-----
Von: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-bounces at lists.adiscon.com] Im Auftrag von Rainer Gerhards
Gesendet: Freitag, 20. Mai 2011 09:30
An: rsyslog-users
Betreff: Re: [rsyslog] Heavy stability problems when using TLS

Consolidated from both messages (so read in full ;))

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- 
> bounces at lists.adiscon.com] On Behalf Of Ole.Rahn at t-systems.com
> Sent: Friday, May 20, 2011 9:12 AM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] Heavy stability problems when using TLS
> 
> ... still crashes with 5.8.1 ...
> 
> In the core dumps (as far as I can read something in there - had to 
> switch them to minimal content, otherwise they rapidly fill up my HD, 
> unfortunately that makes them unreadable to pstack), I see messages
> like:
> * errno=50: Cannot assign requested address. socket: 3
> * Urign requested address. socket (...binary rubbish...) Bind error,
> errno=50: Cannot assign requested address. socket: 3
> 
> 
> No luck in capturing a crash, when running rsyslog in debug mode, so 
> far, will keep on trying...
> 
> Any advice?
> 
> We aim to use rsyslog for a centralized log server and log 
> transmission should be as safe and secure as possible. In our case, 
> that means we want to use syslog via TCP/TLS wherever possible and 
> fall back to TCP or UDP for those nodes that don't support TLS (or even TCP).
> 
> I sent a post in April, already, in order to find out, if it is 
> possible to run plain TCP and TCP/TLS on the same server instance 
> (seems it does not work, so far).

Can you elaborate on what you exactly desire. I think it is possible even without modification.

> At this time, we managed to connect a
> few nodes via TCP/TLS, but the server and apparently even the clients 
> keep crashing several times a day, causing a loss messages.
> 
> For obvious reasons, this is not what we want, at all.
> 
> I already found a bug report, that seems to point in the right 
> direction, but I don't see a solution even though the status is
> "resolved":
> 
> *         http://bugzilla.adiscon.com/show_bug.cgi?id=194
> 
> *         and its spin off
> http://bugzilla.adiscon.com/show_bug.cgi?id=222
> 
> What was the solution?
You need to read the tracker in full. Unfortunately, some new bug was mangled into it when the real one was fixed. Bug 222 handles this new problem. 222 has solution information for it towards the end. 

>From what I read, I think your problem is not related to these two trackers (except that it seems to occur with TLS as well, which is not really a relationship ;)).

> What can be done to prevent the crashes?
> 
> I can provide whatever information you want (at least in regard to the 
> server), but I would rather not post our config to the public.
> 
> The server is using rsyslog 5.8.0 on Solaris and the clients that are 
> currently connected run version 4.4.2 on Red Hat.

I'd suggest to upgrade the clients first, and see if they continue crashing.
Based on the results, let's look further. Alternatively, move the server to a Linux box and check if it keep crashing. Solaris is very different in regard to threading, so this may give us a clue.

HTH
Rainer
> 
> Best regards
> Ole
> 
> 
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Fri May 20 12:00:38 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 20 May 2011 12:00:38 +0200
Subject: [rsyslog] Heavy stability problems when using TLS
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DE1AC@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Ole.Rahn at t-systems.com
> Sent: Friday, May 20, 2011 11:49 AM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] Heavy stability problems when using TLS
> 
> Hi Rainer,
> 
> thank you for your reply, unfortunately my options are quite limited,
here...
> 
> But, first things first:
> What I am trying to achieve is this:
> > ... use rsyslog for a centralized log server and log transmission
> > should be as safe and secure as possible. In our case, that means we
> > want to use syslog via TCP/TLS wherever possible and fall back to TCP
> > or UDP for those nodes that don't support TLS (or even TCP).
> 
> In more detail, our plan is to have rsyslog listening on 3 ports:
> * 514 for syslog/UDP
> * 10514 for syslog/plain TCP
> * 10515 for syslog/TLS/TCP
> 
> UDP seems fine (in order to debug the crashes, I disabled it recently, but
> until then it worked well).
> For the TCP stuff, I tried to first start a listener on port 10514 (plain
TCP), set
> all the TLS parameters and then start the TLS listener on port 10515. That
> resulted in port 10515 accepting plain TCP messages, but no TLS. Not
starting
> the listener on port 10514 in the first place made port 10515 run as
expected
> (TLS). Since, all nodes that were configured to forward logs to the central
> server so far support TLS, I decided to "postpone" this problem for now - I
> guess it is some misconfiguration. To be more specific the relevant part of
> the configuration was:
> 
> $ModLoad immark # provides --MARK-- message capability #$ModLoad
> imudp # provides UDP syslog reception $ModLoad imtcp # provides TCP
> syslog reception and GSS-API (if compiled to support it) $ModLoad imsolaris
> # Reads local Solaris log messages including the kernel log.
> 
> #############################
> # plain TCP                 #
> #############################
> 
> #$InputTCPServerRun 10514
> 
> #############################
> # plain UDP                 #
> #############################
> 
> #$UDPServerRun 514
> 
> #############################
> # TCP/TLS                   #
> #############################
> 
> # make gtls driver the default
> $DefaultNetstreamDriver gtls
> 
> # certificate files
> $DefaultNetstreamDriverCAFile /etc/rsyslog/CA/ca-cert.pem
> $DefaultNetstreamDriverCertFile
> /etc/rsyslog/machine_certificates/machine-cert.pem
> $DefaultNetstreamDriverKeyFile
> /etc/rsyslog/machine_certificates/machine-key.pem
> 
> # Hosts
> $InputTCPServerStreamDriverPermittedPeer host1
> $InputTCPServerStreamDriverPermittedPeer host2
> (...)
> 
> $InputTCPServerStreamDriverAuthMode x509/name
> $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
> $InputTCPServerRun 10515 # start up listener at port 10515
> 
> If I can not get it working like this, I could still imagine to start more
instances
> of rsyslog, where each run another port, or something like that - I am not
> really concerned about this topic, so far.

I think imtcp has the plumbing to use TLS and plain concurrently, but needs
some code changes to actually do that. The obvious option is to use imptcp
for plain tcp, which offers better performance as well. But it is a
Linux-centric module, I am not  sure if it will run at all on Solaris (this
module is strictly targeted towards platforms that support the required APIs,
in order to gain speed).

> 
> 
> About the crashes:
> This problem cannot be postponed, as it has the potential to become a show-
> stopper for rsyslog in our environment.
> I am working in a quite big environment and while I have the server in my
> hands (however, can't change it to another OS due to company policies -
> solaris would not have been my first choice!), the clients are managed by a
> vendor. Since logging is not their main business, they don't support any
> version changes or alike.
> 
> At the bottom line:
> * I can change the rsyslog installation/configuration on the server.
> * I can collect any logs etc. on the server.
> * I can set up a Linux box, but can not put it in the real network.
> * I can not change anything on the clients, but I can request configuration
> changes
> * I might be able to have the vendor collect logs on the client(s).
> 
> I know (very well!!), that this makes it quite difficult to identify the
problem,
> but I can't help it... :-(

This makes finding a solution really hard. I have worked in similar
environments on custom contracts, but even then it is pretty hard to find the
problem if we can't reproduce it with sufficient effort. 

> 
> What would you suggest me to do in order to find the problem under this
> conditions?

Well, I think the key thing is try to setup some lab where the crashes happen
as well. I guess it is either message-induced or induced by some issues in
Solaris threading. Maybe it helps if you record traffic (from sufficient
sources) and replay it against a Linux box. If that one crashes, we have more
options. If it does not crash, this does not directly point us to something,
unfortunately...

Rainer
> 
> Best regards
> Ole
> 
> 
> -----Urspr?ngliche Nachricht-----
> Von: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] Im Auftrag von Rainer Gerhards
> Gesendet: Freitag, 20. Mai 2011 09:30
> An: rsyslog-users
> Betreff: Re: [rsyslog] Heavy stability problems when using TLS
> 
> Consolidated from both messages (so read in full ;))
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Ole.Rahn at t-systems.com
> > Sent: Friday, May 20, 2011 9:12 AM
> > To: rsyslog at lists.adiscon.com
> > Subject: Re: [rsyslog] Heavy stability problems when using TLS
> >
> > ... still crashes with 5.8.1 ...
> >
> > In the core dumps (as far as I can read something in there - had to
> > switch them to minimal content, otherwise they rapidly fill up my HD,
> > unfortunately that makes them unreadable to pstack), I see messages
> > like:
> > * errno=50: Cannot assign requested address. socket: 3
> > * Urign requested address. socket (...binary rubbish...) Bind error,
> > errno=50: Cannot assign requested address. socket: 3
> >
> >
> > No luck in capturing a crash, when running rsyslog in debug mode, so
> > far, will keep on trying...
> >
> > Any advice?
> >
> > We aim to use rsyslog for a centralized log server and log
> > transmission should be as safe and secure as possible. In our case,
> > that means we want to use syslog via TCP/TLS wherever possible and
> > fall back to TCP or UDP for those nodes that don't support TLS (or even
> TCP).
> >
> > I sent a post in April, already, in order to find out, if it is
> > possible to run plain TCP and TCP/TLS on the same server instance
> > (seems it does not work, so far).
> 
> Can you elaborate on what you exactly desire. I think it is possible even
> without modification.
> 
> > At this time, we managed to connect a
> > few nodes via TCP/TLS, but the server and apparently even the clients
> > keep crashing several times a day, causing a loss messages.
> >
> > For obvious reasons, this is not what we want, at all.
> >
> > I already found a bug report, that seems to point in the right
> > direction, but I don't see a solution even though the status is
> > "resolved":
> >
> > *         http://bugzilla.adiscon.com/show_bug.cgi?id=194
> >
> > *         and its spin off
> > http://bugzilla.adiscon.com/show_bug.cgi?id=222
> >
> > What was the solution?
> You need to read the tracker in full. Unfortunately, some new bug was
> mangled into it when the real one was fixed. Bug 222 handles this new
> problem. 222 has solution information for it towards the end.
> 
> From what I read, I think your problem is not related to these two trackers
> (except that it seems to occur with TLS as well, which is not really a
> relationship ;)).
> 
> > What can be done to prevent the crashes?
> >
> > I can provide whatever information you want (at least in regard to the
> > server), but I would rather not post our config to the public.
> >
> > The server is using rsyslog 5.8.0 on Solaris and the clients that are
> > currently connected run version 4.4.2 on Red Hat.
> 
> I'd suggest to upgrade the clients first, and see if they continue
crashing.
> Based on the results, let's look further. Alternatively, move the server to
a
> Linux box and check if it keep crashing. Solaris is very different in
regard to
> threading, so this may give us a clue.
> 
> HTH
> Rainer
> >
> > Best regards
> > Ole
> >
> >
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Fri May 20 12:02:24 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 20 May 2011 12:02:24 +0200
Subject: [rsyslog] Heavy stability problems when using TLS
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DE1AC@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA71DE1AC@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DE1AD@GRFEXC.intern.adiscon.com>

I forgot to mention:

> Well, I think the key thing is try to setup some lab where the crashes
happen
> as well. I guess it is either message-induced or induced by some issues in
> Solaris threading. Maybe it helps if you record traffic (from sufficient
> sources) and replay it against a Linux box. If that one crashes, we have
more
> options. If it does not crash, this does not directly point us to
something,
> unfortunately...

What I am looking for is some better information on when the crash happens.
So valgrind is often a good choice to check if there are some violations. Not
sure, though, if that's available on Solaris.

Rainer

From tbergfeld at hq.adiscon.com  Fri May 20 14:45:48 2011
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Fri, 20 May 2011 14:45:48 +0200
Subject: [rsyslog] rsyslog 6.1.8 (beta) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DE1B1@GRFEXC.intern.adiscon.com>

This release begins the first beta branch for the v6 branch of rsyslog.

Note that the 6.1.7 version, released roughly a month ago was not
functionally enhanced since then. We already begun a new branch that will
become 6.3.0. So technically, 6.1.7 can be considered beta as well.

As such, we expect a quicker than usual beta cycle for version 6.1.

This release imports some bug fixes done to older versions as well as some
minor enhancements (not affecting stability). An update to this version is
useful for users of the 6.1 versions, but if you experience no problems, it
the update is not urgent.

ChangeLog:

http://www.rsyslog.com/changelog-for-6-1-8-beta/

Download:

http://www.rsyslog.com/rsyslog-6-1-8-beta/

As always, feedback is appreciated.

Best regards,
Tom Bergfeld
--

Support
=======
Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html.

From Ole.Rahn at t-systems.com  Fri May 20 15:25:54 2011
From: Ole.Rahn at t-systems.com (Ole.Rahn at t-systems.com)
Date: Fri, 20 May 2011 15:25:54 +0200
Subject: [rsyslog]  Heavy stability problems when using TLS
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DE1AD@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA71DE1AC@GRFEXC.intern.adiscon.com>
	<9B6E2A8877C38245BFB15CC491A11DA71DE1AD@GRFEXC.intern.adiscon.com>
Message-ID: <50CCFFD1B9C6424389383F7A13A03446013CFA43A043@HE101451.emea1.cds.t-internal.com>

Hi Rainer,

it will take me some time to set this up, but I guess that is the way to go...

In the meantime syslog crashed while the debug log was running, but unfortunately it seems like there is nothing interesting in there.
I still post it, just in case it tells you something...

The core dump again contains "(...)ITICAL> Socket bind error, errno=50: Cannot assign requested add(...)"

Best regards
Ole


(...)
5401.988709000:4: --------<NSDSEL_PTCP> calling select, active fds (max 36): 8 9 11 19 32 36
5401.988737200:4: hasRcvInBuffer on nsd 138268: pszRcvBuf 0, lenRcvBuf 0
5401.988745400:4: hasRcvInBuffer on nsd 856f0: pszRcvBuf 0, lenRcvBuf 0
5401.988753400:4: hasRcvInBuffer on nsd 160ff8: pszRcvBuf 6b9e28, lenRcvBuf -1
5401.988761300:4: hasRcvInBuffer on nsd 177408: pszRcvBuf 1be080, lenRcvBuf -1
5401.988768800:4: GnuTLS requested retry of 2 operation - executing
5401.988776100:4: retrying gtls recv, nsd: 177408
5401.988785800:4: GnuTLS receive requires a retry (this most probably is OK and no error condition)
5401.988793900:4: gtlsRecordRecv return. nsd 177408, iRet -2100, lenRcvd -28, lenRcvBuf -1, ptrRcvBuf 359
5401.988804900:4: hasRcvInBuffer on nsd 138268: pszRcvBuf 0, lenRcvBuf 0
5401.988812900:4: hasRcvInBuffer on nsd 856f0: pszRcvBuf 0, lenRcvBuf 0
5401.988820700:4: hasRcvInBuffer on nsd 160ff8: pszRcvBuf 6b9e28, lenRcvBuf -1
5401.988828500:4: hasRcvInBuffer on nsd 177408: pszRcvBuf 1be080, lenRcvBuf -1
5401.988836400:4: hasRcvInBuffer on nsd 15ecd0: pszRcvBuf 607150, lenRcvBuf -1
5401.988844200:4: hasRcvInBuffer on nsd 1409a8: pszRcvBuf 1b9040, lenRcvBuf -1
5401.988852100:4: --------<NSDSEL_PTCP> calling select, active fds (max 36): 8 9 11 19 32 36
5401.988880100:4: hasRcvInBuffer on nsd 138268: pszRcvBuf 0, lenRcvBuf 0
5401.988888300:4: hasRcvInBuffer on nsd 856f0: pszRcvBuf 0, lenRcvBuf 0
5401.988896300:4: hasRcvInBuffer on nsd 160ff8: pszRcvBuf 6b9e28, lenRcvBuf -1
5401.988904300:4: hasRcvInBuffer on nsd 177408: pszRcvBuf 1be080, lenRcvBuf -1
5401.988911700:4: GnuTLS requested retry of 2 operation - executing
5401.988919000:4: retrying gtls recv, nsd: 177408
5401.988928700:4: GnuTLS receive requires a retry (this most probably is OK and no error condition)
5401.988936900:4: gtlsRecordRecv return. nsd 177408, iRet -2100, lenRcvd -28, lenRcvBuf -1, ptrRcvBuf 359
5401.988947700:4: hasRcvInBuffer on nsd 138268: pszRcvBuf 0, lenRcvBuf 0
5401.988955700:4: hasRcvInBuffer on nsd 856f0: pszRcvBuf 0, lenRcvBuf 0
5401.988963500:4: hasRcvInBuffer on nsd 160ff8: pszRcvBuf 6b9e28, lenRcvBuf -1
5401.988971300:4: hasRcvInBuffer on nsd 177408: pszRcvBuf 1be080, lenRcvBuf -1
5401.988979100:4: hasRcvInBuffer on nsd 15ecd0: pszRcvBuf 607150, lenRcvBuf -1
5401.988986900:4: hasRcvInBuffer on nsd 1409a8: pszRcvBuf 1b9040, lenRcvBuf -1
5401.988994800:4: --------<NSDSEL_PTCP> calling select, active fds (max 36): 8 9 11 19 32 36
5401.989022800:4: hasRcvInBuffer on nsd 138268: pszRcvBuf 0, lenRcvBuf 0
5401.989031000:4: hasRcvInBuffer on nsd 856f0: pszRcvBuf 0, lenRcvBuf 0
5401.989038900:4: hasRcvInBuffer on nsd 160ff8: pszRcvBuf 6b9e28, lenRcvBuf -1
5401.989046


-----Urspr?ngliche Nachricht-----
Von: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-bounces at lists.adiscon.com] Im Auftrag von Rainer Gerhards
Gesendet: Freitag, 20. Mai 2011 12:02
An: rsyslog-users
Betreff: Re: [rsyslog] Heavy stability problems when using TLS

I forgot to mention:

> Well, I think the key thing is try to setup some lab where the crashes
happen
> as well. I guess it is either message-induced or induced by some 
> issues in Solaris threading. Maybe it helps if you record traffic 
> (from sufficient
> sources) and replay it against a Linux box. If that one crashes, we 
> have
more
> options. If it does not crash, this does not directly point us to
something,
> unfortunately...

What I am looking for is some better information on when the crash happens.
So valgrind is often a good choice to check if there are some violations. Not sure, though, if that's available on Solaris.

Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Fri May 20 15:38:13 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 20 May 2011 15:38:13 +0200
Subject: [rsyslog] Heavy stability problems when using TLS
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DE1B2@GRFEXC.intern.adiscon.com>

The problem location may actually be far from the actual abort. Could you
mail me the complete debug log?

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Ole.Rahn at t-systems.com
> Sent: Friday, May 20, 2011 3:26 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] Heavy stability problems when using TLS
> 
> Hi Rainer,
> 
> it will take me some time to set this up, but I guess that is the way to
go...
> 
> In the meantime syslog crashed while the debug log was running, but
> unfortunately it seems like there is nothing interesting in there.
> I still post it, just in case it tells you something...
> 
> The core dump again contains "(...)ITICAL> Socket bind error, errno=50:
> Cannot assign requested add(...)"
> 
> Best regards
> Ole
> 
> 
> (...)
> 5401.988709000:4: --------<NSDSEL_PTCP> calling select, active fds (max
36): 8
> 9 11 19 32 36
> 5401.988737200:4: hasRcvInBuffer on nsd 138268: pszRcvBuf 0, lenRcvBuf 0
> 5401.988745400:4: hasRcvInBuffer on nsd 856f0: pszRcvBuf 0, lenRcvBuf 0
> 5401.988753400:4: hasRcvInBuffer on nsd 160ff8: pszRcvBuf 6b9e28,
> lenRcvBuf -1
> 5401.988761300:4: hasRcvInBuffer on nsd 177408: pszRcvBuf 1be080,
> lenRcvBuf -1
> 5401.988768800:4: GnuTLS requested retry of 2 operation - executing
> 5401.988776100:4: retrying gtls recv, nsd: 177408
> 5401.988785800:4: GnuTLS receive requires a retry (this most probably is OK
> and no error condition)
> 5401.988793900:4: gtlsRecordRecv return. nsd 177408, iRet -2100, lenRcvd
-28,
> lenRcvBuf -1, ptrRcvBuf 359
> 5401.988804900:4: hasRcvInBuffer on nsd 138268: pszRcvBuf 0, lenRcvBuf 0
> 5401.988812900:4: hasRcvInBuffer on nsd 856f0: pszRcvBuf 0, lenRcvBuf 0
> 5401.988820700:4: hasRcvInBuffer on nsd 160ff8: pszRcvBuf 6b9e28,
> lenRcvBuf -1
> 5401.988828500:4: hasRcvInBuffer on nsd 177408: pszRcvBuf 1be080,
> lenRcvBuf -1
> 5401.988836400:4: hasRcvInBuffer on nsd 15ecd0: pszRcvBuf 607150,
> lenRcvBuf -1
> 5401.988844200:4: hasRcvInBuffer on nsd 1409a8: pszRcvBuf 1b9040,
> lenRcvBuf -1
> 5401.988852100:4: --------<NSDSEL_PTCP> calling select, active fds (max
36): 8
> 9 11 19 32 36
> 5401.988880100:4: hasRcvInBuffer on nsd 138268: pszRcvBuf 0, lenRcvBuf 0
> 5401.988888300:4: hasRcvInBuffer on nsd 856f0: pszRcvBuf 0, lenRcvBuf 0
> 5401.988896300:4: hasRcvInBuffer on nsd 160ff8: pszRcvBuf 6b9e28,
> lenRcvBuf -1
> 5401.988904300:4: hasRcvInBuffer on nsd 177408: pszRcvBuf 1be080,
> lenRcvBuf -1
> 5401.988911700:4: GnuTLS requested retry of 2 operation - executing
> 5401.988919000:4: retrying gtls recv, nsd: 177408
> 5401.988928700:4: GnuTLS receive requires a retry (this most probably is OK
> and no error condition)
> 5401.988936900:4: gtlsRecordRecv return. nsd 177408, iRet -2100, lenRcvd
-28,
> lenRcvBuf -1, ptrRcvBuf 359
> 5401.988947700:4: hasRcvInBuffer on nsd 138268: pszRcvBuf 0, lenRcvBuf 0
> 5401.988955700:4: hasRcvInBuffer on nsd 856f0: pszRcvBuf 0, lenRcvBuf 0
> 5401.988963500:4: hasRcvInBuffer on nsd 160ff8: pszRcvBuf 6b9e28,
> lenRcvBuf -1
> 5401.988971300:4: hasRcvInBuffer on nsd 177408: pszRcvBuf 1be080,
> lenRcvBuf -1
> 5401.988979100:4: hasRcvInBuffer on nsd 15ecd0: pszRcvBuf 607150,
> lenRcvBuf -1
> 5401.988986900:4: hasRcvInBuffer on nsd 1409a8: pszRcvBuf 1b9040,
> lenRcvBuf -1
> 5401.988994800:4: --------<NSDSEL_PTCP> calling select, active fds (max
36): 8
> 9 11 19 32 36
> 5401.989022800:4: hasRcvInBuffer on nsd 138268: pszRcvBuf 0, lenRcvBuf 0
> 5401.989031000:4: hasRcvInBuffer on nsd 856f0: pszRcvBuf 0, lenRcvBuf 0
> 5401.989038900:4: hasRcvInBuffer on nsd 160ff8: pszRcvBuf 6b9e28,
> lenRcvBuf -1
> 5401.989046
> 
> 
> -----Urspr?ngliche Nachricht-----
> Von: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] Im Auftrag von Rainer Gerhards
> Gesendet: Freitag, 20. Mai 2011 12:02
> An: rsyslog-users
> Betreff: Re: [rsyslog] Heavy stability problems when using TLS
> 
> I forgot to mention:
> 
> > Well, I think the key thing is try to setup some lab where the crashes
> happen
> > as well. I guess it is either message-induced or induced by some
> > issues in Solaris threading. Maybe it helps if you record traffic
> > (from sufficient
> > sources) and replay it against a Linux box. If that one crashes, we
> > have
> more
> > options. If it does not crash, this does not directly point us to
> something,
> > unfortunately...
> 
> What I am looking for is some better information on when the crash
> happens.
> So valgrind is often a good choice to check if there are some violations.
Not
> sure, though, if that's available on Solaris.
> 
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From Ole.Rahn at t-systems.com  Fri May 20 17:10:52 2011
From: Ole.Rahn at t-systems.com (Ole.Rahn at t-systems.com)
Date: Fri, 20 May 2011 17:10:52 +0200
Subject: [rsyslog] Heavy stability problems when using TLS
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DE1B2@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA71DE1B2@GRFEXC.intern.adiscon.com>
Message-ID: <50CCFFD1B9C6424389383F7A13A03446013CFA43A17F@HE101451.emea1.cds.t-internal.com>

I would absolutely be willing to do that, but the file is 6.2 GB and even my access to the log server will barely allow me copy over that file to my machine (even when it is compressed - it is rather slow remote connection which will be auto-closed after a couple of MB, very secure, though...).

Can I somehow reduce the amount by e.g. sending you the first 10,000 lines and the last 10,000 lines?

Ole


-----Urspr?ngliche Nachricht-----
Von: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-bounces at lists.adiscon.com] Im Auftrag von Rainer Gerhards
Gesendet: Freitag, 20. Mai 2011 15:38
An: rsyslog-users
Betreff: Re: [rsyslog] Heavy stability problems when using TLS

The problem location may actually be far from the actual abort. Could you mail me the complete debug log?

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- 
> bounces at lists.adiscon.com] On Behalf Of Ole.Rahn at t-systems.com
> Sent: Friday, May 20, 2011 3:26 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] Heavy stability problems when using TLS
> 
> Hi Rainer,
> 
> it will take me some time to set this up, but I guess that is the way 
> to
go...
> 
> In the meantime syslog crashed while the debug log was running, but 
> unfortunately it seems like there is nothing interesting in there.
> I still post it, just in case it tells you something...
> 
> The core dump again contains "(...)ITICAL> Socket bind error, errno=50:
> Cannot assign requested add(...)"
> 
> Best regards
> Ole
> 
> 
> (...)
> 5401.988709000:4: --------<NSDSEL_PTCP> calling select, active fds 
> (max
36): 8
> 9 11 19 32 36
> 5401.988737200:4: hasRcvInBuffer on nsd 138268: pszRcvBuf 0, lenRcvBuf 
> 0
> 5401.988745400:4: hasRcvInBuffer on nsd 856f0: pszRcvBuf 0, lenRcvBuf 
> 0
> 5401.988753400:4: hasRcvInBuffer on nsd 160ff8: pszRcvBuf 6b9e28, 
> lenRcvBuf -1
> 5401.988761300:4: hasRcvInBuffer on nsd 177408: pszRcvBuf 1be080, 
> lenRcvBuf -1
> 5401.988768800:4: GnuTLS requested retry of 2 operation - executing
> 5401.988776100:4: retrying gtls recv, nsd: 177408
> 5401.988785800:4: GnuTLS receive requires a retry (this most probably 
> is OK and no error condition)
> 5401.988793900:4: gtlsRecordRecv return. nsd 177408, iRet -2100, 
> lenRcvd
-28,
> lenRcvBuf -1, ptrRcvBuf 359
> 5401.988804900:4: hasRcvInBuffer on nsd 138268: pszRcvBuf 0, lenRcvBuf 
> 0
> 5401.988812900:4: hasRcvInBuffer on nsd 856f0: pszRcvBuf 0, lenRcvBuf 
> 0
> 5401.988820700:4: hasRcvInBuffer on nsd 160ff8: pszRcvBuf 6b9e28, 
> lenRcvBuf -1
> 5401.988828500:4: hasRcvInBuffer on nsd 177408: pszRcvBuf 1be080, 
> lenRcvBuf -1
> 5401.988836400:4: hasRcvInBuffer on nsd 15ecd0: pszRcvBuf 607150, 
> lenRcvBuf -1
> 5401.988844200:4: hasRcvInBuffer on nsd 1409a8: pszRcvBuf 1b9040, 
> lenRcvBuf -1
> 5401.988852100:4: --------<NSDSEL_PTCP> calling select, active fds 
> (max
36): 8
> 9 11 19 32 36
> 5401.988880100:4: hasRcvInBuffer on nsd 138268: pszRcvBuf 0, lenRcvBuf 
> 0
> 5401.988888300:4: hasRcvInBuffer on nsd 856f0: pszRcvBuf 0, lenRcvBuf 
> 0
> 5401.988896300:4: hasRcvInBuffer on nsd 160ff8: pszRcvBuf 6b9e28, 
> lenRcvBuf -1
> 5401.988904300:4: hasRcvInBuffer on nsd 177408: pszRcvBuf 1be080, 
> lenRcvBuf -1
> 5401.988911700:4: GnuTLS requested retry of 2 operation - executing
> 5401.988919000:4: retrying gtls recv, nsd: 177408
> 5401.988928700:4: GnuTLS receive requires a retry (this most probably 
> is OK and no error condition)
> 5401.988936900:4: gtlsRecordRecv return. nsd 177408, iRet -2100, 
> lenRcvd
-28,
> lenRcvBuf -1, ptrRcvBuf 359
> 5401.988947700:4: hasRcvInBuffer on nsd 138268: pszRcvBuf 0, lenRcvBuf 
> 0
> 5401.988955700:4: hasRcvInBuffer on nsd 856f0: pszRcvBuf 0, lenRcvBuf 
> 0
> 5401.988963500:4: hasRcvInBuffer on nsd 160ff8: pszRcvBuf 6b9e28, 
> lenRcvBuf -1
> 5401.988971300:4: hasRcvInBuffer on nsd 177408: pszRcvBuf 1be080, 
> lenRcvBuf -1
> 5401.988979100:4: hasRcvInBuffer on nsd 15ecd0: pszRcvBuf 607150, 
> lenRcvBuf -1
> 5401.988986900:4: hasRcvInBuffer on nsd 1409a8: pszRcvBuf 1b9040, 
> lenRcvBuf -1
> 5401.988994800:4: --------<NSDSEL_PTCP> calling select, active fds 
> (max
36): 8
> 9 11 19 32 36
> 5401.989022800:4: hasRcvInBuffer on nsd 138268: pszRcvBuf 0, lenRcvBuf 
> 0
> 5401.989031000:4: hasRcvInBuffer on nsd 856f0: pszRcvBuf 0, lenRcvBuf 
> 0
> 5401.989038900:4: hasRcvInBuffer on nsd 160ff8: pszRcvBuf 6b9e28, 
> lenRcvBuf -1
> 5401.989046
> 
> 
> -----Urspr?ngliche Nachricht-----
> Von: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- 
> bounces at lists.adiscon.com] Im Auftrag von Rainer Gerhards
> Gesendet: Freitag, 20. Mai 2011 12:02
> An: rsyslog-users
> Betreff: Re: [rsyslog] Heavy stability problems when using TLS
> 
> I forgot to mention:
> 
> > Well, I think the key thing is try to setup some lab where the 
> > crashes
> happen
> > as well. I guess it is either message-induced or induced by some 
> > issues in Solaris threading. Maybe it helps if you record traffic 
> > (from sufficient
> > sources) and replay it against a Linux box. If that one crashes, we 
> > have
> more
> > options. If it does not crash, this does not directly point us to
> something,
> > unfortunately...
> 
> What I am looking for is some better information on when the crash 
> happens.
> So valgrind is often a good choice to check if there are some violations.
Not
> sure, though, if that's available on Solaris.
> 
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

From iamsayan at gmail.com  Fri May 20 21:58:36 2011
From: iamsayan at gmail.com (Sayan Chowdhury)
Date: Fri, 20 May 2011 15:58:36 -0400
Subject: [rsyslog] (no subject)
Message-ID: <BANLkTimTzooraQHTm4bRWPqWrYMkrOfadw@mail.gmail.com>



From mhalligan at pagerduty.com  Sat May 21 03:22:56 2011
From: mhalligan at pagerduty.com (Michael Halligan)
Date: Fri, 20 May 2011 18:22:56 -0700
Subject: [rsyslog] imfile broken?
Message-ID: <BANLkTin0rVvj6VojzkR4Q5yroAe67jGV1g@mail.gmail.com>

In my setup, imfile causes rsyslog to go into an infinite loop of restarts,
rendering logging unusable. Is this expected behavior?

From rgerhards at hq.adiscon.com  Sat May 21 11:24:56 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Sat, 21 May 2011 11:24:56 +0200
Subject: [rsyslog] imfile broken?
In-Reply-To: <BANLkTin0rVvj6VojzkR4Q5yroAe67jGV1g@mail.gmail.com>
References: <BANLkTin0rVvj6VojzkR4Q5yroAe67jGV1g@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DE1BA@GRFEXC.intern.adiscon.com>

Obviously no, please provide details.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Halligan
> Sent: Saturday, May 21, 2011 3:23 AM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] imfile broken?
> 
> In my setup, imfile causes rsyslog to go into an infinite loop of
> restarts,
> rendering logging unusable. Is this expected behavior?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From david at lang.hm  Sat May 21 11:52:33 2011
From: david at lang.hm (david at lang.hm)
Date: Sat, 21 May 2011 02:52:33 -0700 (PDT)
Subject: [rsyslog] HOSTNAME and programname extraction failure,
 when rsyslog as receiver, stock syslog as sender
In-Reply-To: <BANLkTikeV5MFt1LEdgG5vPOTS=daXYYEnA@mail.gmail.com>
References: <BANLkTimy_GcSTjVu+DUZKO6oZhuGiOy+UQ@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA71DE199@GRFEXC.intern.adiscon.com>
	<BANLkTinttx=2TEhW=KsczJhwrzczCxEPUg@mail.gmail.com>
	<alpine.DEB.2.02.1105191824300.30095@asgard.lang.hm>
	<BANLkTikeV5MFt1LEdgG5vPOTS=daXYYEnA@mail.gmail.com>
Message-ID: <alpine.DEB.2.02.1105210250300.8167@asgard.lang.hm>

Rainer,
   if the message is missing a timestamp, why does the default parser 
assume that there is a hostname there?

I would assume that if there is no timestamp there isn't a hostname either 
(which would address this particular issue)

did you have some experiance in the past that pushed you to the current 
implementation?

David Lang

On Fri, 20 May 2011, Kaiwang Chen wrote:

> Date: Fri, 20 May 2011 09:35:15 +0800
> From: Kaiwang Chen <kaiwang.chen at gmail.com>
> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> To: rsyslog-users <rsyslog at lists.adiscon.com>
> Subject: Re: [rsyslog] HOSTNAME and programname extraction failure,
>     when rsyslog as receiver, stock syslog as sender
> 
> Yes, it's
> 3c 34 36 3e 65 78 69 74  69 6e 67 20 6f 6e 20 73 69 67 6e 61 6c 20 31 35  0a
> <  4   6  >   e  x   i    t    i    n   g   _  o   n  _   s   i   g
> n   a  l    _   1   5   \n
> where spaces are represented by underscores.
>
> Thanks,
> Kaiwang
>
> 2011/5/20  <david at lang.hm>:
>> if it's being generated without a timestamp, rsyslog should be able to
>> detect that. can you get a raw log and verify that?
>>
>> David Lang
>>
>> On Fri, 20 May 2011, Kaiwang Chen wrote:
>>
>>> Date: Fri, 20 May 2011 00:10:27 +0800
>>> From: Kaiwang Chen <kaiwang.chen at gmail.com>
>>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>>> To: rsyslog-users <rsyslog at lists.adiscon.com>
>>> Subject: Re: [rsyslog] HOSTNAME and programname extraction failure,
>>> ? ?when rsyslog as receiver, stock syslog as sender
>>>
>>> Looks like it's the stock sysklog in CentOS5.6 that generated bad
>>> formatted logs(without timestamp and syslogtag), and with a rsyslog
>>> 3.x installation I have no choice but not use stock sysklog.
>>>
>>> Thanks,
>>> Kaiwang
>>>
>>> 2011/5/19 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>>>>
>>>> I addition to what David already said:
>>>>
>>>> http://www.rsyslog.com/doc/syslog_parsing.html
>>>>
>>>> Rainer
>>>>
>>>>> -----Original Message-----
>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>>> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
>>>>> Sent: Wednesday, May 18, 2011 2:02 PM
>>>>> To: rsyslog at lists.adiscon.com
>>>>> Subject: [rsyslog] HOSTNAME and programname extraction failure, when
>>>>> rsyslog as receiver, stock syslog as sender
>>>>>
>>>>> Hello,
>>>>>
>>>>> I was trying to configure rsyslog(rsyslog-3.22.1-3.el5_5.1) as
>>>>> receiver, stock syslog(sysklogd-1.4.1-46.el5) as sender.
>>>>>
>>>>> The rsyslogd listened on udp/514, and used dynamic filenames with
>>>>> protocol23 ?message formatting:
>>>>> $ModLoad imudp
>>>>> $UDPServerRun 514
>>>>> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
>>>>> $template
>>>>> DynFile,"/var/log/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/rsyslog.log"
>>>>> *.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??DynFile
>>>>>
>>>>> The sender generated several entries(3rd, 4th) violating the
>>>>> "syslogtag: message" convention
>>>>> May 18 19:40:17 dns1 kernel: Kernel logging (proc) stopped.
>>>>> May 18 19:40:17 dns1 kernel: Kernel log daemon terminating.
>>>>> May 18 19:40:18 dns1 exiting on signal 15
>>>>> May 18 19:40:18 dns1 syslogd 1.4.1: restart.
>>>>> May 18 19:40:18 dns1 kernel: klogd 1.4.1, log source = /proc/kmsg
>>>>> started.
>>>>>
>>>>> Take 3rd entry for example, the receiver mistook 'exiting' and 'on' to
>>>>> be %HOSTNAME% and %programname%:
>>>>> <6>1 2011-05-18T19:40:12.592370+08:00 bogon kernel - - - ?Kernel
>>>>> logging (proc) stopped.
>>>>> <6>1 2011-05-18T19:40:12.592693+08:00 bogon kernel - - - ?Kernel log
>>>>> daemon terminating.
>>>>> <46>1 2011-05-18T19:40:13.697115+08:00 exiting on - - - ?signal 15
>>>>> <46>1 2011-05-18T19:40:13.806302+08:00 syslogd 1.4.1 - - - ?restart.
>>>>> <6>1 2011-05-18T19:40:13.811331+08:00 bogon kernel - - - ?klogd 1.4.1,
>>>>> log source = /proc/kmsg started.
>>>>> So, it went to /var/log/hosts/exiting/2011/05/18/rsyslog.log, and
>>>>> that's definitely the wrong place.
>>>>>
>>>>> How to deal with this case?
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Kaiwang
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com
>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From rgerhards at hq.adiscon.com  Sat May 21 15:19:16 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Sat, 21 May 2011 15:19:16 +0200
Subject: [rsyslog] HOSTNAME and programname extraction failure,
	when rsyslog as receiver, stock syslog as sender
In-Reply-To: <alpine.DEB.2.02.1105210250300.8167@asgard.lang.hm>
References: <BANLkTimy_GcSTjVu+DUZKO6oZhuGiOy+UQ@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA71DE199@GRFEXC.intern.adiscon.com><BANLkTinttx=2TEhW=KsczJhwrzczCxEPUg@mail.gmail.com><alpine.DEB.2.02.1105191824300.30095@asgard.lang.hm><BANLkTikeV5MFt1LEdgG5vPOTS=daXYYEnA@mail.gmail.com>
	<alpine.DEB.2.02.1105210250300.8167@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DE1BB@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Saturday, May 21, 2011 11:53 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] HOSTNAME and programname extraction failure,
> when rsyslog as receiver, stock syslog as sender
> 
> Rainer,
>    if the message is missing a timestamp, why does the default parser
> assume that there is a hostname there?
> 
> I would assume that if there is no timestamp there isn't a hostname
> either
> (which would address this particular issue)
> 
> did you have some experiance in the past that pushed you to the current
> implementation?
>

I don't have the specifics at hand, but as far as I remember there were cases
where absence of timestamp does NOT indicate absence of tag and/or hostname.
I am very hesitant to touch the default legacy parser, as the heuristics
works pretty well since > 2 years. All other malformed messages reported were
really badly malformed. So I think the clean path would be to write a parser
module for such dateless but otherwise correct messages...

Rainer
 
> David Lang
> 
> On Fri, 20 May 2011, Kaiwang Chen wrote:
> 
> > Date: Fri, 20 May 2011 09:35:15 +0800
> > From: Kaiwang Chen <kaiwang.chen at gmail.com>
> > Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> > To: rsyslog-users <rsyslog at lists.adiscon.com>
> > Subject: Re: [rsyslog] HOSTNAME and programname extraction failure,
> >     when rsyslog as receiver, stock syslog as sender
> >
> > Yes, it's
> > 3c 34 36 3e 65 78 69 74  69 6e 67 20 6f 6e 20 73 69 67 6e 61 6c 20 31
> 35  0a
> > <  4   6  >   e  x   i    t    i    n   g   _  o   n  _   s   i   g
> > n   a  l    _   1   5   \n
> > where spaces are represented by underscores.
> >
> > Thanks,
> > Kaiwang
> >
> > 2011/5/20  <david at lang.hm>:
> >> if it's being generated without a timestamp, rsyslog should be able
> to
> >> detect that. can you get a raw log and verify that?
> >>
> >> David Lang
> >>
> >> On Fri, 20 May 2011, Kaiwang Chen wrote:
> >>
> >>> Date: Fri, 20 May 2011 00:10:27 +0800
> >>> From: Kaiwang Chen <kaiwang.chen at gmail.com>
> >>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> >>> To: rsyslog-users <rsyslog at lists.adiscon.com>
> >>> Subject: Re: [rsyslog] HOSTNAME and programname extraction failure,
> >>> ? ?when rsyslog as receiver, stock syslog as sender
> >>>
> >>> Looks like it's the stock sysklog in CentOS5.6 that generated bad
> >>> formatted logs(without timestamp and syslogtag), and with a rsyslog
> >>> 3.x installation I have no choice but not use stock sysklog.
> >>>
> >>> Thanks,
> >>> Kaiwang
> >>>
> >>> 2011/5/19 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> >>>>
> >>>> I addition to what David already said:
> >>>>
> >>>> http://www.rsyslog.com/doc/syslog_parsing.html
> >>>>
> >>>> Rainer
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >>>>> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
> >>>>> Sent: Wednesday, May 18, 2011 2:02 PM
> >>>>> To: rsyslog at lists.adiscon.com
> >>>>> Subject: [rsyslog] HOSTNAME and programname extraction failure,
> when
> >>>>> rsyslog as receiver, stock syslog as sender
> >>>>>
> >>>>> Hello,
> >>>>>
> >>>>> I was trying to configure rsyslog(rsyslog-3.22.1-3.el5_5.1) as
> >>>>> receiver, stock syslog(sysklogd-1.4.1-46.el5) as sender.
> >>>>>
> >>>>> The rsyslogd listened on udp/514, and used dynamic filenames with
> >>>>> protocol23 ?message formatting:
> >>>>> $ModLoad imudp
> >>>>> $UDPServerRun 514
> >>>>> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
> >>>>> $template
> >>>>>
> DynFile,"/var/log/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/rsyslog.log"
> >>>>> *.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??DynFile
> >>>>>
> >>>>> The sender generated several entries(3rd, 4th) violating the
> >>>>> "syslogtag: message" convention
> >>>>> May 18 19:40:17 dns1 kernel: Kernel logging (proc) stopped.
> >>>>> May 18 19:40:17 dns1 kernel: Kernel log daemon terminating.
> >>>>> May 18 19:40:18 dns1 exiting on signal 15
> >>>>> May 18 19:40:18 dns1 syslogd 1.4.1: restart.
> >>>>> May 18 19:40:18 dns1 kernel: klogd 1.4.1, log source = /proc/kmsg
> >>>>> started.
> >>>>>
> >>>>> Take 3rd entry for example, the receiver mistook 'exiting' and
> 'on' to
> >>>>> be %HOSTNAME% and %programname%:
> >>>>> <6>1 2011-05-18T19:40:12.592370+08:00 bogon kernel - - - ?Kernel
> >>>>> logging (proc) stopped.
> >>>>> <6>1 2011-05-18T19:40:12.592693+08:00 bogon kernel - - - ?Kernel
> log
> >>>>> daemon terminating.
> >>>>> <46>1 2011-05-18T19:40:13.697115+08:00 exiting on - - - ?signal
> 15
> >>>>> <46>1 2011-05-18T19:40:13.806302+08:00 syslogd 1.4.1 - - -
> ?restart.
> >>>>> <6>1 2011-05-18T19:40:13.811331+08:00 bogon kernel - - - ?klogd
> 1.4.1,
> >>>>> log source = /proc/kmsg started.
> >>>>> So, it went to /var/log/hosts/exiting/2011/05/18/rsyslog.log, and
> >>>>> that's definitely the wrong place.
> >>>>>
> >>>>> How to deal with this case?
> >>>>>
> >>>>>
> >>>>> Thanks,
> >>>>> Kaiwang
> >>>>> _______________________________________________
> >>>>> rsyslog mailing list
> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>> http://www.rsyslog.com
> >>>>
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.com
> >>>>
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >>
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >>
> >>
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >

From mhalligan at pagerduty.com  Sat May 21 18:08:32 2011
From: mhalligan at pagerduty.com (Michael Halligan)
Date: Sat, 21 May 2011 09:08:32 -0700
Subject: [rsyslog] imfile broken?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DE1BA@GRFEXC.intern.adiscon.com>
References: <BANLkTin0rVvj6VojzkR4Q5yroAe67jGV1g@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA71DE1BA@GRFEXC.intern.adiscon.com>
Message-ID: <BANLkTik2QcN+CUJo0iRFKxW3Wd3vrXF+8A@mail.gmail.com>

Sorry, I just don't have the time to debug this. I'm going back to
syslog-ng. Thanks anyways.

On Sat, May 21, 2011 at 2:24 AM, Rainer Gerhards
<rgerhards at hq.adiscon.com>wrote:

> Obviously no, please provide details.
>
> Rainer
>
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Michael Halligan
> > Sent: Saturday, May 21, 2011 3:23 AM
> > To: rsyslog at lists.adiscon.com
> > Subject: [rsyslog] imfile broken?
> >
> > In my setup, imfile causes rsyslog to go into an infinite loop of
> > restarts,
> > rendering logging unusable. Is this expected behavior?
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From kaiwang.chen at gmail.com  Sat May 21 18:24:17 2011
From: kaiwang.chen at gmail.com (Kaiwang Chen)
Date: Sun, 22 May 2011 00:24:17 +0800
Subject: [rsyslog] HOSTNAME and programname extraction failure,
 when rsyslog as receiver, stock syslog as sender
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DE1BB@GRFEXC.intern.adiscon.com>
References: <BANLkTimy_GcSTjVu+DUZKO6oZhuGiOy+UQ@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA71DE199@GRFEXC.intern.adiscon.com>
	<BANLkTinttx=2TEhW=KsczJhwrzczCxEPUg@mail.gmail.com>
	<alpine.DEB.2.02.1105191824300.30095@asgard.lang.hm>
	<BANLkTikeV5MFt1LEdgG5vPOTS=daXYYEnA@mail.gmail.com>
	<alpine.DEB.2.02.1105210250300.8167@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA71DE1BB@GRFEXC.intern.adiscon.com>
Message-ID: <BANLkTimJ1JViw0THCGweFzfuWNYS=rORbw@mail.gmail.com>

So rsyslogd is currently lame in the case of missing both timestamp
and hostname. Is it possible to assign a different parser for specific
log source to get things right in output module?

Thanks,
Kaiwang

2011/5/21 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Saturday, May 21, 2011 11:53 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] HOSTNAME and programname extraction failure,
>> when rsyslog as receiver, stock syslog as sender
>>
>> Rainer,
>> ? ?if the message is missing a timestamp, why does the default parser
>> assume that there is a hostname there?
>>
>> I would assume that if there is no timestamp there isn't a hostname
>> either
>> (which would address this particular issue)
>>
>> did you have some experiance in the past that pushed you to the current
>> implementation?
>>
>
> I don't have the specifics at hand, but as far as I remember there were cases
> where absence of timestamp does NOT indicate absence of tag and/or hostname.
> I am very hesitant to touch the default legacy parser, as the heuristics
> works pretty well since > 2 years. All other malformed messages reported were
> really badly malformed. So I think the clean path would be to write a parser
> module for such dateless but otherwise correct messages...
>
> Rainer
>
>> David Lang
>>
>> On Fri, 20 May 2011, Kaiwang Chen wrote:
>>
>> > Date: Fri, 20 May 2011 09:35:15 +0800
>> > From: Kaiwang Chen <kaiwang.chen at gmail.com>
>> > Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>> > To: rsyslog-users <rsyslog at lists.adiscon.com>
>> > Subject: Re: [rsyslog] HOSTNAME and programname extraction failure,
>> > ? ? when rsyslog as receiver, stock syslog as sender
>> >
>> > Yes, it's
>> > 3c 34 36 3e 65 78 69 74 ?69 6e 67 20 6f 6e 20 73 69 67 6e 61 6c 20 31
>> 35 ?0a
>> > < ?4 ? 6 ?> ? e ?x ? i ? ?t ? ?i ? ?n ? g ? _ ?o ? n ?_ ? s ? i ? g
>> > n ? a ?l ? ?_ ? 1 ? 5 ? \n
>> > where spaces are represented by underscores.
>> >
>> > Thanks,
>> > Kaiwang
>> >
>> > 2011/5/20 ?<david at lang.hm>:
>> >> if it's being generated without a timestamp, rsyslog should be able
>> to
>> >> detect that. can you get a raw log and verify that?
>> >>
>> >> David Lang
>> >>
>> >> On Fri, 20 May 2011, Kaiwang Chen wrote:
>> >>
>> >>> Date: Fri, 20 May 2011 00:10:27 +0800
>> >>> From: Kaiwang Chen <kaiwang.chen at gmail.com>
>> >>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>> >>> To: rsyslog-users <rsyslog at lists.adiscon.com>
>> >>> Subject: Re: [rsyslog] HOSTNAME and programname extraction failure,
>> >>> ? ?when rsyslog as receiver, stock syslog as sender
>> >>>
>> >>> Looks like it's the stock sysklog in CentOS5.6 that generated bad
>> >>> formatted logs(without timestamp and syslogtag), and with a rsyslog
>> >>> 3.x installation I have no choice but not use stock sysklog.
>> >>>
>> >>> Thanks,
>> >>> Kaiwang
>> >>>
>> >>> 2011/5/19 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>> >>>>
>> >>>> I addition to what David already said:
>> >>>>
>> >>>> http://www.rsyslog.com/doc/syslog_parsing.html
>> >>>>
>> >>>> Rainer
>> >>>>
>> >>>>> -----Original Message-----
>> >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> >>>>> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
>> >>>>> Sent: Wednesday, May 18, 2011 2:02 PM
>> >>>>> To: rsyslog at lists.adiscon.com
>> >>>>> Subject: [rsyslog] HOSTNAME and programname extraction failure,
>> when
>> >>>>> rsyslog as receiver, stock syslog as sender
>> >>>>>
>> >>>>> Hello,
>> >>>>>
>> >>>>> I was trying to configure rsyslog(rsyslog-3.22.1-3.el5_5.1) as
>> >>>>> receiver, stock syslog(sysklogd-1.4.1-46.el5) as sender.
>> >>>>>
>> >>>>> The rsyslogd listened on udp/514, and used dynamic filenames with
>> >>>>> protocol23 ?message formatting:
>> >>>>> $ModLoad imudp
>> >>>>> $UDPServerRun 514
>> >>>>> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
>> >>>>> $template
>> >>>>>
>> DynFile,"/var/log/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/rsyslog.log"
>> >>>>> *.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??DynFile
>> >>>>>
>> >>>>> The sender generated several entries(3rd, 4th) violating the
>> >>>>> "syslogtag: message" convention
>> >>>>> May 18 19:40:17 dns1 kernel: Kernel logging (proc) stopped.
>> >>>>> May 18 19:40:17 dns1 kernel: Kernel log daemon terminating.
>> >>>>> May 18 19:40:18 dns1 exiting on signal 15
>> >>>>> May 18 19:40:18 dns1 syslogd 1.4.1: restart.
>> >>>>> May 18 19:40:18 dns1 kernel: klogd 1.4.1, log source = /proc/kmsg
>> >>>>> started.
>> >>>>>
>> >>>>> Take 3rd entry for example, the receiver mistook 'exiting' and
>> 'on' to
>> >>>>> be %HOSTNAME% and %programname%:
>> >>>>> <6>1 2011-05-18T19:40:12.592370+08:00 bogon kernel - - - ?Kernel
>> >>>>> logging (proc) stopped.
>> >>>>> <6>1 2011-05-18T19:40:12.592693+08:00 bogon kernel - - - ?Kernel
>> log
>> >>>>> daemon terminating.
>> >>>>> <46>1 2011-05-18T19:40:13.697115+08:00 exiting on - - - ?signal
>> 15
>> >>>>> <46>1 2011-05-18T19:40:13.806302+08:00 syslogd 1.4.1 - - -
>> ?restart.
>> >>>>> <6>1 2011-05-18T19:40:13.811331+08:00 bogon kernel - - - ?klogd
>> 1.4.1,
>> >>>>> log source = /proc/kmsg started.
>> >>>>> So, it went to /var/log/hosts/exiting/2011/05/18/rsyslog.log, and
>> >>>>> that's definitely the wrong place.
>> >>>>>
>> >>>>> How to deal with this case?
>> >>>>>
>> >>>>>
>> >>>>> Thanks,
>> >>>>> Kaiwang
>> >>>>> _______________________________________________
>> >>>>> rsyslog mailing list
>> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>>>> http://www.rsyslog.com
>> >>>>
>> >>>> _______________________________________________
>> >>>> rsyslog mailing list
>> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>>> http://www.rsyslog.com
>> >>>>
>> >>> _______________________________________________
>> >>> rsyslog mailing list
>> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>> http://www.rsyslog.com
>> >>
>> >> _______________________________________________
>> >> rsyslog mailing list
>> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> http://www.rsyslog.com
>> >>
>> >>
>> > _______________________________________________
>> > rsyslog mailing list
>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > http://www.rsyslog.com
>> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From rgerhards at hq.adiscon.com  Sat May 21 18:25:25 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Sat, 21 May 2011 18:25:25 +0200
Subject: [rsyslog] HOSTNAME and programname extraction failure,
	when rsyslog as receiver, stock syslog as sender
In-Reply-To: <BANLkTimJ1JViw0THCGweFzfuWNYS=rORbw@mail.gmail.com>
References: <BANLkTimy_GcSTjVu+DUZKO6oZhuGiOy+UQ@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA71DE199@GRFEXC.intern.adiscon.com><BANLkTinttx=2TEhW=KsczJhwrzczCxEPUg@mail.gmail.com><alpine.DEB.2.02.1105191824300.30095@asgard.lang.hm><BANLkTikeV5MFt1LEdgG5vPOTS=daXYYEnA@mail.gmail.com><alpine.DEB.2.02.1105210250300.8167@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA71DE1BB@GRFEXC.intern.adiscon.com>
	<BANLkTimJ1JViw0THCGweFzfuWNYS=rORbw@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DE1BD@GRFEXC.intern.adiscon.com>

yes

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
> Sent: Saturday, May 21, 2011 6:24 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] HOSTNAME and programname extraction failure,
> when rsyslog as receiver, stock syslog as sender
> 
> So rsyslogd is currently lame in the case of missing both timestamp
> and hostname. Is it possible to assign a different parser for specific
> log source to get things right in output module?
> 
> Thanks,
> Kaiwang
> 
> 2011/5/21 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >> Sent: Saturday, May 21, 2011 11:53 AM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] HOSTNAME and programname extraction failure,
> >> when rsyslog as receiver, stock syslog as sender
> >>
> >> Rainer,
> >> ? ?if the message is missing a timestamp, why does the default
> parser
> >> assume that there is a hostname there?
> >>
> >> I would assume that if there is no timestamp there isn't a hostname
> >> either
> >> (which would address this particular issue)
> >>
> >> did you have some experiance in the past that pushed you to the
> current
> >> implementation?
> >>
> >
> > I don't have the specifics at hand, but as far as I remember there
> were cases
> > where absence of timestamp does NOT indicate absence of tag and/or
> hostname.
> > I am very hesitant to touch the default legacy parser, as the
> heuristics
> > works pretty well since > 2 years. All other malformed messages
> reported were
> > really badly malformed. So I think the clean path would be to write a
> parser
> > module for such dateless but otherwise correct messages...
> >
> > Rainer
> >
> >> David Lang
> >>
> >> On Fri, 20 May 2011, Kaiwang Chen wrote:
> >>
> >> > Date: Fri, 20 May 2011 09:35:15 +0800
> >> > From: Kaiwang Chen <kaiwang.chen at gmail.com>
> >> > Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> >> > To: rsyslog-users <rsyslog at lists.adiscon.com>
> >> > Subject: Re: [rsyslog] HOSTNAME and programname extraction
> failure,
> >> > ? ? when rsyslog as receiver, stock syslog as sender
> >> >
> >> > Yes, it's
> >> > 3c 34 36 3e 65 78 69 74 ?69 6e 67 20 6f 6e 20 73 69 67 6e 61 6c 20
> 31
> >> 35 ?0a
> >> > < ?4 ? 6 ?> ? e ?x ? i ? ?t ? ?i ? ?n ? g ? _ ?o ? n ?_ ? s ? i
> g
> >> > n ? a ?l ? ?_ ? 1 ? 5 ? \n
> >> > where spaces are represented by underscores.
> >> >
> >> > Thanks,
> >> > Kaiwang
> >> >
> >> > 2011/5/20 ?<david at lang.hm>:
> >> >> if it's being generated without a timestamp, rsyslog should be
> able
> >> to
> >> >> detect that. can you get a raw log and verify that?
> >> >>
> >> >> David Lang
> >> >>
> >> >> On Fri, 20 May 2011, Kaiwang Chen wrote:
> >> >>
> >> >>> Date: Fri, 20 May 2011 00:10:27 +0800
> >> >>> From: Kaiwang Chen <kaiwang.chen at gmail.com>
> >> >>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> >> >>> To: rsyslog-users <rsyslog at lists.adiscon.com>
> >> >>> Subject: Re: [rsyslog] HOSTNAME and programname extraction
> failure,
> >> >>> ? ?when rsyslog as receiver, stock syslog as sender
> >> >>>
> >> >>> Looks like it's the stock sysklog in CentOS5.6 that generated
> bad
> >> >>> formatted logs(without timestamp and syslogtag), and with a
> rsyslog
> >> >>> 3.x installation I have no choice but not use stock sysklog.
> >> >>>
> >> >>> Thanks,
> >> >>> Kaiwang
> >> >>>
> >> >>> 2011/5/19 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> >> >>>>
> >> >>>> I addition to what David already said:
> >> >>>>
> >> >>>> http://www.rsyslog.com/doc/syslog_parsing.html
> >> >>>>
> >> >>>> Rainer
> >> >>>>
> >> >>>>> -----Original Message-----
> >> >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> >>>>> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
> >> >>>>> Sent: Wednesday, May 18, 2011 2:02 PM
> >> >>>>> To: rsyslog at lists.adiscon.com
> >> >>>>> Subject: [rsyslog] HOSTNAME and programname extraction
> failure,
> >> when
> >> >>>>> rsyslog as receiver, stock syslog as sender
> >> >>>>>
> >> >>>>> Hello,
> >> >>>>>
> >> >>>>> I was trying to configure rsyslog(rsyslog-3.22.1-3.el5_5.1) as
> >> >>>>> receiver, stock syslog(sysklogd-1.4.1-46.el5) as sender.
> >> >>>>>
> >> >>>>> The rsyslogd listened on udp/514, and used dynamic filenames
> with
> >> >>>>> protocol23 ?message formatting:
> >> >>>>> $ModLoad imudp
> >> >>>>> $UDPServerRun 514
> >> >>>>> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
> >> >>>>> $template
> >> >>>>>
> >>
> DynFile,"/var/log/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/rsyslog.log"
> >> >>>>> *.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??DynFile
> >> >>>>>
> >> >>>>> The sender generated several entries(3rd, 4th) violating the
> >> >>>>> "syslogtag: message" convention
> >> >>>>> May 18 19:40:17 dns1 kernel: Kernel logging (proc) stopped.
> >> >>>>> May 18 19:40:17 dns1 kernel: Kernel log daemon terminating.
> >> >>>>> May 18 19:40:18 dns1 exiting on signal 15
> >> >>>>> May 18 19:40:18 dns1 syslogd 1.4.1: restart.
> >> >>>>> May 18 19:40:18 dns1 kernel: klogd 1.4.1, log source =
> /proc/kmsg
> >> >>>>> started.
> >> >>>>>
> >> >>>>> Take 3rd entry for example, the receiver mistook 'exiting' and
> >> 'on' to
> >> >>>>> be %HOSTNAME% and %programname%:
> >> >>>>> <6>1 2011-05-18T19:40:12.592370+08:00 bogon kernel - - -
> ?Kernel
> >> >>>>> logging (proc) stopped.
> >> >>>>> <6>1 2011-05-18T19:40:12.592693+08:00 bogon kernel - - -
> ?Kernel
> >> log
> >> >>>>> daemon terminating.
> >> >>>>> <46>1 2011-05-18T19:40:13.697115+08:00 exiting on - - -
> ?signal
> >> 15
> >> >>>>> <46>1 2011-05-18T19:40:13.806302+08:00 syslogd 1.4.1 - - -
> >> ?restart.
> >> >>>>> <6>1 2011-05-18T19:40:13.811331+08:00 bogon kernel - - -
> ?klogd
> >> 1.4.1,
> >> >>>>> log source = /proc/kmsg started.
> >> >>>>> So, it went to /var/log/hosts/exiting/2011/05/18/rsyslog.log,
> and
> >> >>>>> that's definitely the wrong place.
> >> >>>>>
> >> >>>>> How to deal with this case?
> >> >>>>>
> >> >>>>>
> >> >>>>> Thanks,
> >> >>>>> Kaiwang
> >> >>>>> _______________________________________________
> >> >>>>> rsyslog mailing list
> >> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> >>>>> http://www.rsyslog.com
> >> >>>>
> >> >>>> _______________________________________________
> >> >>>> rsyslog mailing list
> >> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> >>>> http://www.rsyslog.com
> >> >>>>
> >> >>> _______________________________________________
> >> >>> rsyslog mailing list
> >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> >>> http://www.rsyslog.com
> >> >>
> >> >> _______________________________________________
> >> >> rsyslog mailing list
> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> >> http://www.rsyslog.com
> >> >>
> >> >>
> >> > _______________________________________________
> >> > rsyslog mailing list
> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> > http://www.rsyslog.com
> >> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From kaiwang.chen at gmail.com  Sat May 21 19:58:27 2011
From: kaiwang.chen at gmail.com (Kaiwang Chen)
Date: Sun, 22 May 2011 01:58:27 +0800
Subject: [rsyslog] HOSTNAME and programname extraction failure,
 when rsyslog as receiver, stock syslog as sender
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DE1BD@GRFEXC.intern.adiscon.com>
References: <BANLkTimy_GcSTjVu+DUZKO6oZhuGiOy+UQ@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA71DE199@GRFEXC.intern.adiscon.com>
	<BANLkTinttx=2TEhW=KsczJhwrzczCxEPUg@mail.gmail.com>
	<alpine.DEB.2.02.1105191824300.30095@asgard.lang.hm>
	<BANLkTikeV5MFt1LEdgG5vPOTS=daXYYEnA@mail.gmail.com>
	<alpine.DEB.2.02.1105210250300.8167@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA71DE1BB@GRFEXC.intern.adiscon.com>
	<BANLkTimJ1JViw0THCGweFzfuWNYS=rORbw@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA71DE1BD@GRFEXC.intern.adiscon.com>
Message-ID: <BANLkTino0FkWLuHMCP+RwFFaowSXxfcrbw@mail.gmail.com>

After a quick scan through Message parsers in rsyslog,
http://www.rsyslog.com/doc/messageparser.html, I was able to locate a
$RulesetParser directive to manipulate the parser chain bound a
specific ruleset. Also noticed the statements under the section "Can I
use different parser chains for different devices":

Note that it is currently under evaluation if rsyslog will support
binding parser chains to specific inputs directly, without depending
on the ruleset. There are some concerns that this may not be necessary
but adds considerable complexity to the configuration. So this may or
may not be possible in the future. In any case, if we decide to add
it, input modules need to support it, so this functionality would
require some time to implement.

Bad news is that
$RulesetParser(http://www.rsyslog.com/doc/rsconf1_rulesetparser.html)
is a 5.3.4+ feature. And multiple
ruleset(http://www.rsyslog.com/doc/multi_ruleset.html) is a 4.5.0+ or
5.1.1+ feature.

I can't figure out whether imudp, which is the only choice for legacy
syslogd sender, supports multi-ruleset or not. Looks like it is
impossible to register multiple rulsets to receive from single input,
say tcp/514.

As for my case, a quick solution would be upgrade all CentOS5 server
to use rsyslogd, and if any bridge device generates mailformated logs,
I would setup a relay and use output module to reformat it.


Thanks,
Kaiwang

2011/5/22 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> yes
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
>> Sent: Saturday, May 21, 2011 6:24 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] HOSTNAME and programname extraction failure,
>> when rsyslog as receiver, stock syslog as sender
>>
>> So rsyslogd is currently lame in the case of missing both timestamp
>> and hostname. Is it possible to assign a different parser for specific
>> log source to get things right in output module?
>>
>> Thanks,
>> Kaiwang
>>
>> 2011/5/21 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>> >> -----Original Message-----
>> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> >> Sent: Saturday, May 21, 2011 11:53 AM
>> >> To: rsyslog-users
>> >> Subject: Re: [rsyslog] HOSTNAME and programname extraction failure,
>> >> when rsyslog as receiver, stock syslog as sender
>> >>
>> >> Rainer,
>> >> ? ?if the message is missing a timestamp, why does the default
>> parser
>> >> assume that there is a hostname there?
>> >>
>> >> I would assume that if there is no timestamp there isn't a hostname
>> >> either
>> >> (which would address this particular issue)
>> >>
>> >> did you have some experiance in the past that pushed you to the
>> current
>> >> implementation?
>> >>
>> >
>> > I don't have the specifics at hand, but as far as I remember there
>> were cases
>> > where absence of timestamp does NOT indicate absence of tag and/or
>> hostname.
>> > I am very hesitant to touch the default legacy parser, as the
>> heuristics
>> > works pretty well since > 2 years. All other malformed messages
>> reported were
>> > really badly malformed. So I think the clean path would be to write a
>> parser
>> > module for such dateless but otherwise correct messages...
>> >
>> > Rainer
>> >
>> >> David Lang
>> >>
>> >> On Fri, 20 May 2011, Kaiwang Chen wrote:
>> >>
>> >> > Date: Fri, 20 May 2011 09:35:15 +0800
>> >> > From: Kaiwang Chen <kaiwang.chen at gmail.com>
>> >> > Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>> >> > To: rsyslog-users <rsyslog at lists.adiscon.com>
>> >> > Subject: Re: [rsyslog] HOSTNAME and programname extraction
>> failure,
>> >> > ? ? when rsyslog as receiver, stock syslog as sender
>> >> >
>> >> > Yes, it's
>> >> > 3c 34 36 3e 65 78 69 74 ?69 6e 67 20 6f 6e 20 73 69 67 6e 61 6c 20
>> 31
>> >> 35 ?0a
>> >> > < ?4 ? 6 ?> ? e ?x ? i ? ?t ? ?i ? ?n ? g ? _ ?o ? n ?_ ? s ? i
>> g
>> >> > n ? a ?l ? ?_ ? 1 ? 5 ? \n
>> >> > where spaces are represented by underscores.
>> >> >
>> >> > Thanks,
>> >> > Kaiwang
>> >> >
>> >> > 2011/5/20 ?<david at lang.hm>:
>> >> >> if it's being generated without a timestamp, rsyslog should be
>> able
>> >> to
>> >> >> detect that. can you get a raw log and verify that?
>> >> >>
>> >> >> David Lang
>> >> >>
>> >> >> On Fri, 20 May 2011, Kaiwang Chen wrote:
>> >> >>
>> >> >>> Date: Fri, 20 May 2011 00:10:27 +0800
>> >> >>> From: Kaiwang Chen <kaiwang.chen at gmail.com>
>> >> >>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>> >> >>> To: rsyslog-users <rsyslog at lists.adiscon.com>
>> >> >>> Subject: Re: [rsyslog] HOSTNAME and programname extraction
>> failure,
>> >> >>> ? ?when rsyslog as receiver, stock syslog as sender
>> >> >>>
>> >> >>> Looks like it's the stock sysklog in CentOS5.6 that generated
>> bad
>> >> >>> formatted logs(without timestamp and syslogtag), and with a
>> rsyslog
>> >> >>> 3.x installation I have no choice but not use stock sysklog.
>> >> >>>
>> >> >>> Thanks,
>> >> >>> Kaiwang
>> >> >>>
>> >> >>> 2011/5/19 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>> >> >>>>
>> >> >>>> I addition to what David already said:
>> >> >>>>
>> >> >>>> http://www.rsyslog.com/doc/syslog_parsing.html
>> >> >>>>
>> >> >>>> Rainer
>> >> >>>>
>> >> >>>>> -----Original Message-----
>> >> >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> >> >>>>> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
>> >> >>>>> Sent: Wednesday, May 18, 2011 2:02 PM
>> >> >>>>> To: rsyslog at lists.adiscon.com
>> >> >>>>> Subject: [rsyslog] HOSTNAME and programname extraction
>> failure,
>> >> when
>> >> >>>>> rsyslog as receiver, stock syslog as sender
>> >> >>>>>
>> >> >>>>> Hello,
>> >> >>>>>
>> >> >>>>> I was trying to configure rsyslog(rsyslog-3.22.1-3.el5_5.1) as
>> >> >>>>> receiver, stock syslog(sysklogd-1.4.1-46.el5) as sender.
>> >> >>>>>
>> >> >>>>> The rsyslogd listened on udp/514, and used dynamic filenames
>> with
>> >> >>>>> protocol23 ?message formatting:
>> >> >>>>> $ModLoad imudp
>> >> >>>>> $UDPServerRun 514
>> >> >>>>> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
>> >> >>>>> $template
>> >> >>>>>
>> >>
>> DynFile,"/var/log/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/rsyslog.log"
>> >> >>>>> *.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??DynFile
>> >> >>>>>
>> >> >>>>> The sender generated several entries(3rd, 4th) violating the
>> >> >>>>> "syslogtag: message" convention
>> >> >>>>> May 18 19:40:17 dns1 kernel: Kernel logging (proc) stopped.
>> >> >>>>> May 18 19:40:17 dns1 kernel: Kernel log daemon terminating.
>> >> >>>>> May 18 19:40:18 dns1 exiting on signal 15
>> >> >>>>> May 18 19:40:18 dns1 syslogd 1.4.1: restart.
>> >> >>>>> May 18 19:40:18 dns1 kernel: klogd 1.4.1, log source =
>> /proc/kmsg
>> >> >>>>> started.
>> >> >>>>>
>> >> >>>>> Take 3rd entry for example, the receiver mistook 'exiting' and
>> >> 'on' to
>> >> >>>>> be %HOSTNAME% and %programname%:
>> >> >>>>> <6>1 2011-05-18T19:40:12.592370+08:00 bogon kernel - - -
>> ?Kernel
>> >> >>>>> logging (proc) stopped.
>> >> >>>>> <6>1 2011-05-18T19:40:12.592693+08:00 bogon kernel - - -
>> ?Kernel
>> >> log
>> >> >>>>> daemon terminating.
>> >> >>>>> <46>1 2011-05-18T19:40:13.697115+08:00 exiting on - - -
>> ?signal
>> >> 15
>> >> >>>>> <46>1 2011-05-18T19:40:13.806302+08:00 syslogd 1.4.1 - - -
>> >> ?restart.
>> >> >>>>> <6>1 2011-05-18T19:40:13.811331+08:00 bogon kernel - - -
>> ?klogd
>> >> 1.4.1,
>> >> >>>>> log source = /proc/kmsg started.
>> >> >>>>> So, it went to /var/log/hosts/exiting/2011/05/18/rsyslog.log,
>> and
>> >> >>>>> that's definitely the wrong place.
>> >> >>>>>
>> >> >>>>> How to deal with this case?
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Thanks,
>> >> >>>>> Kaiwang
>> >> >>>>> _______________________________________________
>> >> >>>>> rsyslog mailing list
>> >> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >>>>> http://www.rsyslog.com
>> >> >>>>
>> >> >>>> _______________________________________________
>> >> >>>> rsyslog mailing list
>> >> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >>>> http://www.rsyslog.com
>> >> >>>>
>> >> >>> _______________________________________________
>> >> >>> rsyslog mailing list
>> >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >>> http://www.rsyslog.com
>> >> >>
>> >> >> _______________________________________________
>> >> >> rsyslog mailing list
>> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >> http://www.rsyslog.com
>> >> >>
>> >> >>
>> >> > _______________________________________________
>> >> > rsyslog mailing list
>> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> > http://www.rsyslog.com
>> >> >
>> > _______________________________________________
>> > rsyslog mailing list
>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > http://www.rsyslog.com
>> >
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From champ at softwink.com  Sat May 21 21:36:21 2011
From: champ at softwink.com (Champ Clark III [Softwink])
Date: Sat, 21 May 2011 15:36:21 -0400
Subject: [rsyslog] imfile broken?
In-Reply-To: <BANLkTik2QcN+CUJo0iRFKxW3Wd3vrXF+8A@mail.gmail.com>
References: <BANLkTin0rVvj6VojzkR4Q5yroAe67jGV1g@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA71DE1BA@GRFEXC.intern.adiscon.com>
	<BANLkTik2QcN+CUJo0iRFKxW3Wd3vrXF+8A@mail.gmail.com>
Message-ID: <20110521193621.GA24067@bundy.vistech.net>

On Sat, May 21, 2011 at 09:08:32AM -0700, Michael Halligan wrote:
> Sorry, I just don't have the time to debug this. I'm going back to
> syslog-ng. Thanks anyways.

	You never supplied any details on the problem,  so it's a bit
hard to help you.  Sounds like _you_ have a issue and not rsyslog.  
Of course,  we'll never know I suppose. 

-- 
        Champ Clark III | Softwink, Inc | 800-538-9357 x 101
                     http://www.softwink.com

GPG Key ID: 58A2A58F
Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
If it wasn't for C, we'd be using BASI, PASAL and OBOL.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20110521/42378108/attachment.pgp>

From david at lang.hm  Sun May 22 07:53:14 2011
From: david at lang.hm (david at lang.hm)
Date: Sat, 21 May 2011 22:53:14 -0700 (PDT)
Subject: [rsyslog] HOSTNAME and programname extraction failure,
 when rsyslog as receiver, stock syslog as sender
In-Reply-To: <BANLkTino0FkWLuHMCP+RwFFaowSXxfcrbw@mail.gmail.com>
References: <BANLkTimy_GcSTjVu+DUZKO6oZhuGiOy+UQ@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA71DE199@GRFEXC.intern.adiscon.com>
	<BANLkTinttx=2TEhW=KsczJhwrzczCxEPUg@mail.gmail.com>
	<alpine.DEB.2.02.1105191824300.30095@asgard.lang.hm>
	<BANLkTikeV5MFt1LEdgG5vPOTS=daXYYEnA@mail.gmail.com>
	<alpine.DEB.2.02.1105210250300.8167@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA71DE1BB@GRFEXC.intern.adiscon.com>
	<BANLkTimJ1JViw0THCGweFzfuWNYS=rORbw@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA71DE1BD@GRFEXC.intern.adiscon.com>
	<BANLkTino0FkWLuHMCP+RwFFaowSXxfcrbw@mail.gmail.com>
Message-ID: <alpine.DEB.2.02.1105212247420.8167@asgard.lang.hm>

switching to rsyslog everywhere is definantly the most flexible option 
(and probably the best from a long-term support point of view).

what Rainer was talking about wasn't the ruleset parser capibility, but 
rather the parser module capibility

http://www.rsyslog.com/doc/messageparser.html

and an example one is at http://www.rsyslog.com/doc/pmlastmsg.html

there isn't currently a module that will fix up this malformed input, it's 
something that probably isn't that hard to do (it does take learning how 
to insert data into a log entry rather than just changeing or deleting it 
like the prior parsers I've written, but I've got instructions on how to 
do that, I just haven't done it yet)

David Lang

  On Sun, 22 May 2011, Kaiwang Chen wrote:

> After a quick scan through Message parsers in rsyslog,
> http://www.rsyslog.com/doc/messageparser.html, I was able to locate a
> $RulesetParser directive to manipulate the parser chain bound a
> specific ruleset. Also noticed the statements under the section "Can I
> use different parser chains for different devices":
>
> Note that it is currently under evaluation if rsyslog will support
> binding parser chains to specific inputs directly, without depending
> on the ruleset. There are some concerns that this may not be necessary
> but adds considerable complexity to the configuration. So this may or
> may not be possible in the future. In any case, if we decide to add
> it, input modules need to support it, so this functionality would
> require some time to implement.
>
> Bad news is that
> $RulesetParser(http://www.rsyslog.com/doc/rsconf1_rulesetparser.html)
> is a 5.3.4+ feature. And multiple
> ruleset(http://www.rsyslog.com/doc/multi_ruleset.html) is a 4.5.0+ or
> 5.1.1+ feature.
>
> I can't figure out whether imudp, which is the only choice for legacy
> syslogd sender, supports multi-ruleset or not. Looks like it is
> impossible to register multiple rulsets to receive from single input,
> say tcp/514.
>
> As for my case, a quick solution would be upgrade all CentOS5 server
> to use rsyslogd, and if any bridge device generates mailformated logs,
> I would setup a relay and use output module to reformat it.
>
>
> Thanks,
> Kaiwang
>
> 2011/5/22 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>> yes
>>
>>> -----Original Message-----
>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
>>> Sent: Saturday, May 21, 2011 6:24 PM
>>> To: rsyslog-users
>>> Subject: Re: [rsyslog] HOSTNAME and programname extraction failure,
>>> when rsyslog as receiver, stock syslog as sender
>>>
>>> So rsyslogd is currently lame in the case of missing both timestamp
>>> and hostname. Is it possible to assign a different parser for specific
>>> log source to get things right in output module?
>>>
>>> Thanks,
>>> Kaiwang
>>>
>>> 2011/5/21 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>>>>> -----Original Message-----
>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>>>> Sent: Saturday, May 21, 2011 11:53 AM
>>>>> To: rsyslog-users
>>>>> Subject: Re: [rsyslog] HOSTNAME and programname extraction failure,
>>>>> when rsyslog as receiver, stock syslog as sender
>>>>>
>>>>> Rainer,
>>>>> ? ?if the message is missing a timestamp, why does the default
>>> parser
>>>>> assume that there is a hostname there?
>>>>>
>>>>> I would assume that if there is no timestamp there isn't a hostname
>>>>> either
>>>>> (which would address this particular issue)
>>>>>
>>>>> did you have some experiance in the past that pushed you to the
>>> current
>>>>> implementation?
>>>>>
>>>>
>>>> I don't have the specifics at hand, but as far as I remember there
>>> were cases
>>>> where absence of timestamp does NOT indicate absence of tag and/or
>>> hostname.
>>>> I am very hesitant to touch the default legacy parser, as the
>>> heuristics
>>>> works pretty well since > 2 years. All other malformed messages
>>> reported were
>>>> really badly malformed. So I think the clean path would be to write a
>>> parser
>>>> module for such dateless but otherwise correct messages...
>>>>
>>>> Rainer
>>>>
>>>>> David Lang
>>>>>
>>>>> On Fri, 20 May 2011, Kaiwang Chen wrote:
>>>>>
>>>>>> Date: Fri, 20 May 2011 09:35:15 +0800
>>>>>> From: Kaiwang Chen <kaiwang.chen at gmail.com>
>>>>>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>>>>>> To: rsyslog-users <rsyslog at lists.adiscon.com>
>>>>>> Subject: Re: [rsyslog] HOSTNAME and programname extraction
>>> failure,
>>>>>> ? ? when rsyslog as receiver, stock syslog as sender
>>>>>>
>>>>>> Yes, it's
>>>>>> 3c 34 36 3e 65 78 69 74 ?69 6e 67 20 6f 6e 20 73 69 67 6e 61 6c 20
>>> 31
>>>>> 35 ?0a
>>>>>> < ?4 ? 6 ?> ? e ?x ? i ? ?t ? ?i ? ?n ? g ? _ ?o ? n ?_ ? s ? i
>>> g
>>>>>> n ? a ?l ? ?_ ? 1 ? 5 ? \n
>>>>>> where spaces are represented by underscores.
>>>>>>
>>>>>> Thanks,
>>>>>> Kaiwang
>>>>>>
>>>>>> 2011/5/20 ?<david at lang.hm>:
>>>>>>> if it's being generated without a timestamp, rsyslog should be
>>> able
>>>>> to
>>>>>>> detect that. can you get a raw log and verify that?
>>>>>>>
>>>>>>> David Lang
>>>>>>>
>>>>>>> On Fri, 20 May 2011, Kaiwang Chen wrote:
>>>>>>>
>>>>>>>> Date: Fri, 20 May 2011 00:10:27 +0800
>>>>>>>> From: Kaiwang Chen <kaiwang.chen at gmail.com>
>>>>>>>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>>>>>>>> To: rsyslog-users <rsyslog at lists.adiscon.com>
>>>>>>>> Subject: Re: [rsyslog] HOSTNAME and programname extraction
>>> failure,
>>>>>>>> ? ?when rsyslog as receiver, stock syslog as sender
>>>>>>>>
>>>>>>>> Looks like it's the stock sysklog in CentOS5.6 that generated
>>> bad
>>>>>>>> formatted logs(without timestamp and syslogtag), and with a
>>> rsyslog
>>>>>>>> 3.x installation I have no choice but not use stock sysklog.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Kaiwang
>>>>>>>>
>>>>>>>> 2011/5/19 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>>>>>>>>>
>>>>>>>>> I addition to what David already said:
>>>>>>>>>
>>>>>>>>> http://www.rsyslog.com/doc/syslog_parsing.html
>>>>>>>>>
>>>>>>>>> Rainer
>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>>>>>>>> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
>>>>>>>>>> Sent: Wednesday, May 18, 2011 2:02 PM
>>>>>>>>>> To: rsyslog at lists.adiscon.com
>>>>>>>>>> Subject: [rsyslog] HOSTNAME and programname extraction
>>> failure,
>>>>> when
>>>>>>>>>> rsyslog as receiver, stock syslog as sender
>>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> I was trying to configure rsyslog(rsyslog-3.22.1-3.el5_5.1) as
>>>>>>>>>> receiver, stock syslog(sysklogd-1.4.1-46.el5) as sender.
>>>>>>>>>>
>>>>>>>>>> The rsyslogd listened on udp/514, and used dynamic filenames
>>> with
>>>>>>>>>> protocol23 ?message formatting:
>>>>>>>>>> $ModLoad imudp
>>>>>>>>>> $UDPServerRun 514
>>>>>>>>>> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
>>>>>>>>>> $template
>>>>>>>>>>
>>>>>
>>> DynFile,"/var/log/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/rsyslog.log"
>>>>>>>>>> *.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??DynFile
>>>>>>>>>>
>>>>>>>>>> The sender generated several entries(3rd, 4th) violating the
>>>>>>>>>> "syslogtag: message" convention
>>>>>>>>>> May 18 19:40:17 dns1 kernel: Kernel logging (proc) stopped.
>>>>>>>>>> May 18 19:40:17 dns1 kernel: Kernel log daemon terminating.
>>>>>>>>>> May 18 19:40:18 dns1 exiting on signal 15
>>>>>>>>>> May 18 19:40:18 dns1 syslogd 1.4.1: restart.
>>>>>>>>>> May 18 19:40:18 dns1 kernel: klogd 1.4.1, log source =
>>> /proc/kmsg
>>>>>>>>>> started.
>>>>>>>>>>
>>>>>>>>>> Take 3rd entry for example, the receiver mistook 'exiting' and
>>>>> 'on' to
>>>>>>>>>> be %HOSTNAME% and %programname%:
>>>>>>>>>> <6>1 2011-05-18T19:40:12.592370+08:00 bogon kernel - - -
>>> ?Kernel
>>>>>>>>>> logging (proc) stopped.
>>>>>>>>>> <6>1 2011-05-18T19:40:12.592693+08:00 bogon kernel - - -
>>> ?Kernel
>>>>> log
>>>>>>>>>> daemon terminating.
>>>>>>>>>> <46>1 2011-05-18T19:40:13.697115+08:00 exiting on - - -
>>> ?signal
>>>>> 15
>>>>>>>>>> <46>1 2011-05-18T19:40:13.806302+08:00 syslogd 1.4.1 - - -
>>>>> ?restart.
>>>>>>>>>> <6>1 2011-05-18T19:40:13.811331+08:00 bogon kernel - - -
>>> ?klogd
>>>>> 1.4.1,
>>>>>>>>>> log source = /proc/kmsg started.
>>>>>>>>>> So, it went to /var/log/hosts/exiting/2011/05/18/rsyslog.log,
>>> and
>>>>>>>>>> that's definitely the wrong place.
>>>>>>>>>>
>>>>>>>>>> How to deal with this case?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Kaiwang
>>>>>>>>>> _______________________________________________
>>>>>>>>>> rsyslog mailing list
>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>>> http://www.rsyslog.com
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> rsyslog mailing list
>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>> http://www.rsyslog.com
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> rsyslog mailing list
>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>> http://www.rsyslog.com
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> rsyslog mailing list
>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>> http://www.rsyslog.com
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com
>>>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From christian.lete at gmail.com  Mon May 23 10:28:39 2011
From: christian.lete at gmail.com (Christian Lete)
Date: Mon, 23 May 2011 10:28:39 +0200
Subject: [rsyslog] High availability on rsyslog (cluster)
Message-ID: <BANLkTin88E7L-4cohhfP+6ihVSe2DvkbQg@mail.gmail.com>

Hi,

I have a small question,  I would need to setup an rsyslog
receiver/forwarder, listening on udp port, since some clients, only
support this option. I would need this service to be highly
available(I don't want to have two machines and having duplicated
information), but since this udp, I can't be for sure if the service
is running fine. What I thought is to indirectly check it, by having
another port listening on tcp and checking the tcp service, if the
service is not running on tcp I would assume the whole system is down
and would failover to the other instance of the cluster, that's the
only way I could think of, do you currently have another way?


thank you very much,

Regards,

Christian

From rory at ooma.com  Mon May 23 21:57:27 2011
From: rory at ooma.com (Rory Toma)
Date: Mon, 23 May 2011 12:57:27 -0700
Subject: [rsyslog] Question on host failover
Message-ID: <4DDABC27.7080806@ooma.com>

I have the following for my clients' rsyslog.conf files:

$DefaultNetStreamDriverCAFile /etc/ca.pem

$DefaultNetStreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode anon

$ActionResumeInterval 29

$WorkDirectory /var/log

$ModLoad imuxsock
$SystemLogSocketName /var/log/log
$OptimizeForUniprocessor on

*.* @@<machine>:110
$ActionExecOnlyWhenPreviousIsSuspended on
*.* @@<machine>:143
$ActionExecOnlyWhenPreviousIsSuspended off



If I block port 110 from the client, I would expect that it would 
failover to port 143. I am not seeing this. Do I have to do something 
different here? It doesn't work if I remove the ActionExec directive, 
either. In either case, it seems to keep sending to port 110, no matter 
what. In this case, <machine> is the same physical machine, with the 
same IP address, it's just a different syslog receiver process.

This is rsyslog-5.6.2

thx

From david at lang.hm  Tue May 24 00:30:21 2011
From: david at lang.hm (david at lang.hm)
Date: Mon, 23 May 2011 15:30:21 -0700 (PDT)
Subject: [rsyslog] High availability on rsyslog (cluster)
In-Reply-To: <BANLkTin88E7L-4cohhfP+6ihVSe2DvkbQg@mail.gmail.com>
References: <BANLkTin88E7L-4cohhfP+6ihVSe2DvkbQg@mail.gmail.com>
Message-ID: <alpine.DEB.2.02.1105231529010.8167@asgard.lang.hm>

depending on how active your logging is, you could watch the logs and say 
that if you don't receive any logs for 1 min (or whatever time is 
approprate), somthing is wrong.

you could also generate known UDP logs to yourself and alert if they don't 
show up.

David Lang

  On Mon, 23 May 2011, Christian Lete wrote:

> Hi,
>
> I have a small question,  I would need to setup an rsyslog
> receiver/forwarder, listening on udp port, since some clients, only
> support this option. I would need this service to be highly
> available(I don't want to have two machines and having duplicated
> information), but since this udp, I can't be for sure if the service
> is running fine. What I thought is to indirectly check it, by having
> another port listening on tcp and checking the tcp service, if the
> service is not running on tcp I would assume the whole system is down
> and would failover to the other instance of the cluster, that's the
> only way I could think of, do you currently have another way?
>
>
> thank you very much,
>
> Regards,
>
> Christian
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From rgerhards at hq.adiscon.com  Tue May 24 07:24:48 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 24 May 2011 07:24:48 +0200
Subject: [rsyslog] Question on host failover
In-Reply-To: <4DDABC27.7080806@ooma.com>
References: <4DDABC27.7080806@ooma.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280D20@GRFEXC.intern.adiscon.com>

This sounds like a bug that has been fixed in current builds. I suggest
updating to 5.8.1.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rory Toma
> Sent: Monday, May 23, 2011 9:57 PM
> To: rsyslog-users
> Subject: [rsyslog] Question on host failover
> 
> I have the following for my clients' rsyslog.conf files:
> 
> $DefaultNetStreamDriverCAFile /etc/ca.pem
> 
> $DefaultNetStreamDriver gtls
> $ActionSendStreamDriverMode 1
> $ActionSendStreamDriverAuthMode anon
> 
> $ActionResumeInterval 29
> 
> $WorkDirectory /var/log
> 
> $ModLoad imuxsock
> $SystemLogSocketName /var/log/log
> $OptimizeForUniprocessor on
> 
> *.* @@<machine>:110
> $ActionExecOnlyWhenPreviousIsSuspended on
> *.* @@<machine>:143
> $ActionExecOnlyWhenPreviousIsSuspended off
> 
> 
> 
> If I block port 110 from the client, I would expect that it would
> failover to port 143. I am not seeing this. Do I have to do something
> different here? It doesn't work if I remove the ActionExec directive,
> either. In either case, it seems to keep sending to port 110, no matter
> what. In this case, <machine> is the same physical machine, with the
> same IP address, it's just a different syslog receiver process.
> 
> This is rsyslog-5.6.2
> 
> thx
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Tue May 24 07:26:10 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 24 May 2011 07:26:10 +0200
Subject: [rsyslog] High availability on rsyslog (cluster)
In-Reply-To: <alpine.DEB.2.02.1105231529010.8167@asgard.lang.hm>
References: <BANLkTin88E7L-4cohhfP+6ihVSe2DvkbQg@mail.gmail.com>
	<alpine.DEB.2.02.1105231529010.8167@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280D21@GRFEXC.intern.adiscon.com>

David and all,

are you aware of any high availability APIs that would enable rsyslog to do
some kind of automatic failover in a cluster environment? I have never
specifically programmed for that and wonder if there are any options.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, May 24, 2011 12:30 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
> 
> depending on how active your logging is, you could watch the logs and
> say
> that if you don't receive any logs for 1 min (or whatever time is
> approprate), somthing is wrong.
> 
> you could also generate known UDP logs to yourself and alert if they
> don't
> show up.
> 
> David Lang
> 
>   On Mon, 23 May 2011, Christian Lete wrote:
> 
> > Hi,
> >
> > I have a small question,  I would need to setup an rsyslog
> > receiver/forwarder, listening on udp port, since some clients, only
> > support this option. I would need this service to be highly
> > available(I don't want to have two machines and having duplicated
> > information), but since this udp, I can't be for sure if the service
> > is running fine. What I thought is to indirectly check it, by having
> > another port listening on tcp and checking the tcp service, if the
> > service is not running on tcp I would assume the whole system is down
> > and would failover to the other instance of the cluster, that's the
> > only way I could think of, do you currently have another way?
> >
> >
> > thank you very much,
> >
> > Regards,
> >
> > Christian
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From david at lang.hm  Tue May 24 08:08:18 2011
From: david at lang.hm (david at lang.hm)
Date: Mon, 23 May 2011 23:08:18 -0700 (PDT)
Subject: [rsyslog] High availability on rsyslog (cluster)
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280D21@GRFEXC.intern.adiscon.com>
References: <BANLkTin88E7L-4cohhfP+6ihVSe2DvkbQg@mail.gmail.com>
	<alpine.DEB.2.02.1105231529010.8167@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7280D21@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.2.02.1105232304290.8167@asgard.lang.hm>

take a look at linux-ha

It's a framework to manage HA (including active/active load sharing, 
quorums, etc)

it extends the traditional init.d startup scripts to also include a 
'status' call to tell if the service is active or not. the framework calls 
this service periodically and if the service fails, it does a failover. 
With the correct configuration (and software), it can do sub-second 
failover.

David Lang


  On 
Tue, 24 May 2011, Rainer Gerhards wrote:

> David and all,
>
> are you aware of any high availability APIs that would enable rsyslog to do
> some kind of automatic failover in a cluster environment? I have never
> specifically programmed for that and wonder if there are any options.
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Tuesday, May 24, 2011 12:30 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>
>> depending on how active your logging is, you could watch the logs and
>> say
>> that if you don't receive any logs for 1 min (or whatever time is
>> approprate), somthing is wrong.
>>
>> you could also generate known UDP logs to yourself and alert if they
>> don't
>> show up.
>>
>> David Lang
>>
>>   On Mon, 23 May 2011, Christian Lete wrote:
>>
>>> Hi,
>>>
>>> I have a small question,  I would need to setup an rsyslog
>>> receiver/forwarder, listening on udp port, since some clients, only
>>> support this option. I would need this service to be highly
>>> available(I don't want to have two machines and having duplicated
>>> information), but since this udp, I can't be for sure if the service
>>> is running fine. What I thought is to indirectly check it, by having
>>> another port listening on tcp and checking the tcp service, if the
>>> service is not running on tcp I would assume the whole system is down
>>> and would failover to the other instance of the cluster, that's the
>>> only way I could think of, do you currently have another way?
>>>
>>>
>>> thank you very much,
>>>
>>> Regards,
>>>
>>> Christian
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From rgerhards at hq.adiscon.com  Tue May 24 08:09:28 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 24 May 2011 08:09:28 +0200
Subject: [rsyslog] High availability on rsyslog (cluster)
In-Reply-To: <alpine.DEB.2.02.1105232304290.8167@asgard.lang.hm>
References: <BANLkTin88E7L-4cohhfP+6ihVSe2DvkbQg@mail.gmail.com><alpine.DEB.2.02.1105231529010.8167@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA7280D21@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.02.1105232304290.8167@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280D22@GRFEXC.intern.adiscon.com>

Thx -- sounds interesting and probably not too much work to do...

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, May 24, 2011 8:08 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
> 
> take a look at linux-ha
> 
> It's a framework to manage HA (including active/active load sharing,
> quorums, etc)
> 
> it extends the traditional init.d startup scripts to also include a
> 'status' call to tell if the service is active or not. the framework
> calls
> this service periodically and if the service fails, it does a failover.
> With the correct configuration (and software), it can do sub-second
> failover.
> 
> David Lang
> 
> 
>   On
> Tue, 24 May 2011, Rainer Gerhards wrote:
> 
> > David and all,
> >
> > are you aware of any high availability APIs that would enable rsyslog
> to do
> > some kind of automatic failover in a cluster environment? I have
> never
> > specifically programmed for that and wonder if there are any options.
> >
> > Rainer
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >> Sent: Tuesday, May 24, 2011 12:30 AM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
> >>
> >> depending on how active your logging is, you could watch the logs
> and
> >> say
> >> that if you don't receive any logs for 1 min (or whatever time is
> >> approprate), somthing is wrong.
> >>
> >> you could also generate known UDP logs to yourself and alert if they
> >> don't
> >> show up.
> >>
> >> David Lang
> >>
> >>   On Mon, 23 May 2011, Christian Lete wrote:
> >>
> >>> Hi,
> >>>
> >>> I have a small question,  I would need to setup an rsyslog
> >>> receiver/forwarder, listening on udp port, since some clients, only
> >>> support this option. I would need this service to be highly
> >>> available(I don't want to have two machines and having duplicated
> >>> information), but since this udp, I can't be for sure if the
> service
> >>> is running fine. What I thought is to indirectly check it, by
> having
> >>> another port listening on tcp and checking the tcp service, if the
> >>> service is not running on tcp I would assume the whole system is
> down
> >>> and would failover to the other instance of the cluster, that's the
> >>> only way I could think of, do you currently have another way?
> >>>
> >>>
> >>> thank you very much,
> >>>
> >>> Regards,
> >>>
> >>> Christian
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >>>
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From david at lang.hm  Tue May 24 08:12:41 2011
From: david at lang.hm (david at lang.hm)
Date: Mon, 23 May 2011 23:12:41 -0700 (PDT)
Subject: [rsyslog] High availability on rsyslog (cluster)
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280D22@GRFEXC.intern.adiscon.com>
References: <BANLkTin88E7L-4cohhfP+6ihVSe2DvkbQg@mail.gmail.com><alpine.DEB.2.02.1105231529010.8167@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA7280D21@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.02.1105232304290.8167@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7280D22@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.2.02.1105232312020.8167@asgard.lang.hm>

take a look at

http://linux-ha.org/wiki/Resource_Agents

and

http://www.linux-ha.org/doc/dev-guides/ra-dev-guide.html

David Lang

On Tue, 24 May 2011, Rainer Gerhards wrote:

> Date: Tue, 24 May 2011 08:09:28 +0200
> From: Rainer Gerhards <rgerhards at hq.adiscon.com>
> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> To: rsyslog-users <rsyslog at lists.adiscon.com>
> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
> 
> Thx -- sounds interesting and probably not too much work to do...
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Tuesday, May 24, 2011 8:08 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>
>> take a look at linux-ha
>>
>> It's a framework to manage HA (including active/active load sharing,
>> quorums, etc)
>>
>> it extends the traditional init.d startup scripts to also include a
>> 'status' call to tell if the service is active or not. the framework
>> calls
>> this service periodically and if the service fails, it does a failover.
>> With the correct configuration (and software), it can do sub-second
>> failover.
>>
>> David Lang
>>
>>
>>   On
>> Tue, 24 May 2011, Rainer Gerhards wrote:
>>
>>> David and all,
>>>
>>> are you aware of any high availability APIs that would enable rsyslog
>> to do
>>> some kind of automatic failover in a cluster environment? I have
>> never
>>> specifically programmed for that and wonder if there are any options.
>>>
>>> Rainer
>>>
>>>> -----Original Message-----
>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>>> Sent: Tuesday, May 24, 2011 12:30 AM
>>>> To: rsyslog-users
>>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>>>
>>>> depending on how active your logging is, you could watch the logs
>> and
>>>> say
>>>> that if you don't receive any logs for 1 min (or whatever time is
>>>> approprate), somthing is wrong.
>>>>
>>>> you could also generate known UDP logs to yourself and alert if they
>>>> don't
>>>> show up.
>>>>
>>>> David Lang
>>>>
>>>>   On Mon, 23 May 2011, Christian Lete wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have a small question,  I would need to setup an rsyslog
>>>>> receiver/forwarder, listening on udp port, since some clients, only
>>>>> support this option. I would need this service to be highly
>>>>> available(I don't want to have two machines and having duplicated
>>>>> information), but since this udp, I can't be for sure if the
>> service
>>>>> is running fine. What I thought is to indirectly check it, by
>> having
>>>>> another port listening on tcp and checking the tcp service, if the
>>>>> service is not running on tcp I would assume the whole system is
>> down
>>>>> and would failover to the other instance of the cluster, that's the
>>>>> only way I could think of, do you currently have another way?
>>>>>
>>>>>
>>>>> thank you very much,
>>>>>
>>>>> Regards,
>>>>>
>>>>> Christian
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com
>>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From david at lang.hm  Tue May 24 08:14:33 2011
From: david at lang.hm (david at lang.hm)
Date: Mon, 23 May 2011 23:14:33 -0700 (PDT)
Subject: [rsyslog] High availability on rsyslog (cluster)
In-Reply-To: <alpine.DEB.2.02.1105232312020.8167@asgard.lang.hm>
References: <BANLkTin88E7L-4cohhfP+6ihVSe2DvkbQg@mail.gmail.com><alpine.DEB.2.02.1105231529010.8167@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA7280D21@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.02.1105232304290.8167@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7280D22@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.02.1105232312020.8167@asgard.lang.hm>
Message-ID: <alpine.DEB.2.02.1105232312590.8167@asgard.lang.hm>

One interesting twist to deal with, when a system is not active, it 
probably still wants to send it's logs somewhere.

two ways of addresssing this

1. two configs, switching between active and passive changes which config 
you use

2. two instances, one used by the local box to send it's logs, the other 
to receive logs from elsewhere.

David Lang

On Mon, 23 May 2011, david at lang.hm wrote:

> take a look at
>
> http://linux-ha.org/wiki/Resource_Agents
>
> and
>
> http://www.linux-ha.org/doc/dev-guides/ra-dev-guide.html
>
> David Lang
>
> On Tue, 24 May 2011, Rainer Gerhards wrote:
>
>> Date: Tue, 24 May 2011 08:09:28 +0200
>> From: Rainer Gerhards <rgerhards at hq.adiscon.com>
>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>> To: rsyslog-users <rsyslog at lists.adiscon.com>
>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>> 
>> Thx -- sounds interesting and probably not too much work to do...
>> 
>> Rainer
>> 
>>> -----Original Message-----
>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>> Sent: Tuesday, May 24, 2011 8:08 AM
>>> To: rsyslog-users
>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>> 
>>> take a look at linux-ha
>>> 
>>> It's a framework to manage HA (including active/active load sharing,
>>> quorums, etc)
>>> 
>>> it extends the traditional init.d startup scripts to also include a
>>> 'status' call to tell if the service is active or not. the framework
>>> calls
>>> this service periodically and if the service fails, it does a failover.
>>> With the correct configuration (and software), it can do sub-second
>>> failover.
>>> 
>>> David Lang
>>> 
>>>
>>>   On
>>> Tue, 24 May 2011, Rainer Gerhards wrote:
>>> 
>>>> David and all,
>>>> 
>>>> are you aware of any high availability APIs that would enable rsyslog
>>> to do
>>>> some kind of automatic failover in a cluster environment? I have
>>> never
>>>> specifically programmed for that and wonder if there are any options.
>>>> 
>>>> Rainer
>>>> 
>>>>> -----Original Message-----
>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>>>> Sent: Tuesday, May 24, 2011 12:30 AM
>>>>> To: rsyslog-users
>>>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>>>> 
>>>>> depending on how active your logging is, you could watch the logs
>>> and
>>>>> say
>>>>> that if you don't receive any logs for 1 min (or whatever time is
>>>>> approprate), somthing is wrong.
>>>>> 
>>>>> you could also generate known UDP logs to yourself and alert if they
>>>>> don't
>>>>> show up.
>>>>> 
>>>>> David Lang
>>>>>
>>>>>   On Mon, 23 May 2011, Christian Lete wrote:
>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> I have a small question,  I would need to setup an rsyslog
>>>>>> receiver/forwarder, listening on udp port, since some clients, only
>>>>>> support this option. I would need this service to be highly
>>>>>> available(I don't want to have two machines and having duplicated
>>>>>> information), but since this udp, I can't be for sure if the
>>> service
>>>>>> is running fine. What I thought is to indirectly check it, by
>>> having
>>>>>> another port listening on tcp and checking the tcp service, if the
>>>>>> service is not running on tcp I would assume the whole system is
>>> down
>>>>>> and would failover to the other instance of the cluster, that's the
>>>>>> only way I could think of, do you currently have another way?
>>>>>> 
>>>>>> 
>>>>>> thank you very much,
>>>>>> 
>>>>>> Regards,
>>>>>> 
>>>>>> Christian
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com
>>>>>> 
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>> 
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From Ole.Rahn at t-systems.com  Wed May 25 09:26:47 2011
From: Ole.Rahn at t-systems.com (Ole.Rahn at t-systems.com)
Date: Wed, 25 May 2011 09:26:47 +0200
Subject: [rsyslog] Heavy stability problems when using TLS
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DE1AD@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA71DE1AC@GRFEXC.intern.adiscon.com>
	<9B6E2A8877C38245BFB15CC491A11DA71DE1AD@GRFEXC.intern.adiscon.com>
Message-ID: <50CCFFD1B9C6424389383F7A13A03446013CFA56003C@HE101451.emea1.cds.t-internal.com>

> What I am looking for is some better information on when the crash happens.
> So valgrind is often a good choice to check if there are some violations. Not sure, though, if that's available on Solaris.
> 
> Rainer

Nope, valgrind is not available on Solaris/Sparc ...

Can't compile it:
(...)
checking for a supported CPU... no (sparc)
configure: error: Unsupported host architecture. Sorry

see also:
http://valgrind.org/info/platforms.html


Ole

From christian.lete at gmail.com  Wed May 25 10:10:05 2011
From: christian.lete at gmail.com (Christian Lete)
Date: Wed, 25 May 2011 10:10:05 +0200
Subject: [rsyslog] High availability on rsyslog (cluster)
In-Reply-To: <alpine.DEB.2.02.1105232312590.8167@asgard.lang.hm>
References: <BANLkTin88E7L-4cohhfP+6ihVSe2DvkbQg@mail.gmail.com>
	<alpine.DEB.2.02.1105231529010.8167@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7280D21@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.02.1105232304290.8167@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7280D22@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.02.1105232312020.8167@asgard.lang.hm>
	<alpine.DEB.2.02.1105232312590.8167@asgard.lang.hm>
Message-ID: <BANLkTi=2rwbrCWKqi3wqw_xcOuXE0UYvsw@mail.gmail.com>

Hi,

Thank you all for you help and suggestions, I will do some tests and
let you know how works for me.

Regards,

Christian

On 24 May 2011 08:14,  <david at lang.hm> wrote:
> One interesting twist to deal with, when a system is not active, it probably
> still wants to send it's logs somewhere.
>
> two ways of addresssing this
>
> 1. two configs, switching between active and passive changes which config
> you use
>
> 2. two instances, one used by the local box to send it's logs, the other to
> receive logs from elsewhere.
>
> David Lang
>
> On Mon, 23 May 2011, david at lang.hm wrote:
>
>> take a look at
>>
>> http://linux-ha.org/wiki/Resource_Agents
>>
>> and
>>
>> http://www.linux-ha.org/doc/dev-guides/ra-dev-guide.html
>>
>> David Lang
>>
>> On Tue, 24 May 2011, Rainer Gerhards wrote:
>>
>>> Date: Tue, 24 May 2011 08:09:28 +0200
>>> From: Rainer Gerhards <rgerhards at hq.adiscon.com>
>>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>>> To: rsyslog-users <rsyslog at lists.adiscon.com>
>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>>
>>> Thx -- sounds interesting and probably not too much work to do...
>>>
>>> Rainer
>>>
>>>> -----Original Message-----
>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>>> Sent: Tuesday, May 24, 2011 8:08 AM
>>>> To: rsyslog-users
>>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>>>
>>>> take a look at linux-ha
>>>>
>>>> It's a framework to manage HA (including active/active load sharing,
>>>> quorums, etc)
>>>>
>>>> it extends the traditional init.d startup scripts to also include a
>>>> 'status' call to tell if the service is active or not. the framework
>>>> calls
>>>> this service periodically and if the service fails, it does a failover.
>>>> With the correct configuration (and software), it can do sub-second
>>>> failover.
>>>>
>>>> David Lang
>>>>
>>>>
>>>> ?On
>>>> Tue, 24 May 2011, Rainer Gerhards wrote:
>>>>
>>>>> David and all,
>>>>>
>>>>> are you aware of any high availability APIs that would enable rsyslog
>>>>
>>>> to do
>>>>>
>>>>> some kind of automatic failover in a cluster environment? I have
>>>>
>>>> never
>>>>>
>>>>> specifically programmed for that and wonder if there are any options.
>>>>>
>>>>> Rainer
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>>>>> Sent: Tuesday, May 24, 2011 12:30 AM
>>>>>> To: rsyslog-users
>>>>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>>>>>
>>>>>> depending on how active your logging is, you could watch the logs
>>>>
>>>> and
>>>>>>
>>>>>> say
>>>>>> that if you don't receive any logs for 1 min (or whatever time is
>>>>>> approprate), somthing is wrong.
>>>>>>
>>>>>> you could also generate known UDP logs to yourself and alert if they
>>>>>> don't
>>>>>> show up.
>>>>>>
>>>>>> David Lang
>>>>>>
>>>>>> ?On Mon, 23 May 2011, Christian Lete wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I have a small question, ?I would need to setup an rsyslog
>>>>>>> receiver/forwarder, listening on udp port, since some clients, only
>>>>>>> support this option. I would need this service to be highly
>>>>>>> available(I don't want to have two machines and having duplicated
>>>>>>> information), but since this udp, I can't be for sure if the
>>>>
>>>> service
>>>>>>>
>>>>>>> is running fine. What I thought is to indirectly check it, by
>>>>
>>>> having
>>>>>>>
>>>>>>> another port listening on tcp and checking the tcp service, if the
>>>>>>> service is not running on tcp I would assume the whole system is
>>>>
>>>> down
>>>>>>>
>>>>>>> and would failover to the other instance of the cluster, that's the
>>>>>>> only way I could think of, do you currently have another way?
>>>>>>>
>>>>>>>
>>>>>>> thank you very much,
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Christian
>>>>>>> _______________________________________________
>>>>>>> rsyslog mailing list
>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>> http://www.rsyslog.com
>>>>>>>
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com
>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com
>>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From rgerhards at hq.adiscon.com  Wed May 25 10:12:37 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 25 May 2011 10:12:37 +0200
Subject: [rsyslog] High availability on rsyslog (cluster)
In-Reply-To: <BANLkTi=2rwbrCWKqi3wqw_xcOuXE0UYvsw@mail.gmail.com>
References: <BANLkTin88E7L-4cohhfP+6ihVSe2DvkbQg@mail.gmail.com><alpine.DEB.2.02.1105231529010.8167@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA7280D21@GRFEXC.intern.adiscon.com><alpine.DEB.2.02.1105232304290.8167@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA7280D22@GRFEXC.intern.adiscon.com><alpine.DEB.2.02.1105232312020.8167@asgard.lang.hm><alpine.DEB.2.02.1105232312590.8167@asgard.lang.hm>
	<BANLkTi=2rwbrCWKqi3wqw_xcOuXE0UYvsw@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280D43@GRFEXC.intern.adiscon.com>

Christian,

would you also be available for some tests *if* I find time to look into the
HA specific calls? I don't have an environment yet and it would probably take
me too long to set one up for a quick test. No promise thought that I can
implement the support quickly, depends on what else is going on...

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Christian Lete
> Sent: Wednesday, May 25, 2011 10:10 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
> 
> Hi,
> 
> Thank you all for you help and suggestions, I will do some tests and
> let you know how works for me.
> 
> Regards,
> 
> Christian
> 
> On 24 May 2011 08:14,  <david at lang.hm> wrote:
> > One interesting twist to deal with, when a system is not active, it
> probably
> > still wants to send it's logs somewhere.
> >
> > two ways of addresssing this
> >
> > 1. two configs, switching between active and passive changes which
> config
> > you use
> >
> > 2. two instances, one used by the local box to send it's logs, the
> other to
> > receive logs from elsewhere.
> >
> > David Lang
> >
> > On Mon, 23 May 2011, david at lang.hm wrote:
> >
> >> take a look at
> >>
> >> http://linux-ha.org/wiki/Resource_Agents
> >>
> >> and
> >>
> >> http://www.linux-ha.org/doc/dev-guides/ra-dev-guide.html
> >>
> >> David Lang
> >>
> >> On Tue, 24 May 2011, Rainer Gerhards wrote:
> >>
> >>> Date: Tue, 24 May 2011 08:09:28 +0200
> >>> From: Rainer Gerhards <rgerhards at hq.adiscon.com>
> >>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> >>> To: rsyslog-users <rsyslog at lists.adiscon.com>
> >>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
> >>>
> >>> Thx -- sounds interesting and probably not too much work to do...
> >>>
> >>> Rainer
> >>>
> >>>> -----Original Message-----
> >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >>>> Sent: Tuesday, May 24, 2011 8:08 AM
> >>>> To: rsyslog-users
> >>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
> >>>>
> >>>> take a look at linux-ha
> >>>>
> >>>> It's a framework to manage HA (including active/active load
> sharing,
> >>>> quorums, etc)
> >>>>
> >>>> it extends the traditional init.d startup scripts to also include
> a
> >>>> 'status' call to tell if the service is active or not. the
> framework
> >>>> calls
> >>>> this service periodically and if the service fails, it does a
> failover.
> >>>> With the correct configuration (and software), it can do sub-
> second
> >>>> failover.
> >>>>
> >>>> David Lang
> >>>>
> >>>>
> >>>> ?On
> >>>> Tue, 24 May 2011, Rainer Gerhards wrote:
> >>>>
> >>>>> David and all,
> >>>>>
> >>>>> are you aware of any high availability APIs that would enable
> rsyslog
> >>>>
> >>>> to do
> >>>>>
> >>>>> some kind of automatic failover in a cluster environment? I have
> >>>>
> >>>> never
> >>>>>
> >>>>> specifically programmed for that and wonder if there are any
> options.
> >>>>>
> >>>>> Rainer
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >>>>>> Sent: Tuesday, May 24, 2011 12:30 AM
> >>>>>> To: rsyslog-users
> >>>>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
> >>>>>>
> >>>>>> depending on how active your logging is, you could watch the
> logs
> >>>>
> >>>> and
> >>>>>>
> >>>>>> say
> >>>>>> that if you don't receive any logs for 1 min (or whatever time
> is
> >>>>>> approprate), somthing is wrong.
> >>>>>>
> >>>>>> you could also generate known UDP logs to yourself and alert if
> they
> >>>>>> don't
> >>>>>> show up.
> >>>>>>
> >>>>>> David Lang
> >>>>>>
> >>>>>> ?On Mon, 23 May 2011, Christian Lete wrote:
> >>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> I have a small question, ?I would need to setup an rsyslog
> >>>>>>> receiver/forwarder, listening on udp port, since some clients,
> only
> >>>>>>> support this option. I would need this service to be highly
> >>>>>>> available(I don't want to have two machines and having
> duplicated
> >>>>>>> information), but since this udp, I can't be for sure if the
> >>>>
> >>>> service
> >>>>>>>
> >>>>>>> is running fine. What I thought is to indirectly check it, by
> >>>>
> >>>> having
> >>>>>>>
> >>>>>>> another port listening on tcp and checking the tcp service, if
> the
> >>>>>>> service is not running on tcp I would assume the whole system
> is
> >>>>
> >>>> down
> >>>>>>>
> >>>>>>> and would failover to the other instance of the cluster, that's
> the
> >>>>>>> only way I could think of, do you currently have another way?
> >>>>>>>
> >>>>>>>
> >>>>>>> thank you very much,
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>>
> >>>>>>> Christian
> >>>>>>> _______________________________________________
> >>>>>>> rsyslog mailing list
> >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>>>> http://www.rsyslog.com
> >>>>>>>
> >>>>>> _______________________________________________
> >>>>>> rsyslog mailing list
> >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>>> http://www.rsyslog.com
> >>>>>
> >>>>> _______________________________________________
> >>>>> rsyslog mailing list
> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>> http://www.rsyslog.com
> >>>>>
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.com
> >>>
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >>>
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >>
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Wed May 25 10:13:14 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 25 May 2011 10:13:14 +0200
Subject: [rsyslog] Heavy stability problems when using TLS
In-Reply-To: <50CCFFD1B9C6424389383F7A13A03446013CFA56003C@HE101451.emea1.cds.t-internal.com>
References: <9B6E2A8877C38245BFB15CC491A11DA71DE1AC@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DE1AD@GRFEXC.intern.adiscon.com>
	<50CCFFD1B9C6424389383F7A13A03446013CFA56003C@HE101451.emea1.cds.t-internal.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280D44@GRFEXC.intern.adiscon.com>

OK, that makes it a bit harder ;)

Rainer
PS: I got the debug log, will try to make some sense out of it today

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Ole.Rahn at t-systems.com
> Sent: Wednesday, May 25, 2011 9:27 AM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] Heavy stability problems when using TLS
> 
> > What I am looking for is some better information on when the crash
> happens.
> > So valgrind is often a good choice to check if there are some
> violations. Not sure, though, if that's available on Solaris.
> >
> > Rainer
> 
> Nope, valgrind is not available on Solaris/Sparc ...
> 
> Can't compile it:
> (...)
> checking for a supported CPU... no (sparc)
> configure: error: Unsupported host architecture. Sorry
> 
> see also:
> http://valgrind.org/info/platforms.html
> 
> 
> Ole
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From christian.lete at gmail.com  Wed May 25 10:14:32 2011
From: christian.lete at gmail.com (Christian Lete)
Date: Wed, 25 May 2011 10:14:32 +0200
Subject: [rsyslog] High availability on rsyslog (cluster)
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280D43@GRFEXC.intern.adiscon.com>
References: <BANLkTin88E7L-4cohhfP+6ihVSe2DvkbQg@mail.gmail.com>
	<alpine.DEB.2.02.1105231529010.8167@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7280D21@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.02.1105232304290.8167@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7280D22@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.02.1105232312020.8167@asgard.lang.hm>
	<alpine.DEB.2.02.1105232312590.8167@asgard.lang.hm>
	<BANLkTi=2rwbrCWKqi3wqw_xcOuXE0UYvsw@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280D43@GRFEXC.intern.adiscon.com>
Message-ID: <BANLkTinGYhUiMdMRb7-T=h+bFmYJdoWvug@mail.gmail.com>

Hi Rainer,


Yes, I would be very  glad to help if you have the time to develop that!

Thanks!

Christian

On 25 May 2011 10:12, Rainer Gerhards <rgerhards at hq.adiscon.com> wrote:
> Christian,
>
> would you also be available for some tests *if* I find time to look into the
> HA specific calls? I don't have an environment yet and it would probably take
> me too long to set one up for a quick test. No promise thought that I can
> implement the support quickly, depends on what else is going on...
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Christian Lete
>> Sent: Wednesday, May 25, 2011 10:10 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>
>> Hi,
>>
>> Thank you all for you help and suggestions, I will do some tests and
>> let you know how works for me.
>>
>> Regards,
>>
>> Christian
>>
>> On 24 May 2011 08:14, ?<david at lang.hm> wrote:
>> > One interesting twist to deal with, when a system is not active, it
>> probably
>> > still wants to send it's logs somewhere.
>> >
>> > two ways of addresssing this
>> >
>> > 1. two configs, switching between active and passive changes which
>> config
>> > you use
>> >
>> > 2. two instances, one used by the local box to send it's logs, the
>> other to
>> > receive logs from elsewhere.
>> >
>> > David Lang
>> >
>> > On Mon, 23 May 2011, david at lang.hm wrote:
>> >
>> >> take a look at
>> >>
>> >> http://linux-ha.org/wiki/Resource_Agents
>> >>
>> >> and
>> >>
>> >> http://www.linux-ha.org/doc/dev-guides/ra-dev-guide.html
>> >>
>> >> David Lang
>> >>
>> >> On Tue, 24 May 2011, Rainer Gerhards wrote:
>> >>
>> >>> Date: Tue, 24 May 2011 08:09:28 +0200
>> >>> From: Rainer Gerhards <rgerhards at hq.adiscon.com>
>> >>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>> >>> To: rsyslog-users <rsyslog at lists.adiscon.com>
>> >>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>> >>>
>> >>> Thx -- sounds interesting and probably not too much work to do...
>> >>>
>> >>> Rainer
>> >>>
>> >>>> -----Original Message-----
>> >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> >>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> >>>> Sent: Tuesday, May 24, 2011 8:08 AM
>> >>>> To: rsyslog-users
>> >>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>> >>>>
>> >>>> take a look at linux-ha
>> >>>>
>> >>>> It's a framework to manage HA (including active/active load
>> sharing,
>> >>>> quorums, etc)
>> >>>>
>> >>>> it extends the traditional init.d startup scripts to also include
>> a
>> >>>> 'status' call to tell if the service is active or not. the
>> framework
>> >>>> calls
>> >>>> this service periodically and if the service fails, it does a
>> failover.
>> >>>> With the correct configuration (and software), it can do sub-
>> second
>> >>>> failover.
>> >>>>
>> >>>> David Lang
>> >>>>
>> >>>>
>> >>>> ?On
>> >>>> Tue, 24 May 2011, Rainer Gerhards wrote:
>> >>>>
>> >>>>> David and all,
>> >>>>>
>> >>>>> are you aware of any high availability APIs that would enable
>> rsyslog
>> >>>>
>> >>>> to do
>> >>>>>
>> >>>>> some kind of automatic failover in a cluster environment? I have
>> >>>>
>> >>>> never
>> >>>>>
>> >>>>> specifically programmed for that and wonder if there are any
>> options.
>> >>>>>
>> >>>>> Rainer
>> >>>>>
>> >>>>>> -----Original Message-----
>> >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> >>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> >>>>>> Sent: Tuesday, May 24, 2011 12:30 AM
>> >>>>>> To: rsyslog-users
>> >>>>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>> >>>>>>
>> >>>>>> depending on how active your logging is, you could watch the
>> logs
>> >>>>
>> >>>> and
>> >>>>>>
>> >>>>>> say
>> >>>>>> that if you don't receive any logs for 1 min (or whatever time
>> is
>> >>>>>> approprate), somthing is wrong.
>> >>>>>>
>> >>>>>> you could also generate known UDP logs to yourself and alert if
>> they
>> >>>>>> don't
>> >>>>>> show up.
>> >>>>>>
>> >>>>>> David Lang
>> >>>>>>
>> >>>>>> ?On Mon, 23 May 2011, Christian Lete wrote:
>> >>>>>>
>> >>>>>>> Hi,
>> >>>>>>>
>> >>>>>>> I have a small question, ?I would need to setup an rsyslog
>> >>>>>>> receiver/forwarder, listening on udp port, since some clients,
>> only
>> >>>>>>> support this option. I would need this service to be highly
>> >>>>>>> available(I don't want to have two machines and having
>> duplicated
>> >>>>>>> information), but since this udp, I can't be for sure if the
>> >>>>
>> >>>> service
>> >>>>>>>
>> >>>>>>> is running fine. What I thought is to indirectly check it, by
>> >>>>
>> >>>> having
>> >>>>>>>
>> >>>>>>> another port listening on tcp and checking the tcp service, if
>> the
>> >>>>>>> service is not running on tcp I would assume the whole system
>> is
>> >>>>
>> >>>> down
>> >>>>>>>
>> >>>>>>> and would failover to the other instance of the cluster, that's
>> the
>> >>>>>>> only way I could think of, do you currently have another way?
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> thank you very much,
>> >>>>>>>
>> >>>>>>> Regards,
>> >>>>>>>
>> >>>>>>> Christian
>> >>>>>>> _______________________________________________
>> >>>>>>> rsyslog mailing list
>> >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>>>>>> http://www.rsyslog.com
>> >>>>>>>
>> >>>>>> _______________________________________________
>> >>>>>> rsyslog mailing list
>> >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>>>>> http://www.rsyslog.com
>> >>>>>
>> >>>>> _______________________________________________
>> >>>>> rsyslog mailing list
>> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>>>> http://www.rsyslog.com
>> >>>>>
>> >>>> _______________________________________________
>> >>>> rsyslog mailing list
>> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>>> http://www.rsyslog.com
>> >>>
>> >>> _______________________________________________
>> >>> rsyslog mailing list
>> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>> http://www.rsyslog.com
>> >>>
>> >> _______________________________________________
>> >> rsyslog mailing list
>> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> http://www.rsyslog.com
>> >>
>> > _______________________________________________
>> > rsyslog mailing list
>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > http://www.rsyslog.com
>> >
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From rgerhards at hq.adiscon.com  Wed May 25 12:11:25 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 25 May 2011 12:11:25 +0200
Subject: [rsyslog] feedback request: imklog symbol translation
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280D4F@GRFEXC.intern.adiscon.com>

Hi all,

currently, imklog has some plumbing to do kernel symbol translation. However,
as of my understanding, even not-so-totally-recent kernel do this themselves.
It looks to me like the symbol translation support in imklog is a left-over
that really has no need today. In order to reduce code complexity, I am very
tempted to remove it in v6.

Comments on that plan would be appreciated.

Thanks,
Rainer

From Ole.Rahn at t-systems.com  Wed May 25 17:14:23 2011
From: Ole.Rahn at t-systems.com (Ole.Rahn at t-systems.com)
Date: Wed, 25 May 2011 17:14:23 +0200
Subject: [rsyslog] Heavy stability problems when using TLS
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280D44@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA71DE1AC@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DE1AD@GRFEXC.intern.adiscon.com>
	<50CCFFD1B9C6424389383F7A13A03446013CFA56003C@HE101451.emea1.cds.t-internal.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280D44@GRFEXC.intern.adiscon.com>
Message-ID: <50CCFFD1B9C6424389383F7A13A03446013CFA5C2B5A@HE101451.emea1.cds.t-internal.com>

Thank you very much for you support, Rainer!

I noticed one more thing, which might be related to the crashes: 
It seems like rsyslog is comsuming quite a bit memory, at times. While it allocates a handful of MB, most of the time, sometimes uses humungous amounts of memory (just like right now):

   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
 (...)
  2663 root     1455M 1452M sleep   59    0   0:01:07 0,0% rsyslogd/9 

If a shortage in available memory would cause the crashes, it would also explain, why the process dependent core (dump) files that solaris generates for rsyslog always are several GB, I guess.

If memory is short the server gets quite unresponsive (no wonder), which is a phenomenon that is also reported to me about the clients by the vendor who is running them.

Ole


P.S.
Taking even a bit more memory, now:
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
 (...)
  2663 root     1461M 1458M sleep   59    0   0:01:08 0,0% rsyslogd/9


-----Urspr?ngliche Nachricht-----
Von: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-bounces at lists.adiscon.com] 
Gesendet: Mittwoch, 25. Mai 2011 10:13
An: rsyslog-users
Betreff: Re: [rsyslog] Heavy stability problems when using TLS

OK, that makes it a bit harder ;)

Rainer
PS: I got the debug log, will try to make some sense out of it today


From rgerhards at hq.adiscon.com  Wed May 25 17:24:46 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 25 May 2011 17:24:46 +0200
Subject: [rsyslog] Heavy stability problems when using TLS
In-Reply-To: <50CCFFD1B9C6424389383F7A13A03446013CFA5C2B5A@HE101451.emea1.cds.t-internal.com>
References: <9B6E2A8877C38245BFB15CC491A11DA71DE1AC@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DE1AD@GRFEXC.intern.adiscon.com><50CCFFD1B9C6424389383F7A13A03446013CFA56003C@HE101451.emea1.cds.t-internal.com><9B6E2A8877C38245BFB15CC491A11DA7280D44@GRFEXC.intern.adiscon.com>
	<50CCFFD1B9C6424389383F7A13A03446013CFA5C2B5A@HE101451.emea1.cds.t-internal.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280D57@GRFEXC.intern.adiscon.com>

What are your queue sizes?
Rainer
PS: I am off to a conference soon for the rest of the week

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Ole.Rahn at t-systems.com
> Sent: Wednesday, May 25, 2011 5:14 PM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] Heavy stability problems when using TLS
> 
> Thank you very much for you support, Rainer!
> 
> I noticed one more thing, which might be related to the crashes:
> It seems like rsyslog is comsuming quite a bit memory, at times. While
> it allocates a handful of MB, most of the time, sometimes uses
> humungous amounts of memory (just like right now):
> 
>    PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
>  (...)
>   2663 root     1455M 1452M sleep   59    0   0:01:07 0,0% rsyslogd/9
> 
> If a shortage in available memory would cause the crashes, it would
> also explain, why the process dependent core (dump) files that solaris
> generates for rsyslog always are several GB, I guess.
> 
> If memory is short the server gets quite unresponsive (no wonder),
> which is a phenomenon that is also reported to me about the clients by
> the vendor who is running them.
> 
> Ole
> 
> 
> P.S.
> Taking even a bit more memory, now:
>    PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
>  (...)
>   2663 root     1461M 1458M sleep   59    0   0:01:08 0,0% rsyslogd/9
> 
> 
> -----Urspr?ngliche Nachricht-----
> Von: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com]
> Gesendet: Mittwoch, 25. Mai 2011 10:13
> An: rsyslog-users
> Betreff: Re: [rsyslog] Heavy stability problems when using TLS
> 
> OK, that makes it a bit harder ;)
> 
> Rainer
> PS: I got the debug log, will try to make some sense out of it today
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From Ole.Rahn at t-systems.com  Wed May 25 17:34:25 2011
From: Ole.Rahn at t-systems.com (Ole.Rahn at t-systems.com)
Date: Wed, 25 May 2011 17:34:25 +0200
Subject: [rsyslog] Heavy stability problems when using TLS
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280D57@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA71DE1AC@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DE1AD@GRFEXC.intern.adiscon.com><50CCFFD1B9C6424389383F7A13A03446013CFA56003C@HE101451.emea1.cds.t-internal.com><9B6E2A8877C38245BFB15CC491A11DA7280D44@GRFEXC.intern.adiscon.com>
	<50CCFFD1B9C6424389383F7A13A03446013CFA5C2B5A@HE101451.emea1.cds.t-internal.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280D57@GRFEXC.intern.adiscon.com>
Message-ID: <50CCFFD1B9C6424389383F7A13A03446013CFA5C2B89@HE101451.emea1.cds.t-internal.com>

So far it is just this line on the server in regard queues:
$MainMsgQueueSize 100000 # 100,000 may be a value to handle burst traffic

Will try to find out the client settings.


Ole


-----Urspr?ngliche Nachricht-----
Von: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-bounces at lists.adiscon.com] Im Auftrag von Rainer Gerhards
Gesendet: Mittwoch, 25. Mai 2011 17:25
An: rsyslog-users
Betreff: Re: [rsyslog] Heavy stability problems when using TLS

What are your queue sizes?
Rainer
PS: I am off to a conference soon for the rest of the week

From rgerhards at hq.adiscon.com  Wed May 25 18:07:17 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 25 May 2011 18:07:17 +0200
Subject: [rsyslog] Heavy stability problems when using TLS
In-Reply-To: <50CCFFD1B9C6424389383F7A13A03446013CFA5C2B89@HE101451.emea1.cds.t-internal.com>
References: <9B6E2A8877C38245BFB15CC491A11DA71DE1AC@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DE1AD@GRFEXC.intern.adiscon.com><50CCFFD1B9C6424389383F7A13A03446013CFA56003C@HE101451.emea1.cds.t-internal.com><9B6E2A8877C38245BFB15CC491A11DA7280D44@GRFEXC.intern.adiscon.com><50CCFFD1B9C6424389383F7A13A03446013CFA5C2B5A@HE101451.emea1.cds.t-internal.com><9B6E2A8877C38245BFB15CC491A11DA7280D57@GRFEXC.intern.adiscon.com>
	<50CCFFD1B9C6424389383F7A13A03446013CFA5C2B89@HE101451.emea1.cds.t-internal.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280D5B@GRFEXC.intern.adiscon.com>

Doesnt sound too much. Where did you see the big memory footprint? Client or
server? Maybe it would be useful to use impstats to see queue sizes.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Ole.Rahn at t-systems.com
> Sent: Wednesday, May 25, 2011 5:34 PM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] Heavy stability problems when using TLS
> 
> So far it is just this line on the server in regard queues:
> $MainMsgQueueSize 100000 # 100,000 may be a value to handle burst
> traffic
> 
> Will try to find out the client settings.
> 
> 
> Ole
> 
> 
> -----Urspr?ngliche Nachricht-----
> Von: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] Im Auftrag von Rainer Gerhards
> Gesendet: Mittwoch, 25. Mai 2011 17:25
> An: rsyslog-users
> Betreff: Re: [rsyslog] Heavy stability problems when using TLS
> 
> What are your queue sizes?
> Rainer
> PS: I am off to a conference soon for the rest of the week
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From david at lang.hm  Wed May 25 19:22:19 2011
From: david at lang.hm (david at lang.hm)
Date: Wed, 25 May 2011 10:22:19 -0700 (PDT)
Subject: [rsyslog] High availability on rsyslog (cluster)
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280D43@GRFEXC.intern.adiscon.com>
References: <BANLkTin88E7L-4cohhfP+6ihVSe2DvkbQg@mail.gmail.com><alpine.DEB.2.02.1105231529010.8167@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA7280D21@GRFEXC.intern.adiscon.com><alpine.DEB.2.02.1105232304290.8167@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA7280D22@GRFEXC.intern.adiscon.com><alpine.DEB.2.02.1105232312020.8167@asgard.lang.hm><alpine.DEB.2.02.1105232312590.8167@asgard.lang.hm>
	<BANLkTi=2rwbrCWKqi3wqw_xcOuXE0UYvsw@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280D43@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.2.02.1105251021320.23692@asgard.lang.hm>

I have many HA environments available for testing

What I am doing right now is to just run rsyslog normally on each system 
and move the IP address, but it would be nice to have service checking.

David Lang

On Wed, 25 May 2011, Rainer Gerhards wrote:

> Date: Wed, 25 May 2011 10:12:37 +0200
> From: Rainer Gerhards <rgerhards at hq.adiscon.com>
> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> To: rsyslog-users <rsyslog at lists.adiscon.com>
> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
> 
> Christian,
>
> would you also be available for some tests *if* I find time to look into the
> HA specific calls? I don't have an environment yet and it would probably take
> me too long to set one up for a quick test. No promise thought that I can
> implement the support quickly, depends on what else is going on...
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Christian Lete
>> Sent: Wednesday, May 25, 2011 10:10 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>
>> Hi,
>>
>> Thank you all for you help and suggestions, I will do some tests and
>> let you know how works for me.
>>
>> Regards,
>>
>> Christian
>>
>> On 24 May 2011 08:14,  <david at lang.hm> wrote:
>>> One interesting twist to deal with, when a system is not active, it
>> probably
>>> still wants to send it's logs somewhere.
>>>
>>> two ways of addresssing this
>>>
>>> 1. two configs, switching between active and passive changes which
>> config
>>> you use
>>>
>>> 2. two instances, one used by the local box to send it's logs, the
>> other to
>>> receive logs from elsewhere.
>>>
>>> David Lang
>>>
>>> On Mon, 23 May 2011, david at lang.hm wrote:
>>>
>>>> take a look at
>>>>
>>>> http://linux-ha.org/wiki/Resource_Agents
>>>>
>>>> and
>>>>
>>>> http://www.linux-ha.org/doc/dev-guides/ra-dev-guide.html
>>>>
>>>> David Lang
>>>>
>>>> On Tue, 24 May 2011, Rainer Gerhards wrote:
>>>>
>>>>> Date: Tue, 24 May 2011 08:09:28 +0200
>>>>> From: Rainer Gerhards <rgerhards at hq.adiscon.com>
>>>>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>>>>> To: rsyslog-users <rsyslog at lists.adiscon.com>
>>>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>>>>
>>>>> Thx -- sounds interesting and probably not too much work to do...
>>>>>
>>>>> Rainer
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>>>>> Sent: Tuesday, May 24, 2011 8:08 AM
>>>>>> To: rsyslog-users
>>>>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>>>>>
>>>>>> take a look at linux-ha
>>>>>>
>>>>>> It's a framework to manage HA (including active/active load
>> sharing,
>>>>>> quorums, etc)
>>>>>>
>>>>>> it extends the traditional init.d startup scripts to also include
>> a
>>>>>> 'status' call to tell if the service is active or not. the
>> framework
>>>>>> calls
>>>>>> this service periodically and if the service fails, it does a
>> failover.
>>>>>> With the correct configuration (and software), it can do sub-
>> second
>>>>>> failover.
>>>>>>
>>>>>> David Lang
>>>>>>
>>>>>>
>>>>>> ?On
>>>>>> Tue, 24 May 2011, Rainer Gerhards wrote:
>>>>>>
>>>>>>> David and all,
>>>>>>>
>>>>>>> are you aware of any high availability APIs that would enable
>> rsyslog
>>>>>>
>>>>>> to do
>>>>>>>
>>>>>>> some kind of automatic failover in a cluster environment? I have
>>>>>>
>>>>>> never
>>>>>>>
>>>>>>> specifically programmed for that and wonder if there are any
>> options.
>>>>>>>
>>>>>>> Rainer
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>>>>>>> Sent: Tuesday, May 24, 2011 12:30 AM
>>>>>>>> To: rsyslog-users
>>>>>>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>>>>>>>
>>>>>>>> depending on how active your logging is, you could watch the
>> logs
>>>>>>
>>>>>> and
>>>>>>>>
>>>>>>>> say
>>>>>>>> that if you don't receive any logs for 1 min (or whatever time
>> is
>>>>>>>> approprate), somthing is wrong.
>>>>>>>>
>>>>>>>> you could also generate known UDP logs to yourself and alert if
>> they
>>>>>>>> don't
>>>>>>>> show up.
>>>>>>>>
>>>>>>>> David Lang
>>>>>>>>
>>>>>>>> ?On Mon, 23 May 2011, Christian Lete wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I have a small question, ?I would need to setup an rsyslog
>>>>>>>>> receiver/forwarder, listening on udp port, since some clients,
>> only
>>>>>>>>> support this option. I would need this service to be highly
>>>>>>>>> available(I don't want to have two machines and having
>> duplicated
>>>>>>>>> information), but since this udp, I can't be for sure if the
>>>>>>
>>>>>> service
>>>>>>>>>
>>>>>>>>> is running fine. What I thought is to indirectly check it, by
>>>>>>
>>>>>> having
>>>>>>>>>
>>>>>>>>> another port listening on tcp and checking the tcp service, if
>> the
>>>>>>>>> service is not running on tcp I would assume the whole system
>> is
>>>>>>
>>>>>> down
>>>>>>>>>
>>>>>>>>> and would failover to the other instance of the cluster, that's
>> the
>>>>>>>>> only way I could think of, do you currently have another way?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> thank you very much,
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Christian
>>>>>>>>> _______________________________________________
>>>>>>>>> rsyslog mailing list
>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>> http://www.rsyslog.com
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> rsyslog mailing list
>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>> http://www.rsyslog.com
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> rsyslog mailing list
>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>> http://www.rsyslog.com
>>>>>>>
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com
>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com
>>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From david at lang.hm  Wed May 25 19:24:10 2011
From: david at lang.hm (david at lang.hm)
Date: Wed, 25 May 2011 10:24:10 -0700 (PDT)
Subject: [rsyslog] feedback request: imklog symbol translation
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280D4F@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA7280D4F@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.2.02.1105251023530.23692@asgard.lang.hm>

how old are the kernels that do it themselves?

David Lang

On Wed, 25 May 2011, Rainer Gerhards wrote:

> Date: Wed, 25 May 2011 12:11:25 +0200
> From: Rainer Gerhards <rgerhards at hq.adiscon.com>
> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] feedback request: imklog symbol translation
> 
> Hi all,
>
> currently, imklog has some plumbing to do kernel symbol translation. However,
> as of my understanding, even not-so-totally-recent kernel do this themselves.
> It looks to me like the symbol translation support in imklog is a left-over
> that really has no need today. In order to reduce code complexity, I am very
> tempted to remove it in v6.
>
> Comments on that plan would be appreciated.
>
> Thanks,
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From Ole.Rahn at t-systems.com  Thu May 26 10:49:52 2011
From: Ole.Rahn at t-systems.com (Ole.Rahn at t-systems.com)
Date: Thu, 26 May 2011 10:49:52 +0200
Subject: [rsyslog] Heavy stability problems when using TLS
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280D5B@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA71DE1AC@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DE1AD@GRFEXC.intern.adiscon.com><50CCFFD1B9C6424389383F7A13A03446013CFA56003C@HE101451.emea1.cds.t-internal.com><9B6E2A8877C38245BFB15CC491A11DA7280D44@GRFEXC.intern.adiscon.com><50CCFFD1B9C6424389383F7A13A03446013CFA5C2B5A@HE101451.emea1.cds.t-internal.com><9B6E2A8877C38245BFB15CC491A11DA7280D57@GRFEXC.intern.adiscon.com>
	<50CCFFD1B9C6424389383F7A13A03446013CFA5C2B89@HE101451.emea1.cds.t-internal.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280D5B@GRFEXC.intern.adiscon.com>
Message-ID: <50CCFFD1B9C6424389383F7A13A03446013CFA5C3138@HE101451.emea1.cds.t-internal.com>

Hi Rainer,

I saw the big memory footprint on the server.

In the meantime I got the queue config of the clients. They use:

$WorkDirectory                  /var/spool/rsyslog
$ActionQueueType                LinkedList
$ActionQueueFilename            main.queue
#Infinite retries on failure
$ActionResumeRetryCount         -1
$ActionQueueSaveOnShutdown      on

$ActionQueueDiscardMark         9000
$ActionQueueDiscardSeverity     4

Will take a look at impstats, next!

Best regards
Ole


-----Urspr?ngliche Nachricht-----

Doesnt sound too much. Where did you see the big memory footprint? Client or server? Maybe it would be useful to use impstats to see queue sizes.

Rainer

> -----Original Message-----
> 
> So far it is just this line on the server in regard queues:
> $MainMsgQueueSize 100000 # 100,000 may be a value to handle burst 
> traffic
> 
> Will try to find out the client settings.
> 
> 
> Ole


From christian.lete at gmail.com  Thu May 26 14:01:34 2011
From: christian.lete at gmail.com (Christian Lete)
Date: Thu, 26 May 2011 14:01:34 +0200
Subject: [rsyslog] High availability on rsyslog (cluster)
In-Reply-To: <alpine.DEB.2.02.1105251021320.23692@asgard.lang.hm>
References: <BANLkTin88E7L-4cohhfP+6ihVSe2DvkbQg@mail.gmail.com>
	<alpine.DEB.2.02.1105231529010.8167@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7280D21@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.02.1105232304290.8167@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7280D22@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.02.1105232312020.8167@asgard.lang.hm>
	<alpine.DEB.2.02.1105232312590.8167@asgard.lang.hm>
	<BANLkTi=2rwbrCWKqi3wqw_xcOuXE0UYvsw@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280D43@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.02.1105251021320.23692@asgard.lang.hm>
Message-ID: <BANLkTi=+k_ffY0XHbw9qxryXb9wNiyHa6A@mail.gmail.com>

Hi,

I also thought that maybe, it  would be possible to return some
timestamp or some string once given a test string is sent to the port?

Cheers,

Christian

On 25 May 2011 19:22,  <david at lang.hm> wrote:
> I have many HA environments available for testing
>
> What I am doing right now is to just run rsyslog normally on each system and
> move the IP address, but it would be nice to have service checking.
>
> David Lang
>
> On Wed, 25 May 2011, Rainer Gerhards wrote:
>
>> Date: Wed, 25 May 2011 10:12:37 +0200
>> From: Rainer Gerhards <rgerhards at hq.adiscon.com>
>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>> To: rsyslog-users <rsyslog at lists.adiscon.com>
>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>
>> Christian,
>>
>> would you also be available for some tests *if* I find time to look into
>> the
>> HA specific calls? I don't have an environment yet and it would probably
>> take
>> me too long to set one up for a quick test. No promise thought that I can
>> implement the support quickly, depends on what else is going on...
>>
>> Rainer
>>
>>> -----Original Message-----
>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>> bounces at lists.adiscon.com] On Behalf Of Christian Lete
>>> Sent: Wednesday, May 25, 2011 10:10 AM
>>> To: rsyslog-users
>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>>
>>> Hi,
>>>
>>> Thank you all for you help and suggestions, I will do some tests and
>>> let you know how works for me.
>>>
>>> Regards,
>>>
>>> Christian
>>>
>>> On 24 May 2011 08:14, ?<david at lang.hm> wrote:
>>>>
>>>> One interesting twist to deal with, when a system is not active, it
>>>
>>> probably
>>>>
>>>> still wants to send it's logs somewhere.
>>>>
>>>> two ways of addresssing this
>>>>
>>>> 1. two configs, switching between active and passive changes which
>>>
>>> config
>>>>
>>>> you use
>>>>
>>>> 2. two instances, one used by the local box to send it's logs, the
>>>
>>> other to
>>>>
>>>> receive logs from elsewhere.
>>>>
>>>> David Lang
>>>>
>>>> On Mon, 23 May 2011, david at lang.hm wrote:
>>>>
>>>>> take a look at
>>>>>
>>>>> http://linux-ha.org/wiki/Resource_Agents
>>>>>
>>>>> and
>>>>>
>>>>> http://www.linux-ha.org/doc/dev-guides/ra-dev-guide.html
>>>>>
>>>>> David Lang
>>>>>
>>>>> On Tue, 24 May 2011, Rainer Gerhards wrote:
>>>>>
>>>>>> Date: Tue, 24 May 2011 08:09:28 +0200
>>>>>> From: Rainer Gerhards <rgerhards at hq.adiscon.com>
>>>>>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>>>>>> To: rsyslog-users <rsyslog at lists.adiscon.com>
>>>>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>>>>>
>>>>>> Thx -- sounds interesting and probably not too much work to do...
>>>>>>
>>>>>> Rainer
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>>>>>> Sent: Tuesday, May 24, 2011 8:08 AM
>>>>>>> To: rsyslog-users
>>>>>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>>>>>>
>>>>>>> take a look at linux-ha
>>>>>>>
>>>>>>> It's a framework to manage HA (including active/active load
>>>
>>> sharing,
>>>>>>>
>>>>>>> quorums, etc)
>>>>>>>
>>>>>>> it extends the traditional init.d startup scripts to also include
>>>
>>> a
>>>>>>>
>>>>>>> 'status' call to tell if the service is active or not. the
>>>
>>> framework
>>>>>>>
>>>>>>> calls
>>>>>>> this service periodically and if the service fails, it does a
>>>
>>> failover.
>>>>>>>
>>>>>>> With the correct configuration (and software), it can do sub-
>>>
>>> second
>>>>>>>
>>>>>>> failover.
>>>>>>>
>>>>>>> David Lang
>>>>>>>
>>>>>>>
>>>>>>> ?On
>>>>>>> Tue, 24 May 2011, Rainer Gerhards wrote:
>>>>>>>
>>>>>>>> David and all,
>>>>>>>>
>>>>>>>> are you aware of any high availability APIs that would enable
>>>
>>> rsyslog
>>>>>>>
>>>>>>> to do
>>>>>>>>
>>>>>>>> some kind of automatic failover in a cluster environment? I have
>>>>>>>
>>>>>>> never
>>>>>>>>
>>>>>>>> specifically programmed for that and wonder if there are any
>>>
>>> options.
>>>>>>>>
>>>>>>>> Rainer
>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>>>>>>>> Sent: Tuesday, May 24, 2011 12:30 AM
>>>>>>>>> To: rsyslog-users
>>>>>>>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>>>>>>>>
>>>>>>>>> depending on how active your logging is, you could watch the
>>>
>>> logs
>>>>>>>
>>>>>>> and
>>>>>>>>>
>>>>>>>>> say
>>>>>>>>> that if you don't receive any logs for 1 min (or whatever time
>>>
>>> is
>>>>>>>>>
>>>>>>>>> approprate), somthing is wrong.
>>>>>>>>>
>>>>>>>>> you could also generate known UDP logs to yourself and alert if
>>>
>>> they
>>>>>>>>>
>>>>>>>>> don't
>>>>>>>>> show up.
>>>>>>>>>
>>>>>>>>> David Lang
>>>>>>>>>
>>>>>>>>> ?On Mon, 23 May 2011, Christian Lete wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I have a small question, ?I would need to setup an rsyslog
>>>>>>>>>> receiver/forwarder, listening on udp port, since some clients,
>>>
>>> only
>>>>>>>>>>
>>>>>>>>>> support this option. I would need this service to be highly
>>>>>>>>>> available(I don't want to have two machines and having
>>>
>>> duplicated
>>>>>>>>>>
>>>>>>>>>> information), but since this udp, I can't be for sure if the
>>>>>>>
>>>>>>> service
>>>>>>>>>>
>>>>>>>>>> is running fine. What I thought is to indirectly check it, by
>>>>>>>
>>>>>>> having
>>>>>>>>>>
>>>>>>>>>> another port listening on tcp and checking the tcp service, if
>>>
>>> the
>>>>>>>>>>
>>>>>>>>>> service is not running on tcp I would assume the whole system
>>>
>>> is
>>>>>>>
>>>>>>> down
>>>>>>>>>>
>>>>>>>>>> and would failover to the other instance of the cluster, that's
>>>
>>> the
>>>>>>>>>>
>>>>>>>>>> only way I could think of, do you currently have another way?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> thank you very much,
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> Christian
>>>>>>>>>> _______________________________________________
>>>>>>>>>> rsyslog mailing list
>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>>> http://www.rsyslog.com
>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> rsyslog mailing list
>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>> http://www.rsyslog.com
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> rsyslog mailing list
>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>> http://www.rsyslog.com
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> rsyslog mailing list
>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>> http://www.rsyslog.com
>>>>>>
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com
>>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com
>>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>
>

From rgerhards at hq.adiscon.com  Thu May 26 17:53:03 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 26 May 2011 17:53:03 +0200
Subject: [rsyslog] Heavy stability problems when using TLS
References: <9B6E2A8877C38245BFB15CC491A11DA71DE1AC@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DE1AD@GRFEXC.intern.adiscon.com><50CCFFD1B9C6424389383F7A13A03446013CFA56003C@HE101451.emea1.cds.t-internal.com><9B6E2A8877C38245BFB15CC491A11DA7280D44@GRFEXC.intern.adiscon.com><50CCFFD1B9C6424389383F7A13A03446013CFA5C2B5A@HE101451.emea1.cds.t-internal.com><9B6E2A8877C38245BFB15CC491A11DA7280D57@GRFEXC.intern.adiscon.com><50CCFFD1B9C6424389383F7A13A03446013CFA5C2B89@HE101451.emea1.cds.t-internal.com><9B6E2A8877C38245BFB15CC491A11DA7280D5B@GRFEXC.intern.adiscon.com>
	<50CCFFD1B9C6424389383F7A13A03446013CFA5C3138@HE101451.emea1.cds.t-internal.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA701075E@GRFEXC.intern.adiscon.com>

Do you know how many queues they have? Can you post the full (mangeled for
anonymity) config? Queues may sum up.

impstats is a good idea ;)

RAiner

-----Urspr?ngliche Nachricht-----
Von: rsyslog-bounces at lists.adiscon.com im Auftrag von Ole.Rahn at t-systems.com
Gesendet: Do 26.05.2011 10:49
An: rsyslog at lists.adiscon.com
Betreff: Re: [rsyslog] Heavy stability problems when using TLS
 
Hi Rainer,

I saw the big memory footprint on the server.

In the meantime I got the queue config of the clients. They use:

$WorkDirectory                  /var/spool/rsyslog
$ActionQueueType                LinkedList
$ActionQueueFilename            main.queue
#Infinite retries on failure
$ActionResumeRetryCount         -1
$ActionQueueSaveOnShutdown      on

$ActionQueueDiscardMark         9000
$ActionQueueDiscardSeverity     4

Will take a look at impstats, next!

Best regards
Ole


-----Urspr?ngliche Nachricht-----

Doesnt sound too much. Where did you see the big memory footprint? Client or
server? Maybe it would be useful to use impstats to see queue sizes.

Rainer

> -----Original Message-----
> 
> So far it is just this line on the server in regard queues:
> $MainMsgQueueSize 100000 # 100,000 may be a value to handle burst 
> traffic
> 
> Will try to find out the client settings.
> 
> 
> Ole

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com


From david at lang.hm  Thu May 26 20:51:53 2011
From: david at lang.hm (david at lang.hm)
Date: Thu, 26 May 2011 11:51:53 -0700 (PDT)
Subject: [rsyslog] High availability on rsyslog (cluster)
In-Reply-To: <BANLkTi=+k_ffY0XHbw9qxryXb9wNiyHa6A@mail.gmail.com>
References: <BANLkTin88E7L-4cohhfP+6ihVSe2DvkbQg@mail.gmail.com>
	<alpine.DEB.2.02.1105231529010.8167@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7280D21@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.02.1105232304290.8167@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7280D22@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.02.1105232312020.8167@asgard.lang.hm>
	<alpine.DEB.2.02.1105232312590.8167@asgard.lang.hm>
	<BANLkTi=2rwbrCWKqi3wqw_xcOuXE0UYvsw@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280D43@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.02.1105251021320.23692@asgard.lang.hm>
	<BANLkTi=+k_ffY0XHbw9qxryXb9wNiyHa6A@mail.gmail.com>
Message-ID: <alpine.DEB.2.02.1105261150130.4413@asgard.lang.hm>

it wouldn't be easy to have the port return a response, but you could 
easily send some unique string to the port and then have a rule that did 
something special with the log that had that string in it (write it to a 
named pipe so that some other app can read it, write it to a special 
logfile, etc) and that can have the timestamp that the log was received in 
it.

David Lang

  On Thu, 26 May 2011, Christian Lete wrote:

> Hi,
>
> I also thought that maybe, it  would be possible to return some
> timestamp or some string once given a test string is sent to the port?
>
> Cheers,
>
> Christian
>
> On 25 May 2011 19:22,  <david at lang.hm> wrote:
>> I have many HA environments available for testing
>>
>> What I am doing right now is to just run rsyslog normally on each system and
>> move the IP address, but it would be nice to have service checking.
>>
>> David Lang
>>
>> On Wed, 25 May 2011, Rainer Gerhards wrote:
>>
>>> Date: Wed, 25 May 2011 10:12:37 +0200
>>> From: Rainer Gerhards <rgerhards at hq.adiscon.com>
>>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>>> To: rsyslog-users <rsyslog at lists.adiscon.com>
>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>>
>>> Christian,
>>>
>>> would you also be available for some tests *if* I find time to look into
>>> the
>>> HA specific calls? I don't have an environment yet and it would probably
>>> take
>>> me too long to set one up for a quick test. No promise thought that I can
>>> implement the support quickly, depends on what else is going on...
>>>
>>> Rainer
>>>
>>>> -----Original Message-----
>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>> bounces at lists.adiscon.com] On Behalf Of Christian Lete
>>>> Sent: Wednesday, May 25, 2011 10:10 AM
>>>> To: rsyslog-users
>>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>>>
>>>> Hi,
>>>>
>>>> Thank you all for you help and suggestions, I will do some tests and
>>>> let you know how works for me.
>>>>
>>>> Regards,
>>>>
>>>> Christian
>>>>
>>>> On 24 May 2011 08:14, ?<david at lang.hm> wrote:
>>>>>
>>>>> One interesting twist to deal with, when a system is not active, it
>>>>
>>>> probably
>>>>>
>>>>> still wants to send it's logs somewhere.
>>>>>
>>>>> two ways of addresssing this
>>>>>
>>>>> 1. two configs, switching between active and passive changes which
>>>>
>>>> config
>>>>>
>>>>> you use
>>>>>
>>>>> 2. two instances, one used by the local box to send it's logs, the
>>>>
>>>> other to
>>>>>
>>>>> receive logs from elsewhere.
>>>>>
>>>>> David Lang
>>>>>
>>>>> On Mon, 23 May 2011, david at lang.hm wrote:
>>>>>
>>>>>> take a look at
>>>>>>
>>>>>> http://linux-ha.org/wiki/Resource_Agents
>>>>>>
>>>>>> and
>>>>>>
>>>>>> http://www.linux-ha.org/doc/dev-guides/ra-dev-guide.html
>>>>>>
>>>>>> David Lang
>>>>>>
>>>>>> On Tue, 24 May 2011, Rainer Gerhards wrote:
>>>>>>
>>>>>>> Date: Tue, 24 May 2011 08:09:28 +0200
>>>>>>> From: Rainer Gerhards <rgerhards at hq.adiscon.com>
>>>>>>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>>>>>>> To: rsyslog-users <rsyslog at lists.adiscon.com>
>>>>>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>>>>>>
>>>>>>> Thx -- sounds interesting and probably not too much work to do...
>>>>>>>
>>>>>>> Rainer
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>>>>>>> Sent: Tuesday, May 24, 2011 8:08 AM
>>>>>>>> To: rsyslog-users
>>>>>>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>>>>>>>
>>>>>>>> take a look at linux-ha
>>>>>>>>
>>>>>>>> It's a framework to manage HA (including active/active load
>>>>
>>>> sharing,
>>>>>>>>
>>>>>>>> quorums, etc)
>>>>>>>>
>>>>>>>> it extends the traditional init.d startup scripts to also include
>>>>
>>>> a
>>>>>>>>
>>>>>>>> 'status' call to tell if the service is active or not. the
>>>>
>>>> framework
>>>>>>>>
>>>>>>>> calls
>>>>>>>> this service periodically and if the service fails, it does a
>>>>
>>>> failover.
>>>>>>>>
>>>>>>>> With the correct configuration (and software), it can do sub-
>>>>
>>>> second
>>>>>>>>
>>>>>>>> failover.
>>>>>>>>
>>>>>>>> David Lang
>>>>>>>>
>>>>>>>>
>>>>>>>> ?On
>>>>>>>> Tue, 24 May 2011, Rainer Gerhards wrote:
>>>>>>>>
>>>>>>>>> David and all,
>>>>>>>>>
>>>>>>>>> are you aware of any high availability APIs that would enable
>>>>
>>>> rsyslog
>>>>>>>>
>>>>>>>> to do
>>>>>>>>>
>>>>>>>>> some kind of automatic failover in a cluster environment? I have
>>>>>>>>
>>>>>>>> never
>>>>>>>>>
>>>>>>>>> specifically programmed for that and wonder if there are any
>>>>
>>>> options.
>>>>>>>>>
>>>>>>>>> Rainer
>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>>>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>>>>>>>>> Sent: Tuesday, May 24, 2011 12:30 AM
>>>>>>>>>> To: rsyslog-users
>>>>>>>>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>>>>>>>>>
>>>>>>>>>> depending on how active your logging is, you could watch the
>>>>
>>>> logs
>>>>>>>>
>>>>>>>> and
>>>>>>>>>>
>>>>>>>>>> say
>>>>>>>>>> that if you don't receive any logs for 1 min (or whatever time
>>>>
>>>> is
>>>>>>>>>>
>>>>>>>>>> approprate), somthing is wrong.
>>>>>>>>>>
>>>>>>>>>> you could also generate known UDP logs to yourself and alert if
>>>>
>>>> they
>>>>>>>>>>
>>>>>>>>>> don't
>>>>>>>>>> show up.
>>>>>>>>>>
>>>>>>>>>> David Lang
>>>>>>>>>>
>>>>>>>>>> ?On Mon, 23 May 2011, Christian Lete wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I have a small question, ?I would need to setup an rsyslog
>>>>>>>>>>> receiver/forwarder, listening on udp port, since some clients,
>>>>
>>>> only
>>>>>>>>>>>
>>>>>>>>>>> support this option. I would need this service to be highly
>>>>>>>>>>> available(I don't want to have two machines and having
>>>>
>>>> duplicated
>>>>>>>>>>>
>>>>>>>>>>> information), but since this udp, I can't be for sure if the
>>>>>>>>
>>>>>>>> service
>>>>>>>>>>>
>>>>>>>>>>> is running fine. What I thought is to indirectly check it, by
>>>>>>>>
>>>>>>>> having
>>>>>>>>>>>
>>>>>>>>>>> another port listening on tcp and checking the tcp service, if
>>>>
>>>> the
>>>>>>>>>>>
>>>>>>>>>>> service is not running on tcp I would assume the whole system
>>>>
>>>> is
>>>>>>>>
>>>>>>>> down
>>>>>>>>>>>
>>>>>>>>>>> and would failover to the other instance of the cluster, that's
>>>>
>>>> the
>>>>>>>>>>>
>>>>>>>>>>> only way I could think of, do you currently have another way?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> thank you very much,
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>>
>>>>>>>>>>> Christian
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> rsyslog mailing list
>>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>>>> http://www.rsyslog.com
>>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> rsyslog mailing list
>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>>> http://www.rsyslog.com
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> rsyslog mailing list
>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>> http://www.rsyslog.com
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> rsyslog mailing list
>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>> http://www.rsyslog.com
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> rsyslog mailing list
>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>> http://www.rsyslog.com
>>>>>>>
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com
>>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com
>>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From taotetek at gmail.com  Fri May 27 02:26:59 2011
From: taotetek at gmail.com (Brian Knox)
Date: Thu, 26 May 2011 20:26:59 -0400
Subject: [rsyslog] ZeroMQ input and output plugins
Message-ID: <BANLkTimgLOSsfXuvNAyS+8SHOH_UcbC6rA@mail.gmail.com>

If anyone has any interest - we released zeromq input and output plugins for
rsyslog today.  They are relatively new and there's work to do on them, but
we have them up and running in our lab.  We'd love other people to take a
look and provide us with feedback!

Thanks

https://github.com/aggregateknowledge/rsyslog-zeromq

From rgerhards at hq.adiscon.com  Mon May 30 09:15:00 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 30 May 2011 09:15:00 +0200
Subject: [rsyslog] ZeroMQ input and output plugins
In-Reply-To: <BANLkTimgLOSsfXuvNAyS+8SHOH_UcbC6rA@mail.gmail.com>
References: <BANLkTimgLOSsfXuvNAyS+8SHOH_UcbC6rA@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280D6F@GRFEXC.intern.adiscon.com>

Hi all,

congrats, this sounds like a useful addition :)

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Brian Knox
> Sent: Friday, May 27, 2011 2:27 AM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] ZeroMQ input and output plugins
> 
> If anyone has any interest - we released zeromq input and output plugins
for
> rsyslog today.  They are relatively new and there's work to do on them, but
> we have them up and running in our lab.  We'd love other people to take a
> look and provide us with feedback!
> 
> Thanks
> 
> https://github.com/aggregateknowledge/rsyslog-zeromq
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From chb at muc.de  Tue May 31 22:43:30 2011
From: chb at muc.de (Christian Brunner)
Date: Tue, 31 May 2011 22:43:30 +0200
Subject: [rsyslog] ZeroMQ input and output plugins
In-Reply-To: <BANLkTimgLOSsfXuvNAyS+8SHOH_UcbC6rA@mail.gmail.com>
References: <BANLkTimgLOSsfXuvNAyS+8SHOH_UcbC6rA@mail.gmail.com>
Message-ID: <BANLkTi=d5cpZm4Z+wF-Jz0Tpy=vd1o0MOw@mail.gmail.com>

This is great! I started an omzeromq module some time ago, but never
really finished it.

I've now managed to get it running with the current rsyslog master
branch and have added a configuration option for the messaging pattern
(please see the following patches).

Christian


2011/5/27 Brian Knox <taotetek at gmail.com>:
> If anyone has any interest - we released zeromq input and output plugins for
> rsyslog today. ?They are relatively new and there's work to do on them, but
> we have them up and running in our lab. ?We'd love other people to take a
> look and provide us with feedback!
>
> Thanks
>
> https://github.com/aggregateknowledge/rsyslog-zeromq
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From chb at muc.de  Tue May 31 22:44:25 2011
From: chb at muc.de (Christian Brunner)
Date: Tue, 31 May 2011 22:44:25 +0200
Subject: [rsyslog] [PATCH rsyslog-zeromq 1/3] use zmq_connect in omzeromq
Message-ID: <20110531204425.GA18110@sir.fritz.box>

The omzeromq module uses zmq_bind instead of zmq_connect. This doesn't work.

Thanks,
Christian
---
 omzeromq/omzeromq.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/omzeromq/omzeromq.c b/omzeromq/omzeromq.c
index 12501d6..58c835c 100644
--- a/omzeromq/omzeromq.c
+++ b/omzeromq/omzeromq.c
@@ -117,7 +117,7 @@ static rsRetVal init_zeromq(instanceData *pData, int bSilent)
     pData->socket = zmq_socket(pData->context, pData->pattern);
 
     if (pData->connstr)
-        zmq_bind(pData->socket, (char *) pData->connstr);
+        zmq_connect(pData->socket, (char *) pData->connstr);
     else
         zmq_bind(pData->socket, (char *) pData->bindstr);
 
-- 
1.7.1


From chb at muc.de  Tue May 31 22:45:42 2011
From: chb at muc.de (Christian Brunner)
Date: Tue, 31 May 2011 22:45:42 +0200
Subject: [rsyslog] [PATCH rsyslog-zeromq 2/3] merge with the current master
	branch
Message-ID: <20110531204542.GB18110@sir.fritz.box>

To compile with the current master branch of rsyslog I had to do some
changes. (2011-05-31)

I'm not sure if everything is correct, but at least it seems to work.

Christian
---
 imzeromq/imzeromq.c |   14 +++++++-------
 omzeromq/omzeromq.c |   15 ++++++++++++++-
 2 files changed, 21 insertions(+), 8 deletions(-)

diff --git a/imzeromq/imzeromq.c b/imzeromq/imzeromq.c
index f275473..b5d9a45 100644
--- a/imzeromq/imzeromq.c
+++ b/imzeromq/imzeromq.c
@@ -80,7 +80,7 @@ set_ruleset(void __attribute__((unused)) *pVal, uchar *pszName)
 	rsRetVal localRet;
 	DEFiRet;
 
-	localRet = ruleset.GetRuleset(&pRuleset, pszName);
+	localRet = ruleset.GetRuleset(ourConf, &pRuleset, pszName);
 	if(localRet == RS_RET_NOT_FOUND) {
 		errmsg.LogError(0, NO_ERRCODE, "error: "
                         "ruleset '%s' not found - ignored", pszName);
@@ -447,14 +447,14 @@ CODEmodInit_QueryRegCFSLineHdlr
 
 	/* register config file handlers */
 	CHKiRet(omsdRegCFSLineHdlr((uchar *)"inputzeromqserverbindruleset",
-                               0, eCmdHdlrGetWord,
-		set_ruleset, NULL, STD_LOADABLE_MODULE_ID));
+                0, eCmdHdlrGetWord, set_ruleset, NULL,
+		STD_LOADABLE_MODULE_ID, eConfObjGlobal));
 	CHKiRet(omsdRegCFSLineHdlr((uchar *)"inputzeromqserverrun",
-                               0, eCmdHdlrGetWord,
-		add_endpoint, NULL, STD_LOADABLE_MODULE_ID));
+                0, eCmdHdlrGetWord, add_endpoint, NULL,
+		STD_LOADABLE_MODULE_ID, eConfObjGlobal));
 	CHKiRet(omsdRegCFSLineHdlr((uchar *)"resetconfigvariables",
-                               1, eCmdHdlrCustomHandler,
-		resetConfigVariables, NULL, STD_LOADABLE_MODULE_ID));
+                1, eCmdHdlrCustomHandler, resetConfigVariables, NULL,
+		STD_LOADABLE_MODULE_ID, eConfObjGlobal));
 ENDmodInit
 /* vim:set ai:
  */
diff --git a/omzeromq/omzeromq.c b/omzeromq/omzeromq.c
index 81f8dca..12501d6 100644
--- a/omzeromq/omzeromq.c
+++ b/omzeromq/omzeromq.c
@@ -48,6 +48,8 @@
 MODULE_TYPE_OUTPUT
 MODULE_TYPE_NOKEEP
 
+static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __attribute__((unused)) *pVal);
+
 /* internal structures
  */
 DEF_OMOD_STATIC_DATA
@@ -66,6 +68,16 @@ typedef struct _instanceData {
     void *		socket;
 } instanceData;
 
+typedef struct configSettings_s {
+} configSettings_t;
+
+SCOPING_SUPPORT;
+
+BEGINinitConfVars              /* (re)set config variables to default values */
+CODESTARTinitConfVars 
+       resetConfigVariables(NULL, NULL);
+ENDinitConfVars
+
 BEGINcreateInstance
 CODESTARTcreateInstance
 ENDcreateInstance
@@ -326,10 +338,11 @@ static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __a
 
 BEGINmodInit()
 CODESTARTmodInit
+SCOPINGmodInit
 	*ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */
 CODEmodInit_QueryRegCFSLineHdlr
 	CHKiRet(objUse(errmsg, CORE_COMPONENT));
-	CHKiRet(omsdRegCFSLineHdlr((uchar *)"resetconfigvariables", 1, eCmdHdlrCustomHandler, resetConfigVariables, NULL, STD_LOADABLE_MODULE_ID));
+	CHKiRet(omsdRegCFSLineHdlr((uchar *)"resetconfigvariables", 1, eCmdHdlrCustomHandler, resetConfigVariables, NULL, STD_LOADABLE_MODULE_ID, eConfObjAction));
 CODEmodInit_QueryRegCFSLineHdlr
 ENDmodInit
 
-- 
1.7.1


From chb at muc.de  Tue May 31 22:46:50 2011
From: chb at muc.de (Christian Brunner)
Date: Tue, 31 May 2011 22:46:50 +0200
Subject: [rsyslog] [PATCH rsyslog-zeromq 3/3] make the messaging pattern a
	config option
Message-ID: <20110531204650.GC18110@sir.fritz.box>

Not only PUSH/PULL can be a usefull pattern, but also PUB/SUB. Although
PUSH/PULL makes perfectly sense for the default.

Thanks,
Christian
---
 README.markdown     |    2 ++
 imzeromq/imzeromq.c |   26 +++++++++++++++++++++++++-
 omzeromq/omzeromq.c |   21 ++++++++++++++++++++-
 3 files changed, 47 insertions(+), 2 deletions(-)

diff --git a/README.markdown b/README.markdown
index 6a1dbfa..75c4bf8 100644
--- a/README.markdown
+++ b/README.markdown
@@ -50,6 +50,7 @@ The :omzeromq: selector takes the following parameter components:
 * hwm=&lt;NNN&gt;             Sets the high water mark of the socket.
 * swap=&lt;NNN&gt;            Sets the swap value for the socket.
 * threads=&lt;N&gt;           Sets the number of zeromq context threads.
+* pattern=&lt;patternstr&gt;  Sets the messaging pattern (push or pub).
 
 The format for the selector may be specified in the standard way with
 a trailing ";&lt;FORMAT&gt;" specifier.
@@ -71,6 +72,7 @@ The $InputZeroMQServerRun directive takes the following parameter components:
 * connect=&lt;endpoint&gt;    Connect to the specified endpoint.
 * bind=&lt;endpoint&gt;       Bind to the specified endpoint.
 * identity=&lt;identstr&gt;   Sets the identity of the socket.
+* pattern=&lt;patternstr&gt;  Sets the messaging pattern (pull or sub).
 
 Examples:
 
diff --git a/imzeromq/imzeromq.c b/imzeromq/imzeromq.c
index 783559e..f275473 100644
--- a/imzeromq/imzeromq.c
+++ b/imzeromq/imzeromq.c
@@ -107,6 +107,8 @@ static rsRetVal add_endpoint(void __attribute__((unused)) * oldp, uchar * valp)
     char * connstr = NULL;
     char * bindstr = NULL;
     char * identstr = NULL;
+    char * patternstr = NULL;
+    int pattern = ZMQ_PULL;
 
     char * ptr1;
     char * binding;
@@ -142,6 +144,10 @@ static rsRetVal add_endpoint(void __attribute__((unused)) * oldp, uchar * valp)
         {
             CHKmalloc(identstr = strdup(val));
         }
+        else if (strcmp(binding, "pattern") == 0)
+        {
+            CHKmalloc(patternstr = strdup(val));
+        }
         else
         {
             errmsg.LogError(0, NO_ERRCODE, "Unknown argument %s", binding);
@@ -167,6 +173,24 @@ static rsRetVal add_endpoint(void __attribute__((unused)) * oldp, uchar * valp)
 		ABORT_FINALIZE(RS_RET_INVALID_PARAMS);
     }
 
+    // check for valid patterns (pull is default)
+    if (patternstr) {
+        if (strcmp(binding, "pull") == 0) {
+            pattern = ZMQ_PULL;
+        }
+        else if (strcmp(binding, "sub") == 0)
+        {
+            pattern = ZMQ_SUB;
+        }
+        else
+	{
+            errmsg.LogError(0,
+                RS_RET_INVALID_PARAMS, "error: "
+                "invalid messaging pattern - use 'pull' or 'sub'");
+            ABORT_FINALIZE(RS_RET_INVALID_PARAMS);
+	}
+    }
+
     if (!s_context)
         s_context = zmq_init(1);
     if (!s_context)
@@ -178,7 +202,7 @@ static rsRetVal add_endpoint(void __attribute__((unused)) * oldp, uchar * valp)
 		ABORT_FINALIZE(RS_RET_INVALID_PARAMS);
     }
 
-    void * sock = zmq_socket(s_context, ZMQ_PULL);
+    void * sock = zmq_socket(s_context, pattern);
     if (!sock)
     {
 		errmsg.LogError(0,
diff --git a/omzeromq/omzeromq.c b/omzeromq/omzeromq.c
index 55865a6..81f8dca 100644
--- a/omzeromq/omzeromq.c
+++ b/omzeromq/omzeromq.c
@@ -60,6 +60,7 @@ typedef struct _instanceData {
     int64		swapsz;
     uchar *		identstr;
     int64		threads;
+    int			pattern;
 
     void *		context;
     void *		socket;
@@ -101,7 +102,7 @@ static rsRetVal init_zeromq(instanceData *pData, int bSilent)
 	ASSERT(pData->socket == NULL);
 
     pData->context = zmq_init(pData->threads);
-    pData->socket = zmq_socket(pData->context, ZMQ_PUSH);
+    pData->socket = zmq_socket(pData->context, pData->pattern);
 
     if (pData->connstr)
         zmq_bind(pData->socket, (char *) pData->connstr);
@@ -174,6 +175,7 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1)
 
     pData->connstr = NULL;
     pData->bindstr = NULL;
+    pData->pattern = ZMQ_PUSH;
     pData->hwmsz = -1;
 	pData->swapsz = -1;
     pData->identstr = NULL;
@@ -242,6 +244,23 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1)
                 ABORT_FINALIZE(RS_RET_INVALID_PARAMS);
             }
         }
+        else if (strcmp(binding, "pattern") == 0)
+        {
+	    if (strcmp(val, "push") == 0) {
+                pData->pattern = ZMQ_PUSH;
+            }
+            else if (strcmp(val, "pub") == 0)
+            {
+                pData->pattern = ZMQ_SUB;
+            }
+	    else
+            {
+                errmsg.LogError(0,
+                    RS_RET_INVALID_PARAMS,
+		    "error: invalid messaging pattern - use 'push' or 'pub'");
+                ABORT_FINALIZE(RS_RET_INVALID_PARAMS);
+            }
+        }
         else
         {
             errmsg.LogError(0, NO_ERRCODE, "Unknown argument %s", binding);
-- 
1.7.1