From tbergfeld at hq.adiscon.com Mon May 3 13:15:02 2010 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Mon, 3 May 2010 13:15:02 +0200 Subject: [rsyslog] rsyslog 4.7.2 (v4-devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103D29@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 4.7.2, a member of the v4-devel branch. 4.7.2 is a bugfixing-release. Its primary bugfix solves problems with atomic instruction emulation. Users who have compiled rsyslog for older CPUs (like Intel 386) or CPUs for which gcc lacks atomic instruction support (like Sparc) are strongly encouraged to upgrade to the new versions. For all others, an update is optional. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article457.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-202.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From tbergfeld at hq.adiscon.com Mon May 3 15:02:25 2010 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Mon, 3 May 2010 15:02:25 +0200 Subject: [rsyslog] rsyslog 5.5.4 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103D2F@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.5.4, a member of the devel branch. Rsyslog has become the de-facto standard on modern Linux operating systems. It's high-performance log processing, database integration, modularity and support for multiple logging protocols make it the sysadmin's logging daemon of choice. The project was started in 2004 and has since then evolved rapidly. Starting with today, rsyslog is not only available on Linux and BSD, but also on Sun Solaris. Both Intel and Sparc machines are fully supported under Solaris. Depending on operator need, rsyslog can replace stock Solaris syslogd or be used in conjunction with it. The later case provides enhanced rsyslog functionality without the need to change the system infrastructure. Solaris is now a tier-one target platform. That means that all testing for major releases will be carried out on Solaris as well as on other platforms. The Solaris port was done very careful taking into account Sun's somewhat specific syslogd handling via door files and preserving the full power of rsyslog. So it not only compiles and runs on Solaris but rsyslog is a good citizen in the Solaris environment. As of usual rsyslog project policies, the project does not make installation packages other than the source distribution available. However, we work closely together with the Solaris community be able to provide them. We expect additional announcements soon. The versions with initial solid Solaris support are 4.7.2 and 5.5.4. Rsyslog's Solaris port was made possible by a generous contribution of hardware and some development funding by a sponsor which preferred to remain anonymous. We from the rsyslog project would like to express our sincere appreciation. Contributions of any kind are always very welcome. ChangeLog: http://www.rsyslog.com/Article459.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-203.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From pgollucci at freebsd.org Wed May 5 01:17:57 2010 From: pgollucci at freebsd.org (Philip M. Gollucci) Date: Tue, 4 May 2010 19:17:57 -0400 (EDT) Subject: [rsyslog] Create rsyslog55 and friends to track -devel Message-ID: <201005042317.o44NHvwa036646@frieza.p6m7g8.net> >Submitter-Id: current-users >Originator: Philip M. Gollucci >Organization: RideCharge Inc. >Confidential: no >Synopsis: Create rsyslog55 and friends to track -devel >Severity: non-critical >Priority: low >Category: ports >Class: change-request >Release: FreeBSD 9.0-CURRENT amd64 >Environment: System: FreeBSD frieza.p6m7g8.net 9.0-CURRENT FreeBSD 9.0-CURRENT #0: Mon Apr 26 16:20:00 EDT 2010 root at frieza.p6m7g8.net:/usr/obj/usr/src/sys/FRIEZA amd64 >Description: Repo copies needed: sysutils/rsyslog5 -> sysutils/rsyslog55 sysutils/rsyslog5-dbi -> sysutils/rsyslog55-dbi sysutils/rsyslog5-gnutls -> sysutils/rsyslog55-gnutls sysutils/rsyslog5-gssapi -> sysutils/rsyslog55-gssapi sysutils/rsyslog5-mysql -> sysutils/rsyslog55-mysql sysutils/rsyslog5-pgsql -> sysutils/rsyslog55-pgsql sysutils/rsyslog5-relp -> sysutils/rsyslog55-relp sysutils/rsyslog5-rfc3195 -> sysutils/rsyslog55-rfc3195 sysutils/rsyslog5-snmp -> sysutils/rsyslog55-snmp Sponsored by: RideCharge Inc. / Taxi Magic >How-To-Repeat: >Fix: --- rsyslog55.diff begins here --- diff --git a/sysutils/Makefile b/sysutils/Makefile index e3b17be..8d02e35 100644 --- a/sysutils/Makefile +++ b/sysutils/Makefile @@ -711,6 +711,15 @@ SUBDIR += rsyslog5-relp SUBDIR += rsyslog5-rfc3195 SUBDIR += rsyslog5-snmp + SUBDIR += rsyslog55 + SUBDIR += rsyslog55-dbi + SUBDIR += rsyslog55-gnutls + SUBDIR += rsyslog55-gssapi + SUBDIR += rsyslog55-mysql + SUBDIR += rsyslog55-pgsql + SUBDIR += rsyslog55-relp + SUBDIR += rsyslog55-rfc3195 + SUBDIR += rsyslog55-snmp SUBDIR += rtty SUBDIR += ruby-log4r SUBDIR += ruby-quota diff --git a/sysutils/rsyslog55-dbi/Makefile b/sysutils/rsyslog55-dbi/Makefile index 6e4ee57..c9d145a 100644 --- a/sysutils/rsyslog55-dbi/Makefile +++ b/sysutils/rsyslog55-dbi/Makefile @@ -5,7 +5,7 @@ # $FreeBSD: ports/sysutils/rsyslog5-dbi/Makefile,v 1.4 2009/12/18 20:44:28 miwi Exp $ COMMENT= LibDBI output module for rsyslog -MASTERDIR= ${.CURDIR}/../rsyslog5 +MASTERDIR= ${.CURDIR}/../rsyslog55 MNAME= libdbi LIB_DEPENDS= dbi.0:${PORTSDIR}/databases/libdbi diff --git a/sysutils/rsyslog55-gnutls/Makefile b/sysutils/rsyslog55-gnutls/Makefile index 3deb756..2f22ddb 100644 --- a/sysutils/rsyslog55-gnutls/Makefile +++ b/sysutils/rsyslog55-gnutls/Makefile @@ -6,7 +6,7 @@ # COMMENT= GNUTLS module for rsyslog -MASTERDIR= ${.CURDIR}/../rsyslog5 +MASTERDIR= ${.CURDIR}/../rsyslog55 MNAME= gnutls LIB_DEPENDS+= gnutls.40:${PORTSDIR}/security/gnutls diff --git a/sysutils/rsyslog55-gssapi/Makefile b/sysutils/rsyslog55-gssapi/Makefile index 6452ebc..9fa9ccb 100644 --- a/sysutils/rsyslog55-gssapi/Makefile +++ b/sysutils/rsyslog55-gssapi/Makefile @@ -6,7 +6,7 @@ # COMMENT= GSS API input/output module for rsyslog -MASTERDIR= ${.CURDIR}/../rsyslog5 +MASTERDIR= ${.CURDIR}/../rsyslog55 MNAME= gssapi diff --git a/sysutils/rsyslog55-mysql/Makefile b/sysutils/rsyslog55-mysql/Makefile index 3b9eca1..682588c 100644 --- a/sysutils/rsyslog55-mysql/Makefile +++ b/sysutils/rsyslog55-mysql/Makefile @@ -6,7 +6,7 @@ # COMMENT= MySQL output module for rsyslog -MASTERDIR= ${.CURDIR}/../rsyslog5 +MASTERDIR= ${.CURDIR}/../rsyslog55 MNAME= mysql USE_MYSQL= yes diff --git a/sysutils/rsyslog55-pgsql/Makefile b/sysutils/rsyslog55-pgsql/Makefile index 470009e..c9a08e6 100644 --- a/sysutils/rsyslog55-pgsql/Makefile +++ b/sysutils/rsyslog55-pgsql/Makefile @@ -6,7 +6,7 @@ # COMMENT= PostgreSQL output module for rsyslog -MASTERDIR= ${.CURDIR}/../rsyslog5 +MASTERDIR= ${.CURDIR}/../rsyslog55 MNAME= pgsql USE_PGSQL= yes diff --git a/sysutils/rsyslog55-relp/Makefile b/sysutils/rsyslog55-relp/Makefile index 8a915c1..86c7894 100644 --- a/sysutils/rsyslog55-relp/Makefile +++ b/sysutils/rsyslog55-relp/Makefile @@ -6,7 +6,7 @@ # COMMENT= RELP input/output module for rsyslog -MASTERDIR= ${.CURDIR}/../rsyslog5 +MASTERDIR= ${.CURDIR}/../rsyslog55 MNAME= relp BUILD_DEPENDS+= pkg-config:${PORTSDIR}/devel/pkg-config diff --git a/sysutils/rsyslog55-rfc3195/Makefile b/sysutils/rsyslog55-rfc3195/Makefile index 733daa4..db6fe57 100644 --- a/sysutils/rsyslog55-rfc3195/Makefile +++ b/sysutils/rsyslog55-rfc3195/Makefile @@ -6,7 +6,7 @@ # COMMENT= RFC3195 input support for rsyslog -MASTERDIR= ${.CURDIR}/../rsyslog5 +MASTERDIR= ${.CURDIR}/../rsyslog55 MNAME= rfc3195 BUILD_DEPENDS+= pkg-config:${PORTSDIR}/devel/pkg-config diff --git a/sysutils/rsyslog55-snmp/Makefile b/sysutils/rsyslog55-snmp/Makefile index 8116cf1..466d5b8 100644 --- a/sysutils/rsyslog55-snmp/Makefile +++ b/sysutils/rsyslog55-snmp/Makefile @@ -6,7 +6,7 @@ # COMMENT= SNMP trap sender for rsyslog -MASTERDIR= ${.CURDIR}/../rsyslog5 +MASTERDIR= ${.CURDIR}/../rsyslog55 MNAME= snmp LIB_DEPENDS= netsnmp.16:${PORTSDIR}/net-mgmt/net-snmp diff --git a/sysutils/rsyslog55/Makefile b/sysutils/rsyslog55/Makefile index 679ff97..080dd91 100644 --- a/sysutils/rsyslog55/Makefile +++ b/sysutils/rsyslog55/Makefile @@ -6,7 +6,7 @@ # PORTNAME= rsyslog -PORTVERSION= 5.4.0 +PORTVERSION= 5.5.4 CATEGORIES= sysutils MASTER_SITES= http://download.rsyslog.com/rsyslog/ .ifdef MNAME @@ -17,12 +17,20 @@ MAINTAINER= cristianorolim at hotmail.com COMMENT?= Syslogd supporting SQL, TCP and TLS .ifdef MNAME -RUN_DEPENDS= rsyslog>=5:${PORTSDIR}/sysutils/rsyslog5 +RUN_DEPENDS= rsyslog>=5.5.0:${PORTSDIR}/sysutils/rsyslog55 PLIST= ${.CURDIR}/pkg-plist .endif -CONFLICTS= rsyslog-[!5].[0-9]* +.ifdef WITH_MYSQL_MICROSECONDS +EXTRA_PATCHES+= ${FILESDIR}/extra-patch-mysql-microseconds +.endif + +.ifdef WITH_SANE_HOSTNAME +EXTRA_PATCHES+= ${FILESDIR}/extra-patch-sane-hostname +.endif + +CONFLICTS= rsyslog-[!5].[0-9]* rsyslog-5.4.* CPPFLAGS+= -I${LOCALBASE}/include LDFLAGS+= -L${LOCALBASE}/lib GNU_CONFIGURE= yes @@ -52,6 +60,7 @@ post-patch: ${WRKSRC}/tools/syslogd.c @${GREP} -rl '/etc/rsyslog.conf' ${WRKSRC}|${XARGS} ${REINPLACE_CMD} -e\ 's|/etc/rsyslog.conf|${PREFIX}/etc/rsyslog.conf|' + @${REINPLACE_CMD} -e 's,/lib/rsyslog,${PREFIX}/lib/rsyslog,' ${WRKSRC}/tools/syslogd.c @${FIND} ${WRKSRC} -name '*.bak' -delete post-install: @@ -75,8 +84,8 @@ IGNORE= with gssapi module is only supported on FreeBSD 7.x or later CONFIGURE_ARGS+= --disable-rsyslogd --disable-klog -DESCR?= ${.CURDIR}/../rsyslog5/pkg-descr -MD5_FILE?= ${.CURDIR}/../rsyslog5/distinfo +DESCR?= ${.CURDIR}/../rsyslog55/pkg-descr +MD5_FILE?= ${.CURDIR}/../rsyslog55/distinfo .endif .if ${OSVERSION} < 700042 diff --git a/sysutils/rsyslog55/distinfo b/sysutils/rsyslog55/distinfo index a452749..bbda583 100644 --- a/sysutils/rsyslog55/distinfo +++ b/sysutils/rsyslog55/distinfo @@ -1,3 +1,3 @@ -MD5 (rsyslog-5.4.0.tar.gz) = 291882229d50496f42bd63174076dd37 -SHA256 (rsyslog-5.4.0.tar.gz) = d9cd21d2fcd45fcae65eb0a51927c40315cca02afdc62478abd950febfcf7228 -SIZE (rsyslog-5.4.0.tar.gz) = 2124201 +MD5 (rsyslog-5.5.4.tar.gz) = 824df2504955df1619e5ec2915d783aa +SHA256 (rsyslog-5.5.4.tar.gz) = 31853a551ea7ca960c59c9e33406b1748bdf311059c9d8a4ce98816d51b17cac +SIZE (rsyslog-5.5.4.tar.gz) = 2200136 diff --git a/sysutils/rsyslog55/files/extra-patch-mysql-microseconds b/sysutils/rsyslog55/files/extra-patch-mysql-microseconds new file mode 100644 index 0000000..ec248b0 --- /dev/null +++ b/sysutils/rsyslog55/files/extra-patch-mysql-microseconds @@ -0,0 +1,56 @@ +--- ./runtime/datetime.c.orig 2010-05-04 18:57:25.588028725 -0400 ++++ ./runtime/datetime.c 2010-05-04 18:59:12.390680038 -0400 +@@ -644,18 +644,30 @@ + pBuf[1] = (ts->year / 100) % 10 + '0'; + pBuf[2] = (ts->year / 10) % 10 + '0'; + pBuf[3] = ts->year % 10 + '0'; +- pBuf[4] = (ts->month / 10) % 10 + '0'; +- pBuf[5] = ts->month % 10 + '0'; +- pBuf[6] = (ts->day / 10) % 10 + '0'; +- pBuf[7] = ts->day % 10 + '0'; +- pBuf[8] = (ts->hour / 10) % 10 + '0'; +- pBuf[9] = ts->hour % 10 + '0'; +- pBuf[10] = (ts->minute / 10) % 10 + '0'; +- pBuf[11] = ts->minute % 10 + '0'; +- pBuf[12] = (ts->second / 10) % 10 + '0'; +- pBuf[13] = ts->second % 10 + '0'; +- pBuf[14] = '\0'; +- return 15; ++ pBuf[4] = '-'; ++ pBuf[5] = (ts->month / 10) % 10 + '0'; ++ pBuf[6] = ts->month % 10 + '0'; ++ pBuf[7] = '-'; ++ pBuf[8] = (ts->day / 10) % 10 + '0'; ++ pBuf[9] = ts->day % 10 + '0'; ++ pBuf[10] = ' '; ++ pBuf[11] = (ts->hour / 10) % 10 + '0'; ++ pBuf[12] = ts->hour % 10 + '0'; ++ pBuf[13] = ':'; ++ pBuf[14] = (ts->minute / 10) % 10 + '0'; ++ pBuf[15] = ts->minute % 10 + '0'; ++ pBuf[16] = ':'; ++ pBuf[17] = (ts->second / 10) % 10 + '0'; ++ pBuf[18] = ts->second % 10 + '0'; ++ pBuf[19] = '.'; ++ pBuf[20] = (ts->secfrac / 100000) % 10 + '0'; ++ pBuf[21] = (ts->secfrac / 10000) % 10 + '0'; ++ pBuf[22] = (ts->secfrac / 1000) % 10 + '0'; ++ pBuf[23] = (ts->secfrac / 100) % 10 + '0'; ++ pBuf[24] = (ts->secfrac / 10) % 10 + '0'; ++ pBuf[25] = ts->secfrac % 10 + '0'; ++ pBuf[26] = '\0'; ++ return 26; + + } + +--- ./runtime/msg.c.orig 2010-05-04 19:00:20.241528788 -0400 ++++ ./runtime/msg.c 2010-05-04 19:00:06.136349680 -0400 +@@ -1293,7 +1293,7 @@ + case tplFmtMySQLDate: + MsgLock(pM); + if(pM->pszTIMESTAMP_MySQL == NULL) { +- if((pM->pszTIMESTAMP_MySQL = MALLOC(15)) == NULL) { ++ if((pM->pszTIMESTAMP_MySQL = MALLOC(26)) == NULL) { + MsgUnlock(pM); + return ""; + } diff --git a/sysutils/rsyslog55/files/extra-patch-sane-hostname b/sysutils/rsyslog55/files/extra-patch-sane-hostname new file mode 100644 index 0000000..bc72514 --- /dev/null +++ b/sysutils/rsyslog55/files/extra-patch-sane-hostname @@ -0,0 +1,40 @@ +--- ./tools/syslogd.c.orig 2010-05-04 19:02:05.548362478 -0400 ++++ ./tools/syslogd.c 2010-05-04 19:02:27.452450741 -0400 +@@ -2611,37 +2611,6 @@ + net.getLocalHostname(&LocalFQDNName); + CHKmalloc(LocalHostName = (uchar*) strdup((char*)LocalFQDNName)); + glbl.SetLocalFQDNName(LocalFQDNName); /* set the FQDN before we modify it */ +- if((p = (uchar*)strchr((char*)LocalHostName, '.'))) { +- *p++ = '\0'; +- LocalDomain = p; +- } else { +- LocalDomain = (uchar*)""; +- +- /* It's not clearly defined whether gethostname() +- * should return the simple hostname or the fqdn. A +- * good piece of software should be aware of both and +- * we want to distribute good software. Joey +- * +- * Good software also always checks its return values... +- * If syslogd starts up before DNS is up & /etc/hosts +- * doesn't have LocalHostName listed, gethostbyname will +- * return NULL. +- */ +- /* TODO: gethostbyname() is not thread-safe, but replacing it is +- * not urgent as we do not run on multiple threads here. rgerhards, 2007-09-25 +- */ +- hent = gethostbyname((char*)LocalHostName); +- if(hent) { +- free(LocalHostName); +- CHKmalloc(LocalHostName = (uchar*)strdup(hent->h_name)); +- +- if((p = (uchar*)strchr((char*)LocalHostName, '.'))) +- { +- *p++ = '\0'; +- LocalDomain = p; +- } +- } +- } + + /* Convert to lower case to recognize the correct domain laterly */ + for(p = LocalDomain ; *p ; p++) diff --git a/sysutils/rsyslog55/pkg-plist b/sysutils/rsyslog55/pkg-plist index 4120023..3d534e8 100644 --- a/sysutils/rsyslog55/pkg-plist +++ b/sysutils/rsyslog55/pkg-plist @@ -52,6 +52,7 @@ sbin/rsyslogd %%PORTDOCS%%%%DOCSDIR%%/imgssapi.html %%PORTDOCS%%%%DOCSDIR%%/imklog.html %%PORTDOCS%%%%DOCSDIR%%/imrelp.html +%%PORTDOCS%%%%DOCSDIR%%/imsolaris.html %%PORTDOCS%%%%DOCSDIR%%/imtcp.html %%PORTDOCS%%%%DOCSDIR%%/imuxsock.html %%PORTDOCS%%%%DOCSDIR%%/index.html @@ -93,6 +94,7 @@ sbin/rsyslogd %%PORTDOCS%%%%DOCSDIR%%/rsconf1_dropmsgswithmaliciousdnsptrrecords.html %%PORTDOCS%%%%DOCSDIR%%/rsconf1_droptrailinglfonreception.html %%PORTDOCS%%%%DOCSDIR%%/rsconf1_dynafilecachesize.html +%%PORTDOCS%%%%DOCSDIR%%/rsconf1_escape8bitcharsonreceive.html %%PORTDOCS%%%%DOCSDIR%%/rsconf1_escapecontrolcharactersonreceive.html %%PORTDOCS%%%%DOCSDIR%%/rsconf1_failonchownfailure.html %%PORTDOCS%%%%DOCSDIR%%/rsconf1_filecreatemode.html --- rsyslog55.diff ends here --- From rgerhards at hq.adiscon.com Tue May 18 18:11:37 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 18 May 2010 18:11:37 +0200 Subject: [rsyslog] Feedback requested: fast queue mode Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103DE5@GRFEXC.intern.adiscon.com> Hi all, I am currently thinking about various optimizations for even more throughput inside the v5 engine. During this effort, I am also doing some review of existing literature on lock-free and wait-free algorithms. One of my former ideas re-appeared, and I wanted to get feedback if that would be a useful addition. Even if it is seen as valuable, I will probably not implement it immediately, but I would try to incorporate it into the new design, which I plan to implement later this year. Looking at the queue modes, we have some overhead inside queues because queues need to do a lot of things. Things like race-limiting, blocking on full queue, even going to disk if the queue fills up too much. Also, support for thread pools and all that is needed. Note that all of this overhead is also necessary if the queue is used to run an action asynchronously. I think I could implement a considerably faster queue, if I limit its features. Most importantly, that means: - no support for going to the disk (that should not be an issue, I think) - no support for race-limiting - capability to accept message loss if queue is full - message loss victim not selected based on priority - message loss on shutdown acceptable In short, the queue would provide simple in-memory queueing services including synchronization between multiple producers and consumers, but no advanced services at all. At this price, it could probably reduced the queue overhead very considerably. I think such a queue could be useful for the (common) case when data needs to be shuffled to files, and some loss is acceptable (e.g. UDP is the input). I'd say we could probably improve the performance for this use case by a factor of two. Would this be useful? Thoughts are appreciated. Rainer From rory at ooma.com Tue May 18 18:14:00 2010 From: rory at ooma.com (Rory Toma) Date: Tue, 18 May 2010 09:14:00 -0700 Subject: [rsyslog] Feedback requested: fast queue mode In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103DE5@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103DE5@GRFEXC.intern.adiscon.com> Message-ID: <4BF2BCC8.2060004@ooma.com> I could see this being very useful in a relay situation. As a side question, has anyone done injection from rsyslog into MongoDB? On 5/18/10 9:11 AM, Rainer Gerhards wrote: > Hi all, > > I am currently thinking about various optimizations for even more throughput > inside the v5 engine. During this effort, I am also doing some review of > existing literature on lock-free and wait-free algorithms. One of my former > ideas re-appeared, and I wanted to get feedback if that would be a useful > addition. Even if it is seen as valuable, I will probably not implement it > immediately, but I would try to incorporate it into the new design, which I > plan to implement later this year. > > Looking at the queue modes, we have some overhead inside queues because > queues need to do a lot of things. Things like race-limiting, blocking on > full queue, even going to disk if the queue fills up too much. Also, support > for thread pools and all that is needed. Note that all of this overhead is > also necessary if the queue is used to run an action asynchronously. > > I think I could implement a considerably faster queue, if I limit its > features. Most importantly, that means: > > - no support for going to the disk (that should not be an issue, I think) > - no support for race-limiting > - capability to accept message loss if queue is full > - message loss victim not selected based on priority > - message loss on shutdown acceptable > > In short, the queue would provide simple in-memory queueing services > including synchronization between multiple producers and consumers, but no > advanced services at all. At this price, it could probably reduced the queue > overhead very considerably. > > I think such a queue could be useful for the (common) case when data needs to > be shuffled to files, and some loss is acceptable (e.g. UDP is the input). > I'd say we could probably improve the performance for this use case by a > factor of two. > > Would this be useful? Thoughts are appreciated. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rory at ooma.com Wed May 19 22:48:20 2010 From: rory at ooma.com (Rory Toma) Date: Wed, 19 May 2010 13:48:20 -0700 Subject: [rsyslog] MongoDB Message-ID: <4BF44E94.8010204@ooma.com> Has anyone done a MongoDB insertion engine for rsyslog yet? From tbergfeld at hq.adiscon.com Thu May 20 15:52:45 2010 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Thu, 20 May 2010 15:52:45 +0200 Subject: [rsyslog] rsyslog 5.5.5 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103E08@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.5.5, a member of the devel branch. This is a bug-fixing release which contains a single fix that solves a potential hang condition on system shutdown when infinite action retries are configured for an asynchronous action using a queue in disk-assisted mode and the action was suspended. This is probably not a very common case, but a configuration recommended by our doc samples. If you do not use such a configuration, there is no need to update at this time. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article461.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-204.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From jean.mousinho at ist.utl.pt Thu May 20 19:28:52 2010 From: jean.mousinho at ist.utl.pt (Jean F. Mousinho) Date: Thu, 20 May 2010 18:28:52 +0100 Subject: [rsyslog] rsyslog + tls + debian Message-ID: <1274376532.14504.2.camel@muse.ist.utl.pt> Hello, Was anyone successful to get rsyslog working with TLS module in Debian? rsyslogd 4.4.2, compiled with: FEATURE_REGEXP: Yes FEATURE_LARGEFILE: Yes FEATURE_NETZIP (message compression): Yes GSSAPI Kerberos 5 support: Yes FEATURE_DEBUG (debug build, slow code): No Atomic operations supported: Yes Runtime Instrumentation (slow code): No Output is: rsyslogd: could not load module '/usr/lib/rsyslog/lmnsd_gtls.so', rsyslog error -2078 [try http://www.rsyslog.com/e/2068 ] Module file exists: -rw-r--r-- 1 root root 27196 2010-05-17 16:12 /usr/lib/rsyslog/lmnsd_gtls.so I've used the packages provided in backports. Thanks for your time. Jean Mousinho From james at linux-source.org Fri May 21 07:04:59 2010 From: james at linux-source.org (James Corteciano) Date: Fri, 21 May 2010 13:04:59 +0800 Subject: [rsyslog] Centralized rsyslog server and httpd vhost Message-ID: Hi All, My goal is to centralized all system/apps logs from different web farm servers. I have the following setups: node0 - Centralized rsyslog server web_farm1 - web farm 1 server running RHEL, rsyslog, and httpd with different vhost. web_farm2 - web farm 2 server running RHEL, rsyslog, and httpd with different vhost. 1) How can I configure rsyslog from node0 to capture all vhost logs from web_farm servers and all logs will be directly placed like: /var/log/syslog/web_farm1/-error.log /var/log/syslog/web_farm1/-access.log 2) How to configure the httpd service in web_farm servers to push all logs to node0 syslog server? I saw from internet like the following. httpd.conf: ErrorLog "|/bin/logger -p local5.err" CustomLog "|/bin/logger -p local6.info" rsyslog.conf: *.* @node0 Thank you. Regards, James From david at lang.hm Fri May 21 07:31:52 2010 From: david at lang.hm (david at lang.hm) Date: Thu, 20 May 2010 22:31:52 -0700 (PDT) Subject: [rsyslog] Feedback requested: fast queue mode In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103DE5@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103DE5@GRFEXC.intern.adiscon.com> Message-ID: On Tue, 18 May 2010, Rainer Gerhards wrote: > Hi all, > > I am currently thinking about various optimizations for even more throughput > inside the v5 engine. During this effort, I am also doing some review of > existing literature on lock-free and wait-free algorithms. One of my former > ideas re-appeared, and I wanted to get feedback if that would be a useful > addition. Even if it is seen as valuable, I will probably not implement it > immediately, but I would try to incorporate it into the new design, which I > plan to implement later this year. > > Looking at the queue modes, we have some overhead inside queues because > queues need to do a lot of things. Things like race-limiting, blocking on > full queue, even going to disk if the queue fills up too much. Also, support > for thread pools and all that is needed. Note that all of this overhead is > also necessary if the queue is used to run an action asynchronously. > > I think I could implement a considerably faster queue, if I limit its > features. Most importantly, that means: > > - no support for going to the disk (that should not be an issue, I think) > - no support for race-limiting > - capability to accept message loss if queue is full > - message loss victim not selected based on priority > - message loss on shutdown acceptable > > In short, the queue would provide simple in-memory queueing services > including synchronization between multiple producers and consumers, but no > advanced services at all. At this price, it could probably reduced the queue > overhead very considerably. > > I think such a queue could be useful for the (common) case when data needs to > be shuffled to files, and some loss is acceptable (e.g. UDP is the input). > I'd say we could probably improve the performance for this use case by a > factor of two. > > Would this be useful? Thoughts are appreciated. yes, this sounds useful. David Lang From david at lang.hm Fri May 21 07:33:25 2010 From: david at lang.hm (david at lang.hm) Date: Thu, 20 May 2010 22:33:25 -0700 (PDT) Subject: [rsyslog] MongoDB In-Reply-To: <4BF44E94.8010204@ooma.com> References: <4BF44E94.8010204@ooma.com> Message-ID: On Wed, 19 May 2010, Rory Toma wrote: > Has anyone done a MongoDB insertion engine for rsyslog yet? not that I am aware of. In fact, as far as I know, only the postgres and possibly oracle modules take full advantage of the vector mode that allows them to efficiantly batch the message inserts. lots of room for work in this area. David Lang From david at lang.hm Fri May 21 07:37:55 2010 From: david at lang.hm (david at lang.hm) Date: Thu, 20 May 2010 22:37:55 -0700 (PDT) Subject: [rsyslog] Centralized rsyslog server and httpd vhost In-Reply-To: References: Message-ID: On Fri, 21 May 2010, James Corteciano wrote: > Hi All, > > My goal is to centralized all system/apps logs from different web farm > servers. I have the following setups: > > node0 - Centralized rsyslog server > web_farm1 - web farm 1 server running RHEL, rsyslog, and httpd with > different vhost. > web_farm2 - web farm 2 server running RHEL, rsyslog, and httpd with > different vhost. > > 1) How can I configure rsyslog from node0 to capture all vhost logs from > web_farm servers and all logs will be directly placed like: > > /var/log/syslog/web_farm1/-error.log > /var/log/syslog/web_farm1/-access.log > > 2) How to configure the httpd service in web_farm servers to push all logs > to node0 syslog server? I saw from internet like the following. > > httpd.conf: > ErrorLog "|/bin/logger -p local5.err" > CustomLog "|/bin/logger -p local6.info" > > rsyslog.conf: > *.* @node0 when logging from apache you can have log commands inside each vhost, or if you don't the logs will be handled by the main server. what I do is to have the access logs handled by the main server and create a custom format that includes the vhost as part of the format (I also reorder things so that data I really care about is near the beginning of the log and data that can be long is later in the message, so if it becomes extremely long and overflows the max log length I don't loose data I consider critical) then I run it through a perl script that reformats the message to put the vhost name in the server field and sends it out via UDP to my syslog server. I don't have access to that file at the moment (it's at work), I'll try to get a copy tomorrow and post it. David Lang From james at linux-source.org Fri May 21 08:57:50 2010 From: james at linux-source.org (James Corteciano) Date: Fri, 21 May 2010 14:57:50 +0800 Subject: [rsyslog] Centralized rsyslog server and httpd vhost In-Reply-To: References: Message-ID: Hi David, Thanks for your reply and I'm looking forward about it. Regards, James On Fri, May 21, 2010 at 1:37 PM, wrote: > On Fri, 21 May 2010, James Corteciano wrote: > > > Hi All, > > > > My goal is to centralized all system/apps logs from different web farm > > servers. I have the following setups: > > > > node0 - Centralized rsyslog server > > web_farm1 - web farm 1 server running RHEL, rsyslog, and httpd with > > different vhost. > > web_farm2 - web farm 2 server running RHEL, rsyslog, and httpd with > > different vhost. > > > > 1) How can I configure rsyslog from node0 to capture all vhost logs from > > web_farm servers and all logs will be directly placed like: > > > > /var/log/syslog/web_farm1/-error.log > > /var/log/syslog/web_farm1/-access.log > > > > 2) How to configure the httpd service in web_farm servers to push all > logs > > to node0 syslog server? I saw from internet like the following. > > > > httpd.conf: > > ErrorLog "|/bin/logger -p local5.err" > > CustomLog "|/bin/logger -p local6.info" > > > > rsyslog.conf: > > *.* @node0 > > when logging from apache you can have log commands inside each vhost, or > if you don't the logs will be handled by the main server. > > what I do is to have the access logs handled by the main server and create > a custom format that includes the vhost as part of the format (I also > reorder things so that data I really care about is near the beginning of > the log and data that can be long is later in the message, so if it > becomes extremely long and overflows the max log length I don't loose data > I consider critical) > > then I run it through a perl script that reformats the message to put the > vhost name in the server field and sends it out via UDP to my syslog > server. > > I don't have access to that file at the moment (it's at work), I'll try to > get a copy tomorrow and post it. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From james at linux-source.org Fri May 21 09:49:04 2010 From: james at linux-source.org (James Corteciano) Date: Fri, 21 May 2010 15:49:04 +0800 Subject: [rsyslog] Centralized rsyslog server and httpd vhost In-Reply-To: References: Message-ID: Currently, I have this kind of setup. [node0] rsyslog.conf: $template MsgFormat,"%msg%\n" $template ApacheRemoteCustom,"/var/log/httpd/web_farm1/%msg:F,32:2%.log" if $syslogfacility-text == 'local6' and $programname == 'rhcs-node1' then -?ApacheRemoteCustom;MsgFormat [web_farm1] httpd.conf: LogLevel warn ErrorLog "|/bin/logger -p local5.err" LogFormat "%v %h %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" vcombined CustomLog "|/bin/logger -p local6.info -t rhcs-node1" vcombined" It works for getting vhost access logs. However, it doesn't work for error logs because apache ErrorLog is not possible to customize the error log by adding or removing information . Regards, James On Fri, May 21, 2010 at 1:37 PM, wrote: > On Fri, 21 May 2010, James Corteciano wrote: > > > Hi All, > > > > My goal is to centralized all system/apps logs from different web farm > > servers. I have the following setups: > > > > node0 - Centralized rsyslog server > > web_farm1 - web farm 1 server running RHEL, rsyslog, and httpd with > > different vhost. > > web_farm2 - web farm 2 server running RHEL, rsyslog, and httpd with > > different vhost. > > > > 1) How can I configure rsyslog from node0 to capture all vhost logs from > > web_farm servers and all logs will be directly placed like: > > > > /var/log/syslog/web_farm1/-error.log > > /var/log/syslog/web_farm1/-access.log > > > > 2) How to configure the httpd service in web_farm servers to push all > logs > > to node0 syslog server? I saw from internet like the following. > > > > httpd.conf: > > ErrorLog "|/bin/logger -p local5.err" > > CustomLog "|/bin/logger -p local6.info" > > > > rsyslog.conf: > > *.* @node0 > > when logging from apache you can have log commands inside each vhost, or > if you don't the logs will be handled by the main server. > > what I do is to have the access logs handled by the main server and create > a custom format that includes the vhost as part of the format (I also > reorder things so that data I really care about is near the beginning of > the log and data that can be long is later in the message, so if it > becomes extremely long and overflows the max log length I don't loose data > I consider critical) > > then I run it through a perl script that reformats the message to put the > vhost name in the server field and sends it out via UDP to my syslog > server. > > I don't have access to that file at the moment (it's at work), I'll try to > get a copy tomorrow and post it. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From iain at shihad.org Fri May 21 18:37:23 2010 From: iain at shihad.org (Iain M Conochie) Date: Fri, 21 May 2010 17:37:23 +0100 Subject: [rsyslog] Splitting rsyslog messages by hostname into MySQL database Message-ID: <4BF6B6C3.1080108@shihad.org> Afternoon all, I have rsyslog sending all my messages into a mysql database and this is working well. Now I want to start to split the remote messages via hostname into separate tables in the database. I have created a new table FaiEvents with the same schema as SystemEvents and also I have created a config file with actions and template like so: $template Fai-Event,"insert into FaiEvents \ (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)\ values ('%msg%',z,\ %syslogfacility%,\ '%HOSTNAME%',\ %syslogpriority%,\ '%timereported:::date-mysql%',\ '%timegenerated:::date-mysql%',\ %iut%,\ '%syslogtag%')",SQL if $hostname == 'faiserver'\ then :ommysql:localhost,Syslog,rsyslog,********;Fai-Event However rsyslog gives me the error: rsyslogd: the last error occured in /etc/rsyslog.d/05-faiservers.conf, line 13 which is the action line. What variable should I be using to test for the hostname? Can I use a regex here? Any help appreciated! Regards Iain Conochie From james at linux-source.org Sat May 22 11:00:38 2010 From: james at linux-source.org (James Corteciano) Date: Sat, 22 May 2010 17:00:38 +0800 Subject: [rsyslog] Centralized rsyslog server and httpd vhost In-Reply-To: References: Message-ID: Hi David, How about your Apache ErrorLogs? How do you get it that the same thing in Access log? Thank you. Regards, James On Fri, May 21, 2010 at 1:37 PM, wrote: > On Fri, 21 May 2010, James Corteciano wrote: > > > Hi All, > > > > My goal is to centralized all system/apps logs from different web farm > > servers. I have the following setups: > > > > node0 - Centralized rsyslog server > > web_farm1 - web farm 1 server running RHEL, rsyslog, and httpd with > > different vhost. > > web_farm2 - web farm 2 server running RHEL, rsyslog, and httpd with > > different vhost. > > > > 1) How can I configure rsyslog from node0 to capture all vhost logs from > > web_farm servers and all logs will be directly placed like: > > > > /var/log/syslog/web_farm1/-error.log > > /var/log/syslog/web_farm1/-access.log > > > > 2) How to configure the httpd service in web_farm servers to push all > logs > > to node0 syslog server? I saw from internet like the following. > > > > httpd.conf: > > ErrorLog "|/bin/logger -p local5.err" > > CustomLog "|/bin/logger -p local6.info" > > > > rsyslog.conf: > > *.* @node0 > > when logging from apache you can have log commands inside each vhost, or > if you don't the logs will be handled by the main server. > > what I do is to have the access logs handled by the main server and create > a custom format that includes the vhost as part of the format (I also > reorder things so that data I really care about is near the beginning of > the log and data that can be long is later in the message, so if it > becomes extremely long and overflows the max log length I don't loose data > I consider critical) > > then I run it through a perl script that reformats the message to put the > vhost name in the server field and sends it out via UDP to my syslog > server. > > I don't have access to that file at the moment (it's at work), I'll try to > get a copy tomorrow and post it. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Sun May 23 00:21:43 2010 From: david at lang.hm (david at lang.hm) Date: Sat, 22 May 2010 15:21:43 -0700 (PDT) Subject: [rsyslog] Centralized rsyslog server and httpd vhost In-Reply-To: References: Message-ID: On Fri, 21 May 2010, James Corteciano wrote: > Currently, I have this kind of setup. > > [node0] > > rsyslog.conf: > $template MsgFormat,"%msg%\n" > $template ApacheRemoteCustom,"/var/log/httpd/web_farm1/%msg:F,32:2%.log" > if $syslogfacility-text == 'local6' and $programname == 'rhcs-node1' then > -?ApacheRemoteCustom;MsgFormat > > > [web_farm1] > > httpd.conf: > LogLevel warn > ErrorLog "|/bin/logger -p local5.err" > LogFormat "%v %h %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" > vcombined > CustomLog "|/bin/logger -p local6.info -t rhcs-node1" vcombined" > > > It works for getting vhost access logs. However, it doesn't work for error > logs because apache ErrorLog is not possible to customize the error log by > adding or removing > information > . personally I would not try to get error logs in directly specificly because of this. (there's also the problem that a single cgi script error could spew out thousands of lines of garbage into the error log) what's needed is an error log that doesn't include stderr from the cgis being run, but would log a single line along the lines of 'output on stderr from X, see Y for details' where Y is another logfile on the server. unfortunantly this will take modifications to apache to do. David Lang > Regards, > James > > > On Fri, May 21, 2010 at 1:37 PM, wrote: > >> On Fri, 21 May 2010, James Corteciano wrote: >> >>> Hi All, >>> >>> My goal is to centralized all system/apps logs from different web farm >>> servers. I have the following setups: >>> >>> node0 - Centralized rsyslog server >>> web_farm1 - web farm 1 server running RHEL, rsyslog, and httpd with >>> different vhost. >>> web_farm2 - web farm 2 server running RHEL, rsyslog, and httpd with >>> different vhost. >>> >>> 1) How can I configure rsyslog from node0 to capture all vhost logs from >>> web_farm servers and all logs will be directly placed like: >>> >>> /var/log/syslog/web_farm1/-error.log >>> /var/log/syslog/web_farm1/-access.log >>> >>> 2) How to configure the httpd service in web_farm servers to push all >> logs >>> to node0 syslog server? I saw from internet like the following. >>> >>> httpd.conf: >>> ErrorLog "|/bin/logger -p local5.err" >>> CustomLog "|/bin/logger -p local6.info" >>> >>> rsyslog.conf: >>> *.* @node0 >> >> when logging from apache you can have log commands inside each vhost, or >> if you don't the logs will be handled by the main server. >> >> what I do is to have the access logs handled by the main server and create >> a custom format that includes the vhost as part of the format (I also >> reorder things so that data I really care about is near the beginning of >> the log and data that can be long is later in the message, so if it >> becomes extremely long and overflows the max log length I don't loose data >> I consider critical) >> >> then I run it through a perl script that reformats the message to put the >> vhost name in the server field and sends it out via UDP to my syslog >> server. >> >> I don't have access to that file at the moment (it's at work), I'll try to >> get a copy tomorrow and post it. >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From jli at jlisbz.com Wed May 26 05:54:57 2010 From: jli at jlisbz.com (John Li) Date: Tue, 25 May 2010 23:54:57 -0400 Subject: [rsyslog] Where is the output module for the udp transportation to remote syslog server Message-ID: Hi, Is the output via udp to remote syslog server implemented as a output module? I could not find it in the plugins folder. Thanks. -- John Jun Li jli at jlisbz.com My Blog: http://www.jlisbz.com My LinkedIn Profile: http://www.linkedin.com/in/johnjunli From fn42551 at fmi.uni-sofia.bg Wed May 26 16:17:07 2010 From: fn42551 at fmi.uni-sofia.bg (Angel Tsankov) Date: Wed, 26 May 2010 17:17:07 +0300 Subject: [rsyslog] Check if rsyslog is running Message-ID: What is the recommended way for an application to check if rsyslogd is running? Angel Tsankov From jli at jlisbz.com Wed May 26 16:41:25 2010 From: jli at jlisbz.com (John Li) Date: Wed, 26 May 2010 10:41:25 -0400 Subject: [rsyslog] Check if rsyslog is running In-Reply-To: References: Message-ID: monit from http://mmonit.com/monit/ shoud be able to handle this easily. And it can do both monitoring and restarting if the process is crashed. -- John Jun Li jli at jlisbz.com My Blog: http://www.jlisbz.com My LinkedIn Profile: http://www.linkedin.com/in/johnjunli My Twitter: http://www.twitter.com/jlisbz My facebook: http://www.facebook.com/profile.php?id=593495282 On Wed, May 26, 2010 at 10:17 AM, Angel Tsankov wrote: > What is the recommended way for an application to check if rsyslogd is > running? > > Angel Tsankov > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed May 26 17:36:56 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 26 May 2010 18:36:56 +0300 Subject: [rsyslog] Where is the output module for the udp transportation toremote syslog server Message-ID: <003401cafce9$44d808a1$100013ac@intern.adiscon.com> This is a built-in module, it does not need to be loaded (it is actually linked into the main executable). HTH Rainer ----- Urspr?ngliche Nachricht ----- Von: John Li Gesendet: Mittwoch, 26. Mai 2010 07:03 An: rsyslog at lists.adiscon.com Betreff: [rsyslog] Where is the output module for the udp transportation toremote syslog server Hi, Is the output via udp to remote syslog server implemented as a output module? I could not find it in the plugins folder. Thanks. -- John Jun Li jli at jlisbz.com My Blog: http://www.jlisbz.com My LinkedIn Profile: http://www.linkedin.com/in/johnjunli _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From jli at jlisbz.com Wed May 26 17:46:03 2010 From: jli at jlisbz.com (John Li) Date: Wed, 26 May 2010 11:46:03 -0400 Subject: [rsyslog] Where is the output module for the udp transportation toremote syslog server In-Reply-To: <003401cafce9$44d808a1$100013ac@intern.adiscon.com> References: <003401cafce9$44d808a1$100013ac@intern.adiscon.com> Message-ID: Thanks. My goal is to change the content of msg and I am planning to use output module to do that. Is this the right approach and do you mind point me to some sample code in output module to do that? -- John Jun Li jli at jlisbz.com My Blog: http://www.jlisbz.com My LinkedIn Profile: http://www.linkedin.com/in/johnjunli On Wed, May 26, 2010 at 11:36 AM, Rainer Gerhards wrote: > This is a built-in module, it does not need to be loaded (it is actually > linked into the main executable). > > HTH > Rainer > > ----- Urspr?ngliche Nachricht ----- > Von: John Li > Gesendet: Mittwoch, 26. Mai 2010 07:03 > An: rsyslog at lists.adiscon.com > Betreff: [rsyslog] Where is the output module for the udp transportation > toremote syslog server > > Hi, > > Is the output via udp to remote syslog server implemented as a output > module? I could not find it in the plugins folder. > > Thanks. > > -- > John Jun Li > jli at jlisbz.com > > My Blog: http://www.jlisbz.com > My LinkedIn Profile: http://www.linkedin.com/in/johnjunli > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed May 26 21:28:06 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 26 May 2010 22:28:06 +0300 Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server Message-ID: <004501cafd09$9027db05$100013ac@intern.adiscon.com> You need to look into templates. It is quite easy to rewrite message content with templates. There are samples in the doc and in the wiki. Rainer ----- Urspr?ngliche Nachricht ----- Von: John Li Gesendet: Mittwoch, 26. Mai 2010 18:53 An: rsyslog-users Betreff: Re: [rsyslog] Where is the output module for the udp transportationtoremote syslog server Thanks. My goal is to change the content of msg and I am planning to use output module to do that. Is this the right approach and do you mind point me to some sample code in output module to do that? -- John Jun Li jli at jlisbz.com My Blog: http://www.jlisbz.com My LinkedIn Profile: http://www.linkedin.com/in/johnjunli On Wed, May 26, 2010 at 11:36 AM, Rainer Gerhards wrote: > This is a built-in module, it does not need to be loaded (it is actually > linked into the main executable). > > HTH > Rainer > > ----- Urspr?ngliche Nachricht ----- > Von: John Li > Gesendet: Mittwoch, 26. Mai 2010 07:03 > An: rsyslog at lists.adiscon.com > Betreff: [rsyslog] Where is the output module for the udp transportation > toremote syslog server > > Hi, > > Is the output via udp to remote syslog server implemented as a output > module? I could not find it in the plugins folder. > > Thanks. > > -- > John Jun Li > jli at jlisbz.com > > My Blog: http://www.jlisbz.com > My LinkedIn Profile: http://www.linkedin.com/in/johnjunli > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From jli at jlisbz.com Wed May 26 22:15:16 2010 From: jli at jlisbz.com (John Li) Date: Wed, 26 May 2010 16:15:16 -0400 Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server In-Reply-To: <004501cafd09$9027db05$100013ac@intern.adiscon.com> References: <004501cafd09$9027db05$100013ac@intern.adiscon.com> Message-ID: This is the RE case I posted in forum last week and output module is the best to achieve it. Here is the case I described: " For example, a typical firewall log: 192.168.20.5 23456 192.168.10.10 80 Accept Web 192.168.20.6 5678 192.168.10.10 22 Deny SSH If I want to have the xml form of them, it could be : 192.168.20.5192.168.10.102345680AcceptWeb 192.168.20.6192.168.10.10567822DenySSH If I understand correctly for template, I had to do RE for 6 times for each log entry and that could cause performance issue in large environment for sure. " So I need to rewrite the msg in the output module, please let me know where to find some sample code or doc. And here is one more question: "One thing I want to make sure is the output plugin which I will make should be still able to use other output method such as syslog/snmp etc with the converted message, right? ." I was able to create my own output module based on the stdout module but could not figure out how to rewrite the msg back to rsyslog so the rewritten msg can be used by other output module. Is this doable? Thanks a lot. -- John On Wed, May 26, 2010 at 3:28 PM, Rainer Gerhards wrote: > You need to look into templates. It is quite easy to rewrite message > content with templates. There are samples in the doc and in the wiki. > > Rainer > > ----- Urspr?ngliche Nachricht ----- > Von: John Li > Gesendet: Mittwoch, 26. Mai 2010 18:53 > An: rsyslog-users > Betreff: Re: [rsyslog] Where is the output module for the udp > transportationtoremote syslog server > > Thanks. > > My goal is to change the content of msg and I am planning to use output > module to do that. Is this the right approach and do you mind point me to > some sample code in output module to do that? > > > -- > John Jun Li > jli at jlisbz.com > > My Blog: http://www.jlisbz.com > My LinkedIn Profile: http://www.linkedin.com/in/johnjunli > > > > On Wed, May 26, 2010 at 11:36 AM, Rainer Gerhards > wrote: > > > This is a built-in module, it does not need to be loaded (it is actually > > linked into the main executable). > > > > HTH > > Rainer > > > > ----- Urspr?ngliche Nachricht ----- > > Von: John Li > > Gesendet: Mittwoch, 26. Mai 2010 07:03 > > An: rsyslog at lists.adiscon.com > > Betreff: [rsyslog] Where is the output module for the udp transportation > > toremote syslog server > > > > Hi, > > > > Is the output via udp to remote syslog server implemented as a output > > module? I could not find it in the plugins folder. > > > > Thanks. > > > > -- > > John Jun Li > > jli at jlisbz.com > > > > My Blog: http://www.jlisbz.com > > My LinkedIn Profile: http://www.linkedin.com/in/johnjunli > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Thu May 27 07:55:25 2010 From: david at lang.hm (david at lang.hm) Date: Wed, 26 May 2010 22:55:25 -0700 (PDT) Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server In-Reply-To: References: <004501cafd09$9027db05$100013ac@intern.adiscon.com> Message-ID: you may want to consider doing this on the input side instead of the output side. see http://www.rsyslog.com/doc-messageparser.html yes, in many ways it's operating backwards, but it may be significantly less work to implement and maintain it this way. David Lang On Wed, 26 May 2010, John Li wrote: > This is the RE case I posted in forum last week and output module is the > best to achieve it. > > Here is the case I described: > " > For example, a typical firewall log: > 192.168.20.5 23456 192.168.10.10 80 Accept Web > 192.168.20.6 5678 192.168.10.10 22 Deny SSH > > If I want to have the xml form of them, it could be : > 192.168.20.5192.168.10.102345680AcceptWeb > 192.168.20.6192.168.10.10567822DenySSH > > If I understand correctly for template, I had to do RE for 6 times for each > log entry and that could cause performance issue in large environment for > sure. " > > So I need to rewrite the msg in the output module, please let me know where > to find some sample code or doc. And here is one more question: > > "One thing I want to make sure is the output plugin which I will make should > be still able to use other output method such as syslog/snmp etc with the > converted message, right? ." > > I was able to create my own output module based on the stdout module but > could not figure out how to rewrite the msg back to rsyslog so the rewritten > msg can be used by other output module. Is this doable? > > Thanks a lot. > > From jli at jlisbz.com Thu May 27 15:29:28 2010 From: jli at jlisbz.com (John Li) Date: Thu, 27 May 2010 09:29:28 -0400 Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server In-Reply-To: References: <004501cafd09$9027db05$100013ac@intern.adiscon.com> Message-ID: Not totally agree. First, your will lose the flexibility in the input side if you put the rewritten code in the input module. Second, parser looks like to target the syslog format validity instead of message rewritten. But if it's not possible to rewrite the msg in the output module, I will have to do that in the parser. Can someone please confirm? Thanks. -- John Jun Li jli at jlisbz.com My Blog: http://www.jlisbz.com My LinkedIn Profile: http://www.linkedin.com/in/johnjunli On Thu, May 27, 2010 at 1:55 AM, wrote: > you may want to consider doing this on the input side instead of the > output side. > > see http://www.rsyslog.com/doc-messageparser.html > > yes, in many ways it's operating backwards, but it may be significantly > less work to implement and maintain it this way. > > David Lang > > On Wed, 26 May 2010, John Li wrote: > > > This is the RE case I posted in forum last week and output module is the > > best to achieve it. > > > > Here is the case I described: > > " > > For example, a typical firewall log: > > 192.168.20.5 23456 192.168.10.10 80 Accept Web > > 192.168.20.6 5678 192.168.10.10 22 Deny SSH > > > > If I want to have the xml form of them, it could be : > > > 192.168.20.5192.168.10.102345680AcceptWeb > > > 192.168.20.6192.168.10.10567822DenySSH > > > > If I understand correctly for template, I had to do RE for 6 times for > each > > log entry and that could cause performance issue in large environment for > > sure. " > > > > So I need to rewrite the msg in the output module, please let me know > where > > to find some sample code or doc. And here is one more question: > > > > "One thing I want to make sure is the output plugin which I will make > should > > be still able to use other output method such as syslog/snmp etc with the > > converted message, right? ." > > > > I was able to create my own output module based on the stdout module but > > could not figure out how to rewrite the msg back to rsyslog so the > rewritten > > msg can be used by other output module. Is this doable? > > > > Thanks a lot. > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Fri May 28 07:25:58 2010 From: david at lang.hm (david at lang.hm) Date: Thu, 27 May 2010 22:25:58 -0700 (PDT) Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server In-Reply-To: References: <004501cafd09$9027db05$100013ac@intern.adiscon.com> Message-ID: On Thu, 27 May 2010, John Li wrote: > Not totally agree. First, your will lose the flexibility in the input side > if you put the rewritten code in the input module. Second, parser looks like > to target the syslog format validity instead of message rewritten. it doesn't just validate the message, it takes the message off the wire, and breaksit into the separate properties that rsyslog handles internally (and are available for the output templates). It already has the option to modify the string as it does this (look at control character re-writing) > But if it's not possible to rewrite the msg in the output module, I will > have to do that in the parser. Can someone please confirm? I agree that doing it in the output would be far better in many ways, but since there isn't a way to do a plugin there (at least not as far as I know, it would be good to get confirmation or a better idea) David Lang > Thanks. > > From rgerhards at hq.adiscon.com Mon May 31 11:35:54 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 31 May 2010 11:35:54 +0200 Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server References: <004501cafd09$9027db05$100013ac@intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103E26@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, May 28, 2010 7:26 AM > To: rsyslog-users > Subject: Re: [rsyslog] Where is the output module for the udp > transportationtoremote syslog server > > On Thu, 27 May 2010, John Li wrote: > > > Not totally agree. First, your will lose the flexibility in the input > side > > if you put the rewritten code in the input module. Second, parser > looks like > > to target the syslog format validity instead of message rewritten. > > it doesn't just validate the message, it takes the message off the > wire, > and breaksit into the separate properties that rsyslog handles > internally > (and are available for the output templates). It already has the option > to > modify the string as it does this (look at control character re- > writing) > > > But if it's not possible to rewrite the msg in the output module, I > will > > have to do that in the parser. Can someone please confirm? John, You can do whatever you like in an output module, including rewriting any part of the message. Of course, you can NOT modify the message strings that *other* output modules see. > I agree that doing it in the output would be far better in many ways, > but > since there isn't a way to do a plugin there (at least not as far as I > know, it would be good to get confirmation or a better idea) David, can you tell me what you have on your mind for this functionality? I have thought a bit about it, and I probably have one approach myself. But I would prefer to hear your idea before I push you into a direction. Thanks, Rainer From david at lang.hm Mon May 31 12:16:32 2010 From: david at lang.hm (david at lang.hm) Date: Mon, 31 May 2010 03:16:32 -0700 (PDT) Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103E26@GRFEXC.intern.adiscon.com> References: <004501cafd09$9027db05$100013ac@intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103E26@GRFEXC.intern.adiscon.com> Message-ID: On Mon, 31 May 2010, Rainer Gerhards wrote: >> I agree that doing it in the output would be far better in many ways, >> but >> since there isn't a way to do a plugin there (at least not as far as I >> know, it would be good to get confirmation or a better idea) > > David, can you tell me what you have on your mind for this functionality? I > have thought a bit about it, and I probably have one approach myself. But I > would prefer to hear your idea before I push you into a direction. two options 1. something that would work similar to the existing format string, but would call a C subroutine that could read the existing properties and would create the output string in a buffer 2. something that could also modify the exisitng properties (more powerful, but also more dangerous and could involve locking to prevent other things from trying to read properties at the same time) we haven't gone too far down the road of researching the output performance (since the input and queue locking has dominated so far), but it is clear that the output currently takes significantly more CPU time than input, it may be that being able to use C to define the output format instead of interpreting the format string may be a noticable improvement. Is there a relativly easy way to test this? (say, hard-code a format or two and test writes to file and network with the hard-coded format vs a format string that produces the same output?) David Lang From david at lang.hm Mon May 31 12:30:22 2010 From: david at lang.hm (david at lang.hm) Date: Mon, 31 May 2010 03:30:22 -0700 (PDT) Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server In-Reply-To: References: <004501cafd09$9027db05$100013ac@intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103E26@GRFEXC.intern.adiscon.com> Message-ID: On Mon, 31 May 2010, david at lang.hm wrote: > On Mon, 31 May 2010, Rainer Gerhards wrote: > >>> I agree that doing it in the output would be far better in many ways, >>> but >>> since there isn't a way to do a plugin there (at least not as far as I >>> know, it would be good to get confirmation or a better idea) >> >> David, can you tell me what you have on your mind for this functionality? I >> have thought a bit about it, and I probably have one approach myself. But I >> would prefer to hear your idea before I push you into a direction. > > > two options > > 1. something that would work similar to the existing format > string, but would call a C subroutine that could read the existing > properties and would create the output string in a buffer > > 2. something that could also modify the exisitng properties (more > powerful, but also more dangerous and could involve locking to prevent > other things from trying to read properties at the same time) > > we haven't gone too far down the road of researching the output > performance (since the input and queue locking has dominated so far), but > it is clear that the output currently takes significantly more CPU time > than input, it may be that being able to use C to define the output format > instead of interpreting the format string may be a noticable improvement. > Is there a relativly easy way to test this? (say, hard-code a format or > two and test writes to file and network with the hard-coded format vs a > format string that produces the same output?) for the traditional output formats the difference may not be that much, but if there is extensive parsing involved (as the initial poster is doing, or what I would expect is common for specific log types into a database) the difference can be much more significant since it can replace multiple regex statements with a much faster single pass that looks for word breaks and inserts standard filler in those spots. With the new syslog format where the data is 'supposed to be' in a series of name=value tuples, something like this would be a pretty efficiant way of extracting particular portions of the data to be output (although the properties could be extended to do this sort of thing by providing something similar to a perl hash) David Lang From jli at jlisbz.com Mon May 31 14:17:02 2010 From: jli at jlisbz.com (John Li) Date: Mon, 31 May 2010 05:17:02 -0700 Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server Message-ID: <2054128934449600685@unknownmsgid> Thanks a lot. Currently i am stucked at the design that output module can not modify the msg to be seen by other output modules. I understand why it's designed that way but just wondering if there is a quick hack to persist the modified msg in output module so other modules can see. Or do you guys have something to handle this scenario better? Thanks David for better describing the problem. Sent from my HTC -----Original Message----- From: david at lang.hm Sent: May 31, 2010 6:30 AM To: rsyslog-users Subject: Re: [rsyslog] Where is the output module for the udp transportationtoremote syslog server On Mon, 31 May 2010, david at lang.hm wrote: > On Mon, 31 May 2010, Rainer Gerhards wrote: > >>> I agree that doing it in the output would be far better in many ways, >>> but >>> since there isn't a way to do a plugin there (at least not as far as I >>> know, it would be good to get confirmation or a better idea) >> >> David, can you tell me what you have on your mind for this functionality? I >> have thought a bit about it, and I probably have one approach myself. But I >> would prefer to hear your idea before I push you into a direction. > > > two options > > 1. something that would work similar to the existing format > string, but would call a C subroutine that could read the existing > properties and would create the output string in a buffer > > 2. something that could also modify the exisitng properties (more > powerful, but also more dangerous and could involve locking to prevent > other things from trying to read properties at the same time) > > we haven't gone too far down the road of researching the output > performance (since the input and queue locking has dominated so far), but > it is clear that the output currently takes significantly more CPU time > than input, it may be that being able to use C to define the output format > instead of interpreting the format string may be a noticable improvement. > Is there a relativly easy way to test this? (say, hard-code a format or > two and test writes to file and network with the hard-coded format vs a > format string that produces the same output?) for the traditional output formats the difference may not be that much, but if there is extensive parsing involved (as the initial poster is doing, or what I would expect is common for specific log types into a database) the difference can be much more significant since it can replace multiple regex statements with a much faster single pass that looks for word breaks and inserts standard filler in those spots. With the new syslog format where the data is 'supposed to be' in a series of name=value tuples, something like this would be a pretty efficiant way of extracting particular portions of the data to be output (although the properties could be extended to do this sort of thing by providing something similar to a perl hash) David Lang _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From jli at jlisbz.com Mon May 31 14:17:24 2010 From: jli at jlisbz.com (John Li) Date: Mon, 31 May 2010 08:17:24 -0400 Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server Message-ID: <7684824564864990291@unknownmsgid> Thanks a lot. Currently i am stucked at the design that output module can not modify the msg to be seen by other output modules. I understand why it's designed that way but just wondering if there is a quick hack to persist the modified msg in output module so other modules can see. Or do you guys have something to handle this scenario better? Thanks David for better describing the problem. Sent from my HTC -----Original Message----- From: david at lang.hm Sent: May 31, 2010 6:30 AM To: rsyslog-users Subject: Re: [rsyslog] Where is the output module for the udp transportationtoremote syslog server On Mon, 31 May 2010, david at lang.hm wrote: > On Mon, 31 May 2010, Rainer Gerhards wrote: > >>> I agree that doing it in the output would be far better in many ways, >>> but >>> since there isn't a way to do a plugin there (at least not as far as I >>> know, it would be good to get confirmation or a better idea) >> >> David, can you tell me what you have on your mind for this functionality? I >> have thought a bit about it, and I probably have one approach myself. But I >> would prefer to hear your idea before I push you into a direction. > > > two options > > 1. something that would work similar to the existing format > string, but would call a C subroutine that could read the existing > properties and would create the output string in a buffer > > 2. something that could also modify the exisitng properties (more > powerful, but also more dangerous and could involve locking to prevent > other things from trying to read properties at the same time) > > we haven't gone too far down the road of researching the output > performance (since the input and queue locking has dominated so far), but > it is clear that the output currently takes significantly more CPU time > than input, it may be that being able to use C to define the output format > instead of interpreting the format string may be a noticable improvement. > Is there a relativly easy way to test this? (say, hard-code a format or > two and test writes to file and network with the hard-coded format vs a > format string that produces the same output?) for the traditional output formats the difference may not be that much, but if there is extensive parsing involved (as the initial poster is doing, or what I would expect is common for specific log types into a database) the difference can be much more significant since it can replace multiple regex statements with a much faster single pass that looks for word breaks and inserts standard filler in those spots. With the new syslog format where the data is 'supposed to be' in a series of name=value tuples, something like this would be a pretty efficiant way of extracting particular portions of the data to be output (although the properties could be extended to do this sort of thing by providing something similar to a perl hash) David Lang _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon May 31 14:24:27 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 31 May 2010 14:24:27 +0200 Subject: [rsyslog] Where is the output module for the udptransportationtoremote syslog server References: <2054128934449600685@unknownmsgid> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103E2E@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of John Li > Sent: Monday, May 31, 2010 2:17 PM > To: david at lang.hm; rsyslog-users > Subject: Re: [rsyslog] Where is the output module for the > udptransportationtoremote syslog server > > Thanks a lot. > Currently i am stucked at the design that output module can not modify > the msg to be seen by other output modules. I understand why it's > designed that way but just wondering if there is a quick hack to > persist the modified msg in output module so other modules can see. You may want to have a look at omruleset. > Or do you guys have something to handle this scenario better? Thanks > David for better describing the problem. I will shortly reply to David's mail, I think the information will be useful for you as well. I just need some more time to prepare that message. Rainer > > Sent from my HTC > > -----Original Message----- > From: david at lang.hm > Sent: May 31, 2010 6:30 AM > To: rsyslog-users > Subject: Re: [rsyslog] Where is the output module for > the udp transportationtoremote syslog server > > > On Mon, 31 May 2010, david at lang.hm wrote: > > > On Mon, 31 May 2010, Rainer Gerhards wrote: > > > >>> I agree that doing it in the output would be far better in many > ways, > >>> but > >>> since there isn't a way to do a plugin there (at least not as far > as I > >>> know, it would be good to get confirmation or a better idea) > >> > >> David, can you tell me what you have on your mind for this > functionality? I > >> have thought a bit about it, and I probably have one approach > myself. But I > >> would prefer to hear your idea before I push you into a direction. > > > > > > two options > > > > 1. something that would work similar to the existing format > > string, but would call a C subroutine that could read the existing > > properties and would create the output string in a buffer > > > > 2. something that could also modify the exisitng properties (more > > powerful, but also more dangerous and could involve locking to > prevent > > other things from trying to read properties at the same time) > > > > we haven't gone too far down the road of researching the output > > performance (since the input and queue locking has dominated so far), > but > > it is clear that the output currently takes significantly more CPU > time > > than input, it may be that being able to use C to define the output > format > > instead of interpreting the format string may be a noticable > improvement. > > Is there a relativly easy way to test this? (say, hard-code a format > or > > two and test writes to file and network with the hard-coded format vs > a > > format string that produces the same output?) > > for the traditional output formats the difference may not be that much, > but if there is extensive parsing involved (as the initial poster is > doing, or what I would expect is common for specific log types into a > database) the difference can be much more significant since it can > replace > multiple regex statements with a much faster single pass that looks for > word breaks and inserts standard filler in those spots. > > With the new syslog format where the data is 'supposed to be' in a > series of name=value tuples, something like this would be a pretty > efficiant way of extracting particular portions of the data to be > output > (although the properties could be extended to do this sort of thing by > providing something similar to a perl hash) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon May 31 15:39:06 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 31 May 2010 15:39:06 +0200 Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server References: <004501cafd09$9027db05$100013ac@intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103E26@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103E37@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Monday, May 31, 2010 12:30 PM > To: rsyslog-users > Subject: Re: [rsyslog] Where is the output module for the udp > transportationtoremote syslog server > > On Mon, 31 May 2010, david at lang.hm wrote: > > > On Mon, 31 May 2010, Rainer Gerhards wrote: > > > >>> I agree that doing it in the output would be far better in many > ways, > >>> but > >>> since there isn't a way to do a plugin there (at least not as far > as I > >>> know, it would be good to get confirmation or a better idea) > >> > >> David, can you tell me what you have on your mind for this > functionality? I > >> have thought a bit about it, and I probably have one approach > myself. But I > >> would prefer to hear your idea before I push you into a direction. > > > > > > two options > > > > 1. something that would work similar to the existing format > > string, but would call a C subroutine that could read the existing > > properties and would create the output string in a buffer > > > > 2. something that could also modify the exisitng properties (more > > powerful, but also more dangerous and could involve locking to > prevent > > other things from trying to read properties at the same time) > > > > we haven't gone too far down the road of researching the output > > performance (since the input and queue locking has dominated so far), > but > > it is clear that the output currently takes significantly more CPU > time > > than input, it may be that being able to use C to define the output > format > > instead of interpreting the format string may be a noticable > improvement. > > Is there a relativly easy way to test this? (say, hard-code a format > or > > two and test writes to file and network with the hard-coded format vs > a > > format string that produces the same output?) > > for the traditional output formats the difference may not be that much, > but if there is extensive parsing involved (as the initial poster is > doing, or what I would expect is common for specific log types into a > database) the difference can be much more significant since it can > replace > multiple regex statements with a much faster single pass that looks for > word breaks and inserts standard filler in those spots. > > With the new syslog format where the data is 'supposed to be' in a > series of name=value tuples, something like this would be a pretty > efficiant way of extracting particular portions of the data to be > output > (although the properties could be extended to do this sort of thing by > providing something similar to a perl hash) You are looking in the same direction I am, and I think this is good news ;) The current engine supports functions coded in C, but not yet as real plugins nor in an easy to see way. It is done via a crude function interface library module, and only within the script engine. My original plan (over a year, or even two, ago) was to generalize these library plugins, so that it is easy to add new code and load them as plugins. Actually, making them available as plugins should not be too much work given the already existing infrastructure. There already exist a handful of "function modules", the control structure is just statically created during compile time, much as some of the output plugins are statically linked. Then the original plan was to enable templates to call scripts and enable scripts to define templates (kind of). Unfortunately, I got distracted by more important things before I could complete all of this. HOWEVER, at this time performance was not a major concern. With what has evolved in the mean time, I do not like the original approach that much any longer. At least the script engine must become much faster before I can take a real look at that capability. Right now, scripts generate a interim code that then is interpreted by a (kind of) virtual machine. A script invocation inside a template would mean that a VM must be instantiated, the script interpreted and the resulting string be used as template contents. Clearly, this is not for high-performance use. Still, however, it may be useful to have that capability for those cases, where performance is not the #1 consideration. But given that everything would need to be implemented, it does make limited sense to look into something known to be too slow in the long run. BTW, this is one reason that I have not yet continued to work on the script engine, knowing that some larger redesign is due to fit it into the now much tighter runtime constraints. On the performance of the output system: I think the system in general is quite fast and efficient, with only ONE important exception: that is, if multiple replacements need to happen. Still, the algorithm is quite efficient, but it is generic and needs to run though a number of steps. Of course, it is definitely faster to permit a C plugin to look at the message and then format, in an "atomic" way the resulting custom string. Thus, you need to write multiple C codes instead of using a generic engine, but can do so in a much higher performance way. I would assume, however, that this approach cannot beat the simple templates we usually use (maybe by less than 5% and, of course, there may be cases where this matters). As you know, my current focus is speed, together with some functional enhancements. I was looking at queue operations improvements, but the potential output speed improvements may be more interesting than the queue mode improvements (and apply to more use cases). So it may make sense to look into these, first. My challenge here is to find something that is a) generic enough to be useful in various (usual) cases b) specific enough to be rather fast and it should also be able to implement within a few weeks at most, because I can probably not spend much more time on a single feature/refactoring. One solution may be to create "template modules". I could envision a template module to be something that generates the template string *as a whole* from the input message. That is, we would have $template current-style,"%msg%\n" but also (**) $modload tplcustom $template custom,tplcustom where tplcustom generates the template string. While this sounds promising, we have some issues. One immediately pops up my mind: we will probably be able to use the same template for file writing or forwarding, but for file writing we need a LF at the end, while for forwarding we do not need it. So the most natural way would be to have the ability to embed a "custom template" into a regular template, like suggested by this syntax: $template both,"%=tplcustom%\n" however, this brings us down to the slippery slope of the original design. As a next thing to be requested, I could ask for using not the msg object (with its fixed unmodified properties), but rather of a transformation of the message object. So we would end up with something like this: $template cmplx,"%=tplcustom(syslogtag & msg)%" Which would require a much more complex logic working behind the scenes. Of course, depending on the format used, the engine could select different processing algorithms. Doing this on the fly seems possible, but requires more work than I can commit in one sequence. Also, it would be useful to have the ability to persist already-generated properties with the message while it is continued to be processed in the rule engine. So far, we do not have this ability, and the reason is processing time (plus, as usual, implementation effort): for that, we would need to maintain a list (or hash, ...) of name/value pairs, store them to disk for disk queues and shuffle them through the rule engine as processing is carried out. As I said, quite doable, but another big addition. So I am somewhat stuck with things that sound interesting, but are a bit interdependent. Doing them all together is too big to be useful, and it will probably fail because I can probably not keep focus on all of the for the next, say, 9 to 12 month that it would require to complete everything. So I am again down to picking what is most useful. Out of this discussion, it looks like the idea I marked with (**), the plain C template generator could be a useful route to take. I am saying this under the assumption that it would be relatively easy to implement and cause at least some speedup in standard cases (contrary to what I expect, I have to admit...). But that approach is highly specialized, requiring a C module for each custom format. So does it really serve the rsyslog community well - or just some very isolated use cases? Thinking more about it, it would probably be useful if it is both a) relatively easy to implement and b) causes some speedup in standard cases But b) cannot be proven without actually implementing the interface. So, in practice, the questions boils down to what we *expect* about the usefulness of this utility. Having said that, I'd appreciate feedback, both on the concrete question of the usefulness of this feature as well as any and all comments on the situation at large. I am trying to put my development resources, which thankfully have been somewhat increased nowadays :) to the area where they provide greatest benefit. Rainer From tbergfeld at hq.adiscon.com Mon May 3 13:15:02 2010 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Mon, 3 May 2010 13:15:02 +0200 Subject: [rsyslog] rsyslog 4.7.2 (v4-devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103D29@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 4.7.2, a member of the v4-devel branch. 4.7.2 is a bugfixing-release. Its primary bugfix solves problems with atomic instruction emulation. Users who have compiled rsyslog for older CPUs (like Intel 386) or CPUs for which gcc lacks atomic instruction support (like Sparc) are strongly encouraged to upgrade to the new versions. For all others, an update is optional. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article457.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-202.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From tbergfeld at hq.adiscon.com Mon May 3 15:02:25 2010 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Mon, 3 May 2010 15:02:25 +0200 Subject: [rsyslog] rsyslog 5.5.4 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103D2F@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.5.4, a member of the devel branch. Rsyslog has become the de-facto standard on modern Linux operating systems. It's high-performance log processing, database integration, modularity and support for multiple logging protocols make it the sysadmin's logging daemon of choice. The project was started in 2004 and has since then evolved rapidly. Starting with today, rsyslog is not only available on Linux and BSD, but also on Sun Solaris. Both Intel and Sparc machines are fully supported under Solaris. Depending on operator need, rsyslog can replace stock Solaris syslogd or be used in conjunction with it. The later case provides enhanced rsyslog functionality without the need to change the system infrastructure. Solaris is now a tier-one target platform. That means that all testing for major releases will be carried out on Solaris as well as on other platforms. The Solaris port was done very careful taking into account Sun's somewhat specific syslogd handling via door files and preserving the full power of rsyslog. So it not only compiles and runs on Solaris but rsyslog is a good citizen in the Solaris environment. As of usual rsyslog project policies, the project does not make installation packages other than the source distribution available. However, we work closely together with the Solaris community be able to provide them. We expect additional announcements soon. The versions with initial solid Solaris support are 4.7.2 and 5.5.4. Rsyslog's Solaris port was made possible by a generous contribution of hardware and some development funding by a sponsor which preferred to remain anonymous. We from the rsyslog project would like to express our sincere appreciation. Contributions of any kind are always very welcome. ChangeLog: http://www.rsyslog.com/Article459.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-203.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From pgollucci at freebsd.org Wed May 5 01:17:57 2010 From: pgollucci at freebsd.org (Philip M. Gollucci) Date: Tue, 4 May 2010 19:17:57 -0400 (EDT) Subject: [rsyslog] Create rsyslog55 and friends to track -devel Message-ID: <201005042317.o44NHvwa036646@frieza.p6m7g8.net> >Submitter-Id: current-users >Originator: Philip M. Gollucci >Organization: RideCharge Inc. >Confidential: no >Synopsis: Create rsyslog55 and friends to track -devel >Severity: non-critical >Priority: low >Category: ports >Class: change-request >Release: FreeBSD 9.0-CURRENT amd64 >Environment: System: FreeBSD frieza.p6m7g8.net 9.0-CURRENT FreeBSD 9.0-CURRENT #0: Mon Apr 26 16:20:00 EDT 2010 root at frieza.p6m7g8.net:/usr/obj/usr/src/sys/FRIEZA amd64 >Description: Repo copies needed: sysutils/rsyslog5 -> sysutils/rsyslog55 sysutils/rsyslog5-dbi -> sysutils/rsyslog55-dbi sysutils/rsyslog5-gnutls -> sysutils/rsyslog55-gnutls sysutils/rsyslog5-gssapi -> sysutils/rsyslog55-gssapi sysutils/rsyslog5-mysql -> sysutils/rsyslog55-mysql sysutils/rsyslog5-pgsql -> sysutils/rsyslog55-pgsql sysutils/rsyslog5-relp -> sysutils/rsyslog55-relp sysutils/rsyslog5-rfc3195 -> sysutils/rsyslog55-rfc3195 sysutils/rsyslog5-snmp -> sysutils/rsyslog55-snmp Sponsored by: RideCharge Inc. / Taxi Magic >How-To-Repeat: >Fix: --- rsyslog55.diff begins here --- diff --git a/sysutils/Makefile b/sysutils/Makefile index e3b17be..8d02e35 100644 --- a/sysutils/Makefile +++ b/sysutils/Makefile @@ -711,6 +711,15 @@ SUBDIR += rsyslog5-relp SUBDIR += rsyslog5-rfc3195 SUBDIR += rsyslog5-snmp + SUBDIR += rsyslog55 + SUBDIR += rsyslog55-dbi + SUBDIR += rsyslog55-gnutls + SUBDIR += rsyslog55-gssapi + SUBDIR += rsyslog55-mysql + SUBDIR += rsyslog55-pgsql + SUBDIR += rsyslog55-relp + SUBDIR += rsyslog55-rfc3195 + SUBDIR += rsyslog55-snmp SUBDIR += rtty SUBDIR += ruby-log4r SUBDIR += ruby-quota diff --git a/sysutils/rsyslog55-dbi/Makefile b/sysutils/rsyslog55-dbi/Makefile index 6e4ee57..c9d145a 100644 --- a/sysutils/rsyslog55-dbi/Makefile +++ b/sysutils/rsyslog55-dbi/Makefile @@ -5,7 +5,7 @@ # $FreeBSD: ports/sysutils/rsyslog5-dbi/Makefile,v 1.4 2009/12/18 20:44:28 miwi Exp $ COMMENT= LibDBI output module for rsyslog -MASTERDIR= ${.CURDIR}/../rsyslog5 +MASTERDIR= ${.CURDIR}/../rsyslog55 MNAME= libdbi LIB_DEPENDS= dbi.0:${PORTSDIR}/databases/libdbi diff --git a/sysutils/rsyslog55-gnutls/Makefile b/sysutils/rsyslog55-gnutls/Makefile index 3deb756..2f22ddb 100644 --- a/sysutils/rsyslog55-gnutls/Makefile +++ b/sysutils/rsyslog55-gnutls/Makefile @@ -6,7 +6,7 @@ # COMMENT= GNUTLS module for rsyslog -MASTERDIR= ${.CURDIR}/../rsyslog5 +MASTERDIR= ${.CURDIR}/../rsyslog55 MNAME= gnutls LIB_DEPENDS+= gnutls.40:${PORTSDIR}/security/gnutls diff --git a/sysutils/rsyslog55-gssapi/Makefile b/sysutils/rsyslog55-gssapi/Makefile index 6452ebc..9fa9ccb 100644 --- a/sysutils/rsyslog55-gssapi/Makefile +++ b/sysutils/rsyslog55-gssapi/Makefile @@ -6,7 +6,7 @@ # COMMENT= GSS API input/output module for rsyslog -MASTERDIR= ${.CURDIR}/../rsyslog5 +MASTERDIR= ${.CURDIR}/../rsyslog55 MNAME= gssapi diff --git a/sysutils/rsyslog55-mysql/Makefile b/sysutils/rsyslog55-mysql/Makefile index 3b9eca1..682588c 100644 --- a/sysutils/rsyslog55-mysql/Makefile +++ b/sysutils/rsyslog55-mysql/Makefile @@ -6,7 +6,7 @@ # COMMENT= MySQL output module for rsyslog -MASTERDIR= ${.CURDIR}/../rsyslog5 +MASTERDIR= ${.CURDIR}/../rsyslog55 MNAME= mysql USE_MYSQL= yes diff --git a/sysutils/rsyslog55-pgsql/Makefile b/sysutils/rsyslog55-pgsql/Makefile index 470009e..c9a08e6 100644 --- a/sysutils/rsyslog55-pgsql/Makefile +++ b/sysutils/rsyslog55-pgsql/Makefile @@ -6,7 +6,7 @@ # COMMENT= PostgreSQL output module for rsyslog -MASTERDIR= ${.CURDIR}/../rsyslog5 +MASTERDIR= ${.CURDIR}/../rsyslog55 MNAME= pgsql USE_PGSQL= yes diff --git a/sysutils/rsyslog55-relp/Makefile b/sysutils/rsyslog55-relp/Makefile index 8a915c1..86c7894 100644 --- a/sysutils/rsyslog55-relp/Makefile +++ b/sysutils/rsyslog55-relp/Makefile @@ -6,7 +6,7 @@ # COMMENT= RELP input/output module for rsyslog -MASTERDIR= ${.CURDIR}/../rsyslog5 +MASTERDIR= ${.CURDIR}/../rsyslog55 MNAME= relp BUILD_DEPENDS+= pkg-config:${PORTSDIR}/devel/pkg-config diff --git a/sysutils/rsyslog55-rfc3195/Makefile b/sysutils/rsyslog55-rfc3195/Makefile index 733daa4..db6fe57 100644 --- a/sysutils/rsyslog55-rfc3195/Makefile +++ b/sysutils/rsyslog55-rfc3195/Makefile @@ -6,7 +6,7 @@ # COMMENT= RFC3195 input support for rsyslog -MASTERDIR= ${.CURDIR}/../rsyslog5 +MASTERDIR= ${.CURDIR}/../rsyslog55 MNAME= rfc3195 BUILD_DEPENDS+= pkg-config:${PORTSDIR}/devel/pkg-config diff --git a/sysutils/rsyslog55-snmp/Makefile b/sysutils/rsyslog55-snmp/Makefile index 8116cf1..466d5b8 100644 --- a/sysutils/rsyslog55-snmp/Makefile +++ b/sysutils/rsyslog55-snmp/Makefile @@ -6,7 +6,7 @@ # COMMENT= SNMP trap sender for rsyslog -MASTERDIR= ${.CURDIR}/../rsyslog5 +MASTERDIR= ${.CURDIR}/../rsyslog55 MNAME= snmp LIB_DEPENDS= netsnmp.16:${PORTSDIR}/net-mgmt/net-snmp diff --git a/sysutils/rsyslog55/Makefile b/sysutils/rsyslog55/Makefile index 679ff97..080dd91 100644 --- a/sysutils/rsyslog55/Makefile +++ b/sysutils/rsyslog55/Makefile @@ -6,7 +6,7 @@ # PORTNAME= rsyslog -PORTVERSION= 5.4.0 +PORTVERSION= 5.5.4 CATEGORIES= sysutils MASTER_SITES= http://download.rsyslog.com/rsyslog/ .ifdef MNAME @@ -17,12 +17,20 @@ MAINTAINER= cristianorolim at hotmail.com COMMENT?= Syslogd supporting SQL, TCP and TLS .ifdef MNAME -RUN_DEPENDS= rsyslog>=5:${PORTSDIR}/sysutils/rsyslog5 +RUN_DEPENDS= rsyslog>=5.5.0:${PORTSDIR}/sysutils/rsyslog55 PLIST= ${.CURDIR}/pkg-plist .endif -CONFLICTS= rsyslog-[!5].[0-9]* +.ifdef WITH_MYSQL_MICROSECONDS +EXTRA_PATCHES+= ${FILESDIR}/extra-patch-mysql-microseconds +.endif + +.ifdef WITH_SANE_HOSTNAME +EXTRA_PATCHES+= ${FILESDIR}/extra-patch-sane-hostname +.endif + +CONFLICTS= rsyslog-[!5].[0-9]* rsyslog-5.4.* CPPFLAGS+= -I${LOCALBASE}/include LDFLAGS+= -L${LOCALBASE}/lib GNU_CONFIGURE= yes @@ -52,6 +60,7 @@ post-patch: ${WRKSRC}/tools/syslogd.c @${GREP} -rl '/etc/rsyslog.conf' ${WRKSRC}|${XARGS} ${REINPLACE_CMD} -e\ 's|/etc/rsyslog.conf|${PREFIX}/etc/rsyslog.conf|' + @${REINPLACE_CMD} -e 's,/lib/rsyslog,${PREFIX}/lib/rsyslog,' ${WRKSRC}/tools/syslogd.c @${FIND} ${WRKSRC} -name '*.bak' -delete post-install: @@ -75,8 +84,8 @@ IGNORE= with gssapi module is only supported on FreeBSD 7.x or later CONFIGURE_ARGS+= --disable-rsyslogd --disable-klog -DESCR?= ${.CURDIR}/../rsyslog5/pkg-descr -MD5_FILE?= ${.CURDIR}/../rsyslog5/distinfo +DESCR?= ${.CURDIR}/../rsyslog55/pkg-descr +MD5_FILE?= ${.CURDIR}/../rsyslog55/distinfo .endif .if ${OSVERSION} < 700042 diff --git a/sysutils/rsyslog55/distinfo b/sysutils/rsyslog55/distinfo index a452749..bbda583 100644 --- a/sysutils/rsyslog55/distinfo +++ b/sysutils/rsyslog55/distinfo @@ -1,3 +1,3 @@ -MD5 (rsyslog-5.4.0.tar.gz) = 291882229d50496f42bd63174076dd37 -SHA256 (rsyslog-5.4.0.tar.gz) = d9cd21d2fcd45fcae65eb0a51927c40315cca02afdc62478abd950febfcf7228 -SIZE (rsyslog-5.4.0.tar.gz) = 2124201 +MD5 (rsyslog-5.5.4.tar.gz) = 824df2504955df1619e5ec2915d783aa +SHA256 (rsyslog-5.5.4.tar.gz) = 31853a551ea7ca960c59c9e33406b1748bdf311059c9d8a4ce98816d51b17cac +SIZE (rsyslog-5.5.4.tar.gz) = 2200136 diff --git a/sysutils/rsyslog55/files/extra-patch-mysql-microseconds b/sysutils/rsyslog55/files/extra-patch-mysql-microseconds new file mode 100644 index 0000000..ec248b0 --- /dev/null +++ b/sysutils/rsyslog55/files/extra-patch-mysql-microseconds @@ -0,0 +1,56 @@ +--- ./runtime/datetime.c.orig 2010-05-04 18:57:25.588028725 -0400 ++++ ./runtime/datetime.c 2010-05-04 18:59:12.390680038 -0400 +@@ -644,18 +644,30 @@ + pBuf[1] = (ts->year / 100) % 10 + '0'; + pBuf[2] = (ts->year / 10) % 10 + '0'; + pBuf[3] = ts->year % 10 + '0'; +- pBuf[4] = (ts->month / 10) % 10 + '0'; +- pBuf[5] = ts->month % 10 + '0'; +- pBuf[6] = (ts->day / 10) % 10 + '0'; +- pBuf[7] = ts->day % 10 + '0'; +- pBuf[8] = (ts->hour / 10) % 10 + '0'; +- pBuf[9] = ts->hour % 10 + '0'; +- pBuf[10] = (ts->minute / 10) % 10 + '0'; +- pBuf[11] = ts->minute % 10 + '0'; +- pBuf[12] = (ts->second / 10) % 10 + '0'; +- pBuf[13] = ts->second % 10 + '0'; +- pBuf[14] = '\0'; +- return 15; ++ pBuf[4] = '-'; ++ pBuf[5] = (ts->month / 10) % 10 + '0'; ++ pBuf[6] = ts->month % 10 + '0'; ++ pBuf[7] = '-'; ++ pBuf[8] = (ts->day / 10) % 10 + '0'; ++ pBuf[9] = ts->day % 10 + '0'; ++ pBuf[10] = ' '; ++ pBuf[11] = (ts->hour / 10) % 10 + '0'; ++ pBuf[12] = ts->hour % 10 + '0'; ++ pBuf[13] = ':'; ++ pBuf[14] = (ts->minute / 10) % 10 + '0'; ++ pBuf[15] = ts->minute % 10 + '0'; ++ pBuf[16] = ':'; ++ pBuf[17] = (ts->second / 10) % 10 + '0'; ++ pBuf[18] = ts->second % 10 + '0'; ++ pBuf[19] = '.'; ++ pBuf[20] = (ts->secfrac / 100000) % 10 + '0'; ++ pBuf[21] = (ts->secfrac / 10000) % 10 + '0'; ++ pBuf[22] = (ts->secfrac / 1000) % 10 + '0'; ++ pBuf[23] = (ts->secfrac / 100) % 10 + '0'; ++ pBuf[24] = (ts->secfrac / 10) % 10 + '0'; ++ pBuf[25] = ts->secfrac % 10 + '0'; ++ pBuf[26] = '\0'; ++ return 26; + + } + +--- ./runtime/msg.c.orig 2010-05-04 19:00:20.241528788 -0400 ++++ ./runtime/msg.c 2010-05-04 19:00:06.136349680 -0400 +@@ -1293,7 +1293,7 @@ + case tplFmtMySQLDate: + MsgLock(pM); + if(pM->pszTIMESTAMP_MySQL == NULL) { +- if((pM->pszTIMESTAMP_MySQL = MALLOC(15)) == NULL) { ++ if((pM->pszTIMESTAMP_MySQL = MALLOC(26)) == NULL) { + MsgUnlock(pM); + return ""; + } diff --git a/sysutils/rsyslog55/files/extra-patch-sane-hostname b/sysutils/rsyslog55/files/extra-patch-sane-hostname new file mode 100644 index 0000000..bc72514 --- /dev/null +++ b/sysutils/rsyslog55/files/extra-patch-sane-hostname @@ -0,0 +1,40 @@ +--- ./tools/syslogd.c.orig 2010-05-04 19:02:05.548362478 -0400 ++++ ./tools/syslogd.c 2010-05-04 19:02:27.452450741 -0400 +@@ -2611,37 +2611,6 @@ + net.getLocalHostname(&LocalFQDNName); + CHKmalloc(LocalHostName = (uchar*) strdup((char*)LocalFQDNName)); + glbl.SetLocalFQDNName(LocalFQDNName); /* set the FQDN before we modify it */ +- if((p = (uchar*)strchr((char*)LocalHostName, '.'))) { +- *p++ = '\0'; +- LocalDomain = p; +- } else { +- LocalDomain = (uchar*)""; +- +- /* It's not clearly defined whether gethostname() +- * should return the simple hostname or the fqdn. A +- * good piece of software should be aware of both and +- * we want to distribute good software. Joey +- * +- * Good software also always checks its return values... +- * If syslogd starts up before DNS is up & /etc/hosts +- * doesn't have LocalHostName listed, gethostbyname will +- * return NULL. +- */ +- /* TODO: gethostbyname() is not thread-safe, but replacing it is +- * not urgent as we do not run on multiple threads here. rgerhards, 2007-09-25 +- */ +- hent = gethostbyname((char*)LocalHostName); +- if(hent) { +- free(LocalHostName); +- CHKmalloc(LocalHostName = (uchar*)strdup(hent->h_name)); +- +- if((p = (uchar*)strchr((char*)LocalHostName, '.'))) +- { +- *p++ = '\0'; +- LocalDomain = p; +- } +- } +- } + + /* Convert to lower case to recognize the correct domain laterly */ + for(p = LocalDomain ; *p ; p++) diff --git a/sysutils/rsyslog55/pkg-plist b/sysutils/rsyslog55/pkg-plist index 4120023..3d534e8 100644 --- a/sysutils/rsyslog55/pkg-plist +++ b/sysutils/rsyslog55/pkg-plist @@ -52,6 +52,7 @@ sbin/rsyslogd %%PORTDOCS%%%%DOCSDIR%%/imgssapi.html %%PORTDOCS%%%%DOCSDIR%%/imklog.html %%PORTDOCS%%%%DOCSDIR%%/imrelp.html +%%PORTDOCS%%%%DOCSDIR%%/imsolaris.html %%PORTDOCS%%%%DOCSDIR%%/imtcp.html %%PORTDOCS%%%%DOCSDIR%%/imuxsock.html %%PORTDOCS%%%%DOCSDIR%%/index.html @@ -93,6 +94,7 @@ sbin/rsyslogd %%PORTDOCS%%%%DOCSDIR%%/rsconf1_dropmsgswithmaliciousdnsptrrecords.html %%PORTDOCS%%%%DOCSDIR%%/rsconf1_droptrailinglfonreception.html %%PORTDOCS%%%%DOCSDIR%%/rsconf1_dynafilecachesize.html +%%PORTDOCS%%%%DOCSDIR%%/rsconf1_escape8bitcharsonreceive.html %%PORTDOCS%%%%DOCSDIR%%/rsconf1_escapecontrolcharactersonreceive.html %%PORTDOCS%%%%DOCSDIR%%/rsconf1_failonchownfailure.html %%PORTDOCS%%%%DOCSDIR%%/rsconf1_filecreatemode.html --- rsyslog55.diff ends here --- From rgerhards at hq.adiscon.com Tue May 18 18:11:37 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 18 May 2010 18:11:37 +0200 Subject: [rsyslog] Feedback requested: fast queue mode Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103DE5@GRFEXC.intern.adiscon.com> Hi all, I am currently thinking about various optimizations for even more throughput inside the v5 engine. During this effort, I am also doing some review of existing literature on lock-free and wait-free algorithms. One of my former ideas re-appeared, and I wanted to get feedback if that would be a useful addition. Even if it is seen as valuable, I will probably not implement it immediately, but I would try to incorporate it into the new design, which I plan to implement later this year. Looking at the queue modes, we have some overhead inside queues because queues need to do a lot of things. Things like race-limiting, blocking on full queue, even going to disk if the queue fills up too much. Also, support for thread pools and all that is needed. Note that all of this overhead is also necessary if the queue is used to run an action asynchronously. I think I could implement a considerably faster queue, if I limit its features. Most importantly, that means: - no support for going to the disk (that should not be an issue, I think) - no support for race-limiting - capability to accept message loss if queue is full - message loss victim not selected based on priority - message loss on shutdown acceptable In short, the queue would provide simple in-memory queueing services including synchronization between multiple producers and consumers, but no advanced services at all. At this price, it could probably reduced the queue overhead very considerably. I think such a queue could be useful for the (common) case when data needs to be shuffled to files, and some loss is acceptable (e.g. UDP is the input). I'd say we could probably improve the performance for this use case by a factor of two. Would this be useful? Thoughts are appreciated. Rainer From rory at ooma.com Tue May 18 18:14:00 2010 From: rory at ooma.com (Rory Toma) Date: Tue, 18 May 2010 09:14:00 -0700 Subject: [rsyslog] Feedback requested: fast queue mode In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103DE5@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103DE5@GRFEXC.intern.adiscon.com> Message-ID: <4BF2BCC8.2060004@ooma.com> I could see this being very useful in a relay situation. As a side question, has anyone done injection from rsyslog into MongoDB? On 5/18/10 9:11 AM, Rainer Gerhards wrote: > Hi all, > > I am currently thinking about various optimizations for even more throughput > inside the v5 engine. During this effort, I am also doing some review of > existing literature on lock-free and wait-free algorithms. One of my former > ideas re-appeared, and I wanted to get feedback if that would be a useful > addition. Even if it is seen as valuable, I will probably not implement it > immediately, but I would try to incorporate it into the new design, which I > plan to implement later this year. > > Looking at the queue modes, we have some overhead inside queues because > queues need to do a lot of things. Things like race-limiting, blocking on > full queue, even going to disk if the queue fills up too much. Also, support > for thread pools and all that is needed. Note that all of this overhead is > also necessary if the queue is used to run an action asynchronously. > > I think I could implement a considerably faster queue, if I limit its > features. Most importantly, that means: > > - no support for going to the disk (that should not be an issue, I think) > - no support for race-limiting > - capability to accept message loss if queue is full > - message loss victim not selected based on priority > - message loss on shutdown acceptable > > In short, the queue would provide simple in-memory queueing services > including synchronization between multiple producers and consumers, but no > advanced services at all. At this price, it could probably reduced the queue > overhead very considerably. > > I think such a queue could be useful for the (common) case when data needs to > be shuffled to files, and some loss is acceptable (e.g. UDP is the input). > I'd say we could probably improve the performance for this use case by a > factor of two. > > Would this be useful? Thoughts are appreciated. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rory at ooma.com Wed May 19 22:48:20 2010 From: rory at ooma.com (Rory Toma) Date: Wed, 19 May 2010 13:48:20 -0700 Subject: [rsyslog] MongoDB Message-ID: <4BF44E94.8010204@ooma.com> Has anyone done a MongoDB insertion engine for rsyslog yet? From tbergfeld at hq.adiscon.com Thu May 20 15:52:45 2010 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Thu, 20 May 2010 15:52:45 +0200 Subject: [rsyslog] rsyslog 5.5.5 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103E08@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.5.5, a member of the devel branch. This is a bug-fixing release which contains a single fix that solves a potential hang condition on system shutdown when infinite action retries are configured for an asynchronous action using a queue in disk-assisted mode and the action was suspended. This is probably not a very common case, but a configuration recommended by our doc samples. If you do not use such a configuration, there is no need to update at this time. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article461.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-204.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From jean.mousinho at ist.utl.pt Thu May 20 19:28:52 2010 From: jean.mousinho at ist.utl.pt (Jean F. Mousinho) Date: Thu, 20 May 2010 18:28:52 +0100 Subject: [rsyslog] rsyslog + tls + debian Message-ID: <1274376532.14504.2.camel@muse.ist.utl.pt> Hello, Was anyone successful to get rsyslog working with TLS module in Debian? rsyslogd 4.4.2, compiled with: FEATURE_REGEXP: Yes FEATURE_LARGEFILE: Yes FEATURE_NETZIP (message compression): Yes GSSAPI Kerberos 5 support: Yes FEATURE_DEBUG (debug build, slow code): No Atomic operations supported: Yes Runtime Instrumentation (slow code): No Output is: rsyslogd: could not load module '/usr/lib/rsyslog/lmnsd_gtls.so', rsyslog error -2078 [try http://www.rsyslog.com/e/2068 ] Module file exists: -rw-r--r-- 1 root root 27196 2010-05-17 16:12 /usr/lib/rsyslog/lmnsd_gtls.so I've used the packages provided in backports. Thanks for your time. Jean Mousinho From james at linux-source.org Fri May 21 07:04:59 2010 From: james at linux-source.org (James Corteciano) Date: Fri, 21 May 2010 13:04:59 +0800 Subject: [rsyslog] Centralized rsyslog server and httpd vhost Message-ID: Hi All, My goal is to centralized all system/apps logs from different web farm servers. I have the following setups: node0 - Centralized rsyslog server web_farm1 - web farm 1 server running RHEL, rsyslog, and httpd with different vhost. web_farm2 - web farm 2 server running RHEL, rsyslog, and httpd with different vhost. 1) How can I configure rsyslog from node0 to capture all vhost logs from web_farm servers and all logs will be directly placed like: /var/log/syslog/web_farm1/-error.log /var/log/syslog/web_farm1/-access.log 2) How to configure the httpd service in web_farm servers to push all logs to node0 syslog server? I saw from internet like the following. httpd.conf: ErrorLog "|/bin/logger -p local5.err" CustomLog "|/bin/logger -p local6.info" rsyslog.conf: *.* @node0 Thank you. Regards, James From david at lang.hm Fri May 21 07:31:52 2010 From: david at lang.hm (david at lang.hm) Date: Thu, 20 May 2010 22:31:52 -0700 (PDT) Subject: [rsyslog] Feedback requested: fast queue mode In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103DE5@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103DE5@GRFEXC.intern.adiscon.com> Message-ID: On Tue, 18 May 2010, Rainer Gerhards wrote: > Hi all, > > I am currently thinking about various optimizations for even more throughput > inside the v5 engine. During this effort, I am also doing some review of > existing literature on lock-free and wait-free algorithms. One of my former > ideas re-appeared, and I wanted to get feedback if that would be a useful > addition. Even if it is seen as valuable, I will probably not implement it > immediately, but I would try to incorporate it into the new design, which I > plan to implement later this year. > > Looking at the queue modes, we have some overhead inside queues because > queues need to do a lot of things. Things like race-limiting, blocking on > full queue, even going to disk if the queue fills up too much. Also, support > for thread pools and all that is needed. Note that all of this overhead is > also necessary if the queue is used to run an action asynchronously. > > I think I could implement a considerably faster queue, if I limit its > features. Most importantly, that means: > > - no support for going to the disk (that should not be an issue, I think) > - no support for race-limiting > - capability to accept message loss if queue is full > - message loss victim not selected based on priority > - message loss on shutdown acceptable > > In short, the queue would provide simple in-memory queueing services > including synchronization between multiple producers and consumers, but no > advanced services at all. At this price, it could probably reduced the queue > overhead very considerably. > > I think such a queue could be useful for the (common) case when data needs to > be shuffled to files, and some loss is acceptable (e.g. UDP is the input). > I'd say we could probably improve the performance for this use case by a > factor of two. > > Would this be useful? Thoughts are appreciated. yes, this sounds useful. David Lang From david at lang.hm Fri May 21 07:33:25 2010 From: david at lang.hm (david at lang.hm) Date: Thu, 20 May 2010 22:33:25 -0700 (PDT) Subject: [rsyslog] MongoDB In-Reply-To: <4BF44E94.8010204@ooma.com> References: <4BF44E94.8010204@ooma.com> Message-ID: On Wed, 19 May 2010, Rory Toma wrote: > Has anyone done a MongoDB insertion engine for rsyslog yet? not that I am aware of. In fact, as far as I know, only the postgres and possibly oracle modules take full advantage of the vector mode that allows them to efficiantly batch the message inserts. lots of room for work in this area. David Lang From david at lang.hm Fri May 21 07:37:55 2010 From: david at lang.hm (david at lang.hm) Date: Thu, 20 May 2010 22:37:55 -0700 (PDT) Subject: [rsyslog] Centralized rsyslog server and httpd vhost In-Reply-To: References: Message-ID: On Fri, 21 May 2010, James Corteciano wrote: > Hi All, > > My goal is to centralized all system/apps logs from different web farm > servers. I have the following setups: > > node0 - Centralized rsyslog server > web_farm1 - web farm 1 server running RHEL, rsyslog, and httpd with > different vhost. > web_farm2 - web farm 2 server running RHEL, rsyslog, and httpd with > different vhost. > > 1) How can I configure rsyslog from node0 to capture all vhost logs from > web_farm servers and all logs will be directly placed like: > > /var/log/syslog/web_farm1/-error.log > /var/log/syslog/web_farm1/-access.log > > 2) How to configure the httpd service in web_farm servers to push all logs > to node0 syslog server? I saw from internet like the following. > > httpd.conf: > ErrorLog "|/bin/logger -p local5.err" > CustomLog "|/bin/logger -p local6.info" > > rsyslog.conf: > *.* @node0 when logging from apache you can have log commands inside each vhost, or if you don't the logs will be handled by the main server. what I do is to have the access logs handled by the main server and create a custom format that includes the vhost as part of the format (I also reorder things so that data I really care about is near the beginning of the log and data that can be long is later in the message, so if it becomes extremely long and overflows the max log length I don't loose data I consider critical) then I run it through a perl script that reformats the message to put the vhost name in the server field and sends it out via UDP to my syslog server. I don't have access to that file at the moment (it's at work), I'll try to get a copy tomorrow and post it. David Lang From james at linux-source.org Fri May 21 08:57:50 2010 From: james at linux-source.org (James Corteciano) Date: Fri, 21 May 2010 14:57:50 +0800 Subject: [rsyslog] Centralized rsyslog server and httpd vhost In-Reply-To: References: Message-ID: Hi David, Thanks for your reply and I'm looking forward about it. Regards, James On Fri, May 21, 2010 at 1:37 PM, wrote: > On Fri, 21 May 2010, James Corteciano wrote: > > > Hi All, > > > > My goal is to centralized all system/apps logs from different web farm > > servers. I have the following setups: > > > > node0 - Centralized rsyslog server > > web_farm1 - web farm 1 server running RHEL, rsyslog, and httpd with > > different vhost. > > web_farm2 - web farm 2 server running RHEL, rsyslog, and httpd with > > different vhost. > > > > 1) How can I configure rsyslog from node0 to capture all vhost logs from > > web_farm servers and all logs will be directly placed like: > > > > /var/log/syslog/web_farm1/-error.log > > /var/log/syslog/web_farm1/-access.log > > > > 2) How to configure the httpd service in web_farm servers to push all > logs > > to node0 syslog server? I saw from internet like the following. > > > > httpd.conf: > > ErrorLog "|/bin/logger -p local5.err" > > CustomLog "|/bin/logger -p local6.info" > > > > rsyslog.conf: > > *.* @node0 > > when logging from apache you can have log commands inside each vhost, or > if you don't the logs will be handled by the main server. > > what I do is to have the access logs handled by the main server and create > a custom format that includes the vhost as part of the format (I also > reorder things so that data I really care about is near the beginning of > the log and data that can be long is later in the message, so if it > becomes extremely long and overflows the max log length I don't loose data > I consider critical) > > then I run it through a perl script that reformats the message to put the > vhost name in the server field and sends it out via UDP to my syslog > server. > > I don't have access to that file at the moment (it's at work), I'll try to > get a copy tomorrow and post it. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From james at linux-source.org Fri May 21 09:49:04 2010 From: james at linux-source.org (James Corteciano) Date: Fri, 21 May 2010 15:49:04 +0800 Subject: [rsyslog] Centralized rsyslog server and httpd vhost In-Reply-To: References: Message-ID: Currently, I have this kind of setup. [node0] rsyslog.conf: $template MsgFormat,"%msg%\n" $template ApacheRemoteCustom,"/var/log/httpd/web_farm1/%msg:F,32:2%.log" if $syslogfacility-text == 'local6' and $programname == 'rhcs-node1' then -?ApacheRemoteCustom;MsgFormat [web_farm1] httpd.conf: LogLevel warn ErrorLog "|/bin/logger -p local5.err" LogFormat "%v %h %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" vcombined CustomLog "|/bin/logger -p local6.info -t rhcs-node1" vcombined" It works for getting vhost access logs. However, it doesn't work for error logs because apache ErrorLog is not possible to customize the error log by adding or removing information . Regards, James On Fri, May 21, 2010 at 1:37 PM, wrote: > On Fri, 21 May 2010, James Corteciano wrote: > > > Hi All, > > > > My goal is to centralized all system/apps logs from different web farm > > servers. I have the following setups: > > > > node0 - Centralized rsyslog server > > web_farm1 - web farm 1 server running RHEL, rsyslog, and httpd with > > different vhost. > > web_farm2 - web farm 2 server running RHEL, rsyslog, and httpd with > > different vhost. > > > > 1) How can I configure rsyslog from node0 to capture all vhost logs from > > web_farm servers and all logs will be directly placed like: > > > > /var/log/syslog/web_farm1/-error.log > > /var/log/syslog/web_farm1/-access.log > > > > 2) How to configure the httpd service in web_farm servers to push all > logs > > to node0 syslog server? I saw from internet like the following. > > > > httpd.conf: > > ErrorLog "|/bin/logger -p local5.err" > > CustomLog "|/bin/logger -p local6.info" > > > > rsyslog.conf: > > *.* @node0 > > when logging from apache you can have log commands inside each vhost, or > if you don't the logs will be handled by the main server. > > what I do is to have the access logs handled by the main server and create > a custom format that includes the vhost as part of the format (I also > reorder things so that data I really care about is near the beginning of > the log and data that can be long is later in the message, so if it > becomes extremely long and overflows the max log length I don't loose data > I consider critical) > > then I run it through a perl script that reformats the message to put the > vhost name in the server field and sends it out via UDP to my syslog > server. > > I don't have access to that file at the moment (it's at work), I'll try to > get a copy tomorrow and post it. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From iain at shihad.org Fri May 21 18:37:23 2010 From: iain at shihad.org (Iain M Conochie) Date: Fri, 21 May 2010 17:37:23 +0100 Subject: [rsyslog] Splitting rsyslog messages by hostname into MySQL database Message-ID: <4BF6B6C3.1080108@shihad.org> Afternoon all, I have rsyslog sending all my messages into a mysql database and this is working well. Now I want to start to split the remote messages via hostname into separate tables in the database. I have created a new table FaiEvents with the same schema as SystemEvents and also I have created a config file with actions and template like so: $template Fai-Event,"insert into FaiEvents \ (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)\ values ('%msg%',z,\ %syslogfacility%,\ '%HOSTNAME%',\ %syslogpriority%,\ '%timereported:::date-mysql%',\ '%timegenerated:::date-mysql%',\ %iut%,\ '%syslogtag%')",SQL if $hostname == 'faiserver'\ then :ommysql:localhost,Syslog,rsyslog,********;Fai-Event However rsyslog gives me the error: rsyslogd: the last error occured in /etc/rsyslog.d/05-faiservers.conf, line 13 which is the action line. What variable should I be using to test for the hostname? Can I use a regex here? Any help appreciated! Regards Iain Conochie From james at linux-source.org Sat May 22 11:00:38 2010 From: james at linux-source.org (James Corteciano) Date: Sat, 22 May 2010 17:00:38 +0800 Subject: [rsyslog] Centralized rsyslog server and httpd vhost In-Reply-To: References: Message-ID: Hi David, How about your Apache ErrorLogs? How do you get it that the same thing in Access log? Thank you. Regards, James On Fri, May 21, 2010 at 1:37 PM, wrote: > On Fri, 21 May 2010, James Corteciano wrote: > > > Hi All, > > > > My goal is to centralized all system/apps logs from different web farm > > servers. I have the following setups: > > > > node0 - Centralized rsyslog server > > web_farm1 - web farm 1 server running RHEL, rsyslog, and httpd with > > different vhost. > > web_farm2 - web farm 2 server running RHEL, rsyslog, and httpd with > > different vhost. > > > > 1) How can I configure rsyslog from node0 to capture all vhost logs from > > web_farm servers and all logs will be directly placed like: > > > > /var/log/syslog/web_farm1/-error.log > > /var/log/syslog/web_farm1/-access.log > > > > 2) How to configure the httpd service in web_farm servers to push all > logs > > to node0 syslog server? I saw from internet like the following. > > > > httpd.conf: > > ErrorLog "|/bin/logger -p local5.err" > > CustomLog "|/bin/logger -p local6.info" > > > > rsyslog.conf: > > *.* @node0 > > when logging from apache you can have log commands inside each vhost, or > if you don't the logs will be handled by the main server. > > what I do is to have the access logs handled by the main server and create > a custom format that includes the vhost as part of the format (I also > reorder things so that data I really care about is near the beginning of > the log and data that can be long is later in the message, so if it > becomes extremely long and overflows the max log length I don't loose data > I consider critical) > > then I run it through a perl script that reformats the message to put the > vhost name in the server field and sends it out via UDP to my syslog > server. > > I don't have access to that file at the moment (it's at work), I'll try to > get a copy tomorrow and post it. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Sun May 23 00:21:43 2010 From: david at lang.hm (david at lang.hm) Date: Sat, 22 May 2010 15:21:43 -0700 (PDT) Subject: [rsyslog] Centralized rsyslog server and httpd vhost In-Reply-To: References: Message-ID: On Fri, 21 May 2010, James Corteciano wrote: > Currently, I have this kind of setup. > > [node0] > > rsyslog.conf: > $template MsgFormat,"%msg%\n" > $template ApacheRemoteCustom,"/var/log/httpd/web_farm1/%msg:F,32:2%.log" > if $syslogfacility-text == 'local6' and $programname == 'rhcs-node1' then > -?ApacheRemoteCustom;MsgFormat > > > [web_farm1] > > httpd.conf: > LogLevel warn > ErrorLog "|/bin/logger -p local5.err" > LogFormat "%v %h %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" > vcombined > CustomLog "|/bin/logger -p local6.info -t rhcs-node1" vcombined" > > > It works for getting vhost access logs. However, it doesn't work for error > logs because apache ErrorLog is not possible to customize the error log by > adding or removing > information > . personally I would not try to get error logs in directly specificly because of this. (there's also the problem that a single cgi script error could spew out thousands of lines of garbage into the error log) what's needed is an error log that doesn't include stderr from the cgis being run, but would log a single line along the lines of 'output on stderr from X, see Y for details' where Y is another logfile on the server. unfortunantly this will take modifications to apache to do. David Lang > Regards, > James > > > On Fri, May 21, 2010 at 1:37 PM, wrote: > >> On Fri, 21 May 2010, James Corteciano wrote: >> >>> Hi All, >>> >>> My goal is to centralized all system/apps logs from different web farm >>> servers. I have the following setups: >>> >>> node0 - Centralized rsyslog server >>> web_farm1 - web farm 1 server running RHEL, rsyslog, and httpd with >>> different vhost. >>> web_farm2 - web farm 2 server running RHEL, rsyslog, and httpd with >>> different vhost. >>> >>> 1) How can I configure rsyslog from node0 to capture all vhost logs from >>> web_farm servers and all logs will be directly placed like: >>> >>> /var/log/syslog/web_farm1/-error.log >>> /var/log/syslog/web_farm1/-access.log >>> >>> 2) How to configure the httpd service in web_farm servers to push all >> logs >>> to node0 syslog server? I saw from internet like the following. >>> >>> httpd.conf: >>> ErrorLog "|/bin/logger -p local5.err" >>> CustomLog "|/bin/logger -p local6.info" >>> >>> rsyslog.conf: >>> *.* @node0 >> >> when logging from apache you can have log commands inside each vhost, or >> if you don't the logs will be handled by the main server. >> >> what I do is to have the access logs handled by the main server and create >> a custom format that includes the vhost as part of the format (I also >> reorder things so that data I really care about is near the beginning of >> the log and data that can be long is later in the message, so if it >> becomes extremely long and overflows the max log length I don't loose data >> I consider critical) >> >> then I run it through a perl script that reformats the message to put the >> vhost name in the server field and sends it out via UDP to my syslog >> server. >> >> I don't have access to that file at the moment (it's at work), I'll try to >> get a copy tomorrow and post it. >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From jli at jlisbz.com Wed May 26 05:54:57 2010 From: jli at jlisbz.com (John Li) Date: Tue, 25 May 2010 23:54:57 -0400 Subject: [rsyslog] Where is the output module for the udp transportation to remote syslog server Message-ID: Hi, Is the output via udp to remote syslog server implemented as a output module? I could not find it in the plugins folder. Thanks. -- John Jun Li jli at jlisbz.com My Blog: http://www.jlisbz.com My LinkedIn Profile: http://www.linkedin.com/in/johnjunli From fn42551 at fmi.uni-sofia.bg Wed May 26 16:17:07 2010 From: fn42551 at fmi.uni-sofia.bg (Angel Tsankov) Date: Wed, 26 May 2010 17:17:07 +0300 Subject: [rsyslog] Check if rsyslog is running Message-ID: What is the recommended way for an application to check if rsyslogd is running? Angel Tsankov From jli at jlisbz.com Wed May 26 16:41:25 2010 From: jli at jlisbz.com (John Li) Date: Wed, 26 May 2010 10:41:25 -0400 Subject: [rsyslog] Check if rsyslog is running In-Reply-To: References: Message-ID: monit from http://mmonit.com/monit/ shoud be able to handle this easily. And it can do both monitoring and restarting if the process is crashed. -- John Jun Li jli at jlisbz.com My Blog: http://www.jlisbz.com My LinkedIn Profile: http://www.linkedin.com/in/johnjunli My Twitter: http://www.twitter.com/jlisbz My facebook: http://www.facebook.com/profile.php?id=593495282 On Wed, May 26, 2010 at 10:17 AM, Angel Tsankov wrote: > What is the recommended way for an application to check if rsyslogd is > running? > > Angel Tsankov > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed May 26 17:36:56 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 26 May 2010 18:36:56 +0300 Subject: [rsyslog] Where is the output module for the udp transportation toremote syslog server Message-ID: <003401cafce9$44d808a1$100013ac@intern.adiscon.com> This is a built-in module, it does not need to be loaded (it is actually linked into the main executable). HTH Rainer ----- Urspr?ngliche Nachricht ----- Von: John Li Gesendet: Mittwoch, 26. Mai 2010 07:03 An: rsyslog at lists.adiscon.com Betreff: [rsyslog] Where is the output module for the udp transportation toremote syslog server Hi, Is the output via udp to remote syslog server implemented as a output module? I could not find it in the plugins folder. Thanks. -- John Jun Li jli at jlisbz.com My Blog: http://www.jlisbz.com My LinkedIn Profile: http://www.linkedin.com/in/johnjunli _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From jli at jlisbz.com Wed May 26 17:46:03 2010 From: jli at jlisbz.com (John Li) Date: Wed, 26 May 2010 11:46:03 -0400 Subject: [rsyslog] Where is the output module for the udp transportation toremote syslog server In-Reply-To: <003401cafce9$44d808a1$100013ac@intern.adiscon.com> References: <003401cafce9$44d808a1$100013ac@intern.adiscon.com> Message-ID: Thanks. My goal is to change the content of msg and I am planning to use output module to do that. Is this the right approach and do you mind point me to some sample code in output module to do that? -- John Jun Li jli at jlisbz.com My Blog: http://www.jlisbz.com My LinkedIn Profile: http://www.linkedin.com/in/johnjunli On Wed, May 26, 2010 at 11:36 AM, Rainer Gerhards wrote: > This is a built-in module, it does not need to be loaded (it is actually > linked into the main executable). > > HTH > Rainer > > ----- Urspr?ngliche Nachricht ----- > Von: John Li > Gesendet: Mittwoch, 26. Mai 2010 07:03 > An: rsyslog at lists.adiscon.com > Betreff: [rsyslog] Where is the output module for the udp transportation > toremote syslog server > > Hi, > > Is the output via udp to remote syslog server implemented as a output > module? I could not find it in the plugins folder. > > Thanks. > > -- > John Jun Li > jli at jlisbz.com > > My Blog: http://www.jlisbz.com > My LinkedIn Profile: http://www.linkedin.com/in/johnjunli > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed May 26 21:28:06 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 26 May 2010 22:28:06 +0300 Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server Message-ID: <004501cafd09$9027db05$100013ac@intern.adiscon.com> You need to look into templates. It is quite easy to rewrite message content with templates. There are samples in the doc and in the wiki. Rainer ----- Urspr?ngliche Nachricht ----- Von: John Li Gesendet: Mittwoch, 26. Mai 2010 18:53 An: rsyslog-users Betreff: Re: [rsyslog] Where is the output module for the udp transportationtoremote syslog server Thanks. My goal is to change the content of msg and I am planning to use output module to do that. Is this the right approach and do you mind point me to some sample code in output module to do that? -- John Jun Li jli at jlisbz.com My Blog: http://www.jlisbz.com My LinkedIn Profile: http://www.linkedin.com/in/johnjunli On Wed, May 26, 2010 at 11:36 AM, Rainer Gerhards wrote: > This is a built-in module, it does not need to be loaded (it is actually > linked into the main executable). > > HTH > Rainer > > ----- Urspr?ngliche Nachricht ----- > Von: John Li > Gesendet: Mittwoch, 26. Mai 2010 07:03 > An: rsyslog at lists.adiscon.com > Betreff: [rsyslog] Where is the output module for the udp transportation > toremote syslog server > > Hi, > > Is the output via udp to remote syslog server implemented as a output > module? I could not find it in the plugins folder. > > Thanks. > > -- > John Jun Li > jli at jlisbz.com > > My Blog: http://www.jlisbz.com > My LinkedIn Profile: http://www.linkedin.com/in/johnjunli > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From jli at jlisbz.com Wed May 26 22:15:16 2010 From: jli at jlisbz.com (John Li) Date: Wed, 26 May 2010 16:15:16 -0400 Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server In-Reply-To: <004501cafd09$9027db05$100013ac@intern.adiscon.com> References: <004501cafd09$9027db05$100013ac@intern.adiscon.com> Message-ID: This is the RE case I posted in forum last week and output module is the best to achieve it. Here is the case I described: " For example, a typical firewall log: 192.168.20.5 23456 192.168.10.10 80 Accept Web 192.168.20.6 5678 192.168.10.10 22 Deny SSH If I want to have the xml form of them, it could be : 192.168.20.5192.168.10.102345680AcceptWeb 192.168.20.6192.168.10.10567822DenySSH If I understand correctly for template, I had to do RE for 6 times for each log entry and that could cause performance issue in large environment for sure. " So I need to rewrite the msg in the output module, please let me know where to find some sample code or doc. And here is one more question: "One thing I want to make sure is the output plugin which I will make should be still able to use other output method such as syslog/snmp etc with the converted message, right? ." I was able to create my own output module based on the stdout module but could not figure out how to rewrite the msg back to rsyslog so the rewritten msg can be used by other output module. Is this doable? Thanks a lot. -- John On Wed, May 26, 2010 at 3:28 PM, Rainer Gerhards wrote: > You need to look into templates. It is quite easy to rewrite message > content with templates. There are samples in the doc and in the wiki. > > Rainer > > ----- Urspr?ngliche Nachricht ----- > Von: John Li > Gesendet: Mittwoch, 26. Mai 2010 18:53 > An: rsyslog-users > Betreff: Re: [rsyslog] Where is the output module for the udp > transportationtoremote syslog server > > Thanks. > > My goal is to change the content of msg and I am planning to use output > module to do that. Is this the right approach and do you mind point me to > some sample code in output module to do that? > > > -- > John Jun Li > jli at jlisbz.com > > My Blog: http://www.jlisbz.com > My LinkedIn Profile: http://www.linkedin.com/in/johnjunli > > > > On Wed, May 26, 2010 at 11:36 AM, Rainer Gerhards > wrote: > > > This is a built-in module, it does not need to be loaded (it is actually > > linked into the main executable). > > > > HTH > > Rainer > > > > ----- Urspr?ngliche Nachricht ----- > > Von: John Li > > Gesendet: Mittwoch, 26. Mai 2010 07:03 > > An: rsyslog at lists.adiscon.com > > Betreff: [rsyslog] Where is the output module for the udp transportation > > toremote syslog server > > > > Hi, > > > > Is the output via udp to remote syslog server implemented as a output > > module? I could not find it in the plugins folder. > > > > Thanks. > > > > -- > > John Jun Li > > jli at jlisbz.com > > > > My Blog: http://www.jlisbz.com > > My LinkedIn Profile: http://www.linkedin.com/in/johnjunli > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Thu May 27 07:55:25 2010 From: david at lang.hm (david at lang.hm) Date: Wed, 26 May 2010 22:55:25 -0700 (PDT) Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server In-Reply-To: References: <004501cafd09$9027db05$100013ac@intern.adiscon.com> Message-ID: you may want to consider doing this on the input side instead of the output side. see http://www.rsyslog.com/doc-messageparser.html yes, in many ways it's operating backwards, but it may be significantly less work to implement and maintain it this way. David Lang On Wed, 26 May 2010, John Li wrote: > This is the RE case I posted in forum last week and output module is the > best to achieve it. > > Here is the case I described: > " > For example, a typical firewall log: > 192.168.20.5 23456 192.168.10.10 80 Accept Web > 192.168.20.6 5678 192.168.10.10 22 Deny SSH > > If I want to have the xml form of them, it could be : > 192.168.20.5192.168.10.102345680AcceptWeb > 192.168.20.6192.168.10.10567822DenySSH > > If I understand correctly for template, I had to do RE for 6 times for each > log entry and that could cause performance issue in large environment for > sure. " > > So I need to rewrite the msg in the output module, please let me know where > to find some sample code or doc. And here is one more question: > > "One thing I want to make sure is the output plugin which I will make should > be still able to use other output method such as syslog/snmp etc with the > converted message, right? ." > > I was able to create my own output module based on the stdout module but > could not figure out how to rewrite the msg back to rsyslog so the rewritten > msg can be used by other output module. Is this doable? > > Thanks a lot. > > From jli at jlisbz.com Thu May 27 15:29:28 2010 From: jli at jlisbz.com (John Li) Date: Thu, 27 May 2010 09:29:28 -0400 Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server In-Reply-To: References: <004501cafd09$9027db05$100013ac@intern.adiscon.com> Message-ID: Not totally agree. First, your will lose the flexibility in the input side if you put the rewritten code in the input module. Second, parser looks like to target the syslog format validity instead of message rewritten. But if it's not possible to rewrite the msg in the output module, I will have to do that in the parser. Can someone please confirm? Thanks. -- John Jun Li jli at jlisbz.com My Blog: http://www.jlisbz.com My LinkedIn Profile: http://www.linkedin.com/in/johnjunli On Thu, May 27, 2010 at 1:55 AM, wrote: > you may want to consider doing this on the input side instead of the > output side. > > see http://www.rsyslog.com/doc-messageparser.html > > yes, in many ways it's operating backwards, but it may be significantly > less work to implement and maintain it this way. > > David Lang > > On Wed, 26 May 2010, John Li wrote: > > > This is the RE case I posted in forum last week and output module is the > > best to achieve it. > > > > Here is the case I described: > > " > > For example, a typical firewall log: > > 192.168.20.5 23456 192.168.10.10 80 Accept Web > > 192.168.20.6 5678 192.168.10.10 22 Deny SSH > > > > If I want to have the xml form of them, it could be : > > > 192.168.20.5192.168.10.102345680AcceptWeb > > > 192.168.20.6192.168.10.10567822DenySSH > > > > If I understand correctly for template, I had to do RE for 6 times for > each > > log entry and that could cause performance issue in large environment for > > sure. " > > > > So I need to rewrite the msg in the output module, please let me know > where > > to find some sample code or doc. And here is one more question: > > > > "One thing I want to make sure is the output plugin which I will make > should > > be still able to use other output method such as syslog/snmp etc with the > > converted message, right? ." > > > > I was able to create my own output module based on the stdout module but > > could not figure out how to rewrite the msg back to rsyslog so the > rewritten > > msg can be used by other output module. Is this doable? > > > > Thanks a lot. > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Fri May 28 07:25:58 2010 From: david at lang.hm (david at lang.hm) Date: Thu, 27 May 2010 22:25:58 -0700 (PDT) Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server In-Reply-To: References: <004501cafd09$9027db05$100013ac@intern.adiscon.com> Message-ID: On Thu, 27 May 2010, John Li wrote: > Not totally agree. First, your will lose the flexibility in the input side > if you put the rewritten code in the input module. Second, parser looks like > to target the syslog format validity instead of message rewritten. it doesn't just validate the message, it takes the message off the wire, and breaksit into the separate properties that rsyslog handles internally (and are available for the output templates). It already has the option to modify the string as it does this (look at control character re-writing) > But if it's not possible to rewrite the msg in the output module, I will > have to do that in the parser. Can someone please confirm? I agree that doing it in the output would be far better in many ways, but since there isn't a way to do a plugin there (at least not as far as I know, it would be good to get confirmation or a better idea) David Lang > Thanks. > > From rgerhards at hq.adiscon.com Mon May 31 11:35:54 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 31 May 2010 11:35:54 +0200 Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server References: <004501cafd09$9027db05$100013ac@intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103E26@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, May 28, 2010 7:26 AM > To: rsyslog-users > Subject: Re: [rsyslog] Where is the output module for the udp > transportationtoremote syslog server > > On Thu, 27 May 2010, John Li wrote: > > > Not totally agree. First, your will lose the flexibility in the input > side > > if you put the rewritten code in the input module. Second, parser > looks like > > to target the syslog format validity instead of message rewritten. > > it doesn't just validate the message, it takes the message off the > wire, > and breaksit into the separate properties that rsyslog handles > internally > (and are available for the output templates). It already has the option > to > modify the string as it does this (look at control character re- > writing) > > > But if it's not possible to rewrite the msg in the output module, I > will > > have to do that in the parser. Can someone please confirm? John, You can do whatever you like in an output module, including rewriting any part of the message. Of course, you can NOT modify the message strings that *other* output modules see. > I agree that doing it in the output would be far better in many ways, > but > since there isn't a way to do a plugin there (at least not as far as I > know, it would be good to get confirmation or a better idea) David, can you tell me what you have on your mind for this functionality? I have thought a bit about it, and I probably have one approach myself. But I would prefer to hear your idea before I push you into a direction. Thanks, Rainer From david at lang.hm Mon May 31 12:16:32 2010 From: david at lang.hm (david at lang.hm) Date: Mon, 31 May 2010 03:16:32 -0700 (PDT) Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103E26@GRFEXC.intern.adiscon.com> References: <004501cafd09$9027db05$100013ac@intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103E26@GRFEXC.intern.adiscon.com> Message-ID: On Mon, 31 May 2010, Rainer Gerhards wrote: >> I agree that doing it in the output would be far better in many ways, >> but >> since there isn't a way to do a plugin there (at least not as far as I >> know, it would be good to get confirmation or a better idea) > > David, can you tell me what you have on your mind for this functionality? I > have thought a bit about it, and I probably have one approach myself. But I > would prefer to hear your idea before I push you into a direction. two options 1. something that would work similar to the existing format string, but would call a C subroutine that could read the existing properties and would create the output string in a buffer 2. something that could also modify the exisitng properties (more powerful, but also more dangerous and could involve locking to prevent other things from trying to read properties at the same time) we haven't gone too far down the road of researching the output performance (since the input and queue locking has dominated so far), but it is clear that the output currently takes significantly more CPU time than input, it may be that being able to use C to define the output format instead of interpreting the format string may be a noticable improvement. Is there a relativly easy way to test this? (say, hard-code a format or two and test writes to file and network with the hard-coded format vs a format string that produces the same output?) David Lang From david at lang.hm Mon May 31 12:30:22 2010 From: david at lang.hm (david at lang.hm) Date: Mon, 31 May 2010 03:30:22 -0700 (PDT) Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server In-Reply-To: References: <004501cafd09$9027db05$100013ac@intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103E26@GRFEXC.intern.adiscon.com> Message-ID: On Mon, 31 May 2010, david at lang.hm wrote: > On Mon, 31 May 2010, Rainer Gerhards wrote: > >>> I agree that doing it in the output would be far better in many ways, >>> but >>> since there isn't a way to do a plugin there (at least not as far as I >>> know, it would be good to get confirmation or a better idea) >> >> David, can you tell me what you have on your mind for this functionality? I >> have thought a bit about it, and I probably have one approach myself. But I >> would prefer to hear your idea before I push you into a direction. > > > two options > > 1. something that would work similar to the existing format > string, but would call a C subroutine that could read the existing > properties and would create the output string in a buffer > > 2. something that could also modify the exisitng properties (more > powerful, but also more dangerous and could involve locking to prevent > other things from trying to read properties at the same time) > > we haven't gone too far down the road of researching the output > performance (since the input and queue locking has dominated so far), but > it is clear that the output currently takes significantly more CPU time > than input, it may be that being able to use C to define the output format > instead of interpreting the format string may be a noticable improvement. > Is there a relativly easy way to test this? (say, hard-code a format or > two and test writes to file and network with the hard-coded format vs a > format string that produces the same output?) for the traditional output formats the difference may not be that much, but if there is extensive parsing involved (as the initial poster is doing, or what I would expect is common for specific log types into a database) the difference can be much more significant since it can replace multiple regex statements with a much faster single pass that looks for word breaks and inserts standard filler in those spots. With the new syslog format where the data is 'supposed to be' in a series of name=value tuples, something like this would be a pretty efficiant way of extracting particular portions of the data to be output (although the properties could be extended to do this sort of thing by providing something similar to a perl hash) David Lang From jli at jlisbz.com Mon May 31 14:17:02 2010 From: jli at jlisbz.com (John Li) Date: Mon, 31 May 2010 05:17:02 -0700 Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server Message-ID: <2054128934449600685@unknownmsgid> Thanks a lot. Currently i am stucked at the design that output module can not modify the msg to be seen by other output modules. I understand why it's designed that way but just wondering if there is a quick hack to persist the modified msg in output module so other modules can see. Or do you guys have something to handle this scenario better? Thanks David for better describing the problem. Sent from my HTC -----Original Message----- From: david at lang.hm Sent: May 31, 2010 6:30 AM To: rsyslog-users Subject: Re: [rsyslog] Where is the output module for the udp transportationtoremote syslog server On Mon, 31 May 2010, david at lang.hm wrote: > On Mon, 31 May 2010, Rainer Gerhards wrote: > >>> I agree that doing it in the output would be far better in many ways, >>> but >>> since there isn't a way to do a plugin there (at least not as far as I >>> know, it would be good to get confirmation or a better idea) >> >> David, can you tell me what you have on your mind for this functionality? I >> have thought a bit about it, and I probably have one approach myself. But I >> would prefer to hear your idea before I push you into a direction. > > > two options > > 1. something that would work similar to the existing format > string, but would call a C subroutine that could read the existing > properties and would create the output string in a buffer > > 2. something that could also modify the exisitng properties (more > powerful, but also more dangerous and could involve locking to prevent > other things from trying to read properties at the same time) > > we haven't gone too far down the road of researching the output > performance (since the input and queue locking has dominated so far), but > it is clear that the output currently takes significantly more CPU time > than input, it may be that being able to use C to define the output format > instead of interpreting the format string may be a noticable improvement. > Is there a relativly easy way to test this? (say, hard-code a format or > two and test writes to file and network with the hard-coded format vs a > format string that produces the same output?) for the traditional output formats the difference may not be that much, but if there is extensive parsing involved (as the initial poster is doing, or what I would expect is common for specific log types into a database) the difference can be much more significant since it can replace multiple regex statements with a much faster single pass that looks for word breaks and inserts standard filler in those spots. With the new syslog format where the data is 'supposed to be' in a series of name=value tuples, something like this would be a pretty efficiant way of extracting particular portions of the data to be output (although the properties could be extended to do this sort of thing by providing something similar to a perl hash) David Lang _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From jli at jlisbz.com Mon May 31 14:17:24 2010 From: jli at jlisbz.com (John Li) Date: Mon, 31 May 2010 08:17:24 -0400 Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server Message-ID: <7684824564864990291@unknownmsgid> Thanks a lot. Currently i am stucked at the design that output module can not modify the msg to be seen by other output modules. I understand why it's designed that way but just wondering if there is a quick hack to persist the modified msg in output module so other modules can see. Or do you guys have something to handle this scenario better? Thanks David for better describing the problem. Sent from my HTC -----Original Message----- From: david at lang.hm Sent: May 31, 2010 6:30 AM To: rsyslog-users Subject: Re: [rsyslog] Where is the output module for the udp transportationtoremote syslog server On Mon, 31 May 2010, david at lang.hm wrote: > On Mon, 31 May 2010, Rainer Gerhards wrote: > >>> I agree that doing it in the output would be far better in many ways, >>> but >>> since there isn't a way to do a plugin there (at least not as far as I >>> know, it would be good to get confirmation or a better idea) >> >> David, can you tell me what you have on your mind for this functionality? I >> have thought a bit about it, and I probably have one approach myself. But I >> would prefer to hear your idea before I push you into a direction. > > > two options > > 1. something that would work similar to the existing format > string, but would call a C subroutine that could read the existing > properties and would create the output string in a buffer > > 2. something that could also modify the exisitng properties (more > powerful, but also more dangerous and could involve locking to prevent > other things from trying to read properties at the same time) > > we haven't gone too far down the road of researching the output > performance (since the input and queue locking has dominated so far), but > it is clear that the output currently takes significantly more CPU time > than input, it may be that being able to use C to define the output format > instead of interpreting the format string may be a noticable improvement. > Is there a relativly easy way to test this? (say, hard-code a format or > two and test writes to file and network with the hard-coded format vs a > format string that produces the same output?) for the traditional output formats the difference may not be that much, but if there is extensive parsing involved (as the initial poster is doing, or what I would expect is common for specific log types into a database) the difference can be much more significant since it can replace multiple regex statements with a much faster single pass that looks for word breaks and inserts standard filler in those spots. With the new syslog format where the data is 'supposed to be' in a series of name=value tuples, something like this would be a pretty efficiant way of extracting particular portions of the data to be output (although the properties could be extended to do this sort of thing by providing something similar to a perl hash) David Lang _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon May 31 14:24:27 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 31 May 2010 14:24:27 +0200 Subject: [rsyslog] Where is the output module for the udptransportationtoremote syslog server References: <2054128934449600685@unknownmsgid> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103E2E@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of John Li > Sent: Monday, May 31, 2010 2:17 PM > To: david at lang.hm; rsyslog-users > Subject: Re: [rsyslog] Where is the output module for the > udptransportationtoremote syslog server > > Thanks a lot. > Currently i am stucked at the design that output module can not modify > the msg to be seen by other output modules. I understand why it's > designed that way but just wondering if there is a quick hack to > persist the modified msg in output module so other modules can see. You may want to have a look at omruleset. > Or do you guys have something to handle this scenario better? Thanks > David for better describing the problem. I will shortly reply to David's mail, I think the information will be useful for you as well. I just need some more time to prepare that message. Rainer > > Sent from my HTC > > -----Original Message----- > From: david at lang.hm > Sent: May 31, 2010 6:30 AM > To: rsyslog-users > Subject: Re: [rsyslog] Where is the output module for > the udp transportationtoremote syslog server > > > On Mon, 31 May 2010, david at lang.hm wrote: > > > On Mon, 31 May 2010, Rainer Gerhards wrote: > > > >>> I agree that doing it in the output would be far better in many > ways, > >>> but > >>> since there isn't a way to do a plugin there (at least not as far > as I > >>> know, it would be good to get confirmation or a better idea) > >> > >> David, can you tell me what you have on your mind for this > functionality? I > >> have thought a bit about it, and I probably have one approach > myself. But I > >> would prefer to hear your idea before I push you into a direction. > > > > > > two options > > > > 1. something that would work similar to the existing format > > string, but would call a C subroutine that could read the existing > > properties and would create the output string in a buffer > > > > 2. something that could also modify the exisitng properties (more > > powerful, but also more dangerous and could involve locking to > prevent > > other things from trying to read properties at the same time) > > > > we haven't gone too far down the road of researching the output > > performance (since the input and queue locking has dominated so far), > but > > it is clear that the output currently takes significantly more CPU > time > > than input, it may be that being able to use C to define the output > format > > instead of interpreting the format string may be a noticable > improvement. > > Is there a relativly easy way to test this? (say, hard-code a format > or > > two and test writes to file and network with the hard-coded format vs > a > > format string that produces the same output?) > > for the traditional output formats the difference may not be that much, > but if there is extensive parsing involved (as the initial poster is > doing, or what I would expect is common for specific log types into a > database) the difference can be much more significant since it can > replace > multiple regex statements with a much faster single pass that looks for > word breaks and inserts standard filler in those spots. > > With the new syslog format where the data is 'supposed to be' in a > series of name=value tuples, something like this would be a pretty > efficiant way of extracting particular portions of the data to be > output > (although the properties could be extended to do this sort of thing by > providing something similar to a perl hash) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon May 31 15:39:06 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 31 May 2010 15:39:06 +0200 Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server References: <004501cafd09$9027db05$100013ac@intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103E26@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103E37@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Monday, May 31, 2010 12:30 PM > To: rsyslog-users > Subject: Re: [rsyslog] Where is the output module for the udp > transportationtoremote syslog server > > On Mon, 31 May 2010, david at lang.hm wrote: > > > On Mon, 31 May 2010, Rainer Gerhards wrote: > > > >>> I agree that doing it in the output would be far better in many > ways, > >>> but > >>> since there isn't a way to do a plugin there (at least not as far > as I > >>> know, it would be good to get confirmation or a better idea) > >> > >> David, can you tell me what you have on your mind for this > functionality? I > >> have thought a bit about it, and I probably have one approach > myself. But I > >> would prefer to hear your idea before I push you into a direction. > > > > > > two options > > > > 1. something that would work similar to the existing format > > string, but would call a C subroutine that could read the existing > > properties and would create the output string in a buffer > > > > 2. something that could also modify the exisitng properties (more > > powerful, but also more dangerous and could involve locking to > prevent > > other things from trying to read properties at the same time) > > > > we haven't gone too far down the road of researching the output > > performance (since the input and queue locking has dominated so far), > but > > it is clear that the output currently takes significantly more CPU > time > > than input, it may be that being able to use C to define the output > format > > instead of interpreting the format string may be a noticable > improvement. > > Is there a relativly easy way to test this? (say, hard-code a format > or > > two and test writes to file and network with the hard-coded format vs > a > > format string that produces the same output?) > > for the traditional output formats the difference may not be that much, > but if there is extensive parsing involved (as the initial poster is > doing, or what I would expect is common for specific log types into a > database) the difference can be much more significant since it can > replace > multiple regex statements with a much faster single pass that looks for > word breaks and inserts standard filler in those spots. > > With the new syslog format where the data is 'supposed to be' in a > series of name=value tuples, something like this would be a pretty > efficiant way of extracting particular portions of the data to be > output > (although the properties could be extended to do this sort of thing by > providing something similar to a perl hash) You are looking in the same direction I am, and I think this is good news ;) The current engine supports functions coded in C, but not yet as real plugins nor in an easy to see way. It is done via a crude function interface library module, and only within the script engine. My original plan (over a year, or even two, ago) was to generalize these library plugins, so that it is easy to add new code and load them as plugins. Actually, making them available as plugins should not be too much work given the already existing infrastructure. There already exist a handful of "function modules", the control structure is just statically created during compile time, much as some of the output plugins are statically linked. Then the original plan was to enable templates to call scripts and enable scripts to define templates (kind of). Unfortunately, I got distracted by more important things before I could complete all of this. HOWEVER, at this time performance was not a major concern. With what has evolved in the mean time, I do not like the original approach that much any longer. At least the script engine must become much faster before I can take a real look at that capability. Right now, scripts generate a interim code that then is interpreted by a (kind of) virtual machine. A script invocation inside a template would mean that a VM must be instantiated, the script interpreted and the resulting string be used as template contents. Clearly, this is not for high-performance use. Still, however, it may be useful to have that capability for those cases, where performance is not the #1 consideration. But given that everything would need to be implemented, it does make limited sense to look into something known to be too slow in the long run. BTW, this is one reason that I have not yet continued to work on the script engine, knowing that some larger redesign is due to fit it into the now much tighter runtime constraints. On the performance of the output system: I think the system in general is quite fast and efficient, with only ONE important exception: that is, if multiple replacements need to happen. Still, the algorithm is quite efficient, but it is generic and needs to run though a number of steps. Of course, it is definitely faster to permit a C plugin to look at the message and then format, in an "atomic" way the resulting custom string. Thus, you need to write multiple C codes instead of using a generic engine, but can do so in a much higher performance way. I would assume, however, that this approach cannot beat the simple templates we usually use (maybe by less than 5% and, of course, there may be cases where this matters). As you know, my current focus is speed, together with some functional enhancements. I was looking at queue operations improvements, but the potential output speed improvements may be more interesting than the queue mode improvements (and apply to more use cases). So it may make sense to look into these, first. My challenge here is to find something that is a) generic enough to be useful in various (usual) cases b) specific enough to be rather fast and it should also be able to implement within a few weeks at most, because I can probably not spend much more time on a single feature/refactoring. One solution may be to create "template modules". I could envision a template module to be something that generates the template string *as a whole* from the input message. That is, we would have $template current-style,"%msg%\n" but also (**) $modload tplcustom $template custom,tplcustom where tplcustom generates the template string. While this sounds promising, we have some issues. One immediately pops up my mind: we will probably be able to use the same template for file writing or forwarding, but for file writing we need a LF at the end, while for forwarding we do not need it. So the most natural way would be to have the ability to embed a "custom template" into a regular template, like suggested by this syntax: $template both,"%=tplcustom%\n" however, this brings us down to the slippery slope of the original design. As a next thing to be requested, I could ask for using not the msg object (with its fixed unmodified properties), but rather of a transformation of the message object. So we would end up with something like this: $template cmplx,"%=tplcustom(syslogtag & msg)%" Which would require a much more complex logic working behind the scenes. Of course, depending on the format used, the engine could select different processing algorithms. Doing this on the fly seems possible, but requires more work than I can commit in one sequence. Also, it would be useful to have the ability to persist already-generated properties with the message while it is continued to be processed in the rule engine. So far, we do not have this ability, and the reason is processing time (plus, as usual, implementation effort): for that, we would need to maintain a list (or hash, ...) of name/value pairs, store them to disk for disk queues and shuffle them through the rule engine as processing is carried out. As I said, quite doable, but another big addition. So I am somewhat stuck with things that sound interesting, but are a bit interdependent. Doing them all together is too big to be useful, and it will probably fail because I can probably not keep focus on all of the for the next, say, 9 to 12 month that it would require to complete everything. So I am again down to picking what is most useful. Out of this discussion, it looks like the idea I marked with (**), the plain C template generator could be a useful route to take. I am saying this under the assumption that it would be relatively easy to implement and cause at least some speedup in standard cases (contrary to what I expect, I have to admit...). But that approach is highly specialized, requiring a C module for each custom format. So does it really serve the rsyslog community well - or just some very isolated use cases? Thinking more about it, it would probably be useful if it is both a) relatively easy to implement and b) causes some speedup in standard cases But b) cannot be proven without actually implementing the interface. So, in practice, the questions boils down to what we *expect* about the usefulness of this utility. Having said that, I'd appreciate feedback, both on the concrete question of the usefulness of this feature as well as any and all comments on the situation at large. I am trying to put my development resources, which thankfully have been somewhat increased nowadays :) to the area where they provide greatest benefit. Rainer From tbergfeld at hq.adiscon.com Mon May 3 13:15:02 2010 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Mon, 3 May 2010 13:15:02 +0200 Subject: [rsyslog] rsyslog 4.7.2 (v4-devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103D29@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 4.7.2, a member of the v4-devel branch. 4.7.2 is a bugfixing-release. Its primary bugfix solves problems with atomic instruction emulation. Users who have compiled rsyslog for older CPUs (like Intel 386) or CPUs for which gcc lacks atomic instruction support (like Sparc) are strongly encouraged to upgrade to the new versions. For all others, an update is optional. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article457.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-202.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From tbergfeld at hq.adiscon.com Mon May 3 15:02:25 2010 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Mon, 3 May 2010 15:02:25 +0200 Subject: [rsyslog] rsyslog 5.5.4 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103D2F@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.5.4, a member of the devel branch. Rsyslog has become the de-facto standard on modern Linux operating systems. It's high-performance log processing, database integration, modularity and support for multiple logging protocols make it the sysadmin's logging daemon of choice. The project was started in 2004 and has since then evolved rapidly. Starting with today, rsyslog is not only available on Linux and BSD, but also on Sun Solaris. Both Intel and Sparc machines are fully supported under Solaris. Depending on operator need, rsyslog can replace stock Solaris syslogd or be used in conjunction with it. The later case provides enhanced rsyslog functionality without the need to change the system infrastructure. Solaris is now a tier-one target platform. That means that all testing for major releases will be carried out on Solaris as well as on other platforms. The Solaris port was done very careful taking into account Sun's somewhat specific syslogd handling via door files and preserving the full power of rsyslog. So it not only compiles and runs on Solaris but rsyslog is a good citizen in the Solaris environment. As of usual rsyslog project policies, the project does not make installation packages other than the source distribution available. However, we work closely together with the Solaris community be able to provide them. We expect additional announcements soon. The versions with initial solid Solaris support are 4.7.2 and 5.5.4. Rsyslog's Solaris port was made possible by a generous contribution of hardware and some development funding by a sponsor which preferred to remain anonymous. We from the rsyslog project would like to express our sincere appreciation. Contributions of any kind are always very welcome. ChangeLog: http://www.rsyslog.com/Article459.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-203.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From pgollucci at freebsd.org Wed May 5 01:17:57 2010 From: pgollucci at freebsd.org (Philip M. Gollucci) Date: Tue, 4 May 2010 19:17:57 -0400 (EDT) Subject: [rsyslog] Create rsyslog55 and friends to track -devel Message-ID: <201005042317.o44NHvwa036646@frieza.p6m7g8.net> >Submitter-Id: current-users >Originator: Philip M. Gollucci >Organization: RideCharge Inc. >Confidential: no >Synopsis: Create rsyslog55 and friends to track -devel >Severity: non-critical >Priority: low >Category: ports >Class: change-request >Release: FreeBSD 9.0-CURRENT amd64 >Environment: System: FreeBSD frieza.p6m7g8.net 9.0-CURRENT FreeBSD 9.0-CURRENT #0: Mon Apr 26 16:20:00 EDT 2010 root at frieza.p6m7g8.net:/usr/obj/usr/src/sys/FRIEZA amd64 >Description: Repo copies needed: sysutils/rsyslog5 -> sysutils/rsyslog55 sysutils/rsyslog5-dbi -> sysutils/rsyslog55-dbi sysutils/rsyslog5-gnutls -> sysutils/rsyslog55-gnutls sysutils/rsyslog5-gssapi -> sysutils/rsyslog55-gssapi sysutils/rsyslog5-mysql -> sysutils/rsyslog55-mysql sysutils/rsyslog5-pgsql -> sysutils/rsyslog55-pgsql sysutils/rsyslog5-relp -> sysutils/rsyslog55-relp sysutils/rsyslog5-rfc3195 -> sysutils/rsyslog55-rfc3195 sysutils/rsyslog5-snmp -> sysutils/rsyslog55-snmp Sponsored by: RideCharge Inc. / Taxi Magic >How-To-Repeat: >Fix: --- rsyslog55.diff begins here --- diff --git a/sysutils/Makefile b/sysutils/Makefile index e3b17be..8d02e35 100644 --- a/sysutils/Makefile +++ b/sysutils/Makefile @@ -711,6 +711,15 @@ SUBDIR += rsyslog5-relp SUBDIR += rsyslog5-rfc3195 SUBDIR += rsyslog5-snmp + SUBDIR += rsyslog55 + SUBDIR += rsyslog55-dbi + SUBDIR += rsyslog55-gnutls + SUBDIR += rsyslog55-gssapi + SUBDIR += rsyslog55-mysql + SUBDIR += rsyslog55-pgsql + SUBDIR += rsyslog55-relp + SUBDIR += rsyslog55-rfc3195 + SUBDIR += rsyslog55-snmp SUBDIR += rtty SUBDIR += ruby-log4r SUBDIR += ruby-quota diff --git a/sysutils/rsyslog55-dbi/Makefile b/sysutils/rsyslog55-dbi/Makefile index 6e4ee57..c9d145a 100644 --- a/sysutils/rsyslog55-dbi/Makefile +++ b/sysutils/rsyslog55-dbi/Makefile @@ -5,7 +5,7 @@ # $FreeBSD: ports/sysutils/rsyslog5-dbi/Makefile,v 1.4 2009/12/18 20:44:28 miwi Exp $ COMMENT= LibDBI output module for rsyslog -MASTERDIR= ${.CURDIR}/../rsyslog5 +MASTERDIR= ${.CURDIR}/../rsyslog55 MNAME= libdbi LIB_DEPENDS= dbi.0:${PORTSDIR}/databases/libdbi diff --git a/sysutils/rsyslog55-gnutls/Makefile b/sysutils/rsyslog55-gnutls/Makefile index 3deb756..2f22ddb 100644 --- a/sysutils/rsyslog55-gnutls/Makefile +++ b/sysutils/rsyslog55-gnutls/Makefile @@ -6,7 +6,7 @@ # COMMENT= GNUTLS module for rsyslog -MASTERDIR= ${.CURDIR}/../rsyslog5 +MASTERDIR= ${.CURDIR}/../rsyslog55 MNAME= gnutls LIB_DEPENDS+= gnutls.40:${PORTSDIR}/security/gnutls diff --git a/sysutils/rsyslog55-gssapi/Makefile b/sysutils/rsyslog55-gssapi/Makefile index 6452ebc..9fa9ccb 100644 --- a/sysutils/rsyslog55-gssapi/Makefile +++ b/sysutils/rsyslog55-gssapi/Makefile @@ -6,7 +6,7 @@ # COMMENT= GSS API input/output module for rsyslog -MASTERDIR= ${.CURDIR}/../rsyslog5 +MASTERDIR= ${.CURDIR}/../rsyslog55 MNAME= gssapi diff --git a/sysutils/rsyslog55-mysql/Makefile b/sysutils/rsyslog55-mysql/Makefile index 3b9eca1..682588c 100644 --- a/sysutils/rsyslog55-mysql/Makefile +++ b/sysutils/rsyslog55-mysql/Makefile @@ -6,7 +6,7 @@ # COMMENT= MySQL output module for rsyslog -MASTERDIR= ${.CURDIR}/../rsyslog5 +MASTERDIR= ${.CURDIR}/../rsyslog55 MNAME= mysql USE_MYSQL= yes diff --git a/sysutils/rsyslog55-pgsql/Makefile b/sysutils/rsyslog55-pgsql/Makefile index 470009e..c9a08e6 100644 --- a/sysutils/rsyslog55-pgsql/Makefile +++ b/sysutils/rsyslog55-pgsql/Makefile @@ -6,7 +6,7 @@ # COMMENT= PostgreSQL output module for rsyslog -MASTERDIR= ${.CURDIR}/../rsyslog5 +MASTERDIR= ${.CURDIR}/../rsyslog55 MNAME= pgsql USE_PGSQL= yes diff --git a/sysutils/rsyslog55-relp/Makefile b/sysutils/rsyslog55-relp/Makefile index 8a915c1..86c7894 100644 --- a/sysutils/rsyslog55-relp/Makefile +++ b/sysutils/rsyslog55-relp/Makefile @@ -6,7 +6,7 @@ # COMMENT= RELP input/output module for rsyslog -MASTERDIR= ${.CURDIR}/../rsyslog5 +MASTERDIR= ${.CURDIR}/../rsyslog55 MNAME= relp BUILD_DEPENDS+= pkg-config:${PORTSDIR}/devel/pkg-config diff --git a/sysutils/rsyslog55-rfc3195/Makefile b/sysutils/rsyslog55-rfc3195/Makefile index 733daa4..db6fe57 100644 --- a/sysutils/rsyslog55-rfc3195/Makefile +++ b/sysutils/rsyslog55-rfc3195/Makefile @@ -6,7 +6,7 @@ # COMMENT= RFC3195 input support for rsyslog -MASTERDIR= ${.CURDIR}/../rsyslog5 +MASTERDIR= ${.CURDIR}/../rsyslog55 MNAME= rfc3195 BUILD_DEPENDS+= pkg-config:${PORTSDIR}/devel/pkg-config diff --git a/sysutils/rsyslog55-snmp/Makefile b/sysutils/rsyslog55-snmp/Makefile index 8116cf1..466d5b8 100644 --- a/sysutils/rsyslog55-snmp/Makefile +++ b/sysutils/rsyslog55-snmp/Makefile @@ -6,7 +6,7 @@ # COMMENT= SNMP trap sender for rsyslog -MASTERDIR= ${.CURDIR}/../rsyslog5 +MASTERDIR= ${.CURDIR}/../rsyslog55 MNAME= snmp LIB_DEPENDS= netsnmp.16:${PORTSDIR}/net-mgmt/net-snmp diff --git a/sysutils/rsyslog55/Makefile b/sysutils/rsyslog55/Makefile index 679ff97..080dd91 100644 --- a/sysutils/rsyslog55/Makefile +++ b/sysutils/rsyslog55/Makefile @@ -6,7 +6,7 @@ # PORTNAME= rsyslog -PORTVERSION= 5.4.0 +PORTVERSION= 5.5.4 CATEGORIES= sysutils MASTER_SITES= http://download.rsyslog.com/rsyslog/ .ifdef MNAME @@ -17,12 +17,20 @@ MAINTAINER= cristianorolim at hotmail.com COMMENT?= Syslogd supporting SQL, TCP and TLS .ifdef MNAME -RUN_DEPENDS= rsyslog>=5:${PORTSDIR}/sysutils/rsyslog5 +RUN_DEPENDS= rsyslog>=5.5.0:${PORTSDIR}/sysutils/rsyslog55 PLIST= ${.CURDIR}/pkg-plist .endif -CONFLICTS= rsyslog-[!5].[0-9]* +.ifdef WITH_MYSQL_MICROSECONDS +EXTRA_PATCHES+= ${FILESDIR}/extra-patch-mysql-microseconds +.endif + +.ifdef WITH_SANE_HOSTNAME +EXTRA_PATCHES+= ${FILESDIR}/extra-patch-sane-hostname +.endif + +CONFLICTS= rsyslog-[!5].[0-9]* rsyslog-5.4.* CPPFLAGS+= -I${LOCALBASE}/include LDFLAGS+= -L${LOCALBASE}/lib GNU_CONFIGURE= yes @@ -52,6 +60,7 @@ post-patch: ${WRKSRC}/tools/syslogd.c @${GREP} -rl '/etc/rsyslog.conf' ${WRKSRC}|${XARGS} ${REINPLACE_CMD} -e\ 's|/etc/rsyslog.conf|${PREFIX}/etc/rsyslog.conf|' + @${REINPLACE_CMD} -e 's,/lib/rsyslog,${PREFIX}/lib/rsyslog,' ${WRKSRC}/tools/syslogd.c @${FIND} ${WRKSRC} -name '*.bak' -delete post-install: @@ -75,8 +84,8 @@ IGNORE= with gssapi module is only supported on FreeBSD 7.x or later CONFIGURE_ARGS+= --disable-rsyslogd --disable-klog -DESCR?= ${.CURDIR}/../rsyslog5/pkg-descr -MD5_FILE?= ${.CURDIR}/../rsyslog5/distinfo +DESCR?= ${.CURDIR}/../rsyslog55/pkg-descr +MD5_FILE?= ${.CURDIR}/../rsyslog55/distinfo .endif .if ${OSVERSION} < 700042 diff --git a/sysutils/rsyslog55/distinfo b/sysutils/rsyslog55/distinfo index a452749..bbda583 100644 --- a/sysutils/rsyslog55/distinfo +++ b/sysutils/rsyslog55/distinfo @@ -1,3 +1,3 @@ -MD5 (rsyslog-5.4.0.tar.gz) = 291882229d50496f42bd63174076dd37 -SHA256 (rsyslog-5.4.0.tar.gz) = d9cd21d2fcd45fcae65eb0a51927c40315cca02afdc62478abd950febfcf7228 -SIZE (rsyslog-5.4.0.tar.gz) = 2124201 +MD5 (rsyslog-5.5.4.tar.gz) = 824df2504955df1619e5ec2915d783aa +SHA256 (rsyslog-5.5.4.tar.gz) = 31853a551ea7ca960c59c9e33406b1748bdf311059c9d8a4ce98816d51b17cac +SIZE (rsyslog-5.5.4.tar.gz) = 2200136 diff --git a/sysutils/rsyslog55/files/extra-patch-mysql-microseconds b/sysutils/rsyslog55/files/extra-patch-mysql-microseconds new file mode 100644 index 0000000..ec248b0 --- /dev/null +++ b/sysutils/rsyslog55/files/extra-patch-mysql-microseconds @@ -0,0 +1,56 @@ +--- ./runtime/datetime.c.orig 2010-05-04 18:57:25.588028725 -0400 ++++ ./runtime/datetime.c 2010-05-04 18:59:12.390680038 -0400 +@@ -644,18 +644,30 @@ + pBuf[1] = (ts->year / 100) % 10 + '0'; + pBuf[2] = (ts->year / 10) % 10 + '0'; + pBuf[3] = ts->year % 10 + '0'; +- pBuf[4] = (ts->month / 10) % 10 + '0'; +- pBuf[5] = ts->month % 10 + '0'; +- pBuf[6] = (ts->day / 10) % 10 + '0'; +- pBuf[7] = ts->day % 10 + '0'; +- pBuf[8] = (ts->hour / 10) % 10 + '0'; +- pBuf[9] = ts->hour % 10 + '0'; +- pBuf[10] = (ts->minute / 10) % 10 + '0'; +- pBuf[11] = ts->minute % 10 + '0'; +- pBuf[12] = (ts->second / 10) % 10 + '0'; +- pBuf[13] = ts->second % 10 + '0'; +- pBuf[14] = '\0'; +- return 15; ++ pBuf[4] = '-'; ++ pBuf[5] = (ts->month / 10) % 10 + '0'; ++ pBuf[6] = ts->month % 10 + '0'; ++ pBuf[7] = '-'; ++ pBuf[8] = (ts->day / 10) % 10 + '0'; ++ pBuf[9] = ts->day % 10 + '0'; ++ pBuf[10] = ' '; ++ pBuf[11] = (ts->hour / 10) % 10 + '0'; ++ pBuf[12] = ts->hour % 10 + '0'; ++ pBuf[13] = ':'; ++ pBuf[14] = (ts->minute / 10) % 10 + '0'; ++ pBuf[15] = ts->minute % 10 + '0'; ++ pBuf[16] = ':'; ++ pBuf[17] = (ts->second / 10) % 10 + '0'; ++ pBuf[18] = ts->second % 10 + '0'; ++ pBuf[19] = '.'; ++ pBuf[20] = (ts->secfrac / 100000) % 10 + '0'; ++ pBuf[21] = (ts->secfrac / 10000) % 10 + '0'; ++ pBuf[22] = (ts->secfrac / 1000) % 10 + '0'; ++ pBuf[23] = (ts->secfrac / 100) % 10 + '0'; ++ pBuf[24] = (ts->secfrac / 10) % 10 + '0'; ++ pBuf[25] = ts->secfrac % 10 + '0'; ++ pBuf[26] = '\0'; ++ return 26; + + } + +--- ./runtime/msg.c.orig 2010-05-04 19:00:20.241528788 -0400 ++++ ./runtime/msg.c 2010-05-04 19:00:06.136349680 -0400 +@@ -1293,7 +1293,7 @@ + case tplFmtMySQLDate: + MsgLock(pM); + if(pM->pszTIMESTAMP_MySQL == NULL) { +- if((pM->pszTIMESTAMP_MySQL = MALLOC(15)) == NULL) { ++ if((pM->pszTIMESTAMP_MySQL = MALLOC(26)) == NULL) { + MsgUnlock(pM); + return ""; + } diff --git a/sysutils/rsyslog55/files/extra-patch-sane-hostname b/sysutils/rsyslog55/files/extra-patch-sane-hostname new file mode 100644 index 0000000..bc72514 --- /dev/null +++ b/sysutils/rsyslog55/files/extra-patch-sane-hostname @@ -0,0 +1,40 @@ +--- ./tools/syslogd.c.orig 2010-05-04 19:02:05.548362478 -0400 ++++ ./tools/syslogd.c 2010-05-04 19:02:27.452450741 -0400 +@@ -2611,37 +2611,6 @@ + net.getLocalHostname(&LocalFQDNName); + CHKmalloc(LocalHostName = (uchar*) strdup((char*)LocalFQDNName)); + glbl.SetLocalFQDNName(LocalFQDNName); /* set the FQDN before we modify it */ +- if((p = (uchar*)strchr((char*)LocalHostName, '.'))) { +- *p++ = '\0'; +- LocalDomain = p; +- } else { +- LocalDomain = (uchar*)""; +- +- /* It's not clearly defined whether gethostname() +- * should return the simple hostname or the fqdn. A +- * good piece of software should be aware of both and +- * we want to distribute good software. Joey +- * +- * Good software also always checks its return values... +- * If syslogd starts up before DNS is up & /etc/hosts +- * doesn't have LocalHostName listed, gethostbyname will +- * return NULL. +- */ +- /* TODO: gethostbyname() is not thread-safe, but replacing it is +- * not urgent as we do not run on multiple threads here. rgerhards, 2007-09-25 +- */ +- hent = gethostbyname((char*)LocalHostName); +- if(hent) { +- free(LocalHostName); +- CHKmalloc(LocalHostName = (uchar*)strdup(hent->h_name)); +- +- if((p = (uchar*)strchr((char*)LocalHostName, '.'))) +- { +- *p++ = '\0'; +- LocalDomain = p; +- } +- } +- } + + /* Convert to lower case to recognize the correct domain laterly */ + for(p = LocalDomain ; *p ; p++) diff --git a/sysutils/rsyslog55/pkg-plist b/sysutils/rsyslog55/pkg-plist index 4120023..3d534e8 100644 --- a/sysutils/rsyslog55/pkg-plist +++ b/sysutils/rsyslog55/pkg-plist @@ -52,6 +52,7 @@ sbin/rsyslogd %%PORTDOCS%%%%DOCSDIR%%/imgssapi.html %%PORTDOCS%%%%DOCSDIR%%/imklog.html %%PORTDOCS%%%%DOCSDIR%%/imrelp.html +%%PORTDOCS%%%%DOCSDIR%%/imsolaris.html %%PORTDOCS%%%%DOCSDIR%%/imtcp.html %%PORTDOCS%%%%DOCSDIR%%/imuxsock.html %%PORTDOCS%%%%DOCSDIR%%/index.html @@ -93,6 +94,7 @@ sbin/rsyslogd %%PORTDOCS%%%%DOCSDIR%%/rsconf1_dropmsgswithmaliciousdnsptrrecords.html %%PORTDOCS%%%%DOCSDIR%%/rsconf1_droptrailinglfonreception.html %%PORTDOCS%%%%DOCSDIR%%/rsconf1_dynafilecachesize.html +%%PORTDOCS%%%%DOCSDIR%%/rsconf1_escape8bitcharsonreceive.html %%PORTDOCS%%%%DOCSDIR%%/rsconf1_escapecontrolcharactersonreceive.html %%PORTDOCS%%%%DOCSDIR%%/rsconf1_failonchownfailure.html %%PORTDOCS%%%%DOCSDIR%%/rsconf1_filecreatemode.html --- rsyslog55.diff ends here --- From rgerhards at hq.adiscon.com Tue May 18 18:11:37 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 18 May 2010 18:11:37 +0200 Subject: [rsyslog] Feedback requested: fast queue mode Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103DE5@GRFEXC.intern.adiscon.com> Hi all, I am currently thinking about various optimizations for even more throughput inside the v5 engine. During this effort, I am also doing some review of existing literature on lock-free and wait-free algorithms. One of my former ideas re-appeared, and I wanted to get feedback if that would be a useful addition. Even if it is seen as valuable, I will probably not implement it immediately, but I would try to incorporate it into the new design, which I plan to implement later this year. Looking at the queue modes, we have some overhead inside queues because queues need to do a lot of things. Things like race-limiting, blocking on full queue, even going to disk if the queue fills up too much. Also, support for thread pools and all that is needed. Note that all of this overhead is also necessary if the queue is used to run an action asynchronously. I think I could implement a considerably faster queue, if I limit its features. Most importantly, that means: - no support for going to the disk (that should not be an issue, I think) - no support for race-limiting - capability to accept message loss if queue is full - message loss victim not selected based on priority - message loss on shutdown acceptable In short, the queue would provide simple in-memory queueing services including synchronization between multiple producers and consumers, but no advanced services at all. At this price, it could probably reduced the queue overhead very considerably. I think such a queue could be useful for the (common) case when data needs to be shuffled to files, and some loss is acceptable (e.g. UDP is the input). I'd say we could probably improve the performance for this use case by a factor of two. Would this be useful? Thoughts are appreciated. Rainer From rory at ooma.com Tue May 18 18:14:00 2010 From: rory at ooma.com (Rory Toma) Date: Tue, 18 May 2010 09:14:00 -0700 Subject: [rsyslog] Feedback requested: fast queue mode In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103DE5@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103DE5@GRFEXC.intern.adiscon.com> Message-ID: <4BF2BCC8.2060004@ooma.com> I could see this being very useful in a relay situation. As a side question, has anyone done injection from rsyslog into MongoDB? On 5/18/10 9:11 AM, Rainer Gerhards wrote: > Hi all, > > I am currently thinking about various optimizations for even more throughput > inside the v5 engine. During this effort, I am also doing some review of > existing literature on lock-free and wait-free algorithms. One of my former > ideas re-appeared, and I wanted to get feedback if that would be a useful > addition. Even if it is seen as valuable, I will probably not implement it > immediately, but I would try to incorporate it into the new design, which I > plan to implement later this year. > > Looking at the queue modes, we have some overhead inside queues because > queues need to do a lot of things. Things like race-limiting, blocking on > full queue, even going to disk if the queue fills up too much. Also, support > for thread pools and all that is needed. Note that all of this overhead is > also necessary if the queue is used to run an action asynchronously. > > I think I could implement a considerably faster queue, if I limit its > features. Most importantly, that means: > > - no support for going to the disk (that should not be an issue, I think) > - no support for race-limiting > - capability to accept message loss if queue is full > - message loss victim not selected based on priority > - message loss on shutdown acceptable > > In short, the queue would provide simple in-memory queueing services > including synchronization between multiple producers and consumers, but no > advanced services at all. At this price, it could probably reduced the queue > overhead very considerably. > > I think such a queue could be useful for the (common) case when data needs to > be shuffled to files, and some loss is acceptable (e.g. UDP is the input). > I'd say we could probably improve the performance for this use case by a > factor of two. > > Would this be useful? Thoughts are appreciated. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rory at ooma.com Wed May 19 22:48:20 2010 From: rory at ooma.com (Rory Toma) Date: Wed, 19 May 2010 13:48:20 -0700 Subject: [rsyslog] MongoDB Message-ID: <4BF44E94.8010204@ooma.com> Has anyone done a MongoDB insertion engine for rsyslog yet? From tbergfeld at hq.adiscon.com Thu May 20 15:52:45 2010 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Thu, 20 May 2010 15:52:45 +0200 Subject: [rsyslog] rsyslog 5.5.5 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103E08@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.5.5, a member of the devel branch. This is a bug-fixing release which contains a single fix that solves a potential hang condition on system shutdown when infinite action retries are configured for an asynchronous action using a queue in disk-assisted mode and the action was suspended. This is probably not a very common case, but a configuration recommended by our doc samples. If you do not use such a configuration, there is no need to update at this time. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article461.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-204.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From jean.mousinho at ist.utl.pt Thu May 20 19:28:52 2010 From: jean.mousinho at ist.utl.pt (Jean F. Mousinho) Date: Thu, 20 May 2010 18:28:52 +0100 Subject: [rsyslog] rsyslog + tls + debian Message-ID: <1274376532.14504.2.camel@muse.ist.utl.pt> Hello, Was anyone successful to get rsyslog working with TLS module in Debian? rsyslogd 4.4.2, compiled with: FEATURE_REGEXP: Yes FEATURE_LARGEFILE: Yes FEATURE_NETZIP (message compression): Yes GSSAPI Kerberos 5 support: Yes FEATURE_DEBUG (debug build, slow code): No Atomic operations supported: Yes Runtime Instrumentation (slow code): No Output is: rsyslogd: could not load module '/usr/lib/rsyslog/lmnsd_gtls.so', rsyslog error -2078 [try http://www.rsyslog.com/e/2068 ] Module file exists: -rw-r--r-- 1 root root 27196 2010-05-17 16:12 /usr/lib/rsyslog/lmnsd_gtls.so I've used the packages provided in backports. Thanks for your time. Jean Mousinho From james at linux-source.org Fri May 21 07:04:59 2010 From: james at linux-source.org (James Corteciano) Date: Fri, 21 May 2010 13:04:59 +0800 Subject: [rsyslog] Centralized rsyslog server and httpd vhost Message-ID: Hi All, My goal is to centralized all system/apps logs from different web farm servers. I have the following setups: node0 - Centralized rsyslog server web_farm1 - web farm 1 server running RHEL, rsyslog, and httpd with different vhost. web_farm2 - web farm 2 server running RHEL, rsyslog, and httpd with different vhost. 1) How can I configure rsyslog from node0 to capture all vhost logs from web_farm servers and all logs will be directly placed like: /var/log/syslog/web_farm1/-error.log /var/log/syslog/web_farm1/-access.log 2) How to configure the httpd service in web_farm servers to push all logs to node0 syslog server? I saw from internet like the following. httpd.conf: ErrorLog "|/bin/logger -p local5.err" CustomLog "|/bin/logger -p local6.info" rsyslog.conf: *.* @node0 Thank you. Regards, James From david at lang.hm Fri May 21 07:31:52 2010 From: david at lang.hm (david at lang.hm) Date: Thu, 20 May 2010 22:31:52 -0700 (PDT) Subject: [rsyslog] Feedback requested: fast queue mode In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103DE5@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103DE5@GRFEXC.intern.adiscon.com> Message-ID: On Tue, 18 May 2010, Rainer Gerhards wrote: > Hi all, > > I am currently thinking about various optimizations for even more throughput > inside the v5 engine. During this effort, I am also doing some review of > existing literature on lock-free and wait-free algorithms. One of my former > ideas re-appeared, and I wanted to get feedback if that would be a useful > addition. Even if it is seen as valuable, I will probably not implement it > immediately, but I would try to incorporate it into the new design, which I > plan to implement later this year. > > Looking at the queue modes, we have some overhead inside queues because > queues need to do a lot of things. Things like race-limiting, blocking on > full queue, even going to disk if the queue fills up too much. Also, support > for thread pools and all that is needed. Note that all of this overhead is > also necessary if the queue is used to run an action asynchronously. > > I think I could implement a considerably faster queue, if I limit its > features. Most importantly, that means: > > - no support for going to the disk (that should not be an issue, I think) > - no support for race-limiting > - capability to accept message loss if queue is full > - message loss victim not selected based on priority > - message loss on shutdown acceptable > > In short, the queue would provide simple in-memory queueing services > including synchronization between multiple producers and consumers, but no > advanced services at all. At this price, it could probably reduced the queue > overhead very considerably. > > I think such a queue could be useful for the (common) case when data needs to > be shuffled to files, and some loss is acceptable (e.g. UDP is the input). > I'd say we could probably improve the performance for this use case by a > factor of two. > > Would this be useful? Thoughts are appreciated. yes, this sounds useful. David Lang From david at lang.hm Fri May 21 07:33:25 2010 From: david at lang.hm (david at lang.hm) Date: Thu, 20 May 2010 22:33:25 -0700 (PDT) Subject: [rsyslog] MongoDB In-Reply-To: <4BF44E94.8010204@ooma.com> References: <4BF44E94.8010204@ooma.com> Message-ID: On Wed, 19 May 2010, Rory Toma wrote: > Has anyone done a MongoDB insertion engine for rsyslog yet? not that I am aware of. In fact, as far as I know, only the postgres and possibly oracle modules take full advantage of the vector mode that allows them to efficiantly batch the message inserts. lots of room for work in this area. David Lang From david at lang.hm Fri May 21 07:37:55 2010 From: david at lang.hm (david at lang.hm) Date: Thu, 20 May 2010 22:37:55 -0700 (PDT) Subject: [rsyslog] Centralized rsyslog server and httpd vhost In-Reply-To: References: Message-ID: On Fri, 21 May 2010, James Corteciano wrote: > Hi All, > > My goal is to centralized all system/apps logs from different web farm > servers. I have the following setups: > > node0 - Centralized rsyslog server > web_farm1 - web farm 1 server running RHEL, rsyslog, and httpd with > different vhost. > web_farm2 - web farm 2 server running RHEL, rsyslog, and httpd with > different vhost. > > 1) How can I configure rsyslog from node0 to capture all vhost logs from > web_farm servers and all logs will be directly placed like: > > /var/log/syslog/web_farm1/-error.log > /var/log/syslog/web_farm1/-access.log > > 2) How to configure the httpd service in web_farm servers to push all logs > to node0 syslog server? I saw from internet like the following. > > httpd.conf: > ErrorLog "|/bin/logger -p local5.err" > CustomLog "|/bin/logger -p local6.info" > > rsyslog.conf: > *.* @node0 when logging from apache you can have log commands inside each vhost, or if you don't the logs will be handled by the main server. what I do is to have the access logs handled by the main server and create a custom format that includes the vhost as part of the format (I also reorder things so that data I really care about is near the beginning of the log and data that can be long is later in the message, so if it becomes extremely long and overflows the max log length I don't loose data I consider critical) then I run it through a perl script that reformats the message to put the vhost name in the server field and sends it out via UDP to my syslog server. I don't have access to that file at the moment (it's at work), I'll try to get a copy tomorrow and post it. David Lang From james at linux-source.org Fri May 21 08:57:50 2010 From: james at linux-source.org (James Corteciano) Date: Fri, 21 May 2010 14:57:50 +0800 Subject: [rsyslog] Centralized rsyslog server and httpd vhost In-Reply-To: References: Message-ID: Hi David, Thanks for your reply and I'm looking forward about it. Regards, James On Fri, May 21, 2010 at 1:37 PM, wrote: > On Fri, 21 May 2010, James Corteciano wrote: > > > Hi All, > > > > My goal is to centralized all system/apps logs from different web farm > > servers. I have the following setups: > > > > node0 - Centralized rsyslog server > > web_farm1 - web farm 1 server running RHEL, rsyslog, and httpd with > > different vhost. > > web_farm2 - web farm 2 server running RHEL, rsyslog, and httpd with > > different vhost. > > > > 1) How can I configure rsyslog from node0 to capture all vhost logs from > > web_farm servers and all logs will be directly placed like: > > > > /var/log/syslog/web_farm1/-error.log > > /var/log/syslog/web_farm1/-access.log > > > > 2) How to configure the httpd service in web_farm servers to push all > logs > > to node0 syslog server? I saw from internet like the following. > > > > httpd.conf: > > ErrorLog "|/bin/logger -p local5.err" > > CustomLog "|/bin/logger -p local6.info" > > > > rsyslog.conf: > > *.* @node0 > > when logging from apache you can have log commands inside each vhost, or > if you don't the logs will be handled by the main server. > > what I do is to have the access logs handled by the main server and create > a custom format that includes the vhost as part of the format (I also > reorder things so that data I really care about is near the beginning of > the log and data that can be long is later in the message, so if it > becomes extremely long and overflows the max log length I don't loose data > I consider critical) > > then I run it through a perl script that reformats the message to put the > vhost name in the server field and sends it out via UDP to my syslog > server. > > I don't have access to that file at the moment (it's at work), I'll try to > get a copy tomorrow and post it. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From james at linux-source.org Fri May 21 09:49:04 2010 From: james at linux-source.org (James Corteciano) Date: Fri, 21 May 2010 15:49:04 +0800 Subject: [rsyslog] Centralized rsyslog server and httpd vhost In-Reply-To: References: Message-ID: Currently, I have this kind of setup. [node0] rsyslog.conf: $template MsgFormat,"%msg%\n" $template ApacheRemoteCustom,"/var/log/httpd/web_farm1/%msg:F,32:2%.log" if $syslogfacility-text == 'local6' and $programname == 'rhcs-node1' then -?ApacheRemoteCustom;MsgFormat [web_farm1] httpd.conf: LogLevel warn ErrorLog "|/bin/logger -p local5.err" LogFormat "%v %h %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" vcombined CustomLog "|/bin/logger -p local6.info -t rhcs-node1" vcombined" It works for getting vhost access logs. However, it doesn't work for error logs because apache ErrorLog is not possible to customize the error log by adding or removing information . Regards, James On Fri, May 21, 2010 at 1:37 PM, wrote: > On Fri, 21 May 2010, James Corteciano wrote: > > > Hi All, > > > > My goal is to centralized all system/apps logs from different web farm > > servers. I have the following setups: > > > > node0 - Centralized rsyslog server > > web_farm1 - web farm 1 server running RHEL, rsyslog, and httpd with > > different vhost. > > web_farm2 - web farm 2 server running RHEL, rsyslog, and httpd with > > different vhost. > > > > 1) How can I configure rsyslog from node0 to capture all vhost logs from > > web_farm servers and all logs will be directly placed like: > > > > /var/log/syslog/web_farm1/-error.log > > /var/log/syslog/web_farm1/-access.log > > > > 2) How to configure the httpd service in web_farm servers to push all > logs > > to node0 syslog server? I saw from internet like the following. > > > > httpd.conf: > > ErrorLog "|/bin/logger -p local5.err" > > CustomLog "|/bin/logger -p local6.info" > > > > rsyslog.conf: > > *.* @node0 > > when logging from apache you can have log commands inside each vhost, or > if you don't the logs will be handled by the main server. > > what I do is to have the access logs handled by the main server and create > a custom format that includes the vhost as part of the format (I also > reorder things so that data I really care about is near the beginning of > the log and data that can be long is later in the message, so if it > becomes extremely long and overflows the max log length I don't loose data > I consider critical) > > then I run it through a perl script that reformats the message to put the > vhost name in the server field and sends it out via UDP to my syslog > server. > > I don't have access to that file at the moment (it's at work), I'll try to > get a copy tomorrow and post it. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From iain at shihad.org Fri May 21 18:37:23 2010 From: iain at shihad.org (Iain M Conochie) Date: Fri, 21 May 2010 17:37:23 +0100 Subject: [rsyslog] Splitting rsyslog messages by hostname into MySQL database Message-ID: <4BF6B6C3.1080108@shihad.org> Afternoon all, I have rsyslog sending all my messages into a mysql database and this is working well. Now I want to start to split the remote messages via hostname into separate tables in the database. I have created a new table FaiEvents with the same schema as SystemEvents and also I have created a config file with actions and template like so: $template Fai-Event,"insert into FaiEvents \ (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)\ values ('%msg%',z,\ %syslogfacility%,\ '%HOSTNAME%',\ %syslogpriority%,\ '%timereported:::date-mysql%',\ '%timegenerated:::date-mysql%',\ %iut%,\ '%syslogtag%')",SQL if $hostname == 'faiserver'\ then :ommysql:localhost,Syslog,rsyslog,********;Fai-Event However rsyslog gives me the error: rsyslogd: the last error occured in /etc/rsyslog.d/05-faiservers.conf, line 13 which is the action line. What variable should I be using to test for the hostname? Can I use a regex here? Any help appreciated! Regards Iain Conochie From james at linux-source.org Sat May 22 11:00:38 2010 From: james at linux-source.org (James Corteciano) Date: Sat, 22 May 2010 17:00:38 +0800 Subject: [rsyslog] Centralized rsyslog server and httpd vhost In-Reply-To: References: Message-ID: Hi David, How about your Apache ErrorLogs? How do you get it that the same thing in Access log? Thank you. Regards, James On Fri, May 21, 2010 at 1:37 PM, wrote: > On Fri, 21 May 2010, James Corteciano wrote: > > > Hi All, > > > > My goal is to centralized all system/apps logs from different web farm > > servers. I have the following setups: > > > > node0 - Centralized rsyslog server > > web_farm1 - web farm 1 server running RHEL, rsyslog, and httpd with > > different vhost. > > web_farm2 - web farm 2 server running RHEL, rsyslog, and httpd with > > different vhost. > > > > 1) How can I configure rsyslog from node0 to capture all vhost logs from > > web_farm servers and all logs will be directly placed like: > > > > /var/log/syslog/web_farm1/-error.log > > /var/log/syslog/web_farm1/-access.log > > > > 2) How to configure the httpd service in web_farm servers to push all > logs > > to node0 syslog server? I saw from internet like the following. > > > > httpd.conf: > > ErrorLog "|/bin/logger -p local5.err" > > CustomLog "|/bin/logger -p local6.info" > > > > rsyslog.conf: > > *.* @node0 > > when logging from apache you can have log commands inside each vhost, or > if you don't the logs will be handled by the main server. > > what I do is to have the access logs handled by the main server and create > a custom format that includes the vhost as part of the format (I also > reorder things so that data I really care about is near the beginning of > the log and data that can be long is later in the message, so if it > becomes extremely long and overflows the max log length I don't loose data > I consider critical) > > then I run it through a perl script that reformats the message to put the > vhost name in the server field and sends it out via UDP to my syslog > server. > > I don't have access to that file at the moment (it's at work), I'll try to > get a copy tomorrow and post it. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Sun May 23 00:21:43 2010 From: david at lang.hm (david at lang.hm) Date: Sat, 22 May 2010 15:21:43 -0700 (PDT) Subject: [rsyslog] Centralized rsyslog server and httpd vhost In-Reply-To: References: Message-ID: On Fri, 21 May 2010, James Corteciano wrote: > Currently, I have this kind of setup. > > [node0] > > rsyslog.conf: > $template MsgFormat,"%msg%\n" > $template ApacheRemoteCustom,"/var/log/httpd/web_farm1/%msg:F,32:2%.log" > if $syslogfacility-text == 'local6' and $programname == 'rhcs-node1' then > -?ApacheRemoteCustom;MsgFormat > > > [web_farm1] > > httpd.conf: > LogLevel warn > ErrorLog "|/bin/logger -p local5.err" > LogFormat "%v %h %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" > vcombined > CustomLog "|/bin/logger -p local6.info -t rhcs-node1" vcombined" > > > It works for getting vhost access logs. However, it doesn't work for error > logs because apache ErrorLog is not possible to customize the error log by > adding or removing > information > . personally I would not try to get error logs in directly specificly because of this. (there's also the problem that a single cgi script error could spew out thousands of lines of garbage into the error log) what's needed is an error log that doesn't include stderr from the cgis being run, but would log a single line along the lines of 'output on stderr from X, see Y for details' where Y is another logfile on the server. unfortunantly this will take modifications to apache to do. David Lang > Regards, > James > > > On Fri, May 21, 2010 at 1:37 PM, wrote: > >> On Fri, 21 May 2010, James Corteciano wrote: >> >>> Hi All, >>> >>> My goal is to centralized all system/apps logs from different web farm >>> servers. I have the following setups: >>> >>> node0 - Centralized rsyslog server >>> web_farm1 - web farm 1 server running RHEL, rsyslog, and httpd with >>> different vhost. >>> web_farm2 - web farm 2 server running RHEL, rsyslog, and httpd with >>> different vhost. >>> >>> 1) How can I configure rsyslog from node0 to capture all vhost logs from >>> web_farm servers and all logs will be directly placed like: >>> >>> /var/log/syslog/web_farm1/-error.log >>> /var/log/syslog/web_farm1/-access.log >>> >>> 2) How to configure the httpd service in web_farm servers to push all >> logs >>> to node0 syslog server? I saw from internet like the following. >>> >>> httpd.conf: >>> ErrorLog "|/bin/logger -p local5.err" >>> CustomLog "|/bin/logger -p local6.info" >>> >>> rsyslog.conf: >>> *.* @node0 >> >> when logging from apache you can have log commands inside each vhost, or >> if you don't the logs will be handled by the main server. >> >> what I do is to have the access logs handled by the main server and create >> a custom format that includes the vhost as part of the format (I also >> reorder things so that data I really care about is near the beginning of >> the log and data that can be long is later in the message, so if it >> becomes extremely long and overflows the max log length I don't loose data >> I consider critical) >> >> then I run it through a perl script that reformats the message to put the >> vhost name in the server field and sends it out via UDP to my syslog >> server. >> >> I don't have access to that file at the moment (it's at work), I'll try to >> get a copy tomorrow and post it. >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From jli at jlisbz.com Wed May 26 05:54:57 2010 From: jli at jlisbz.com (John Li) Date: Tue, 25 May 2010 23:54:57 -0400 Subject: [rsyslog] Where is the output module for the udp transportation to remote syslog server Message-ID: Hi, Is the output via udp to remote syslog server implemented as a output module? I could not find it in the plugins folder. Thanks. -- John Jun Li jli at jlisbz.com My Blog: http://www.jlisbz.com My LinkedIn Profile: http://www.linkedin.com/in/johnjunli From fn42551 at fmi.uni-sofia.bg Wed May 26 16:17:07 2010 From: fn42551 at fmi.uni-sofia.bg (Angel Tsankov) Date: Wed, 26 May 2010 17:17:07 +0300 Subject: [rsyslog] Check if rsyslog is running Message-ID: What is the recommended way for an application to check if rsyslogd is running? Angel Tsankov From jli at jlisbz.com Wed May 26 16:41:25 2010 From: jli at jlisbz.com (John Li) Date: Wed, 26 May 2010 10:41:25 -0400 Subject: [rsyslog] Check if rsyslog is running In-Reply-To: References: Message-ID: monit from http://mmonit.com/monit/ shoud be able to handle this easily. And it can do both monitoring and restarting if the process is crashed. -- John Jun Li jli at jlisbz.com My Blog: http://www.jlisbz.com My LinkedIn Profile: http://www.linkedin.com/in/johnjunli My Twitter: http://www.twitter.com/jlisbz My facebook: http://www.facebook.com/profile.php?id=593495282 On Wed, May 26, 2010 at 10:17 AM, Angel Tsankov wrote: > What is the recommended way for an application to check if rsyslogd is > running? > > Angel Tsankov > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed May 26 17:36:56 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 26 May 2010 18:36:56 +0300 Subject: [rsyslog] Where is the output module for the udp transportation toremote syslog server Message-ID: <003401cafce9$44d808a1$100013ac@intern.adiscon.com> This is a built-in module, it does not need to be loaded (it is actually linked into the main executable). HTH Rainer ----- Urspr?ngliche Nachricht ----- Von: John Li Gesendet: Mittwoch, 26. Mai 2010 07:03 An: rsyslog at lists.adiscon.com Betreff: [rsyslog] Where is the output module for the udp transportation toremote syslog server Hi, Is the output via udp to remote syslog server implemented as a output module? I could not find it in the plugins folder. Thanks. -- John Jun Li jli at jlisbz.com My Blog: http://www.jlisbz.com My LinkedIn Profile: http://www.linkedin.com/in/johnjunli _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From jli at jlisbz.com Wed May 26 17:46:03 2010 From: jli at jlisbz.com (John Li) Date: Wed, 26 May 2010 11:46:03 -0400 Subject: [rsyslog] Where is the output module for the udp transportation toremote syslog server In-Reply-To: <003401cafce9$44d808a1$100013ac@intern.adiscon.com> References: <003401cafce9$44d808a1$100013ac@intern.adiscon.com> Message-ID: Thanks. My goal is to change the content of msg and I am planning to use output module to do that. Is this the right approach and do you mind point me to some sample code in output module to do that? -- John Jun Li jli at jlisbz.com My Blog: http://www.jlisbz.com My LinkedIn Profile: http://www.linkedin.com/in/johnjunli On Wed, May 26, 2010 at 11:36 AM, Rainer Gerhards wrote: > This is a built-in module, it does not need to be loaded (it is actually > linked into the main executable). > > HTH > Rainer > > ----- Urspr?ngliche Nachricht ----- > Von: John Li > Gesendet: Mittwoch, 26. Mai 2010 07:03 > An: rsyslog at lists.adiscon.com > Betreff: [rsyslog] Where is the output module for the udp transportation > toremote syslog server > > Hi, > > Is the output via udp to remote syslog server implemented as a output > module? I could not find it in the plugins folder. > > Thanks. > > -- > John Jun Li > jli at jlisbz.com > > My Blog: http://www.jlisbz.com > My LinkedIn Profile: http://www.linkedin.com/in/johnjunli > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed May 26 21:28:06 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 26 May 2010 22:28:06 +0300 Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server Message-ID: <004501cafd09$9027db05$100013ac@intern.adiscon.com> You need to look into templates. It is quite easy to rewrite message content with templates. There are samples in the doc and in the wiki. Rainer ----- Urspr?ngliche Nachricht ----- Von: John Li Gesendet: Mittwoch, 26. Mai 2010 18:53 An: rsyslog-users Betreff: Re: [rsyslog] Where is the output module for the udp transportationtoremote syslog server Thanks. My goal is to change the content of msg and I am planning to use output module to do that. Is this the right approach and do you mind point me to some sample code in output module to do that? -- John Jun Li jli at jlisbz.com My Blog: http://www.jlisbz.com My LinkedIn Profile: http://www.linkedin.com/in/johnjunli On Wed, May 26, 2010 at 11:36 AM, Rainer Gerhards wrote: > This is a built-in module, it does not need to be loaded (it is actually > linked into the main executable). > > HTH > Rainer > > ----- Urspr?ngliche Nachricht ----- > Von: John Li > Gesendet: Mittwoch, 26. Mai 2010 07:03 > An: rsyslog at lists.adiscon.com > Betreff: [rsyslog] Where is the output module for the udp transportation > toremote syslog server > > Hi, > > Is the output via udp to remote syslog server implemented as a output > module? I could not find it in the plugins folder. > > Thanks. > > -- > John Jun Li > jli at jlisbz.com > > My Blog: http://www.jlisbz.com > My LinkedIn Profile: http://www.linkedin.com/in/johnjunli > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From jli at jlisbz.com Wed May 26 22:15:16 2010 From: jli at jlisbz.com (John Li) Date: Wed, 26 May 2010 16:15:16 -0400 Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server In-Reply-To: <004501cafd09$9027db05$100013ac@intern.adiscon.com> References: <004501cafd09$9027db05$100013ac@intern.adiscon.com> Message-ID: This is the RE case I posted in forum last week and output module is the best to achieve it. Here is the case I described: " For example, a typical firewall log: 192.168.20.5 23456 192.168.10.10 80 Accept Web 192.168.20.6 5678 192.168.10.10 22 Deny SSH If I want to have the xml form of them, it could be : 192.168.20.5192.168.10.102345680AcceptWeb 192.168.20.6192.168.10.10567822DenySSH If I understand correctly for template, I had to do RE for 6 times for each log entry and that could cause performance issue in large environment for sure. " So I need to rewrite the msg in the output module, please let me know where to find some sample code or doc. And here is one more question: "One thing I want to make sure is the output plugin which I will make should be still able to use other output method such as syslog/snmp etc with the converted message, right? ." I was able to create my own output module based on the stdout module but could not figure out how to rewrite the msg back to rsyslog so the rewritten msg can be used by other output module. Is this doable? Thanks a lot. -- John On Wed, May 26, 2010 at 3:28 PM, Rainer Gerhards wrote: > You need to look into templates. It is quite easy to rewrite message > content with templates. There are samples in the doc and in the wiki. > > Rainer > > ----- Urspr?ngliche Nachricht ----- > Von: John Li > Gesendet: Mittwoch, 26. Mai 2010 18:53 > An: rsyslog-users > Betreff: Re: [rsyslog] Where is the output module for the udp > transportationtoremote syslog server > > Thanks. > > My goal is to change the content of msg and I am planning to use output > module to do that. Is this the right approach and do you mind point me to > some sample code in output module to do that? > > > -- > John Jun Li > jli at jlisbz.com > > My Blog: http://www.jlisbz.com > My LinkedIn Profile: http://www.linkedin.com/in/johnjunli > > > > On Wed, May 26, 2010 at 11:36 AM, Rainer Gerhards > wrote: > > > This is a built-in module, it does not need to be loaded (it is actually > > linked into the main executable). > > > > HTH > > Rainer > > > > ----- Urspr?ngliche Nachricht ----- > > Von: John Li > > Gesendet: Mittwoch, 26. Mai 2010 07:03 > > An: rsyslog at lists.adiscon.com > > Betreff: [rsyslog] Where is the output module for the udp transportation > > toremote syslog server > > > > Hi, > > > > Is the output via udp to remote syslog server implemented as a output > > module? I could not find it in the plugins folder. > > > > Thanks. > > > > -- > > John Jun Li > > jli at jlisbz.com > > > > My Blog: http://www.jlisbz.com > > My LinkedIn Profile: http://www.linkedin.com/in/johnjunli > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Thu May 27 07:55:25 2010 From: david at lang.hm (david at lang.hm) Date: Wed, 26 May 2010 22:55:25 -0700 (PDT) Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server In-Reply-To: References: <004501cafd09$9027db05$100013ac@intern.adiscon.com> Message-ID: you may want to consider doing this on the input side instead of the output side. see http://www.rsyslog.com/doc-messageparser.html yes, in many ways it's operating backwards, but it may be significantly less work to implement and maintain it this way. David Lang On Wed, 26 May 2010, John Li wrote: > This is the RE case I posted in forum last week and output module is the > best to achieve it. > > Here is the case I described: > " > For example, a typical firewall log: > 192.168.20.5 23456 192.168.10.10 80 Accept Web > 192.168.20.6 5678 192.168.10.10 22 Deny SSH > > If I want to have the xml form of them, it could be : > 192.168.20.5192.168.10.102345680AcceptWeb > 192.168.20.6192.168.10.10567822DenySSH > > If I understand correctly for template, I had to do RE for 6 times for each > log entry and that could cause performance issue in large environment for > sure. " > > So I need to rewrite the msg in the output module, please let me know where > to find some sample code or doc. And here is one more question: > > "One thing I want to make sure is the output plugin which I will make should > be still able to use other output method such as syslog/snmp etc with the > converted message, right? ." > > I was able to create my own output module based on the stdout module but > could not figure out how to rewrite the msg back to rsyslog so the rewritten > msg can be used by other output module. Is this doable? > > Thanks a lot. > > From jli at jlisbz.com Thu May 27 15:29:28 2010 From: jli at jlisbz.com (John Li) Date: Thu, 27 May 2010 09:29:28 -0400 Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server In-Reply-To: References: <004501cafd09$9027db05$100013ac@intern.adiscon.com> Message-ID: Not totally agree. First, your will lose the flexibility in the input side if you put the rewritten code in the input module. Second, parser looks like to target the syslog format validity instead of message rewritten. But if it's not possible to rewrite the msg in the output module, I will have to do that in the parser. Can someone please confirm? Thanks. -- John Jun Li jli at jlisbz.com My Blog: http://www.jlisbz.com My LinkedIn Profile: http://www.linkedin.com/in/johnjunli On Thu, May 27, 2010 at 1:55 AM, wrote: > you may want to consider doing this on the input side instead of the > output side. > > see http://www.rsyslog.com/doc-messageparser.html > > yes, in many ways it's operating backwards, but it may be significantly > less work to implement and maintain it this way. > > David Lang > > On Wed, 26 May 2010, John Li wrote: > > > This is the RE case I posted in forum last week and output module is the > > best to achieve it. > > > > Here is the case I described: > > " > > For example, a typical firewall log: > > 192.168.20.5 23456 192.168.10.10 80 Accept Web > > 192.168.20.6 5678 192.168.10.10 22 Deny SSH > > > > If I want to have the xml form of them, it could be : > > > 192.168.20.5192.168.10.102345680AcceptWeb > > > 192.168.20.6192.168.10.10567822DenySSH > > > > If I understand correctly for template, I had to do RE for 6 times for > each > > log entry and that could cause performance issue in large environment for > > sure. " > > > > So I need to rewrite the msg in the output module, please let me know > where > > to find some sample code or doc. And here is one more question: > > > > "One thing I want to make sure is the output plugin which I will make > should > > be still able to use other output method such as syslog/snmp etc with the > > converted message, right? ." > > > > I was able to create my own output module based on the stdout module but > > could not figure out how to rewrite the msg back to rsyslog so the > rewritten > > msg can be used by other output module. Is this doable? > > > > Thanks a lot. > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Fri May 28 07:25:58 2010 From: david at lang.hm (david at lang.hm) Date: Thu, 27 May 2010 22:25:58 -0700 (PDT) Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server In-Reply-To: References: <004501cafd09$9027db05$100013ac@intern.adiscon.com> Message-ID: On Thu, 27 May 2010, John Li wrote: > Not totally agree. First, your will lose the flexibility in the input side > if you put the rewritten code in the input module. Second, parser looks like > to target the syslog format validity instead of message rewritten. it doesn't just validate the message, it takes the message off the wire, and breaksit into the separate properties that rsyslog handles internally (and are available for the output templates). It already has the option to modify the string as it does this (look at control character re-writing) > But if it's not possible to rewrite the msg in the output module, I will > have to do that in the parser. Can someone please confirm? I agree that doing it in the output would be far better in many ways, but since there isn't a way to do a plugin there (at least not as far as I know, it would be good to get confirmation or a better idea) David Lang > Thanks. > > From rgerhards at hq.adiscon.com Mon May 31 11:35:54 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 31 May 2010 11:35:54 +0200 Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server References: <004501cafd09$9027db05$100013ac@intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103E26@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, May 28, 2010 7:26 AM > To: rsyslog-users > Subject: Re: [rsyslog] Where is the output module for the udp > transportationtoremote syslog server > > On Thu, 27 May 2010, John Li wrote: > > > Not totally agree. First, your will lose the flexibility in the input > side > > if you put the rewritten code in the input module. Second, parser > looks like > > to target the syslog format validity instead of message rewritten. > > it doesn't just validate the message, it takes the message off the > wire, > and breaksit into the separate properties that rsyslog handles > internally > (and are available for the output templates). It already has the option > to > modify the string as it does this (look at control character re- > writing) > > > But if it's not possible to rewrite the msg in the output module, I > will > > have to do that in the parser. Can someone please confirm? John, You can do whatever you like in an output module, including rewriting any part of the message. Of course, you can NOT modify the message strings that *other* output modules see. > I agree that doing it in the output would be far better in many ways, > but > since there isn't a way to do a plugin there (at least not as far as I > know, it would be good to get confirmation or a better idea) David, can you tell me what you have on your mind for this functionality? I have thought a bit about it, and I probably have one approach myself. But I would prefer to hear your idea before I push you into a direction. Thanks, Rainer From david at lang.hm Mon May 31 12:16:32 2010 From: david at lang.hm (david at lang.hm) Date: Mon, 31 May 2010 03:16:32 -0700 (PDT) Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103E26@GRFEXC.intern.adiscon.com> References: <004501cafd09$9027db05$100013ac@intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103E26@GRFEXC.intern.adiscon.com> Message-ID: On Mon, 31 May 2010, Rainer Gerhards wrote: >> I agree that doing it in the output would be far better in many ways, >> but >> since there isn't a way to do a plugin there (at least not as far as I >> know, it would be good to get confirmation or a better idea) > > David, can you tell me what you have on your mind for this functionality? I > have thought a bit about it, and I probably have one approach myself. But I > would prefer to hear your idea before I push you into a direction. two options 1. something that would work similar to the existing format string, but would call a C subroutine that could read the existing properties and would create the output string in a buffer 2. something that could also modify the exisitng properties (more powerful, but also more dangerous and could involve locking to prevent other things from trying to read properties at the same time) we haven't gone too far down the road of researching the output performance (since the input and queue locking has dominated so far), but it is clear that the output currently takes significantly more CPU time than input, it may be that being able to use C to define the output format instead of interpreting the format string may be a noticable improvement. Is there a relativly easy way to test this? (say, hard-code a format or two and test writes to file and network with the hard-coded format vs a format string that produces the same output?) David Lang From david at lang.hm Mon May 31 12:30:22 2010 From: david at lang.hm (david at lang.hm) Date: Mon, 31 May 2010 03:30:22 -0700 (PDT) Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server In-Reply-To: References: <004501cafd09$9027db05$100013ac@intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103E26@GRFEXC.intern.adiscon.com> Message-ID: On Mon, 31 May 2010, david at lang.hm wrote: > On Mon, 31 May 2010, Rainer Gerhards wrote: > >>> I agree that doing it in the output would be far better in many ways, >>> but >>> since there isn't a way to do a plugin there (at least not as far as I >>> know, it would be good to get confirmation or a better idea) >> >> David, can you tell me what you have on your mind for this functionality? I >> have thought a bit about it, and I probably have one approach myself. But I >> would prefer to hear your idea before I push you into a direction. > > > two options > > 1. something that would work similar to the existing format > string, but would call a C subroutine that could read the existing > properties and would create the output string in a buffer > > 2. something that could also modify the exisitng properties (more > powerful, but also more dangerous and could involve locking to prevent > other things from trying to read properties at the same time) > > we haven't gone too far down the road of researching the output > performance (since the input and queue locking has dominated so far), but > it is clear that the output currently takes significantly more CPU time > than input, it may be that being able to use C to define the output format > instead of interpreting the format string may be a noticable improvement. > Is there a relativly easy way to test this? (say, hard-code a format or > two and test writes to file and network with the hard-coded format vs a > format string that produces the same output?) for the traditional output formats the difference may not be that much, but if there is extensive parsing involved (as the initial poster is doing, or what I would expect is common for specific log types into a database) the difference can be much more significant since it can replace multiple regex statements with a much faster single pass that looks for word breaks and inserts standard filler in those spots. With the new syslog format where the data is 'supposed to be' in a series of name=value tuples, something like this would be a pretty efficiant way of extracting particular portions of the data to be output (although the properties could be extended to do this sort of thing by providing something similar to a perl hash) David Lang From jli at jlisbz.com Mon May 31 14:17:02 2010 From: jli at jlisbz.com (John Li) Date: Mon, 31 May 2010 05:17:02 -0700 Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server Message-ID: <2054128934449600685@unknownmsgid> Thanks a lot. Currently i am stucked at the design that output module can not modify the msg to be seen by other output modules. I understand why it's designed that way but just wondering if there is a quick hack to persist the modified msg in output module so other modules can see. Or do you guys have something to handle this scenario better? Thanks David for better describing the problem. Sent from my HTC -----Original Message----- From: david at lang.hm Sent: May 31, 2010 6:30 AM To: rsyslog-users Subject: Re: [rsyslog] Where is the output module for the udp transportationtoremote syslog server On Mon, 31 May 2010, david at lang.hm wrote: > On Mon, 31 May 2010, Rainer Gerhards wrote: > >>> I agree that doing it in the output would be far better in many ways, >>> but >>> since there isn't a way to do a plugin there (at least not as far as I >>> know, it would be good to get confirmation or a better idea) >> >> David, can you tell me what you have on your mind for this functionality? I >> have thought a bit about it, and I probably have one approach myself. But I >> would prefer to hear your idea before I push you into a direction. > > > two options > > 1. something that would work similar to the existing format > string, but would call a C subroutine that could read the existing > properties and would create the output string in a buffer > > 2. something that could also modify the exisitng properties (more > powerful, but also more dangerous and could involve locking to prevent > other things from trying to read properties at the same time) > > we haven't gone too far down the road of researching the output > performance (since the input and queue locking has dominated so far), but > it is clear that the output currently takes significantly more CPU time > than input, it may be that being able to use C to define the output format > instead of interpreting the format string may be a noticable improvement. > Is there a relativly easy way to test this? (say, hard-code a format or > two and test writes to file and network with the hard-coded format vs a > format string that produces the same output?) for the traditional output formats the difference may not be that much, but if there is extensive parsing involved (as the initial poster is doing, or what I would expect is common for specific log types into a database) the difference can be much more significant since it can replace multiple regex statements with a much faster single pass that looks for word breaks and inserts standard filler in those spots. With the new syslog format where the data is 'supposed to be' in a series of name=value tuples, something like this would be a pretty efficiant way of extracting particular portions of the data to be output (although the properties could be extended to do this sort of thing by providing something similar to a perl hash) David Lang _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From jli at jlisbz.com Mon May 31 14:17:24 2010 From: jli at jlisbz.com (John Li) Date: Mon, 31 May 2010 08:17:24 -0400 Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server Message-ID: <7684824564864990291@unknownmsgid> Thanks a lot. Currently i am stucked at the design that output module can not modify the msg to be seen by other output modules. I understand why it's designed that way but just wondering if there is a quick hack to persist the modified msg in output module so other modules can see. Or do you guys have something to handle this scenario better? Thanks David for better describing the problem. Sent from my HTC -----Original Message----- From: david at lang.hm Sent: May 31, 2010 6:30 AM To: rsyslog-users Subject: Re: [rsyslog] Where is the output module for the udp transportationtoremote syslog server On Mon, 31 May 2010, david at lang.hm wrote: > On Mon, 31 May 2010, Rainer Gerhards wrote: > >>> I agree that doing it in the output would be far better in many ways, >>> but >>> since there isn't a way to do a plugin there (at least not as far as I >>> know, it would be good to get confirmation or a better idea) >> >> David, can you tell me what you have on your mind for this functionality? I >> have thought a bit about it, and I probably have one approach myself. But I >> would prefer to hear your idea before I push you into a direction. > > > two options > > 1. something that would work similar to the existing format > string, but would call a C subroutine that could read the existing > properties and would create the output string in a buffer > > 2. something that could also modify the exisitng properties (more > powerful, but also more dangerous and could involve locking to prevent > other things from trying to read properties at the same time) > > we haven't gone too far down the road of researching the output > performance (since the input and queue locking has dominated so far), but > it is clear that the output currently takes significantly more CPU time > than input, it may be that being able to use C to define the output format > instead of interpreting the format string may be a noticable improvement. > Is there a relativly easy way to test this? (say, hard-code a format or > two and test writes to file and network with the hard-coded format vs a > format string that produces the same output?) for the traditional output formats the difference may not be that much, but if there is extensive parsing involved (as the initial poster is doing, or what I would expect is common for specific log types into a database) the difference can be much more significant since it can replace multiple regex statements with a much faster single pass that looks for word breaks and inserts standard filler in those spots. With the new syslog format where the data is 'supposed to be' in a series of name=value tuples, something like this would be a pretty efficiant way of extracting particular portions of the data to be output (although the properties could be extended to do this sort of thing by providing something similar to a perl hash) David Lang _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon May 31 14:24:27 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 31 May 2010 14:24:27 +0200 Subject: [rsyslog] Where is the output module for the udptransportationtoremote syslog server References: <2054128934449600685@unknownmsgid> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103E2E@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of John Li > Sent: Monday, May 31, 2010 2:17 PM > To: david at lang.hm; rsyslog-users > Subject: Re: [rsyslog] Where is the output module for the > udptransportationtoremote syslog server > > Thanks a lot. > Currently i am stucked at the design that output module can not modify > the msg to be seen by other output modules. I understand why it's > designed that way but just wondering if there is a quick hack to > persist the modified msg in output module so other modules can see. You may want to have a look at omruleset. > Or do you guys have something to handle this scenario better? Thanks > David for better describing the problem. I will shortly reply to David's mail, I think the information will be useful for you as well. I just need some more time to prepare that message. Rainer > > Sent from my HTC > > -----Original Message----- > From: david at lang.hm > Sent: May 31, 2010 6:30 AM > To: rsyslog-users > Subject: Re: [rsyslog] Where is the output module for > the udp transportationtoremote syslog server > > > On Mon, 31 May 2010, david at lang.hm wrote: > > > On Mon, 31 May 2010, Rainer Gerhards wrote: > > > >>> I agree that doing it in the output would be far better in many > ways, > >>> but > >>> since there isn't a way to do a plugin there (at least not as far > as I > >>> know, it would be good to get confirmation or a better idea) > >> > >> David, can you tell me what you have on your mind for this > functionality? I > >> have thought a bit about it, and I probably have one approach > myself. But I > >> would prefer to hear your idea before I push you into a direction. > > > > > > two options > > > > 1. something that would work similar to the existing format > > string, but would call a C subroutine that could read the existing > > properties and would create the output string in a buffer > > > > 2. something that could also modify the exisitng properties (more > > powerful, but also more dangerous and could involve locking to > prevent > > other things from trying to read properties at the same time) > > > > we haven't gone too far down the road of researching the output > > performance (since the input and queue locking has dominated so far), > but > > it is clear that the output currently takes significantly more CPU > time > > than input, it may be that being able to use C to define the output > format > > instead of interpreting the format string may be a noticable > improvement. > > Is there a relativly easy way to test this? (say, hard-code a format > or > > two and test writes to file and network with the hard-coded format vs > a > > format string that produces the same output?) > > for the traditional output formats the difference may not be that much, > but if there is extensive parsing involved (as the initial poster is > doing, or what I would expect is common for specific log types into a > database) the difference can be much more significant since it can > replace > multiple regex statements with a much faster single pass that looks for > word breaks and inserts standard filler in those spots. > > With the new syslog format where the data is 'supposed to be' in a > series of name=value tuples, something like this would be a pretty > efficiant way of extracting particular portions of the data to be > output > (although the properties could be extended to do this sort of thing by > providing something similar to a perl hash) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon May 31 15:39:06 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 31 May 2010 15:39:06 +0200 Subject: [rsyslog] Where is the output module for the udp transportationtoremote syslog server References: <004501cafd09$9027db05$100013ac@intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103E26@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103E37@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Monday, May 31, 2010 12:30 PM > To: rsyslog-users > Subject: Re: [rsyslog] Where is the output module for the udp > transportationtoremote syslog server > > On Mon, 31 May 2010, david at lang.hm wrote: > > > On Mon, 31 May 2010, Rainer Gerhards wrote: > > > >>> I agree that doing it in the output would be far better in many > ways, > >>> but > >>> since there isn't a way to do a plugin there (at least not as far > as I > >>> know, it would be good to get confirmation or a better idea) > >> > >> David, can you tell me what you have on your mind for this > functionality? I > >> have thought a bit about it, and I probably have one approach > myself. But I > >> would prefer to hear your idea before I push you into a direction. > > > > > > two options > > > > 1. something that would work similar to the existing format > > string, but would call a C subroutine that could read the existing > > properties and would create the output string in a buffer > > > > 2. something that could also modify the exisitng properties (more > > powerful, but also more dangerous and could involve locking to > prevent > > other things from trying to read properties at the same time) > > > > we haven't gone too far down the road of researching the output > > performance (since the input and queue locking has dominated so far), > but > > it is clear that the output currently takes significantly more CPU > time > > than input, it may be that being able to use C to define the output > format > > instead of interpreting the format string may be a noticable > improvement. > > Is there a relativly easy way to test this? (say, hard-code a format > or > > two and test writes to file and network with the hard-coded format vs > a > > format string that produces the same output?) > > for the traditional output formats the difference may not be that much, > but if there is extensive parsing involved (as the initial poster is > doing, or what I would expect is common for specific log types into a > database) the difference can be much more significant since it can > replace > multiple regex statements with a much faster single pass that looks for > word breaks and inserts standard filler in those spots. > > With the new syslog format where the data is 'supposed to be' in a > series of name=value tuples, something like this would be a pretty > efficiant way of extracting particular portions of the data to be > output > (although the properties could be extended to do this sort of thing by > providing something similar to a perl hash) You are looking in the same direction I am, and I think this is good news ;) The current engine supports functions coded in C, but not yet as real plugins nor in an easy to see way. It is done via a crude function interface library module, and only within the script engine. My original plan (over a year, or even two, ago) was to generalize these library plugins, so that it is easy to add new code and load them as plugins. Actually, making them available as plugins should not be too much work given the already existing infrastructure. There already exist a handful of "function modules", the control structure is just statically created during compile time, much as some of the output plugins are statically linked. Then the original plan was to enable templates to call scripts and enable scripts to define templates (kind of). Unfortunately, I got distracted by more important things before I could complete all of this. HOWEVER, at this time performance was not a major concern. With what has evolved in the mean time, I do not like the original approach that much any longer. At least the script engine must become much faster before I can take a real look at that capability. Right now, scripts generate a interim code that then is interpreted by a (kind of) virtual machine. A script invocation inside a template would mean that a VM must be instantiated, the script interpreted and the resulting string be used as template contents. Clearly, this is not for high-performance use. Still, however, it may be useful to have that capability for those cases, where performance is not the #1 consideration. But given that everything would need to be implemented, it does make limited sense to look into something known to be too slow in the long run. BTW, this is one reason that I have not yet continued to work on the script engine, knowing that some larger redesign is due to fit it into the now much tighter runtime constraints. On the performance of the output system: I think the system in general is quite fast and efficient, with only ONE important exception: that is, if multiple replacements need to happen. Still, the algorithm is quite efficient, but it is generic and needs to run though a number of steps. Of course, it is definitely faster to permit a C plugin to look at the message and then format, in an "atomic" way the resulting custom string. Thus, you need to write multiple C codes instead of using a generic engine, but can do so in a much higher performance way. I would assume, however, that this approach cannot beat the simple templates we usually use (maybe by less than 5% and, of course, there may be cases where this matters). As you know, my current focus is speed, together with some functional enhancements. I was looking at queue operations improvements, but the potential output speed improvements may be more interesting than the queue mode improvements (and apply to more use cases). So it may make sense to look into these, first. My challenge here is to find something that is a) generic enough to be useful in various (usual) cases b) specific enough to be rather fast and it should also be able to implement within a few weeks at most, because I can probably not spend much more time on a single feature/refactoring. One solution may be to create "template modules". I could envision a template module to be something that generates the template string *as a whole* from the input message. That is, we would have $template current-style,"%msg%\n" but also (**) $modload tplcustom $template custom,tplcustom where tplcustom generates the template string. While this sounds promising, we have some issues. One immediately pops up my mind: we will probably be able to use the same template for file writing or forwarding, but for file writing we need a LF at the end, while for forwarding we do not need it. So the most natural way would be to have the ability to embed a "custom template" into a regular template, like suggested by this syntax: $template both,"%=tplcustom%\n" however, this brings us down to the slippery slope of the original design. As a next thing to be requested, I could ask for using not the msg object (with its fixed unmodified properties), but rather of a transformation of the message object. So we would end up with something like this: $template cmplx,"%=tplcustom(syslogtag & msg)%" Which would require a much more complex logic working behind the scenes. Of course, depending on the format used, the engine could select different processing algorithms. Doing this on the fly seems possible, but requires more work than I can commit in one sequence. Also, it would be useful to have the ability to persist already-generated properties with the message while it is continued to be processed in the rule engine. So far, we do not have this ability, and the reason is processing time (plus, as usual, implementation effort): for that, we would need to maintain a list (or hash, ...) of name/value pairs, store them to disk for disk queues and shuffle them through the rule engine as processing is carried out. As I said, quite doable, but another big addition. So I am somewhat stuck with things that sound interesting, but are a bit interdependent. Doing them all together is too big to be useful, and it will probably fail because I can probably not keep focus on all of the for the next, say, 9 to 12 month that it would require to complete everything. So I am again down to picking what is most useful. Out of this discussion, it looks like the idea I marked with (**), the plain C template generator could be a useful route to take. I am saying this under the assumption that it would be relatively easy to implement and cause at least some speedup in standard cases (contrary to what I expect, I have to admit...). But that approach is highly specialized, requiring a C module for each custom format. So does it really serve the rsyslog community well - or just some very isolated use cases? Thinking more about it, it would probably be useful if it is both a) relatively easy to implement and b) causes some speedup in standard cases But b) cannot be proven without actually implementing the interface. So, in practice, the questions boils down to what we *expect* about the usefulness of this utility. Having said that, I'd appreciate feedback, both on the concrete question of the usefulness of this feature as well as any and all comments on the situation at large. I am trying to put my development resources, which thankfully have been somewhat increased nowadays :) to the area where they provide greatest benefit. Rainer