From champ at softwink.com Thu Mar 17 14:30:02 2011 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Thu, 17 Mar 2011 09:30:02 -0400 Subject: [Lognorm] Liblognorm - Long time no see :) In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDA0E@GRFEXC.intern.adiscon.com> References: <20110113153409.GA15921@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD9F7@GRFEXC.intern.adiscon.com> <20110113173334.GA19911@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD9FD@GRFEXC.intern.adiscon.com> <20110113183413.GA22721@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DDA04@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDA0E@GRFEXC.intern.adiscon.com> Message-ID: <20110317133002.GA27261@bundy.vistech.net> Hello all! I've been out of the loop for a bit. I've been at a client for several weeks doing a Sagan install. That's since been completed and it's working very well. This really put liblognorm through some tests during this install. The good news is that I didn't really have any issues with liblognorm at all. Granted not _all_ log line go through liblognorm, but a pretty decent amount have been. This site is pushing about 10 million events a day. I'm actually going to be out of town next week, but sort of wanted to restart discussions about liblognorm and log normalization in general. I've since released a new version of Sagan which uses liblognorm a good bit! It's version 0.1.8 and can be found at http://sagan.softiwnk.com Hope things are going well with everyone. -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From champ at softwink.com Mon Mar 21 13:26:19 2011 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Mon, 21 Mar 2011 08:26:19 -0400 Subject: [Lognorm] libee compile issues. Message-ID: <20110321122619.GC25589@bundy.vistech.net> Rainer, I got this e-mail a couple of days ago. I was wondering if you've recieved any similar issues with building libee. ---- Keep in mind, I'm not the author of libee (or libestr/liblognorm). That was done by the same guy that wrote rsyslog. I'll pass this information on to him and see if there is an answer. > Hey, > > I?m having troubles setting Sagans dependencies up properly. > LIBESTR did run through properly. > LIBEE however is being a pain. > > I had quite a few troubles getting past the ./configure , it simply > couldn't find libestr(which was in /usr/lib) even when giving the proper > path on the commandline. > Now I'm past the configure and Make quits me with > make all-recursive > make[1]: Entering directory `/home/ferial/libee-0.1.0' > Making all in tests > make[2]: Entering directory `/home/ferial/libee-0.1.0/tests' > make[2]: Nothing to be done for `all'. > make[2]: Leaving directory `/home/ferial/libee-0.1.0/tests' > Making all in include > make[2]: Entering directory `/home/ferial/libee-0.1.0/include' > Making all in libee > make[3]: Entering directory `/home/ferial/libee-0.1.0/include/libee' > make[3]: Nothing to be done for `all'. > make[3]: Leaving directory `/home/ferial/libee-0.1.0/include/libee' > make[3]: Entering directory `/home/ferial/libee-0.1.0/include' > make[3]: Nothing to be done for `all-am'. > make[3]: Leaving directory `/home/ferial/libee-0.1.0/include' > make[2]: Leaving directory `/home/ferial/libee-0.1.0/include' > Making all in src > make[2]: Entering directory `/home/ferial/libee-0.1.0/src' > CCLD convert > /usr/bin/ld:/usr/lib/pkgconfig/libestr.pc: file format not recognized; > treating as linker script > /usr/bin/ld:/usr/lib/pkgconfig/libestr.pc:2: syntax error > collect2: ld returned 1 exit status > make[2]: *** [convert] Error 1 > make[2]: Leaving directory `/home/ferial/libee-0.1.0/src' > make[1]: *** [all-recursive] Error 1 > make[1]: Leaving directory `/home/ferial/libee-0.1.0' > make: *** [all] Error 2 > > Help appreciated > > regards ----- -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From rgerhards at hq.adiscon.com Mon Mar 21 14:44:09 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 21 Mar 2011 14:44:09 +0100 Subject: [Lognorm] libee compile issues. In-Reply-To: <20110321122619.GC25589@bundy.vistech.net> References: <20110321122619.GC25589@bundy.vistech.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDE48@GRFEXC.intern.adiscon.com> I got problems depending on --libdir and its helpers. At least on 64bit systems the defaults autotools use seem not to match the usual system defaults. Let me know if that helps. If not, we must try to reproduce the issue. Rainer > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink] > Sent: Monday, March 21, 2011 1:26 PM > To: lognorm at lists.adiscon.com > Subject: [Lognorm] libee compile issues. > > > Rainer, > > I got this e-mail a couple of days ago. I was wondering if you've > recieved any similar issues with building libee. > > > ---- > > Keep in mind, I'm not the author of libee (or libestr/liblognorm). That > was done by the same guy that wrote rsyslog. > I'll pass this information on to him and see if there is an answer. > > > Hey, > > > > I?m having troubles setting Sagans dependencies up properly. > > LIBESTR did run through properly. > > LIBEE however is being a pain. > > > > I had quite a few troubles getting past the ./configure , it simply > > couldn't find libestr(which was in /usr/lib) even when giving the > > proper path on the commandline. > > Now I'm past the configure and Make quits me with make all-recursive > > make[1]: Entering directory `/home/ferial/libee-0.1.0' > > Making all in tests > > make[2]: Entering directory `/home/ferial/libee-0.1.0/tests' > > make[2]: Nothing to be done for `all'. > > make[2]: Leaving directory `/home/ferial/libee-0.1.0/tests' > > Making all in include > > make[2]: Entering directory `/home/ferial/libee-0.1.0/include' > > Making all in libee > > make[3]: Entering directory `/home/ferial/libee-0.1.0/include/libee' > > make[3]: Nothing to be done for `all'. > > make[3]: Leaving directory `/home/ferial/libee-0.1.0/include/libee' > > make[3]: Entering directory `/home/ferial/libee-0.1.0/include' > > make[3]: Nothing to be done for `all-am'. > > make[3]: Leaving directory `/home/ferial/libee-0.1.0/include' > > make[2]: Leaving directory `/home/ferial/libee-0.1.0/include' > > Making all in src > > make[2]: Entering directory `/home/ferial/libee-0.1.0/src' > > CCLD convert > > /usr/bin/ld:/usr/lib/pkgconfig/libestr.pc: file format not recognized; > > treating as linker script > > /usr/bin/ld:/usr/lib/pkgconfig/libestr.pc:2: syntax error > > collect2: ld returned 1 exit status > > make[2]: *** [convert] Error 1 > > make[2]: Leaving directory `/home/ferial/libee-0.1.0/src' > > make[1]: *** [all-recursive] Error 1 > > make[1]: Leaving directory `/home/ferial/libee-0.1.0' > > make: *** [all] Error 2 > > > > Help appreciated > > > > regards > > ----- > > > -- > Champ Clark III | Softwink, Inc | 800-538-9357 x 101 > http://www.softwink.com > > GPG Key ID: 58A2A58F > Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it > wasn't for C, we'd be using BASI, PASAL and OBOL. From champ at softwink.com Mon Mar 21 14:47:21 2011 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Mon, 21 Mar 2011 09:47:21 -0400 Subject: [Lognorm] libee compile issues. In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDE48@GRFEXC.intern.adiscon.com> References: <20110321122619.GC25589@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DDE48@GRFEXC.intern.adiscon.com> Message-ID: <20110321134721.GB28360@bundy.vistech.net> On Mon, Mar 21, 2011 at 02:44:09PM +0100, Rainer Gerhards wrote: > I got problems depending on --libdir and its helpers. At least on 64bit > systems the defaults autotools use seem not to match the usual system > defaults. Let me know if that helps. If not, we must try to reproduce the > issue. > > Rainer Thanks.. I'll pass this information along. -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From champ at softwink.com Mon Mar 21 15:38:17 2011 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Mon, 21 Mar 2011 10:38:17 -0400 Subject: [Lognorm] liblognorm - segfault issue - Debian Wheezy In-Reply-To: <20110321122619.GC25589@bundy.vistech.net> References: <20110321122619.GC25589@bundy.vistech.net> Message-ID: <20110321143817.GB30874@bundy.vistech.net> Rainer, I received a report from a Sagan user that loading the normalization files in Sagan is causing a segfault. I've not been able to reproduce it myself and I'm actively using the cisco-normalize.rulebase file in production. I had him run Sagan through gdb, and this is the editted output. It seems to load the first normalization rulebase fine, and blows up on the cisco-normalize.rulebase. Program received signal SIGSEGV, Segmentation fault. 0xb7fa1673 in ln_buildPTree () from /usr/lib/liblognorm.so.0 (gdb) bt #0 0xb7fa1673 in ln_buildPTree () from /usr/lib/liblognorm.so.0 #1 0xb7fa21d9 in ln_sampRead () from /usr/lib/liblognorm.so.0 #2 0xb7fa01c3 in ln_loadSamples () from /usr/lib/liblognorm.so.0 #3 0x0804b4fc in main (argc=1, argv=0xbffffb74) at sagan.c:424 (gdb) This is line 424 in sagan.c (it's the ln_loadSamples line) ----- if (stat(liblognormtoloadstruct[i].filepath, &fileinfo)) sagan_log(1, "%s was not fonnd.", liblognormtoloadstruct[i].filepath); ln_loadSamples(ctx, liblognormtoloadstruct[i].filepath); ------ I've attached the cisco-normalize.rulebase file as well. Any ideas? I'm going to see if I can't somehow reproduce this. This was on a Debian Wheezy box. I'll find out if he built the libestr/libee/liblognorm himself or if he used the Debian package. Be back with more information shortly :) Thanks. -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From champ at softwink.com Mon Mar 21 16:11:36 2011 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Mon, 21 Mar 2011 11:11:36 -0400 Subject: [Lognorm] liblognorm - segfault issue - Debian Wheezy In-Reply-To: <20110321143817.GB30874@bundy.vistech.net> References: <20110321122619.GC25589@bundy.vistech.net> <20110321143817.GB30874@bundy.vistech.net> Message-ID: <20110321151136.GA1029@bundy.vistech.net> Sorry, I forgot to attach the cisco-normalize.rulebase. Here it is. I hope it doesnt get stripped. -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- # Sagan cisco.rulebase # Copyright (c) 2009-2011, Softwink, Inc. # All rights reserved. # # This file is used in conjunction with liblognorm. # # Please submit any custom rules or ideas to sagan-submit at softwink.com or the sagan-sigs mailing list # #************************************************************* # Redistribution and use in source and binary forms, with or without modification, are permitted provided that the # following conditions are met: # # * Redistributions of source code must retain the above copyright notice, this list of conditions and the following # disclaimer. # * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the # following disclaimer in the documentation and/or other materials provided with the distribution. # * Neither the name of the nor the names of its contributors may be used to endorse or promote products derived # from this software without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS AS IS AND ANY EXPRESS OR IMPLIED WARRANTIES, # INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE # DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR # SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, # WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE # USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. # #************************************************************* prefix= # # 1w3d: %SNMP-3-AUTHFAIL: Authentication failure for SNMP req from host 192.168.0.1 rule=: %uptime:word% %authfail:word% Authentication failure for SNMP req from host %src-ip:ipv4% # Access denied URL http://www.example.com/somethings.txt SRC 192.168.0.1 DEST 10.10.10.10 on interface inside rule=: Access denied URL %url:word% SRC %src-ip:ipv4% DEST %dst-ip:ipv4% on interface %interface:word% # Caused by WebVPN or IPSec # AAA user authentication Successful : server = 10.10.10.10 : user = domain\bob rule=: AAA user authentication Successful : server = %ip-src:ipv4% : user = %username:word% rule=: AAA user authentication Rejected : reason = %reason:word% : server = %ip-src:ipv4% : user = %username:word% # User authentication failed: Uname: timothy rule=: User authentication failed: Uname: %username:word% # Space at the end of this line! # %ASA-6-315011: SSH session from 192.168.0.1 on interface Outside2 for user "test" disconnected by SSH server, reason: "Internal error" (0x00) # SSH session from 10.20.10.200 on interface Outside2 for user "root" disconnected by SSH server, reason: "Internal error" (0x00) rule=: SSH session from %src-ip:ipv4% on interface %interface:word% for user %username:quoted-string% disconnected by SSH server, reason: %reason:quoted-string% %code:word% rule=: SSH session from %src-ip:ipv4% on interface %interface:word% for user %username:quoted-string% disconnected by SSH server, reason: %reason:quoted-string% %code:word% rule=: Configured from console by %tty:word:% (%ip:ipv4%) rule=: Authentication failure for %proto:word% req from host %ip:ipv4% rule=: Attempted to connect to %servname:word% from %ip:ipv4% -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From laanwj at gmail.com Mon Mar 21 18:59:53 2011 From: laanwj at gmail.com (Wladimir van der Laan) Date: Mon, 21 Mar 2011 18:59:53 +0100 Subject: [Lognorm] Identifying message types Message-ID: Hello, I have a question about the usage of lognorm. As I understand, the program extracts data fields from log messages in text format, by means of examples from a ruleset file. The output is represented as metadata key/value pairs. But as far as I can see, it outputs no identifier as to what kind of message the log line represents. For automated log processing, one would also need to identify the message, for example, as failed authentication, or dhcp request, etc. Am I overlooking something? Is it possible to add a message type field in a ruleset? Greetings, Wladimir -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan.kleindl at rlb-stmk.raiffeisen.at Tue Mar 22 08:55:40 2011 From: stefan.kleindl at rlb-stmk.raiffeisen.at (stefan.kleindl at rlb-stmk.raiffeisen.at) Date: Tue, 22 Mar 2011 08:55:40 +0100 Subject: [Lognorm] libee compile issues Message-ID: Taking this off champs hands since its my issue after all It's a 64bit RHEL 6.0 machine, Now I'm back to LIBESTR running through and LIBEE not passing configure :/ [root at xltest1204 libee-0.1.0]# ./configure --libdir=/usr/lib --includedir=/usr/include --prefix=/usr --prefix=/usr checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking for gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking for style of include used by make... GNU checking dependency style of gcc... gcc3 checking whether gcc and cc understand -c and -o together... yes checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking for a sed that does not truncate output... /bin/sed checking for grep that handles long lines and -e... /bin/grep checking for egrep... /bin/grep -E checking for fgrep... /bin/grep -F checking for ld used by gcc... /usr/bin/ld checking if the linker (/usr/bin/ld) is GNU ld... yes checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B checking the name lister (/usr/bin/nm -B) interface... BSD nm checking whether ln -s works... yes checking the maximum length of command line arguments... 1966080 checking whether the shell understands some XSI constructs... yes checking whether the shell understands "+="... yes checking for /usr/bin/ld option to reload object files... -r checking for objdump... objdump checking how to recognize dependent libraries... pass_all checking for ar... ar checking for strip... strip checking for ranlib... ranlib checking command to parse /usr/bin/nm -B output from gcc object... ok checking how to run the C preprocessor... gcc -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking for dlfcn.h... yes checking for objdir... .libs checking if gcc supports -fno-rtti -fno-exceptions... no checking for gcc option to produce PIC... -fPIC -DPIC checking if gcc PIC flag -fPIC -DPIC works... yes checking if gcc static flag -static works... no checking if gcc supports -c -o file.o... yes checking if gcc supports -c -o file.o... (cached) yes checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes checking whether -lc should be explicitly linked in... no checking dynamic linker characteristics... GNU/Linux ld.so checking how to hardcode library paths into programs... immediate checking whether stripping libraries is possible... yes checking if libtool supports shared libraries... yes checking whether to build shared libraries... yes checking whether to build static libraries... yes checking for xml2-config... yes checking for xmlReadFile in -lxml2... yes checking for stdlib.h... (cached) yes checking for GNU libc compatible malloc... yes checking for pkg-config... /usr/bin/pkg-config checking pkg-config is at least version 0.9.0... yes checking for LIBESTR... configure: error: Package requirements (libestr >= 0.0.0) were not met: No package 'libestr' found Consider adjusting the PKG_CONFIG_PATH environment variable if you installed software in a non-standard prefix. Alternatively, you may set the environment variables LIBESTR_CFLAGS and LIBESTR_LIBS to avoid the need to call pkg-config. See the pkg-config man page for more details. [root at xltest1204 lib]# cat pkgconfig/libestr.pc prefix=/usr exec_prefix=${prefix} libdir=/usr/lib includedir=/usr/include Name: libestr Description: some essentials for string processing Version: 0.1.0 Libs: -L${libdir} -lestr Cflags: -I${includedir} I?m open to suggestions, would like to see this baby working :) regards ---------------------------------------- Raiffeisen-Landesbank Steiermark AG, Graz, FN 264700 s, Landesgericht f?r Zivilrechtssachen Graz, DVR 0040495 Der Austausch von Nachrichten mit oa. Absender via E-Mail dient ausschliesslich Informationszwecken. Rechtsgeschaeftliche Erklaerungen duerfen ueber dieses Medium nicht ausgetauscht werden. Correspondence with a.m. sender via e-mail is only for information purposes. This medium is not to be used for the exchange of legally-binding communications. ---------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From rgerhards at hq.adiscon.com Tue Mar 22 10:34:19 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 22 Mar 2011 10:34:19 +0100 Subject: [Lognorm] libee compile issues In-Reply-To: References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDE55@GRFEXC.intern.adiscon.com> The install/build order is 1. libestr 2. libee 3. liblognorm They depend on each other in that order. Be sure to use the proper switches for directories especially on 64 bit systems. HTH Rainer > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > bounces at lists.adiscon.com] On Behalf Of stefan.kleindl at rlb- > stmk.raiffeisen.at > Sent: Tuesday, March 22, 2011 8:56 AM > To: lognorm at lists.adiscon.com > Subject: [Lognorm] libee compile issues > > Taking this off champs hands since its my issue after all > It's a 64bit RHEL 6.0 machine, > > Now I'm back to LIBESTR running through and LIBEE not passing configure > :/ > > [root at xltest1204 libee-0.1.0]# ./configure --libdir=/usr/lib -- > includedir=/usr/include --prefix=/usr --prefix=/usr > checking for a BSD-compatible install... /usr/bin/install -c > checking whether build environment is sane... yes > checking for a thread-safe mkdir -p... /bin/mkdir -p > checking for gawk... gawk > checking whether make sets $(MAKE)... yes > checking for gcc... gcc > checking whether the C compiler works... yes > checking for C compiler default output file name... a.out > checking for suffix of executables... > checking whether we are cross compiling... no > checking for suffix of object files... o > checking whether we are using the GNU C compiler... yes > checking whether gcc accepts -g... yes > checking for gcc option to accept ISO C89... none needed > checking for style of include used by make... GNU > checking dependency style of gcc... gcc3 > checking whether gcc and cc understand -c and -o together... yes > checking build system type... x86_64-unknown-linux-gnu > checking host system type... x86_64-unknown-linux-gnu > checking for a sed that does not truncate output... /bin/sed > checking for grep that handles long lines and -e... /bin/grep > checking for egrep... /bin/grep -E > checking for fgrep... /bin/grep -F > checking for ld used by gcc... /usr/bin/ld > checking if the linker (/usr/bin/ld) is GNU ld... yes > checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B > checking the name lister (/usr/bin/nm -B) interface... BSD nm > checking whether ln -s works... yes > checking the maximum length of command line arguments... 1966080 > checking whether the shell understands some XSI constructs... yes > checking whether the shell understands "+="... yes > checking for /usr/bin/ld option to reload object files... -r > checking for objdump... objdump > checking how to recognize dependent libraries... pass_all > checking for ar... ar > checking for strip... strip > checking for ranlib... ranlib > checking command to parse /usr/bin/nm -B output from gcc object... ok > checking how to run the C preprocessor... gcc -E > checking for ANSI C header files... yes > checking for sys/types.h... yes > checking for sys/stat.h... yes > checking for stdlib.h... yes > checking for string.h... yes > checking for memory.h... yes > checking for strings.h... yes > checking for inttypes.h... yes > checking for stdint.h... yes > checking for unistd.h... yes > checking for dlfcn.h... yes > checking for objdir... .libs > checking if gcc supports -fno-rtti -fno-exceptions... no > checking for gcc option to produce PIC... -fPIC -DPIC > checking if gcc PIC flag -fPIC -DPIC works... yes > checking if gcc static flag -static works... no > checking if gcc supports -c -o file.o... yes > checking if gcc supports -c -o file.o... (cached) yes > checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports > shared libraries... yes > checking whether -lc should be explicitly linked in... no > checking dynamic linker characteristics... GNU/Linux ld.so > checking how to hardcode library paths into programs... immediate > checking whether stripping libraries is possible... yes > checking if libtool supports shared libraries... yes > checking whether to build shared libraries... yes > checking whether to build static libraries... yes > checking for xml2-config... yes > checking for xmlReadFile in -lxml2... yes > checking for stdlib.h... (cached) yes > checking for GNU libc compatible malloc... yes > checking for pkg-config... /usr/bin/pkg-config > checking pkg-config is at least version 0.9.0... yes > checking for LIBESTR... configure: error: Package requirements (libestr > >= 0.0.0) were not met: > > No package 'libestr' found > > Consider adjusting the PKG_CONFIG_PATH environment variable if you > installed software in a non-standard prefix. > > Alternatively, you may set the environment variables LIBESTR_CFLAGS > and LIBESTR_LIBS to avoid the need to call pkg-config. > See the pkg-config man page for more details. > > > [root at xltest1204 lib]# cat pkgconfig/libestr.pc > prefix=/usr > exec_prefix=${prefix} > libdir=/usr/lib > includedir=/usr/include > > Name: libestr > Description: some essentials for string processing > Version: 0.1.0 > Libs: -L${libdir} -lestr > Cflags: -I${includedir} > > > I?m open to suggestions, would like to see this baby working :) > > regards > > ---------------------------------------- > Raiffeisen-Landesbank Steiermark AG, Graz, FN 264700 s, Landesgericht > f?r Zivilrechtssachen Graz, DVR 0040495 > > Der Austausch von Nachrichten mit oa. Absender via E-Mail dient > ausschliesslich Informationszwecken. Rechtsgeschaeftliche Erklaerungen > duerfen ueber dieses Medium nicht ausgetauscht werden. > Correspondence with a.m. sender via e-mail is only for information > purposes. This medium is not to be used for the exchange of legally- > binding communications. > ---------------------------------------- From rgerhards at hq.adiscon.com Tue Mar 22 10:37:49 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 22 Mar 2011 10:37:49 +0100 Subject: [Lognorm] Identifying message types In-Reply-To: References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDE56@GRFEXC.intern.adiscon.com> Hi Wladimir, This is a good question and you are abosultely right -- this is currently missing. In fact, the speace in front of the colon inside the rulebase is reserved for tags, which is the classification you are looking for. Liblognorm is in its infancy, though already quite useful in its current state. I have paused development a bit for two reasons: a) CEE needs to sort out some things -- I'd prefer to have some issues solved before continuing (and re-doing some work). b) devel prio -- right now I am working hard on getting a new stable v5 rsyslog out, and this is taking quite some toll The feature you are asking for is definitely on the today list, and I hope to be able to work more on liblognorm within the next couple of weeks (this year has been very busy - and will be - at least until mid-april). Rainer > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > bounces at lists.adiscon.com] On Behalf Of Wladimir van der Laan > Sent: Monday, March 21, 2011 7:00 PM > To: lognorm at lists.adiscon.com > Subject: [Lognorm] Identifying message types > > Hello, > > I have a question about the usage of lognorm. As I understand, the > program extracts data fields from log messages in text format, by means > of examples from a ruleset file. The output is represented as metadata > key/value pairs. > > But as far as I can see, it outputs no identifier as to what kind of > message the log line represents. For automated log processing, one > would also need to identify the message, for example, as failed > authentication, or dhcp request, etc. > > Am I overlooking something? Is it possible to add a message type field > in a ruleset? > > Greetings, > Wladimir > From stefan.kleindl at rlb-stmk.raiffeisen.at Tue Mar 22 11:10:32 2011 From: stefan.kleindl at rlb-stmk.raiffeisen.at (stefan.kleindl at rlb-stmk.raiffeisen.at) Date: Tue, 22 Mar 2011 11:10:32 +0100 Subject: [Lognorm] libee compile issues In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDE55@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DDE55@GRFEXC.intern.adiscon.com> Message-ID: Oh ye, I found the error - human side. :/ ./configure --libdir=/usr/lib64/ --includedir=/usr/include --prefix=/usr Changed it to /lib64/ and the troubles were gone - it seems. All 3 running through nicely now. Regards Von: rgerhards at hq.adiscon.com An: lognorm at lists.adiscon.com Datum: 22.03.2011 10:34 Betreff: Re: [Lognorm] libee compile issues Gesendet von: lognorm-bounces at lists.adiscon.com The install/build order is 1. libestr 2. libee 3. liblognorm They depend on each other in that order. Be sure to use the proper switches for directories especially on 64 bit systems. HTH Rainer > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > bounces at lists.adiscon.com] On Behalf Of stefan.kleindl at rlb- > stmk.raiffeisen.at > Sent: Tuesday, March 22, 2011 8:56 AM > To: lognorm at lists.adiscon.com > Subject: [Lognorm] libee compile issues > > Taking this off champs hands since its my issue after all > It's a 64bit RHEL 6.0 machine, > > Now I'm back to LIBESTR running through and LIBEE not passing configure > :/ > > [root at xltest1204 libee-0.1.0]# ./configure --libdir=/usr/lib -- > includedir=/usr/include --prefix=/usr --prefix=/usr > checking for a BSD-compatible install... /usr/bin/install -c > checking whether build environment is sane... yes > checking for a thread-safe mkdir -p... /bin/mkdir -p > checking for gawk... gawk > checking whether make sets $(MAKE)... yes > checking for gcc... gcc > checking whether the C compiler works... yes > checking for C compiler default output file name... a.out > checking for suffix of executables... > checking whether we are cross compiling... no > checking for suffix of object files... o > checking whether we are using the GNU C compiler... yes > checking whether gcc accepts -g... yes > checking for gcc option to accept ISO C89... none needed > checking for style of include used by make... GNU > checking dependency style of gcc... gcc3 > checking whether gcc and cc understand -c and -o together... yes > checking build system type... x86_64-unknown-linux-gnu > checking host system type... x86_64-unknown-linux-gnu > checking for a sed that does not truncate output... /bin/sed > checking for grep that handles long lines and -e... /bin/grep > checking for egrep... /bin/grep -E > checking for fgrep... /bin/grep -F > checking for ld used by gcc... /usr/bin/ld > checking if the linker (/usr/bin/ld) is GNU ld... yes > checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B > checking the name lister (/usr/bin/nm -B) interface... BSD nm > checking whether ln -s works... yes > checking the maximum length of command line arguments... 1966080 > checking whether the shell understands some XSI constructs... yes > checking whether the shell understands "+="... yes > checking for /usr/bin/ld option to reload object files... -r > checking for objdump... objdump > checking how to recognize dependent libraries... pass_all > checking for ar... ar > checking for strip... strip > checking for ranlib... ranlib > checking command to parse /usr/bin/nm -B output from gcc object... ok > checking how to run the C preprocessor... gcc -E > checking for ANSI C header files... yes > checking for sys/types.h... yes > checking for sys/stat.h... yes > checking for stdlib.h... yes > checking for string.h... yes > checking for memory.h... yes > checking for strings.h... yes > checking for inttypes.h... yes > checking for stdint.h... yes > checking for unistd.h... yes > checking for dlfcn.h... yes > checking for objdir... .libs > checking if gcc supports -fno-rtti -fno-exceptions... no > checking for gcc option to produce PIC... -fPIC -DPIC > checking if gcc PIC flag -fPIC -DPIC works... yes > checking if gcc static flag -static works... no > checking if gcc supports -c -o file.o... yes > checking if gcc supports -c -o file.o... (cached) yes > checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports > shared libraries... yes > checking whether -lc should be explicitly linked in... no > checking dynamic linker characteristics... GNU/Linux ld.so > checking how to hardcode library paths into programs... immediate > checking whether stripping libraries is possible... yes > checking if libtool supports shared libraries... yes > checking whether to build shared libraries... yes > checking whether to build static libraries... yes > checking for xml2-config... yes > checking for xmlReadFile in -lxml2... yes > checking for stdlib.h... (cached) yes > checking for GNU libc compatible malloc... yes > checking for pkg-config... /usr/bin/pkg-config > checking pkg-config is at least version 0.9.0... yes > checking for LIBESTR... configure: error: Package requirements (libestr > >= 0.0.0) were not met: > > No package 'libestr' found > > Consider adjusting the PKG_CONFIG_PATH environment variable if you > installed software in a non-standard prefix. > > Alternatively, you may set the environment variables LIBESTR_CFLAGS > and LIBESTR_LIBS to avoid the need to call pkg-config. > See the pkg-config man page for more details. > > > [root at xltest1204 lib]# cat pkgconfig/libestr.pc > prefix=/usr > exec_prefix=${prefix} > libdir=/usr/lib > includedir=/usr/include > > Name: libestr > Description: some essentials for string processing > Version: 0.1.0 > Libs: -L${libdir} -lestr > Cflags: -I${includedir} > > > I?m open to suggestions, would like to see this baby working :) > > regards > > ---------------------------------------- > Raiffeisen-Landesbank Steiermark AG, Graz, FN 264700 s, Landesgericht > f?r Zivilrechtssachen Graz, DVR 0040495 > > Der Austausch von Nachrichten mit oa. Absender via E-Mail dient > ausschliesslich Informationszwecken. Rechtsgeschaeftliche Erklaerungen > duerfen ueber dieses Medium nicht ausgetauscht werden. > Correspondence with a.m. sender via e-mail is only for information > purposes. This medium is not to be used for the exchange of legally- > binding communications. > ---------------------------------------- _______________________________________________ Lognorm mailing list Lognorm at lists.adiscon.com http://lists.adiscon.net/mailman/listinfo/lognorm ---------------------------------------- Raiffeisen-Landesbank Steiermark AG, Graz, FN 264700 s, Landesgericht f?r Zivilrechtssachen Graz, DVR 0040495 Der Austausch von Nachrichten mit oa. Absender via E-Mail dient ausschliesslich Informationszwecken. Rechtsgeschaeftliche Erklaerungen duerfen ueber dieses Medium nicht ausgetauscht werden. Correspondence with a.m. sender via e-mail is only for information purposes. This medium is not to be used for the exchange of legally-binding communications. ---------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From rgerhards at hq.adiscon.com Tue Mar 22 11:47:23 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 22 Mar 2011 11:47:23 +0100 Subject: [Lognorm] libee compile issues In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71DDE55@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDE59@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > bounces at lists.adiscon.com] On Behalf Of stefan.kleindl at rlb- > stmk.raiffeisen.at > Sent: Tuesday, March 22, 2011 11:11 AM > To: lognorm at lists.adiscon.com > Subject: Re: [Lognorm] libee compile issues > > Oh ye, I found the error - human side. :/ ./configure --libdir=/usr/lib64/ -- > includedir=/usr/include --prefix=/usr Changed it to /lib64/ and the troubles > were gone - it seems. Yeah, that was along the lines I thought. Glad it worked out :) Rainer > > All 3 running through nicely now. > Regards > > Inactive hide details for rgerhards---22.03.2011 10:34:22---The install/build > order is 1. libestrrgerhards---22.03.2011 10:34:22---The install/build order is 1. > libestr > > Von: rgerhards at hq.adiscon.com > An: lognorm at lists.adiscon.com > Datum: 22.03.2011 10:34 > Betreff: Re: [Lognorm] libee compile issues Gesendet von: lognorm- > bounces at lists.adiscon.com > > ________________________________ > > > > > The install/build order is > > 1. libestr > 2. libee > 3. liblognorm > > They depend on each other in that order. Be sure to use the proper switches > for directories especially on 64 bit systems. > > HTH > Rainer > > > -----Original Message----- > > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > > bounces at lists.adiscon.com] On Behalf Of stefan.kleindl at rlb- > > stmk.raiffeisen.at > > Sent: Tuesday, March 22, 2011 8:56 AM > > To: lognorm at lists.adiscon.com > > Subject: [Lognorm] libee compile issues > > > > Taking this off champs hands since its my issue after all It's a 64bit > > RHEL 6.0 machine, > > > > Now I'm back to LIBESTR running through and LIBEE not passing > > configure :/ > > > > [root at xltest1204 libee-0.1.0]# ./configure --libdir=/usr/lib -- > > includedir=/usr/include --prefix=/usr --prefix=/usr checking for a > > BSD-compatible install... /usr/bin/install -c checking whether build > > environment is sane... yes checking for a thread-safe mkdir -p... > > /bin/mkdir -p checking for gawk... gawk checking whether make sets > > $(MAKE)... yes checking for gcc... gcc checking whether the C compiler > > works... yes checking for C compiler default output file name... a.out > > checking for suffix of executables... > > checking whether we are cross compiling... no checking for suffix of > > object files... o checking whether we are using the GNU C compiler... > > yes checking whether gcc accepts -g... yes checking for gcc option to > > accept ISO C89... none needed checking for style of include used by > > make... GNU checking dependency style of gcc... gcc3 checking whether > > gcc and cc understand -c and -o together... yes checking build system > > type... x86_64-unknown-linux-gnu checking host system type... > > x86_64-unknown-linux-gnu checking for a sed that does not truncate > > output... /bin/sed checking for grep that handles long lines and -e... > > /bin/grep checking for egrep... /bin/grep -E checking for fgrep... > > /bin/grep -F checking for ld used by gcc... /usr/bin/ld checking if > > the linker (/usr/bin/ld) is GNU ld... yes checking for BSD- or > > MS-compatible name lister (nm)... /usr/bin/nm -B checking the name > > lister (/usr/bin/nm -B) interface... BSD nm checking whether ln -s > > works... yes checking the maximum length of command line arguments... > > 1966080 checking whether the shell understands some XSI constructs... > > yes checking whether the shell understands "+="... yes checking for > > /usr/bin/ld option to reload object files... -r checking for > > objdump... objdump checking how to recognize dependent libraries... > > pass_all checking for ar... ar checking for strip... strip checking > > for ranlib... ranlib checking command to parse /usr/bin/nm -B output > > from gcc object... ok checking how to run the C preprocessor... gcc -E > > checking for ANSI C header files... yes checking for sys/types.h... > > yes checking for sys/stat.h... yes checking for stdlib.h... yes > > checking for string.h... yes checking for memory.h... yes checking for > > strings.h... yes checking for inttypes.h... yes checking for > > stdint.h... yes checking for unistd.h... yes checking for dlfcn.h... > > yes checking for objdir... .libs checking if gcc supports -fno-rtti > > -fno-exceptions... no checking for gcc option to produce PIC... -fPIC > > -DPIC checking if gcc PIC flag -fPIC -DPIC works... yes checking if > > gcc static flag -static works... no checking if gcc supports -c -o > > file.o... yes checking if gcc supports -c -o file.o... (cached) yes > > checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports > > shared libraries... yes checking whether -lc should be explicitly > > linked in... no checking dynamic linker characteristics... GNU/Linux > > ld.so checking how to hardcode library paths into programs... > > immediate checking whether stripping libraries is possible... yes > > checking if libtool supports shared libraries... yes checking whether > > to build shared libraries... yes checking whether to build static > > libraries... yes checking for xml2-config... yes checking for > > xmlReadFile in -lxml2... yes checking for stdlib.h... (cached) yes > > checking for GNU libc compatible malloc... yes checking for > > pkg-config... /usr/bin/pkg-config checking pkg-config is at least > > version 0.9.0... yes checking for LIBESTR... configure: error: Package > > requirements (libestr > > >= 0.0.0) were not met: > > > > No package 'libestr' found > > > > Consider adjusting the PKG_CONFIG_PATH environment variable if you > > installed software in a non-standard prefix. > > > > Alternatively, you may set the environment variables LIBESTR_CFLAGS > > and LIBESTR_LIBS to avoid the need to call pkg-config. > > See the pkg-config man page for more details. > > > > > > [root at xltest1204 lib]# cat pkgconfig/libestr.pc prefix=/usr > > exec_prefix=${prefix} libdir=/usr/lib includedir=/usr/include > > > > Name: libestr > > Description: some essentials for string processing > > Version: 0.1.0 > > Libs: -L${libdir} -lestr > > Cflags: -I${includedir} > > > > > > I?m open to suggestions, would like to see this baby working :) > > > > regards > > > > ---------------------------------------- > > Raiffeisen-Landesbank Steiermark AG, Graz, FN 264700 s, Landesgericht > > f?r Zivilrechtssachen Graz, DVR 0040495 > > > > Der Austausch von Nachrichten mit oa. Absender via E-Mail dient > > ausschliesslich Informationszwecken. Rechtsgeschaeftliche Erklaerungen > > duerfen ueber dieses Medium nicht ausgetauscht werden. > > Correspondence with a.m. sender via e-mail is only for information > > purposes. This medium is not to be used for the exchange of legally- > > binding communications. > > ---------------------------------------- > > _______________________________________________ > Lognorm mailing list > Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm > > > > ---------------------------------------- > Raiffeisen-Landesbank Steiermark AG, Graz, FN 264700 s, Landesgericht f?r > Zivilrechtssachen Graz, DVR 0040495 > > Der Austausch von Nachrichten mit oa. Absender via E-Mail dient > ausschliesslich Informationszwecken. Rechtsgeschaeftliche Erklaerungen > duerfen ueber dieses Medium nicht ausgetauscht werden. > Correspondence with a.m. sender via e-mail is only for information purposes. > This medium is not to be used for the exchange of legally-binding > communications. > ---------------------------------------- From laanwj at gmail.com Tue Mar 22 12:33:02 2011 From: laanwj at gmail.com (Wladimir van der Laan) Date: Tue, 22 Mar 2011 12:33:02 +0100 Subject: [Lognorm] Identifying message types In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDE56@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DDE56@GRFEXC.intern.adiscon.com> Message-ID: Hello Rainer, Thanks for the explanation. Looks like I was right in my feeling that this was missing. I understand your rationale to wait for CEE on this, though. I read their spec, and they propose that the identification of a message includes object, action and status. But they haven't defined exactly what these should be, neither do they give any examples. They still have quite a lot of definition work to to. Hopefully, it won't take too long, a standard for logging is very badly needed, and the longer it takes, the more developers will yet again come up with their own solutions. I'm currently classifying all kinds of events in Zenoss Core, and realized that when I was defining regexp patterns I could just as well tell it how to extract out the interesting information for analysis and more useful presentation. Which is how I got to this project. Wladimir BTW: great work on rsyslog. On Tue, Mar 22, 2011 at 10:37 AM, Rainer Gerhards wrote: > Hi Wladimir, > > This is a good question and you are abosultely right -- this is currently > missing. In fact, the speace in front of the colon inside the rulebase is > reserved for tags, which is the classification you are looking for. > Liblognorm is in its infancy, though already quite useful in its current > state. I have paused development a bit for two reasons: > > a) CEE needs to sort out some things -- I'd prefer to have some issues > solved > before continuing (and re-doing some work). > b) devel prio -- right now I am working hard on getting a new stable v5 > rsyslog out, and this is taking quite some toll > > The feature you are asking for is definitely on the today list, and I hope > to > be able to work more on liblognorm within the next couple of weeks (this > year > has been very busy - and will be - at least until mid-april). > > Rainer > > > -----Original Message----- > > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > > bounces at lists.adiscon.com] On Behalf Of Wladimir van der Laan > > Sent: Monday, March 21, 2011 7:00 PM > > To: lognorm at lists.adiscon.com > > Subject: [Lognorm] Identifying message types > > > > Hello, > > > > I have a question about the usage of lognorm. As I understand, the > > program extracts data fields from log messages in text format, by means > > of examples from a ruleset file. The output is represented as metadata > > key/value pairs. > > > > But as far as I can see, it outputs no identifier as to what kind of > > message the log line represents. For automated log processing, one > > would also need to identify the message, for example, as failed > > authentication, or dhcp request, etc. > > > > Am I overlooking something? Is it possible to add a message type field > > in a ruleset? > > > > Greetings, > > Wladimir > > > > _______________________________________________ > Lognorm mailing list > Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rgerhards at hq.adiscon.com Tue Mar 22 12:43:48 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 22 Mar 2011 12:43:48 +0100 Subject: [Lognorm] Identifying message types In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71DDE56@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDE60@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > bounces at lists.adiscon.com] On Behalf Of Wladimir van der Laan > Sent: Tuesday, March 22, 2011 12:33 PM > To: lognorm > Subject: Re: [Lognorm] Identifying message types > > Hello Rainer, > > Thanks for the explanation. Looks like I was right in my feeling that this was > missing. > > I understand your rationale to wait for CEE on this, though. I read their spec, > and they propose that the identification of a message includes object, action > and status. But they haven't defined exactly what these should be, neither > do they give any examples. At the root, each of OAS (object, action, status) is a tag, and there is a semantical net that ties tags together. Think of it like a tag cloud. > > They still have quite a lot of definition work to to. Hopefully, it won't take too > long, a standard for logging is very badly needed, and the longer it takes, the > more developers will yet again come up with their own solutions. It's moving, albeit public visibility is slow. I am on the CEE editorial board, so at least I know when the risk to do further coding is bearable (note that this does not imply anything on consensus within CEE -- motivation to code may have various non CEE-aspects). > > I'm currently classifying all kinds of events in Zenoss Core, and realized that > when I was defining regexp patterns I could just as well tell it how to extract > out the interesting information for analysis and more useful presentation. > Which is how I got to this project. Yeah, I think it is a pretty universal requirement. That's the reason I put it into a lib rather than just the rsyslog engine. As soon as my workload ceases a bit, I think I'll at least provide a quick glimpse at the tags. Rainer > > Wladimir > > BTW: great work on rsyslog. > > > On Tue, Mar 22, 2011 at 10:37 AM, Rainer Gerhards > wrote: > > > Hi Wladimir, > > This is a good question and you are abosultely right -- this is currently > missing. In fact, the speace in front of the colon inside the rulebase is > reserved for tags, which is the classification you are looking for. > Liblognorm is in its infancy, though already quite useful in its current > state. I have paused development a bit for two reasons: > > a) CEE needs to sort out some things -- I'd prefer to have some issues > solved > before continuing (and re-doing some work). > b) devel prio -- right now I am working hard on getting a new stable > v5 > rsyslog out, and this is taking quite some toll > > The feature you are asking for is definitely on the today list, and I > hope to > be able to work more on liblognorm within the next couple of weeks > (this year > has been very busy - and will be - at least until mid-april). > > Rainer > > > > -----Original Message----- > > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > > bounces at lists.adiscon.com] On Behalf Of Wladimir van der Laan > > Sent: Monday, March 21, 2011 7:00 PM > > To: lognorm at lists.adiscon.com > > Subject: [Lognorm] Identifying message types > > > > Hello, > > > > I have a question about the usage of lognorm. As I understand, the > > program extracts data fields from log messages in text format, by > means > > of examples from a ruleset file. The output is represented as > metadata > > key/value pairs. > > > > But as far as I can see, it outputs no identifier as to what kind of > > message the log line represents. For automated log processing, one > > would also need to identify the message, for example, as failed > > authentication, or dhcp request, etc. > > > > Am I overlooking something? Is it possible to add a message type > field > > in a ruleset? > > > > Greetings, > > Wladimir > > > > > _______________________________________________ > Lognorm mailing list > Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm > > From rgerhards at hq.adiscon.com Wed Mar 23 17:23:03 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 23 Mar 2011 17:23:03 +0100 Subject: [Lognorm] liblognorm - segfault issue - Debian Wheezy In-Reply-To: <20110321151136.GA1029@bundy.vistech.net> References: <20110321122619.GC25589@bundy.vistech.net><20110321143817.GB30874@bundy.vistech.net> <20110321151136.GA1029@bundy.vistech.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDE80@GRFEXC.intern.adiscon.com> Sorry, I am at a conference starting tomorrow and needed to prepare some things, so I couldn't look into the report. I have asked Tom to try to reproduce the issue in our lab. If that succeeds, I can probably look at a fix Monday :) Rainer > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink] > Sent: Monday, March 21, 2011 4:12 PM > To: lognorm at lists.adiscon.com > Subject: Re: [Lognorm] liblognorm - segfault issue - Debian Wheezy > > > Sorry, I forgot to attach the cisco-normalize.rulebase. Here > it is. I hope it doesnt get stripped. > -- > Champ Clark III | Softwink, Inc | 800-538-9357 x 101 > http://www.softwink.com > > GPG Key ID: 58A2A58F > Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F > If it wasn't for C, we'd be using BASI, PASAL and OBOL. From tbergfeld at hq.adiscon.com Fri Mar 25 15:17:04 2011 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Fri, 25 Mar 2011 15:17:04 +0100 Subject: [Lognorm] liblognorm - segfault issue - Debian Wheezy In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDE80@GRFEXC.intern.adiscon.com> References: <20110321122619.GC25589@bundy.vistech.net><20110321143817.GB30874@bundy.vistech.net><20110321151136.GA1029@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DDE80@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDE87@GRFEXC.intern.adiscon.com> Hi Champ, I was able to reproduce the issue. I think Rainer will have a look at it and contact you ASAP. Tom -----Urspr?ngliche Nachricht----- Von: lognorm-bounces at lists.adiscon.com [mailto:lognorm-bounces at lists.adiscon.com] Im Auftrag von Rainer Gerhards Gesendet: Mittwoch, 23. M?rz 2011 17:23 An: lognorm Betreff: Re: [Lognorm] liblognorm - segfault issue - Debian Wheezy Sorry, I am at a conference starting tomorrow and needed to prepare some things, so I couldn't look into the report. I have asked Tom to try to reproduce the issue in our lab. If that succeeds, I can probably look at a fix Monday :) Rainer > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink] > Sent: Monday, March 21, 2011 4:12 PM > To: lognorm at lists.adiscon.com > Subject: Re: [Lognorm] liblognorm - segfault issue - Debian Wheezy > > > Sorry, I forgot to attach the cisco-normalize.rulebase. Here > it is. I hope it doesnt get stripped. > -- > Champ Clark III | Softwink, Inc | 800-538-9357 x 101 > http://www.softwink.com > > GPG Key ID: 58A2A58F > Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F > If it wasn't for C, we'd be using BASI, PASAL and OBOL. _______________________________________________ Lognorm mailing list Lognorm at lists.adiscon.com http://lists.adiscon.net/mailman/listinfo/lognorm From champ at softwink.com Fri Mar 25 16:52:56 2011 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Fri, 25 Mar 2011 11:52:56 -0400 Subject: [Lognorm] liblognorm - segfault issue - Debian Wheezy In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDE87@GRFEXC.intern.adiscon.com> References: <20110321122619.GC25589@bundy.vistech.net> <20110321143817.GB30874@bundy.vistech.net> <20110321151136.GA1029@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DDE80@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDE87@GRFEXC.intern.adiscon.com> Message-ID: <20110325155256.GA20248@bundy.vistech.net> On Fri, Mar 25, 2011 at 03:17:04PM +0100, Tom Bergfeld wrote: > Hi Champ, > I was able to reproduce the issue. I think Rainer will have a look at it and > contact you ASAP. Thanks for the info. I'll pass it on. -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From rgerhards at hq.adiscon.com Thu Mar 31 12:46:34 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 31 Mar 2011 12:46:34 +0200 Subject: [Lognorm] liblognorm - segfault issue - Debian Wheezy In-Reply-To: <20110321143817.GB30874@bundy.vistech.net> References: <20110321122619.GC25589@bundy.vistech.net> <20110321143817.GB30874@bundy.vistech.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDEF1@GRFEXC.intern.adiscon.com> Champ, Tom had reproduced the issue with Sagan and not with liblognorm tools themselves (as I had expected). So I began the analysis new this morning. It looks like the liblognorm tools do not have any issue loading the rulebase. HOWEVER, I accidently tried an older version of it on one machine, and that one immediately blew up. I wonder if this could be the cause of the problem (too-old libs). But now it comes handy that Tom could reproduce with Sagan. I'll check what he has done and uses. If it is not an too-old version, and so it appears only under Sagan, we need to join forces in order to debug it. Will let you know what I find out! Rainer > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink] > Sent: Monday, March 21, 2011 3:38 PM > To: lognorm at lists.adiscon.com > Subject: [Lognorm] liblognorm - segfault issue - Debian Wheezy > > > Rainer, > > I received a report from a Sagan user that loading the normalization > files in Sagan is causing a segfault. I've not been able to reproduce it myself > and I'm actively using the cisco-normalize.rulebase file in production. I had > him run Sagan through gdb, and this is the editted output. > > It seems to load the first normalization rulebase fine, and blows up > on the cisco-normalize.rulebase. > > Program received signal SIGSEGV, Segmentation fault. > 0xb7fa1673 in ln_buildPTree () from /usr/lib/liblognorm.so.0 > (gdb) bt > #0 0xb7fa1673 in ln_buildPTree () from /usr/lib/liblognorm.so.0 > #1 0xb7fa21d9 in ln_sampRead () from /usr/lib/liblognorm.so.0 > #2 0xb7fa01c3 in ln_loadSamples () from /usr/lib/liblognorm.so.0 > #3 0x0804b4fc in main (argc=1, argv=0xbffffb74) at sagan.c:424 > (gdb) > > > This is line 424 in sagan.c (it's the ln_loadSamples line) > > ----- > if (stat(liblognormtoloadstruct[i].filepath, &fileinfo)) sagan_log(1, "%s was > not fonnd.", liblognormtoloadstruct[i].filepath); > ln_loadSamples(ctx, liblognormtoloadstruct[i].filepath); > ------ > > I've attached the cisco-normalize.rulebase file as well. Any > ideas? I'm going to see if I can't somehow reproduce this. This was on > a Debian Wheezy box. I'll find out if he built the > libestr/libee/liblognorm himself or if he used the Debian package. > > Be back with more information shortly :) > > Thanks. > -- > Champ Clark III | Softwink, Inc | 800-538-9357 x 101 > http://www.softwink.com > > GPG Key ID: 58A2A58F > Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it > wasn't for C, we'd be using BASI, PASAL and OBOL. From rgerhards at hq.adiscon.com Thu Mar 31 16:10:02 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 31 Mar 2011 16:10:02 +0200 Subject: [Lognorm] liblognorm - segfault issue - Debian Wheezy In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDEF1@GRFEXC.intern.adiscon.com> References: <20110321122619.GC25589@bundy.vistech.net><20110321143817.GB30874@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DDEF1@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDEF6@GRFEXC.intern.adiscon.com> OK, this is the problem: Sagan - for good reason ;) - uses the most recent devel branch of liblognorm, e.g. with "quoted-string" support. However, this is only available in git. The currently released 0.1.0 does not support it AND aborts if it encounters it (instead of gracefully erroring out). So the quick cure is to use the git versions of the libs. Of course, I'll see that I do a couple of fresh releases, hopefully tomorrow ;) I have not completed the analysis 100%, but what I said is the cause with 99.9% probability. Will check the rest and post if it makes a difference (but only then). Rainer > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Thursday, March 31, 2011 12:47 PM > To: lognorm > Subject: Re: [Lognorm] liblognorm - segfault issue - Debian Wheezy > > Champ, > > Tom had reproduced the issue with Sagan and not with liblognorm tools > themselves (as I had expected). So I began the analysis new this morning. It > looks like the liblognorm tools do not have any issue loading the rulebase. > HOWEVER, I accidently tried an older version of it on one machine, and that > one immediately blew up. I wonder if this could be the cause of the problem > (too-old libs). But now it comes handy that Tom could reproduce with Sagan. > I'll check what he has done and uses. If it is not an too-old version, and so it > appears only under Sagan, we need to join forces in order to debug it. > Will let you know what I find out! > > Rainer > > > -----Original Message----- > > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink] > > Sent: Monday, March 21, 2011 3:38 PM > > To: lognorm at lists.adiscon.com > > Subject: [Lognorm] liblognorm - segfault issue - Debian Wheezy > > > > > > Rainer, > > > > I received a report from a Sagan user that loading the normalization > > files in Sagan is causing a segfault. I've not been able to reproduce > > it > myself > > and I'm actively using the cisco-normalize.rulebase file in > > production. I > had > > him run Sagan through gdb, and this is the editted output. > > > > It seems to load the first normalization rulebase fine, and blows up > > on the cisco-normalize.rulebase. > > > > Program received signal SIGSEGV, Segmentation fault. > > 0xb7fa1673 in ln_buildPTree () from /usr/lib/liblognorm.so.0 > > (gdb) bt > > #0 0xb7fa1673 in ln_buildPTree () from /usr/lib/liblognorm.so.0 > > #1 0xb7fa21d9 in ln_sampRead () from /usr/lib/liblognorm.so.0 > > #2 0xb7fa01c3 in ln_loadSamples () from /usr/lib/liblognorm.so.0 > > #3 0x0804b4fc in main (argc=1, argv=0xbffffb74) at sagan.c:424 > > (gdb) > > > > > > This is line 424 in sagan.c (it's the ln_loadSamples line) > > > > ----- > > if (stat(liblognormtoloadstruct[i].filepath, &fileinfo)) sagan_log(1, > > "%s > was > > not fonnd.", liblognormtoloadstruct[i].filepath); > > ln_loadSamples(ctx, liblognormtoloadstruct[i].filepath); > > ------ > > > > I've attached the cisco-normalize.rulebase file as well. Any > > ideas? I'm going to see if I can't somehow reproduce this. This was on > > a Debian Wheezy box. I'll find out if he built the > > libestr/libee/liblognorm himself or if he used the Debian package. > > > > Be back with more information shortly :) > > > > Thanks. > > -- > > Champ Clark III | Softwink, Inc | 800-538-9357 x 101 > > http://www.softwink.com > > > > GPG Key ID: 58A2A58F > > Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F > > If it wasn't for C, we'd be using BASI, PASAL and OBOL. > _______________________________________________ > Lognorm mailing list > Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm From champ at softwink.com Thu Mar 31 18:04:52 2011 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Thu, 31 Mar 2011 12:04:52 -0400 Subject: [Lognorm] liblognorm - segfault issue - Debian Wheezy In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDEF6@GRFEXC.intern.adiscon.com> References: <20110321122619.GC25589@bundy.vistech.net> <20110321143817.GB30874@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DDEF1@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDEF6@GRFEXC.intern.adiscon.com> Message-ID: <20110331160452.GA12113@bundy.vistech.net> On Thu, Mar 31, 2011 at 04:10:02PM +0200, Rainer Gerhards wrote: > OK, this is the problem: > > Sagan - for good reason ;) - uses the most recent devel branch of liblognorm, > e.g. with "quoted-string" support. However, this is only available in git. > The currently released 0.1.0 does not support it AND aborts if it encounters > it (instead of gracefully erroring out). So the quick cure is to use the git > versions of the libs. Of course, I'll see that I do a couple of fresh > releases, hopefully tomorrow ;) > > I have not completed the analysis 100%, but what I said is the cause with > 99.9% probability. Will check the rest and post if it makes a difference (but > only then). Thank you very much Rainer/Tom for looking into this. What I'll do is make a quick "git" package for the user to compile/test with. I'll relay this information of to them. Thanks very much. -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From rgerhards at hq.adiscon.com Thu Mar 31 20:01:31 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 31 Mar 2011 20:01:31 +0200 Subject: [Lognorm] liblognorm - segfault issue - Debian Wheezy In-Reply-To: <20110331160452.GA12113@bundy.vistech.net> References: <20110321122619.GC25589@bundy.vistech.net><20110321143817.GB30874@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DDEF1@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDEF6@GRFEXC.intern.adiscon.com> <20110331160452.GA12113@bundy.vistech.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDEFF@GRFEXC.intern.adiscon.com> I'll see that I get new releases out tomorrow. So it may be worth waiting. I have also good news: as it looks, I have more time to look at liblognorm and friends in April/May. I'll focus on things that are not too CEE-related (and thus unaffected by potential changes). Lot's of cool stuff on my mind ;) Let's hope I really get that timeslot... Rainer > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink] > Sent: Thursday, March 31, 2011 6:05 PM > To: lognorm > Subject: Re: [Lognorm] liblognorm - segfault issue - Debian Wheezy > > On Thu, Mar 31, 2011 at 04:10:02PM +0200, Rainer Gerhards wrote: > > OK, this is the problem: > > > > Sagan - for good reason ;) - uses the most recent devel branch of > liblognorm, > > e.g. with "quoted-string" support. However, this is only available in > git. > > The currently released 0.1.0 does not support it AND aborts if it > encounters > > it (instead of gracefully erroring out). So the quick cure is to use > the git > > versions of the libs. Of course, I'll see that I do a couple of fresh > > releases, hopefully tomorrow ;) > > > > I have not completed the analysis 100%, but what I said is the cause > with > > 99.9% probability. Will check the rest and post if it makes a > difference (but > > only then). > > Thank you very much Rainer/Tom for looking into this. What I'll > do is make a quick "git" package for the user to compile/test with. > I'll relay this information of to them. Thanks very much. > > -- > Champ Clark III | Softwink, Inc | 800-538-9357 x 101 > http://www.softwink.com > > GPG Key ID: 58A2A58F > Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F > If it wasn't for C, we'd be using BASI, PASAL and OBOL. From champ at softwink.com Thu Mar 31 21:39:52 2011 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Thu, 31 Mar 2011 15:39:52 -0400 Subject: [Lognorm] liblognorm - segfault issue - Debian Wheezy In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDEFF@GRFEXC.intern.adiscon.com> References: <20110321122619.GC25589@bundy.vistech.net> <20110321143817.GB30874@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DDEF1@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDEF6@GRFEXC.intern.adiscon.com> <20110331160452.GA12113@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DDEFF@GRFEXC.intern.adiscon.com> Message-ID: <20110331193952.GA20205@bundy.vistech.net> On Thu, Mar 31, 2011 at 08:01:31PM +0200, Rainer Gerhards wrote: > I'll see that I get new releases out tomorrow. So it may be worth waiting. > > I have also good news: as it looks, I have more time to look at liblognorm > and friends in April/May. I'll focus on things that are not too CEE-related > (and thus unaffected by potential changes). Lot's of cool stuff on my mind ;) > Let's hope I really get that timeslot... Woo! That's great to hear. I've pass along the instruction to the Sagan user to grab the libraries via 'git'. I think he'll be able to handle it, but if not, I'll throw up a copy for him in a private directory and/or wait till a new release is done :) -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: