[Lognorm] Identifying message types

[Rainer Gerhards]

> -----Original Message-----
> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> bounces at lists.adiscon.com] On Behalf Of Wladimir van der Laan
> Sent: Wednesday, April 06, 2011 3:27 PM
> To: lognorm
> Subject: Re: [Lognorm] Identifying message types
> 
> Awesome. I think the tag-based approach is very good: it allows for
> matching events that are, for example either ssh, login, or fail or a
> combination of them. 

Yes, that is exactly the idea. And it is an idea that comes from CEE and is
*not* invented by me (just to make sure we have proper credits ;)).

> This will be very convenient with a database
> backend such as MongoDB which has a built in query for 'give me the
> records with this and this tag'.
> 

It looks like I really should have a look into MongoDB...

Rainer

> Wladimir
> 
> 
> On Wed, Apr 6, 2011 at 3:09 PM, Rainer Gerhards
> <rgerhards at hq.adiscon.com> wrote:
> 
> 
> 	It's finally done:
> 
> 	http://blog.gerhards.net/2011/04/log-classification-with-
> liblognorm.html
> 
> 	This is based on some older CEE ideas and not necessarily inline
> with what
> 	comes up. Also, I can think of a couple of more things that would
> be good to
> 	add. But at least we now have the core functionality.
> 
> 	Feedback, as usual, appreciated. Official release will follow
> shortly, either
> 	today or tomorrow. Everything already available via git.
> 
> 
> 	Rainer
> 
> 	> -----Original Message-----
> 	> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> 	> bounces at lists.adiscon.com] On Behalf Of Wladimir van der Laan
> 
> 	> Sent: Tuesday, March 22, 2011 12:33 PM
> 	> To: lognorm
> 
> 	> Subject: Re: [Lognorm] Identifying message types
> 	>
> 	> Hello Rainer,
> 	>
> 	> Thanks for the explanation. Looks like I was right in my
> feeling that this
> 	was
> 	> missing.
> 	>
> 	> I understand your rationale to wait for CEE on this, though. I
> read their
> 	spec,
> 	> and they propose that the identification of a message includes
> object,
> 	action
> 	> and status. But they haven't defined exactly what these should
> be, neither
> 	> do they give any examples.
> 	>
> 	> They still have quite a lot of definition work to to.
> Hopefully, it won't
> 	take too
> 	> long, a standard for logging is very badly needed, and the
> longer it takes,
> 	the
> 	> more developers will yet again come up with their own
> solutions.
> 	>
> 	> I'm currently classifying all kinds of events in Zenoss Core,
> and realized
> 	that
> 	> when I was defining regexp patterns I could just as well tell
> it how to
> 	extract
> 	> out the interesting information for analysis and more useful
> presentation.
> 	> Which is how I got to this project.
> 	>
> 	> Wladimir
> 	>
> 	> BTW: great work on rsyslog.
> 	>
> 	>
> 	> On Tue, Mar 22, 2011 at 10:37 AM, Rainer Gerhards
> 	> <rgerhards at hq.adiscon.com> wrote:
> 	>
> 	>
> 	>       Hi Wladimir,
> 	>
> 	>       This is a good question and you are abosultely right --
> this is
> 	currently
> 	>       missing. In fact, the speace in front of the colon inside
> the
> 	rulebase is
> 	>       reserved for tags, which is the classification you are
> looking for.
> 	>       Liblognorm is in its infancy, though already quite useful
> in its
> 	current
> 	>       state. I have paused development a bit for two reasons:
> 	>
> 	>       a) CEE needs to sort out some things -- I'd prefer to
> have some
> 	issues
> 	> solved
> 	>       before continuing (and re-doing some work).
> 	>       b) devel prio -- right now I am working hard on getting a
> new stable
> 	> v5
> 	>       rsyslog out, and this is taking quite some toll
> 	>
> 	>       The feature you are asking for is definitely on the today
> list, and I
> 	> hope to
> 	>       be able to work more on liblognorm within the next couple
> of weeks
> 	> (this year
> 	>       has been very busy - and will be - at least until mid-
> april).
> 	>
> 	>       Rainer
> 	>
> 	>
> 	>       > -----Original Message-----
> 	>       > From: lognorm-bounces at lists.adiscon.com
> [mailto:lognorm-
> 	>       > bounces at lists.adiscon.com] On Behalf Of Wladimir van
> der Laan
> 	>       > Sent: Monday, March 21, 2011 7:00 PM
> 	>       > To: lognorm at lists.adiscon.com
> 	>       > Subject: [Lognorm] Identifying message types
> 	>       >
> 	>       > Hello,
> 	>       >
> 	>       > I have a question about the usage of lognorm. As I
> understand, the
> 	>       > program extracts data fields from log messages in text
> format, by
> 	> means
> 	>       > of examples from a ruleset file. The output is
> represented as
> 	> metadata
> 	>       > key/value pairs.
> 	>       >
> 	>       > But as far as I can see, it outputs no identifier as to
> what kind
> 	of
> 	>       > message the log line represents. For automated log
> processing, one
> 	>       > would also need to identify the message, for example,
> as failed
> 	>       > authentication, or dhcp request, etc.
> 	>       >
> 	>       > Am I overlooking something? Is it possible to add a
> message type
> 	> field
> 	>       > in a ruleset?
> 	>       >
> 	>       > Greetings,
> 	>       > Wladimir
> 	>       >
> 	>
> 	>
> 	>       _______________________________________________
> 	>       Lognorm mailing list
> 	>       Lognorm at lists.adiscon.com
> 	>       http://lists.adiscon.net/mailman/listinfo/lognorm
> 	>
> 	>
> 
> 	_______________________________________________
> 	Lognorm mailing list
> 	Lognorm at lists.adiscon.com
> 	http://lists.adiscon.net/mailman/listinfo/lognorm
> 
>