[Lognorm] Any reason we should not support regex-based field-type?

singh.janmejay singh.janmejay at gmail.com
Tue Nov 4 00:54:11 CET 2014 

Alright, module loading flag it is then.

--
Regards,
Janmejay

PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.

On Nov 4, 2014 3:42 AM, "David Lang" <david at lang.hm> wrote:

> On Mon, 3 Nov 2014, singh.janmejay wrote:
>
>  I think module-loading call having a flag to turn on/off would be better
>> than build-time flag, purely because it allows one to build a package and
>> use it on several boxes, while keeping the feature off when not required.
>> A
>> more fine grained control, in that sense.
>>
>
> I was thinking in terms of something in the config file. We need it to be
> something that the person configuring rsyslog will see so that they know
> they are doing something slow. Most users don't compile rsyslog so they
> won't ever see the compile time flag.
>
> module loading time seems fine.
>
> David Lang
>
>  We can support both levels of enabling, but then a lot of people may be
>> confused as to why it doesn't work when they turned it on while building.
>>
>> 2 levels also seems a little over-protective to me. Kinda like Windows "do
>> you really want to delete this file" prompts.
>>
>> Does module-loading time flag sound ok?
>>
>> --
>> Regards,
>> Janmejay
>>
>> PS: Please blame the typos in this mail on my phone's uncivilized soft
>> keyboard sporting it's not-so-smart-assist technology.
>>
>> On Nov 3, 2014 10:52 PM, "singh.janmejay" <singh.janmejay at gmail.com>
>> wrote:
>>
>>  Hex was just an example. I meant when someone needs to roll out something
>>> very quickly, and it's not supported in the build they have, it's useful
>>> to
>>> be able to do it using regex in rulebase but still completely working in
>>> terms of rules, rather than having to resort to a mix of rules and
>>> exec-template calls with regexp based property extractors etc.
>>>
>>> It's still the same amount of work, if someone is trying to do something
>>> unsupported by lognorm. So performance effects will still show, just not
>>> in
>>> lognorm.
>>>
>>> This allows for flexibility and clean way of parsing logs at possibly
>>> even
>>> slightly lesser cost then the former approach.
>>>
>>> We can support a switch-on flag in conf file(module loading call, may be)
>>> which can be used to enable performance sensitive features.
>>>
>>> I personally think warning in logs would be enough though.
>>>
>>> --
>>> Regards,
>>> Janmejay
>>>
>>> PS: Please blame the typos in this mail on my phone's uncivilized soft
>>> keyboard sporting it's not-so-smart-assist technology.
>>>
>>> On Nov 3, 2014 9:22 PM, "Champ Clark III" <cclark at quadrantsec.com>
>>> wrote:
>>>
>>>
>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>> Hash: SHA1
>>>>
>>>> I think David means something like:
>>>>
>>>> ./configure --enable-preformance-pigs
>>>>
>>>> I'm not a huge fan of this for a couple of reasons.
>>>>
>>>> In your example,  you want to find hex values.  I'd rather see a lognorm
>>>> "parser" created for this purpose.   I'm afraid if we start with regular
>>>> expressions,  we'll end up mixing rules (RE and non-RE) and it will make
>>>> things very confusing.  I really like the way lognorm's "masking" works.
>>>> I'd rather not see that break.
>>>>
>>>>
>>>>
>>>> On 11/03/2014 07:15 AM, singh.janmejay wrote:
>>>>
>>>>>
>>>>> We log a warning?
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Janmejay
>>>>>
>>>>> PS: Please blame the typos in this mail on my phone's uncivilized soft
>>>>>
>>>> keyboard sporting it's not-so-smart-assist technology.
>>>>
>>>>>
>>>>>
>>>>> On Nov 3, 2014 5:37 PM, "David Lang" <david at lang.hm
>>>>>
>>>> <mailto:david at lang.hm> <david at lang.hm>> wrote:
>>>>
>>>>>
>>>>>     On Mon, 3 Nov 2014, singh.janmejay wrote:
>>>>>
>>>>>         I am thinking of it as a 2nd class field-type.
>>>>>
>>>>>         By that I mean, one gets best performance from 1st class
>>>>>
>>>> supported
>>>>
>>>>>         field-types, but if for some reason that is not sufficient for
>>>>>
>>>> someone,
>>>>
>>>>>         they can use a regex-field-type. It may be a little low on
>>>>>
>>>> performance, but
>>>>
>>>>>         then it unblocks people immediately.
>>>>>
>>>>>         I can do it, just need to know we are not ideologically against
>>>>>
>>>> it.
>>>>
>>>>>
>>>>>         Kind of construct im thinking of:
>>>>>
>>>>>         %foo:regex:[a-f0-9]+% to match hex-numbers for instance.
>>>>>
>>>>>         Thoughts?
>>>>>
>>>>>
>>>>>     Since this will be such a performance pig compared to the existing
>>>>>
>>>> parse tree, how about requiring a 'enable low performance types' flag or
>>>> something like that to enable it?
>>>>
>>>>>
>>>>>     There needs to be some good indicator that this is a performance
>>>>>
>>>> problem.
>>>>
>>>>>
>>>>>     David Lang
>>>>>     _______________________________________________
>>>>>     Lognorm mailing list
>>>>>     Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>
>>>>>
>>>> <Lognorm at lists.adiscon.com>
>>>>
>>>>>     http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>
>>>>>     _______________________________________________
>>>>>     Lognorm mailing list
>>>>>     Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>
>>>>>
>>>> <Lognorm at lists.adiscon.com>
>>>>
>>>>>     http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Lognorm mailing list
>>>>> Lognorm at lists.adiscon.com
>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>
>>>>
>>>>
>>>> - --
>>>> - - Quadrant Information Security
>>>>   Champ Clark III
>>>>   o: 800.538.9357 x 101
>>>>   c: 850.443.2440
>>>> -----BEGIN PGP SIGNATURE-----
>>>> Version: GnuPG v1.4.11 (GNU/Linux)
>>>>
>>>> iQEcBAEBAgAGBQJUV6S0AAoJENnmXt7Lmc3KAwEH/jiwhj/nhFRpvRm7DttrvYTE
>>>> U7kHpjToIMlCSiJqiS4bnvOTS4oTG6mQ5myYzAr16ITTIJQLnJSHmVWBgxlGdUlM
>>>> Kq33I+zYjfUK2Go01PSoLjoE3rgdGa1hptFHVUZREuRkzSP6THtXn8XhexmZzdpH
>>>> 1lC+ZXwNZ9k7FsWm3M027I8zGDKnvZLVdf2UJUElyrNxmWW04ieR3lqJ5qh3Uj8l
>>>> OvTzEDYObghwyS6hThNb1oMz2Sr1AD/mWu+sri1BDKqlPyMkQgYe8KNyI4sKBQHL
>>>> AdCi3TXMvYa8WMb5AOVL4tX0QrOgwNj5mbpeWb2vWrV/S2mkN39TZbE406W91T8=
>>>> =rGsk
>>>> -----END PGP SIGNATURE-----
>>>>
>>>>
>>>> _______________________________________________
>>>> Lognorm mailing list
>>>> Lognorm at lists.adiscon.com
>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>
>>>>
>>>>
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141104/97b7e9fc/attachment-0001.html>

More information about the Lognorm mailing list