<div dir="ltr"><div><div><div>Minor change, have made it:<br><br>foreach($.bar in $!foo) do {...}<br><br></div><div>(note the 'do' before body)<br></div><div><br></div>Added do because it kinda keeps syntax similar to if (expr) then {...}<br><br></div>Here is the change: <a href="https://github.com/janmejay/rsyslog/commit/d4229ed42afb759adfc4d44661fda211e37c72a0">https://github.com/janmejay/rsyslog/commit/d4229ed42afb759adfc4d44661fda211e37c72a0</a><br><br></div><div>Need to clarify desired semantics of 'set' in rscript (started separate mail thread for that). Will send a pull request for this once we close it on that thread.<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Nov 12, 2014 at 11:07 AM, David Lang <span dir="ltr"><<a href="mailto:david@lang.hm" target="_blank">david@lang.hm</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Wed, 12 Nov 2014, singh.janmejay wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
How do we want foreach support to be?<br>
<br>
Possible places are:<br>
1. a foreach loop-construct along-side flow-control structures in rscript<br>
</blockquote>
<br></span>
My understanding of how rscript works is that it's mostly parsed at startup time, not at message processing time, so I don't see how it can be doen in rscript.<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
2. a foreach value-producing templateElement in template<br>
<br>
The tradeoff is, someone looking to do some useful re-structuring of<br>
output, may want to call exec-template several times in approach-1, but<br>
they can just emit out what they want by placing property-extractors in<br>
foreach block in approch-2.<br>
<br>
The counter argument is, templates can't assign variables, execute<br>
functions, do arithmetic-operations etc, so usefulness of foreach in a<br>
template will be limited to most basic of message transformation cases.<br>
Additionally, string-template as of now has no clean way of implementing<br>
foreach. It'll require some involved syntax.<br>
</blockquote>
<br></span>
the result of templates can be assigned to variables.<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
To me approach-1 seems better suited for such a thing (purely because<br>
if-else, the other flow control structure is implemented at this level, and<br>
variable assignment, functions etc make arbitrary transformation easier).<br>
<br>
Once we have decided where to place foreach, let us zero in on its syntax.<br>
To seed the thought, how about this:<br>
<br>
set $.grault = "";<br>
foreach($!foo as $.bar) {<br>
set $.baz = $.bar!quux;<br>
foreach($.baz as $.corge) {<br>
if ( $.corge contains "#" ) then {<br>
set $.grault = $.grault & $.corge;<br>
}<br>
}<br>
}<br>
</blockquote>
<br></span>
in spite of everything I said above, if this can be implemented reasonably (without killing performance when it's not used), this would be extremely powerful.<span class=""><br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Alternatively:<br>
foreach($.bar <- $!foo) {...}<br>
foreach($.bar in $!foo) {...}<br>
</blockquote>
<br></span>
I like this a little better than 'as', but they are almost equal and either is far better than the punctuation versions.<span class="HOEnZb"><font color="#888888"><br>
<br>
David Lang</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
foreach($.bar : $!foo) {...}<br>
<br>
<br>
On Fri, Oct 31, 2014 at 6:46 PM, singh.janmejay <<a href="mailto:singh.janmejay@gmail.com" target="_blank">singh.janmejay@gmail.com</a>><br>
wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Sure, I'll fork on github.<br>
<br>
On Fri, Oct 31, 2014 at 6:42 PM, Rainer Gerhards <<a href="mailto:rgerhards@hq.adiscon.com" target="_blank">rgerhards@hq.adiscon.com</a><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
wrote:<br>
</blockquote>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
2014-10-31 14:08 GMT+01:00 singh.janmejay <<a href="mailto:singh.janmejay@gmail.com" target="_blank">singh.janmejay@gmail.com</a>>:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Cool, I'll implement $!foo!bar[0].<br>
<br>
+1<br>
</blockquote>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Let us process this patch-set, because is kinda hard to keep track of<br>
old patches and re-send in one shot.<br>
<br>
<br>
</blockquote>
would you mind cloning on github and maintain a feature branch there?<br>
That would make it much easier for me, as I could merge the branch when you<br>
are done. If not, it's no problem and I'll maintain that branch.<br>
<br>
<br>
i'll send the new patch once done(i'll now only get to work on it on<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
monday).<br>
<br>
<br>
</blockquote>
I haven't had a chance to look as I am now busy building test<br>
environments and looking at the testbench [yes, one guy!] ;) But I see<br>
Pavel has asked some questions. He recently did a lot of work on the lib,so<br>
it is best to coordinate that part with him.<br>
<br>
Rainer<br>
<br>
Do existing patches look ok except for the indexed-addressing feature?<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
On Fri, Oct 31, 2014 at 4:15 PM, David Lang <<a href="mailto:david@lang.hm" target="_blank">david@lang.hm</a>> wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Fri, 31 Oct 2014, singh.janmejay wrote:<br>
<br>
Yes, I didn't have a need to address tokens individually, but you have<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
a<br>
point.<br>
<br>
Any suggestions on what we want to do for addressing array elements?<br>
<br>
I wonder if its possible to do in $!... notation without breaking<br>
backward<br>
compatibility. How about a function?<br>
<br>
I'll be happy to implement support for addressing it in $!... notation<br>
if<br>
don't mind breaking a corner case in backward compatibility. Eg.<br>
$!foo!bar![0] ? Its kinda ugly though, or so I think.<br>
<br>
</blockquote>
<br>
I was thinking just $!foo!bar[0] it's a bit ugly, but not too bad. It<br>
does mean that you can't have '[' in a variable name, but I don't think<br>
that's likely to be a real problem. I don't think there's ever a really<br>
clean way to do something like $!foo[2]!bar[2]!baz no matter what your<br>
syntax, it gets messy.<br>
<br>
for templates, we probably need some sort of foreach(array, pattern)<br>
function that takes the pattern and repeats it for each item in the array.<br>
<br>
David Lang<br>
<br>
<br>
On Fri, Oct 31, 2014 at 3:44 PM, David Lang <<a href="mailto:david@lang.hm" target="_blank">david@lang.hm</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
On Fri, 31 Oct 2014, singh.janmejay wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
It writes it as a json array, here is a fragment from my manual<br>
tests:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
[ "15", "26", "15" ]<br>
<br>
<br>
</blockquote>
right, but how do you access it in rsyslog?<br>
<br>
if you have { 'foo': { 'bar': '10' } } you access this as $!foo!bar<br>
and<br>
get the result '10'<br>
<br>
what would you use to access the value '26' in your example?<br>
<br>
we also don't have anything like foreach() in our template language,<br>
which<br>
makes it hard to make use of these values as anything other than a<br>
JSON<br>
string.<br>
<br>
I'm not saying that it's not useful, but I am pointing out the<br>
problems<br>
that we will have using it.<br>
<br>
David Lang<br>
<br>
<br>
It was using time in hh:mm:ss format and tokening by colon(:). I'll<br>
add<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
tests for it soon, but until then pasting output here is the best I<br>
can<br>
do.<br>
<br>
The idea behind this is to generate structured content from<br>
semi-structured<br>
or unstructured log messages. So array is a good representation for<br>
tokenized-value (it is multi-valued by nature, and array is a good<br>
way to<br>
represent that).<br>
<br>
But eventually we should allow user to register value-transformers<br>
so that<br>
it can be pre-processed before its emitted. May be have a canned set<br>
of<br>
transformers, and allow user to plug in new ones.<br>
<br>
My first instinct was to utilize variable support for this, infact<br>
this<br>
was<br>
the motivator for variable support. But it still leads to a fairly<br>
complex<br>
config for an access log with 15 - 20 fields, especially given those<br>
fields<br>
can have colon separated entries inside comma separated entries etc.<br>
<br>
So I felt the need for a simpler way of doing it, hence this and<br>
other<br>
(recurse) field-type.<br>
<br>
On Fri, Oct 31, 2014 at 3:23 PM, David Lang <<a href="mailto:david@lang.hm" target="_blank">david@lang.hm</a>> wrote:<br>
<br>
On Fri, 31 Oct 2014, singh.janmejay wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Tokenizer followed by tokenizer is something that I have in mind<br>
too.<br>
But<br>
<br>
I<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
promised myself that i'd write a test for that instead of testing<br>
it<br>
manually :-). Will add that patch on this thread once I get a<br>
chance to<br>
work on it.<br>
<br>
<br>
At least in the short term, you can use the ability to call<br>
</blockquote>
mmnormalize<br>
on<br>
a variable to parse subvariables.<br>
<br>
How are the resulting fields addressed? Rsyslog hasn't had array<br>
addressing yet.<br>
<br>
David Lang<br>
<br>
<br>
However, since you are asking about those kind of forms, let met<br>
discuss<br>
<br>
something else that I was thinking about.<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
The idea is to have another field type called recurse.<br>
<br>
Similar to how tokenized uses a ctx to parse matching text,<br>
recurse will<br>
parse it using the current context. AFAIK, the context is<br>
stateless, so<br>
I<br>
don't see any problems with that. I also plan to support tag based<br>
picking<br>
of which rules the text may match, and if it matches something<br>
else, it<br>
should be considered no-match.<br>
<br>
Instead of typing it out here, i'll attach a picture I took after<br>
thinking<br>
through it briefly(i'll attach it to the next mail).<br>
<br>
<br>
______________________________<u></u>_________________<br>
<br>
</blockquote>
Lognorm mailing list<br>
<a href="mailto:Lognorm@lists.adiscon.com" target="_blank">Lognorm@lists.adiscon.com</a><br>
<a href="http://lists.adiscon.net/mailman/listinfo/lognorm" target="_blank">http://lists.adiscon.net/<u></u>mailman/listinfo/lognorm</a><br>
<br>
______________________________<u></u>_________________<br>
Lognorm mailing list<br>
<a href="mailto:Lognorm@lists.adiscon.com" target="_blank">Lognorm@lists.adiscon.com</a><br>
<a href="http://lists.adiscon.net/mailman/listinfo/lognorm" target="_blank">http://lists.adiscon.net/<u></u>mailman/listinfo/lognorm</a><br>
<br>
<br>
<br>
<br>
</blockquote>
<br>
______________________________<u></u>_________________<br>
</blockquote>
Lognorm mailing list<br>
<a href="mailto:Lognorm@lists.adiscon.com" target="_blank">Lognorm@lists.adiscon.com</a><br>
<a href="http://lists.adiscon.net/mailman/listinfo/lognorm" target="_blank">http://lists.adiscon.net/<u></u>mailman/listinfo/lognorm</a><br>
<br>
______________________________<u></u>_________________<br>
Lognorm mailing list<br>
<a href="mailto:Lognorm@lists.adiscon.com" target="_blank">Lognorm@lists.adiscon.com</a><br>
<a href="http://lists.adiscon.net/mailman/listinfo/lognorm" target="_blank">http://lists.adiscon.net/<u></u>mailman/listinfo/lognorm</a><br>
<br>
<br>
<br>
</blockquote>
<br>
<br>
</blockquote>
______________________________<u></u>_________________<br>
Lognorm mailing list<br>
<a href="mailto:Lognorm@lists.adiscon.com" target="_blank">Lognorm@lists.adiscon.com</a><br>
<a href="http://lists.adiscon.net/mailman/listinfo/lognorm" target="_blank">http://lists.adiscon.net/<u></u>mailman/listinfo/lognorm</a><br>
<br>
______________________________<u></u>_________________<br>
Lognorm mailing list<br>
<a href="mailto:Lognorm@lists.adiscon.com" target="_blank">Lognorm@lists.adiscon.com</a><br>
<a href="http://lists.adiscon.net/mailman/listinfo/lognorm" target="_blank">http://lists.adiscon.net/<u></u>mailman/listinfo/lognorm</a><br>
<br>
<br>
</blockquote>
<br>
<br>
--<br>
Regards,<br>
Janmejay<br>
<a href="http://codehunk.wordpress.com" target="_blank">http://codehunk.wordpress.com</a><br>
<br>
______________________________<u></u>_________________<br>
Lognorm mailing list<br>
<a href="mailto:Lognorm@lists.adiscon.com" target="_blank">Lognorm@lists.adiscon.com</a><br>
<a href="http://lists.adiscon.net/mailman/listinfo/lognorm" target="_blank">http://lists.adiscon.net/<u></u>mailman/listinfo/lognorm</a><br>
<br>
<br>
</blockquote>
<br>
______________________________<u></u>_________________<br>
Lognorm mailing list<br>
<a href="mailto:Lognorm@lists.adiscon.com" target="_blank">Lognorm@lists.adiscon.com</a><br>
<a href="http://lists.adiscon.net/mailman/listinfo/lognorm" target="_blank">http://lists.adiscon.net/<u></u>mailman/listinfo/lognorm</a><br>
<br>
<br>
</blockquote>
<br>
<br>
--<br>
Regards,<br>
Janmejay<br>
<a href="http://codehunk.wordpress.com" target="_blank">http://codehunk.wordpress.com</a><br>
<br>
</blockquote>
<br>
<br>
<br>
</blockquote>
</div></div><br>_______________________________________________<br>
Lognorm mailing list<br>
<a href="mailto:Lognorm@lists.adiscon.com">Lognorm@lists.adiscon.com</a><br>
<a href="http://lists.adiscon.net/mailman/listinfo/lognorm" target="_blank">http://lists.adiscon.net/mailman/listinfo/lognorm</a><br>
<br>_______________________________________________<br>
Lognorm mailing list<br>
<a href="mailto:Lognorm@lists.adiscon.com">Lognorm@lists.adiscon.com</a><br>
<a href="http://lists.adiscon.net/mailman/listinfo/lognorm" target="_blank">http://lists.adiscon.net/mailman/listinfo/lognorm</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature">Regards,<br>Janmejay<br><a href="http://codehunk.wordpress.com">http://codehunk.wordpress.com</a><br></div>
</div>