[rsyslog] abort in 4.2.1

[david at lang.hm]

On Fri, 28 Aug 2009, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
>>
>>>
>>> that would be hard to so for a couple reasons
>>>
>>> at 5-10 times slower the system may not be able to keep up (even with
>>> the
>>> 'slower' afternoon traffic)
>>>
>>> this is running on a very hardened production server, getting
>> valgrind
>>> installed there would require permission from the SVP level.
>>>
>>
>> understood. So let me see what else I can come up with :)
>
> I tried a lab yesterday where I sent roughly 1.5 billion messages (based on
> what I saw in the debug logs). Unfortunately, no abort happened. However, my
> traffic patterns was continous traffic of the same message.
>
> So I am now going to create some new tooling that permits me to mimic your
> traffic pattern much better. That will probably require until early next
> week. To make this really work, it would be really useful if you could send
> me some complete messages from your environment. I suggest to forward them
> via private mail. I hope this is possible.
>
> Also, it would be good if you could --enable-rtinst --enable-debug and try
> out that version on your machine. I am a bit concerned about the speed of the
> resulting executable, it may be too slow. You do not need to run it in debug
> mode itself. These option (especially--enable-debug) will activate in-depth
> runtime checks (assert, will abort when something wrong happens) and my hope
> is that they will catch the bug closer to the root cause. If so, I would need
> the gdb abort info (actually enabling debug output would be an option some
> time later).
>
> Please let me know what would be OK with you.

I will give this a try.

I was going to suggest that since we have the message getting corrupted it 
may make sense to make a temporary branch that has multiple message 
buffers and at various times through the message processing it makes a 
copy of the emssage to the buffer. when the system crashes I will be able 
to look at the core and see where the message is getting corrupted.

I will see about doing a tcpdump at the time that I do this and send it to 
you (I'll need to check with management, but since we have a contract in 
place for other reasons I think we can do this)

I can't do this late on a friday, but I should be able to do this monday 
afternoon.

David Lang