[rsyslog] abort in 4.2.1

[david at lang.hm]

On Tue, 25 Aug 2009, david at lang.hm wrote:

> 2. there are fewer systems sending simultaniously to the central box
> (there are ~100 boxes sending to the relay that dies, but only a half
> dozen relay boxes sending to the central box)
>
> two of the other relays handle a _far_ higher rate of logs, but from fewer
> sources (one has one source that spews ~15G of logs/day, the other
> recieves ~100m logs/day from 6 machines). a third relay has more machines
> sending it logs, but at a lower rate than those two (but still
> significantly higher than the one that fails). if there was a problem with
> load or the number of messages being recieved simultaniously I would
> expect one of these other three to have more problems than the one that
> fails on me.

as an idea of the message rates of the various boxes (they are closer than 
I thought

the one that dies (~130 servers sending logs, including the 60 sending 
bursts from the cron job)

23962669 log messages
busiest seconds
1798    Aug 24 13:11:08
1801    Aug 24 15:20:12
1831    Aug 24 21:15:01
1890    Aug 24 00:45:01
1921    Aug 24 17:26:22
1946    Aug 24 14:02:31
1983    Aug 24 21:29:59
2141    Aug 24 17:27:23
2142    Aug 24 16:15:01
2163    Aug 24 15:43:34
2433    Aug 24 13:00:01

busiest min
total logs
32562   Aug 24 14:19
32579   Aug 24 17:07
32633   Aug 24 14:05
32637   Aug 24 13:09
32709   Aug 24 15:03
33317   Aug 24 14:09
33381   Aug 24 16:05
33466   Aug 24 14:07
33653   Aug 24 13:15
33883   Aug 24 14:08
34029   Aug 24 14:06


another one (61 servers)

31338876 log messages
1574    Aug 24 19:57:01
1578    Aug 24 22:07:47
1580    Aug 24 22:56:13
1583    Aug 24 19:56:44
1587    Aug 24 22:56:14
1587    Aug 24 22:07:22
1602    Aug 24 22:07:33
1613    Aug 24 19:56:50
1620    Aug 24 19:15:00
1732    Aug 24 22:55:07
1907    Aug 24 18:00:09

76928   Aug 24 06:57
77623   Aug 24 16:20
78251   Aug 24 19:15
78770   Aug 24 22:55
79803   Aug 24 22:07
81383   Aug 24 22:08
82841   Aug 24 07:42
85746   Aug 24 17:59
85870   Aug 24 16:22
86423   Aug 24 16:21
89161   Aug 24 01:09


another (13 servers)
24184377 log messages
1131    Aug 24 15:28:44
1150    Aug 23 22:00:02
1163    Aug 23 17:00:01
1165    Aug 24 15:28:45
1165    Aug 24 16:00:01
1231    Aug 24 12:00:02
1247    Aug 24 11:00:02
1298    Aug 24 08:00:02
1327    Aug 24 14:00:02
1330    Aug 24 05:00:02
1340    Aug 24 09:00:02
31931   Aug 24 13:48
31975   Aug 24 09:32
32090   Aug 24 10:34
32867   Aug 24 09:30
33246   Aug 24 10:52
33289   Aug 24 11:56
34161   Aug 24 13:47
34504   Aug 24 13:46
34850   Aug 24 13:31
35002   Aug 24 15:28
35287   Aug 24 13:45

another (7 servers)
29764771 log messages
1195    Aug 24 15:00:03
1214    Aug 24 15:28:44
1253    Aug 24 15:28:45
1261    Aug 24 16:00:01
1269    Aug 24 13:13:53
1329    Aug 24 12:00:02
1355    Aug 24 08:00:02
1358    Aug 24 05:00:02
1404    Aug 24 11:00:02
1410    Aug 24 14:00:02
1491    Aug 24 09:00:02
38272   Aug 24 13:47
38797   Aug 24 09:32
39211   Aug 24 11:56
39624   Aug 24 13:46
39666   Aug 24 10:34
39750   Aug 24 09:30
39802   Aug 24 13:13
39814   Aug 24 13:45
40817   Aug 24 10:52
42071   Aug 24 15:28
43968   Aug 24 13:31

another (115 servers)
124469193 log messages
3289    Aug 24 14:05:01
3312    Aug 24 15:44:44
3319    Aug 24 15:01:13
3319    Aug 24 15:05:16
3320    Aug 24 15:14:44
3331    Aug 24 14:15:34
3350    Aug 24 15:17:42
3422    Aug 24 15:54:44
3542    Aug 24 15:00:01
4075    Aug 24 16:38:16
4078    Aug 24 15:05:15
164209  Aug 24 15:13
164247  Aug 24 16:04
164274  Aug 24 14:19
164369  Aug 24 14:37
164581  Aug 24 15:24
164929  Aug 24 15:12
165015  Aug 24 15:35
165385  Aug 24 14:17
165446  Aug 24 15:34
165566  Aug 24 15:00
166864  Aug 24 15:04

central system (~10 servers sending direct)
251236252 log messages
7208    Aug 24 07:45:01
7318    Aug 24 08:00:02
7414    Aug 24 07:00:01
7427    Aug 24 09:00:01
7452    Aug 24 13:45:01
7723    Aug 24 10:00:01
7838    Aug 24 11:00:01
7858    Aug 24 13:00:01
7970    Aug 24 08:00:01
8155    Aug 24 16:00:01
289323  Aug 24 12:18
289376  Aug 24 12:56
290064  Aug 24 09:15
290077  Aug 24 09:27
291036  Aug 24 08:00
291212  Aug 24 09:38
299116  Aug 24 10:37
300814  Aug 24 09:07
301968  Aug 24 12:57
304175  Aug 24 09:14
306308  Aug 24 07:24