[rsyslog-notify] Forum Thread: Fatal error on disk queue - (Mode 'edit_first_post')
noreply at adiscon.com
noreply at adiscon.com
Wed Jan 27 17:00:01 CET 2016
User: Jzeolla
Forumlink: http://kb.monitorware.com/viewtopic.php?p=26298#p26298
Message:
----------
Hi,
I'm having some issues getting rsyslog to write to local files properly,
and when I was investigating I found that I'm getting a fatal error on disk
queue every time I start rsyslogd. I don't actually think this is causing
the issue where it's unable to write to local files, but I still would like
to clean it up if possible.
Some validation I did to make sure it's not jsut a config file issue
[code:b9f89l0m]
user at server:/etc/rsyslog.d$ ls
50-default.conf logstore.conf postfix.conf
user at server:/etc/rsyslog.d$ /usr/sbin/rsyslogd -N6 -f
50-default.conf
rsyslogd: version 8.12.0, config validation run (level 6),
master config 50-default.conf
rsyslogd: End of config validation run. Bye.
user at server:/etc/rsyslog.d$ /usr/sbin/rsyslogd -N6 -f
logstore.conf
rsyslogd: version 8.12.0, config validation run (level 6),
master config logstore.conf
rsyslogd: module name 'mmjsonparse' is unknown [v8.12.0 try
http://www.rsyslog.com/e/2209 ]
rsyslogd: error during parsing file logstore.conf, on or before
line 64: errors occured in file 'logstore.conf' around line 64
[v8.12.0 try http://www.rsyslog.com/e/2207 ]
rsyslogd: module name 'mmjsonparse' is unknown [v8.12.0 try
http://www.rsyslog.com/e/2209 ]
rsyslogd: error during parsing file logstore.conf, on or before
line 91: errors occured in file 'logstore.conf' around line 91
[v8.12.0 try http://www.rsyslog.com/e/2207 ]
user at server:/etc/rsyslog.d$ /usr/sbin/rsyslogd -N6 -f
postfix.conf
rsyslogd: version 8.12.0, config validation run (level 6),
master config postfix.conf
rsyslogd: End of config validation run. Bye.
user at server:/etc/rsyslog.d$ sed '64q;d' logstore.conf
action(type="mmjsonparse")
user at server:/etc/rsyslog.d$ sed '91q;d' logstore.conf
action(type="mmjsonparse")
[/code:b9f89l0m]
The logs I am seeing are:
[code:b9f89l0m]2016-01-27T10:12:10-05:00
172.19.231.50 rsyslogd-2040: fatal error on disk queue
'action 15 queue[DA]', emergency switch to direct mode
[v8.12.0 try http://www.rsyslog.com/e/2040 ]
2016-01-27T10:12:10-05:00 172.19.231.50
rsyslogd-2040: fatal error on disk queue 'action 17 queue[DA]',
emergency switch to direct mode [v8.12.0 try
http://www.rsyslog.com/e/2040 ][/code:b9f89l0m]
[code:b9f89l0m]7513.578017824:main thread : action 14
queue: starting queue
7513.578026959:main thread : action 14 queue: is
disk-assisted, disk will be used on demand
7513.578037388:main thread : action 14 queue:
params: type 1, enq-only 0, disk assisted 1, spoolDir
'/data/spool/rsyslog', maxFileSz 104857600, maxQSize 10240, lqsize 0,
pqsize 0, child 0, full delay 3750, light delay 7140, deq batch size 128,
high wtrmrk 7500, low wtrmrk 7140, discardmrk 10000, max wrkr 1, min msgs
f. wrkr 10240
7513.578050266:main thread : action 14 queue:Reg:
finalizing construction of worker thread pool (numworkerThreads 1)
7513.578060439:main thread : action 14 queue:Reg/w0:
finalizing construction of worker instance data (for 24 actions)
7513.578073757:main thread : action 14
queue:DAwpool: finalizing construction of worker thread pool
(numworkerThreads 1)
7513.578084008:main thread : action 14
queue:DAwpool/w0: finalizing construction of worker instance data
(for 24 actions)
7513.578095464:main thread : action 14
queue[DA]: starting queue
7513.578105516:main thread : action 14
queue[DA]: .qi file name is
'/data/spool/rsyslog/14.qi', len 61
7513.578114905:main thread : action 14
queue[DA]: I am a child
7513.578126213:main thread : action 14
queue[DA]: clean startup, no .qi file found
7513.578135788:main thread : action 14
queue[DA]: state -2040 reading .qi file - can not read
persisted info (if any)
7513.578145335:main thread : file stream N/A params:
flush interval 0, async write 0
7513.578155390:main thread : file stream N/A params:
flush interval 0, async write 0
7513.578165512:main thread : file stream N/A params:
flush interval 0, async write 0
7513.578176356:main thread : action 14
queue[DA]: params: type 2, enq-only 0, disk assisted 0,
spoolDir '/data/spool/rsyslog', maxFileSz 104857600, maxQSize 0, lqsize 0,
pqsize 0, child 1, full delay -1, light delay -1, deq batch size 8, high
wtrmrk 0, low wtrmrk 1, discardmrk 0, max wrkr 1, min msgs f. wrkr 0
7513.578187924:main thread : action 14
queue[DA]:Reg: finalizing construction of worker thread
pool (numworkerThreads 1)
7513.578197624:main thread : action 14
queue[DA]:Reg/w0: finalizing construction of worker
instance data (for 24 actions)
7513.578209176:main thread : action 14
queue[DA]: queue finished initialization
7513.578220600:main thread : action 14 queue: DA queue
initialized, disk queue 0x1f62a30
7513.578229800:main thread : action 14 queue: queue
finished initialization
7513.578239814:main thread : Action
builtin:omfile[0x1f7dab0]: queue 0x1f7e5d0 started
7513.578248466:main thread : iterateAllActions calling into
action 0x1f7ebf0
7513.578257210:main thread : action 15 queue: starting
queue
7513.578266101:main thread : action 15 queue: is
disk-assisted, disk will be used on demand
7513.578276651:main thread : action 15 queue:
params: type 1, enq-only 0, disk assisted 1, spoolDir
'/data/spool/rsyslog', maxFileSz 104857600, maxQSize 10240, lqsize 0,
pqsize 0, child 0, full delay 3750, light delay 7140, deq batch size 128,
high wtrmrk 7500, low wtrmrk 7140, discardmrk 10000, max wrkr 1, min msgs
f. wrkr 10240
7513.578288267:main thread : action 15 queue:Reg:
finalizing construction of worker thread pool (numworkerThreads 1)
7513.578297979:main thread : action 15 queue:Reg/w0:
finalizing construction of worker instance data (for 24 actions)
7513.578308447:main thread : action 15
queue:DAwpool: finalizing construction of worker thread pool
(numworkerThreads 1)
7513.578318014:main thread : action 15
queue:DAwpool/w0: finalizing construction of worker instance data
(for 24 actions)
7513.578330993:main thread : action 15
queue[DA]: starting queue
7513.578341180:main thread : action 15
queue[DA]: .qi file name is
'/data/spool/rsyslog/15.qi', len 43
7513.578350138:main thread : action 15
queue[DA]: I am a child
7513.578363503:main thread : file stream
/data/spool/rsyslog/15.qi params: flush interval 0, async write 0
7513.578380284:main thread : file
'/data/spool/rsyslog/15.qi' opened as #7 with mode 384
7513.578393708:main thread : strm 0x1f8a120: opened file
'/data/spool/rsyslog/15.qi' for READ as 7
7513.578406579:main thread : strm 0x1f8a120: file 7 read
559 bytes
7513.578425984:main thread : file stream 15 params:
flush interval 0, async write 0
7513.578445584:main thread : file stream 15 params:
flush interval 0, async write 0
7513.578458250:main thread : file stream 15 params:
flush interval 0, async write 0
7513.578651644:main thread : file
'/data/spool/rsyslog/15.00000002' opened as #8 with mode 384
7513.578665731:main thread : strm 0x1f8b440: opened file
'/data/spool/rsyslog/15.00000002' for WRITE as 8
7513.578675653:main thread : strm 0x1f8b440: file 8
seek, pos 9193093
7513.578689644:main thread : file
'/data/spool/rsyslog/15.00000002' opened as #9 with mode 384
7513.578701173:main thread : strm 0x1f8c6d0: opened file
'/data/spool/rsyslog/15.00000002' for READ as 9
7513.578710806:main thread : strm 0x1f8c6d0: file 9
seek, pos 9193093
7513.578722700:main thread : file
'/data/spool/rsyslog/15.00000002' opened as #10 with mode 384
7513.578733593:main thread : strm 0x1f8d960: opened file
'/data/spool/rsyslog/15.00000002' for READ as 10
7513.578743029:main thread : strm 0x1f8d960: file 10
seek, pos 9193093
7513.578752015:main thread : strm 0x1f8a120: file
7(/data/spool/rsyslog/15.qi) closing
7513.578765191:main thread : action 15
queue[DA]: params: type 2, enq-only 0, disk assisted 0,
spoolDir '/data/spool/rsyslog', maxFileSz 104857600, maxQSize 0, lqsize 1,
pqsize 1, child 1, full delay -1, light delay -1, deq batch size 8, high
wtrmrk 0, low wtrmrk 1, discardmrk 0, max wrkr 1, min msgs f. wrkr 0
7513.578778043:main thread : action 15
queue[DA]:Reg: finalizing construction of worker thread
pool (numworkerThreads 1)
7513.578788158:main thread : action 15
queue[DA]:Reg/w0: finalizing construction of worker
instance data (for 24 actions)
7513.578798155:main thread : action 15
queue[DA]: queue finished initialization
7513.578807484:main thread : action 15
queue[DA]:Reg: high activity - starting 1 additional worker
thread(s).
7513.578840064:main thread : action 15
queue[DA]:Reg: started with state 0, num workers now 1
7513.578853425:main thread : action 15 queue: DA queue
initialized, disk queue 0x1f89de0
7513.578863016:main thread : action 15 queue: queue
finished initialization
7513.578873856:main thread : Action
builtin:omfile[0x1f7ebf0]: queue 0x1f7f410 started
7513.578886086:main thread : iterateAllActions calling into
action 0x1f7fa30
7513.578899820:main thread : action 16 queue: starting
queue
7513.578910473:main thread : action 16 queue: is
disk-assisted, disk will be used on demand
7513.578925044:main thread : action 16 queue:
params: type 1, enq-only 0, disk assisted 1, spoolDir
'/data/spool/rsyslog', maxFileSz 104857600, maxQSize 10240, lqsize 0,
pqsize 0, child 0, full delay 3750, light delay 7140, deq batch size 128,
high wtrmrk 7500, low wtrmrk 7140, discardmrk 10000, max wrkr 1, min msgs
f. wrkr 10240
7513.578941189:main thread : action 16 queue:Reg:
finalizing construction of worker thread pool (numworkerThreads 1)
7513.578952192:main thread : action 16 queue:Reg/w0:
finalizing construction of worker instance data (for 24 actions)
7513.578965603:main thread : action 16
queue:DAwpool: finalizing construction of worker thread pool
(numworkerThreads 1)
7513.578976942:main thread : action 16
queue:DAwpool/w0: finalizing construction of worker instance data
(for 24 actions)
7513.578992599:7fc9f2e53700: thread created, tid 12517, name
'rs:action 15 queue[DA]:'
7513.579047612:main thread : action 16
queue[DA]: starting queue
7513.579059731:main thread : action 16
queue[DA]: .qi file name is
'/data/spool/rsyslog/16.qi', len 62
7513.579070007:main thread : action 16
queue[DA]: I am a child
7513.579081477:action 15 queue[DA]:Reg/w0: wti
0x1f8a250: worker starting
7513.579095123:main thread : action 16
queue[DA]: clean startup, no .qi file found
7513.579106262:main thread : action 16
queue[DA]: state -2040 reading .qi file - can not read
persisted info (if any)
7513.579117469:action 15 queue[DA]:Reg/w0:
DeleteProcessedBatch: we deleted 0 objects and enqueued 0 objects
7513.579128720:action 15 queue[DA]:Reg/w0:
doDeleteBatch: delete batch from store, new sizes: log 1, phys 1
7513.579140183:main thread : file stream N/A params:
flush interval 0, async write 0
7513.579153846:action 15 queue[DA]:Reg/w0: strm
0x1f8d960: file 10 read 0 bytes
7513.579165435:action 15 queue[DA]:Reg/w0: strm
0x1f8d960: file 10 EOF
7513.579176997:main thread : file stream N/A params:
flush interval 0, async write 0
7513.579189070:action 15 queue[DA]:Reg/w0: strm
0x1f8d960: file 10(15) closing
7513.579201037:main thread : file stream N/A params:
flush interval 0, async write 0
7513.579213437:action 15 queue[DA]:Reg/w0: file
'/data/spool/rsyslog/15.00000003' opened as #-1 with mode 384
7513.579226095:main thread : action 16
queue[DA]: params: type 2, enq-only 0, disk assisted 0,
spoolDir '/data/spool/rsyslog', maxFileSz 104857600, maxQSize 0, lqsize 0,
pqsize 0, child 1, full delay -1, light delay -1, deq batch size 8, high
wtrmrk 0, low wtrmrk 1, discardmrk 0, max wrkr 1, min msgs f. wrkr 0
7513.579239714:action 15 queue[DA]:Reg/w0: strm
0x1f8d960: open error 2, file
'/data/spool/rsyslog/15.00000003': No such file or directory
7513.579251716:action 15 queue[DA]:Reg/w0:
objDeserialize error -2040 during header processing - trying to recover
7513.579263540:main thread : action 16
queue[DA]:Reg: finalizing construction of worker thread
pool (numworkerThreads 1)
7513.579275477:action 15 queue[DA]:Reg/w0: file
'/data/spool/rsyslog/15.00000003' opened as #-1 with mode 384
7513.579287534:main thread : action 16
queue[DA]:Reg/w0: finalizing construction of worker
instance data (for 24 actions)
7513.579299581:action 15 queue[DA]:Reg/w0: strm
0x1f8d960: open error 2, file
'/data/spool/rsyslog/15.00000003': No such file or directory
7513.579310672:action 15 queue[DA]:Reg/w0:
deserializer has possibly been able to re-sync and recover, state -2040
7513.579322155:main thread : action 16
queue[DA]: queue finished initialization
7513.579334157:action 15 queue[DA]:Reg/w0: action
15 queue[DA]: error -2040 dequeueing element - ignoring, but
strange things may happen
7513.579344827:action 15 queue[DA]:Reg/w0: action
15 queue[DA]: got 'file not found' error -2040, queue defunct
7513.579355571:action 15 queue[DA]:Reg/w0: strm
0x1f8b440: file 8(15) closing
7513.579366128:action 15 queue[DA]:Reg/w0: strm
0x1f8b440: file 8(15) flush, buflen 0 (no need to flush)
7513.579377343:main thread : action 16 queue: DA queue
initialized, disk queue 0x1f90f90
7513.579387972:main thread : action 16 queue: queue
finished initialization
7513.579399123:action 15 queue[DA]:Reg/w0: strm
0x1f8d960: file -1(15) closing
7513.579410336:action 15 queue[DA]:Reg/w0: strm
0x1f8c6d0: file 9(15) closing
7513.579421935:main thread : Action
builtin:omfile[0x1f7fa30]: queue 0x1f80480 started
7513.579432950:main thread : iterateAllActions calling into
action 0x1f80aa0
7513.579442701:main thread : action 17 queue: starting
queue
7513.579452840:main thread : action 17 queue: is
disk-assisted, disk will be used on demand
7513.579464324:action 15 queue[DA]:Reg/w0:
strmCloseFile: deleting '/data/spool/rsyslog/15.00000002'
7513.579476433:main thread : action 17 queue:
params: type 1, enq-only 0, disk assisted 1, spoolDir
'/data/spool/rsyslog', maxFileSz 104857600, maxQSize 10240, lqsize 0,
pqsize 0, child 0, full delay 3750, light delay 7140, deq batch size 128,
high wtrmrk 7500, low wtrmrk 7140, discardmrk 10000, max wrkr 1, min msgs
f. wrkr 10240
7513.579489260:main thread : action 17 queue:Reg:
finalizing construction of worker thread pool (numworkerThreads 1)
7513.579499272:main thread : action 17 queue:Reg/w0:
finalizing construction of worker instance data (for 24 actions)
7513.579508548:action 15 queue[DA]:Reg/w0: action
15 queue[DA]: DA queue is in emergency mode, disabling DA in
parent
7513.579523406:main thread : action 17
queue:DAwpool: finalizing construction of worker thread pool
(numworkerThreads 1)
7513.579534702:main thread : action 17
queue:DAwpool/w0: finalizing construction of worker instance data
(for 24 actions)
7513.579563715:action 15 queue[DA]:Reg/w0: Called
LogMsg, msg: fatal error on disk queue 'action 15 queue[DA]',
emergency switch to direct mode
7513.579577614:main thread : action 17
queue[DA]: starting queue
7513.579589151:main thread : action 17
queue[DA]: .qi file name is
'/data/spool/rsyslog/18.qi', len 48
7513.579598669:main thread : action 17
queue[DA]: I am a child
7513.579624812:action 15 queue[DA]:Reg/w0:
rsyslog/glbl: using '127.0.0.1' as localhost IP
7513.579639538:main thread : file stream
/data/spool/rsyslog/18.qi params: flush interval 0, async write 0
rsyslogd: fatal error on disk queue 'action 15 queue[DA]',
emergency switch to direct mode [v8.12.0 try
http://www.rsyslog.com/e/2040 ]
7513.579656818:main thread : file
'/data/spool/rsyslog/18.qi' opened as #7 with mode 384
7513.579669520:action 15 queue[DA]:Reg/w0: regular
consumer finished, iret=-2183, szlog 0 sz phys 0
7513.579680452:action 15 queue[DA]:Reg/w0:
DDDD: wti 0x1f8a250: worker cleanup action instances
7513.579690824:action 15 queue[DA]:Reg/w0: wti
0x1f8a250, action 0, ptr (nil)
7513.579701009:action 15 queue[DA]:Reg/w0: wti
0x1f8a250, action 1, ptr (nil)
7513.579710636:action 15 queue[DA]:Reg/w0: wti
0x1f8a250, action 2, ptr (nil)
7513.579720433:action 15 queue[DA]:Reg/w0: wti
0x1f8a250, action 3, ptr (nil)
7513.579730145:action 15 queue[DA]:Reg/w0: wti
0x1f8a250, action 4, ptr (nil)
7513.579739910:action 15 queue[DA]:Reg/w0: wti
0x1f8a250, action 5, ptr (nil)
7513.579749876:action 15 queue[DA]:Reg/w0: wti
0x1f8a250, action 6, ptr (nil)
7513.579759543:action 15 queue[DA]:Reg/w0: wti
0x1f8a250, action 7, ptr (nil)
7513.579769259:action 15 queue[DA]:Reg/w0: wti
0x1f8a250, action 8, ptr (nil)
7513.579779074:action 15 queue[DA]:Reg/w0: wti
0x1f8a250, action 9, ptr (nil)
7513.579789146:action 15 queue[DA]:Reg/w0: wti
0x1f8a250, action 10, ptr (nil)
7513.579799202:action 15 queue[DA]:Reg/w0: wti
0x1f8a250, action 11, ptr (nil)
7513.579809201:action 15 queue[DA]:Reg/w0: wti
0x1f8a250, action 12, ptr (nil)
7513.579819470:action 15 queue[DA]:Reg/w0: wti
0x1f8a250, action 13, ptr (nil)
7513.579829284:action 15 queue[DA]:Reg/w0: wti
0x1f8a250, action 14, ptr (nil)
7513.579839433:action 15 queue[DA]:Reg/w0: wti
0x1f8a250, action 15, ptr (nil)
7513.579849364:action 15 queue[DA]:Reg/w0: wti
0x1f8a250, action 16, ptr (nil)
7513.579859169:action 15 queue[DA]:Reg/w0: wti
0x1f8a250, action 17, ptr (nil)
7513.579869108:action 15 queue[DA]:Reg/w0: wti
0x1f8a250, action 18, ptr (nil)
7513.579879092:action 15 queue[DA]:Reg/w0: wti
0x1f8a250, action 19, ptr (nil)
7513.579889069:action 15 queue[DA]:Reg/w0: wti
0x1f8a250, action 20, ptr (nil)
7513.579899159:action 15 queue[DA]:Reg/w0: wti
0x1f8a250, action 21, ptr (nil)
7513.579909251:action 15 queue[DA]:Reg/w0: wti
0x1f8a250, action 22, ptr (nil)
7513.579919284:action 15 queue[DA]:Reg/w0: wti
0x1f8a250, action 23, ptr (nil)
7513.579929388:action 15 queue[DA]:Reg/w0: wti
0x1f8a250: worker exiting
7513.579939501:action 15 queue[DA]:Reg/w0: action
15 queue[DA]:Reg: Worker thread 1f8a250, terminated, num
workers now 0
7513.579950449:main thread : strm 0x1f8dc50: opened file
'/data/spool/rsyslog/18.qi' for READ as 7
7513.579962603:main thread : strm 0x1f8dc50: file 7 read
574 bytes
7513.579981377:main thread : file stream 18 params:
flush interval 0, async write 0
7513.580001785:main thread : file stream 18 params:
flush interval 0, async write 0
7513.580016158:main thread : file stream 18 params:
flush interval 0, async write 0
7513.580032478:action 15 queue[DA]:Reg/w0:
destructor for debug call stack 0x7fc9ec0008c0 called
7513.580182453:main thread : file
'/data/spool/rsyslog/18.00000022' opened as #8 with mode 384
7513.580196613:main thread : strm 0x1f8e060: opened file
'/data/spool/rsyslog/18.00000022' for WRITE as 8
7513.580206126:main thread : strm 0x1f8e060: file 8
seek, pos 88029512
7513.580218860:main thread : file
'/data/spool/rsyslog/18.00000022' opened as #9 with mode 384
7513.580230025:main thread : strm 0x1f8e490: opened file
'/data/spool/rsyslog/18.00000022' for READ as 9
7513.580239561:main thread : strm 0x1f8e490: file 9
seek, pos 88029512
7513.580251426:main thread : file
'/data/spool/rsyslog/18.00000022' opened as #10 with mode 384
7513.580262071:main thread : strm 0x1f8e710: opened file
'/data/spool/rsyslog/18.00000022' for READ as 10
7513.580271432:main thread : strm 0x1f8e710: file 10
seek, pos 88029512
7513.580280724:main thread : strm 0x1f8dc50: file
7(/data/spool/rsyslog/18.qi) closing
7513.580293166:main thread : action 17
queue[DA]: params: type 2, enq-only 0, disk assisted 0,
spoolDir '/data/spool/rsyslog', maxFileSz 104857600, maxQSize 0, lqsize 2,
pqsize 2, child 1, full delay -1, light delay -1, deq batch size 8, high
wtrmrk 0, low wtrmrk 1, discardmrk 0, max wrkr 1, min msgs f. wrkr 0
7513.580305954:main thread : action 17
queue[DA]:Reg: finalizing construction of worker thread
pool (numworkerThreads 1)
7513.580316151:main thread : action 17
queue[DA]:Reg/w0: finalizing construction of worker
instance data (for 24 actions)
7513.580326238:main thread : action 17
queue[DA]: queue finished initialization
7513.580335652:main thread : action 17
queue[DA]:Reg: high activity - starting 1 additional worker
thread(s).
7513.580357559:main thread : action 17
queue[DA]:Reg: started with state 0, num workers now 1
7513.580370836:main thread : action 17 queue: DA queue
initialized, disk queue 0x1f8d8a0
7513.580381162:main thread : action 17 queue: queue
finished initialization
7513.580392905:main thread : Action
builtin:omfile[0x1f80aa0]: queue 0x1f813c0 started
7513.580403043:main thread : iterateAllActions calling into
action 0x1f81b20
7513.580412600:main thread : action 18 queue: starting
queue
7513.580422480:main thread : action 18 queue: is
disk-assisted, disk will be used on demand
7513.580433380:main thread : action 18 queue:
params: type 1, enq-only 0, disk assisted 1, spoolDir
'/data/spool/rsyslog', maxFileSz 104857600, maxQSize 10240, lqsize 0,
pqsize 0, child 0, full delay 3750, light delay 7140, deq batch size 128,
high wtrmrk 7500, low wtrmrk 7140, discardmrk 10000, max wrkr 1, min msgs
f. wrkr 10240
7513.580445910:main thread : action 18 queue:Reg:
finalizing construction of worker thread pool (numworkerThreads 1)
7513.580460289:7fc9f2e53700: thread created, tid 12518, name
'rs:action 17 queue[DA]:'
7513.580477040:action 17 queue[DA]:Reg/w0: wti
0x1f8eae0: worker starting
7513.580488667:main thread : action 18 queue:Reg/w0:
finalizing construction of worker instance data (for 24 actions)
7513.580500625:action 17 queue[DA]:Reg/w0:
DeleteProcessedBatch: we deleted 0 objects and enqueued 0 objects
7513.580511401:action 17 queue[DA]:Reg/w0:
doDeleteBatch: delete batch from store, new sizes: log 2, phys 2
7513.580522726:main thread : action 18
queue:DAwpool: finalizing construction of worker thread pool
(numworkerThreads 1)
7513.580536413:action 17 queue[DA]:Reg/w0: strm
0x1f8e710: file 10 read 0 bytes
7513.580547810:action 17 queue[DA]:Reg/w0: strm
0x1f8e710: file 10 EOF
7513.580558849:main thread : action 18
queue:DAwpool/w0: finalizing construction of worker instance data
(for 24 actions)
7513.580570849:action 17 queue[DA]:Reg/w0: strm
0x1f8e710: file 10(18) closing
7513.580582977:main thread : action 18
queue[DA]: starting queue
7513.580594778:main thread : action 18
queue[DA]: .qi file name is
'/data/spool/rsyslog/18.qi', len 54
7513.580604996:main thread : action 18
queue[DA]: I am a child
7513.580616251:action 17 queue[DA]:Reg/w0: file
'/data/spool/rsyslog/18.00000023' opened as #-1 with mode 384
7513.580628813:main thread : action 18
queue[DA]: clean startup, no .qi file found
7513.580639503:main thread : action 18
queue[DA]: state -2040 reading .qi file - can not read
persisted info (if any)
7513.580650879:action 17 queue[DA]:Reg/w0: strm
0x1f8e710: open error 2, file
'/data/spool/rsyslog/18.00000023': No such file or directory
7513.580662118:action 17 queue[DA]:Reg/w0:
objDeserialize error -2040 during header processing - trying to recover
7513.580673919:main thread : file stream N/A params:
flush interval 0, async write 0
7513.580685739:action 17 queue[DA]:Reg/w0: file
'/data/spool/rsyslog/18.00000023' opened as #-1 with mode 384
7513.580697709:main thread : file stream N/A params:
flush interval 0, async write 0
7513.580709596:action 17 queue[DA]:Reg/w0: strm
0x1f8e710: open error 2, file
'/data/spool/rsyslog/18.00000023': No such file or directory
7513.580720404:action 17 queue[DA]:Reg/w0:
deserializer has possibly been able to re-sync and recover, state -2040
7513.580730891:action 17 queue[DA]:Reg/w0: action
17 queue[DA]: error -2040 dequeueing element - ignoring, but
strange things may happen
7513.580741206:action 17 queue[DA]:Reg/w0: action
17 queue[DA]: got 'file not found' error -2040, queue defunct
7513.580751789:action 17 queue[DA]:Reg/w0: strm
0x1f8e060: file 8(18) closing
7513.580762612:action 17 queue[DA]:Reg/w0: strm
0x1f8e060: file 8(18) flush, buflen 0 (no need to flush)
7513.580773901:main thread : file stream N/A params:
flush interval 0, async write 0
7513.580785890:action 17 queue[DA]:Reg/w0: strm
0x1f8e710: file -1(18) closing
7513.580797390:action 17 queue[DA]:Reg/w0: strm
0x1f8e490: file 9(18) closing
7513.580808658:main thread : action 18
queue[DA]: params: type 2, enq-only 0, disk assisted 0,
spoolDir '/data/spool/rsyslog', maxFileSz 104857600, maxQSize 0, lqsize 0,
pqsize 0, child 1, full delay -1, light delay -1, deq batch size 8, high
wtrmrk 0, low wtrmrk 1, discardmrk 0, max wrkr 1, min msgs f. wrkr 0
7513.580821843:action 17 queue[DA]:Reg/w0:
strmCloseFile: deleting '/data/spool/rsyslog/18.00000022'
7513.580833778:main thread : action 18
queue[DA]:Reg: finalizing construction of worker thread
pool (numworkerThreads 1)
7513.580844603:main thread : action 18
queue[DA]:Reg/w0: finalizing construction of worker
instance data (for 24 actions)
7513.580854526:main thread : action 18
queue[DA]: queue finished initialization
7513.580866880:action 17 queue[DA]:Reg/w0: action
17 queue[DA]: DA queue is in emergency mode, disabling DA in
parent
7513.580880699:main thread : action 18 queue: DA queue
initialized, disk queue 0x1f9bcc0
7513.580891052:main thread : action 18 queue: queue
finished initialization
7513.580903614:action 17 queue[DA]:Reg/w0: Called
LogMsg, msg: fatal error on disk queue 'action 17 queue[DA]',
emergency switch to direct mode
7513.580915969:main thread : Action
builtin:omfile[0x1f81b20]: queue 0x1f825f0 started
rsyslogd: fatal error on disk queue 'action 17 queue[DA]',
emergency switch to direct mode [v8.12.0 try
http://www.rsyslog.com/e/2040 ]
[/code:b9f89l0m]
This may also be useful:
[code:b9f89l0m]
7513.411880028:main thread : action 15 queue: parameter
dump:
7513.411889389:main thread : action 15 queue:
queue.filename '<snip>'
7513.411898552:main thread : action 15 queue:
queue.size: 10240
7513.411907151:main thread : action 15 queue:
queue.dequeuebatchsize: 128
7513.411916066:main thread : action 15 queue:
queue.maxdiskspace: 21474836480
7513.411924361:main thread : action 15 queue:
queue.highwatermark: 7500
7513.411932902:main thread : action 15 queue:
queue.lowwatermark: -1
7513.411941417:main thread : action 15 queue:
queue.fulldelaymark: -1
7513.411949954:main thread : action 15 queue:
queue.lightdelaymark: -1
7513.411958758:main thread : action 15 queue:
queue.discardmark: 10000
7513.411967316:main thread : action 15 queue:
queue.discardseverity: 7
7513.411975753:main thread : action 15 queue:
queue.checkpointinterval: 0
7513.411984378:main thread : action 15 queue:
queue.syncqueuefiles: 0
7513.411993007:main thread : action 15 queue:
queue.type: 1 [LinkedList]
7513.412001579:main thread : action 15 queue:
queue.workerthreads: 1
7513.412009955:main thread : action 15 queue:
queue.timeoutshutdown: 0
7513.412018515:main thread : action 15 queue:
queue.timeoutactioncompletion: 1000
7513.412027094:main thread : action 15 queue:
queue.timeoutenqueue: 2000
7513.412035775:main thread : action 15 queue:
queue.timeoutworkerthreadshutdown: 60000
7513.412044254:main thread : action 15 queue:
queue.workerthreadminimummessages: -1
7513.412051456:main thread : action 15 queue:
queue.maxfilesize: 104857600
7513.412058229:main thread : action 15 queue:
queue.saveonshutdown: 1
7513.412065967:main thread : action 15 queue:
queue.dequeueslowdown: 0
7513.412074882:main thread : action 15 queue:
queue.dequeuetimebegin: 0
7513.412083616:main thread : action 15 queue:
queue.dequeuetimeend: 25
...
7513.446769809:main thread : action 17 queue: parameter
dump:
7513.446778941:main thread : action 17 queue:
queue.filename ’<snip>’
7513.446788052:main thread : action 17 queue:
queue.size: 10240
7513.446796286:main thread : action 17 queue:
queue.dequeuebatchsize: 128
7513.446805074:main thread : action 17 queue:
queue.maxdiskspace: 21474836480
7513.446813863:main thread : action 17 queue:
queue.highwatermark: 7500
7513.446822041:main thread : action 17 queue:
queue.lowwatermark: -1
7513.446830720:main thread : action 17 queue:
queue.fulldelaymark: -1
7513.446838525:main thread : action 17 queue:
queue.lightdelaymark: -1
7513.446846936:main thread : action 17 queue:
queue.discardmark: 10000
7513.446854795:main thread : action 17 queue:
queue.discardseverity: 7
7513.446863385:main thread : action 17 queue:
queue.checkpointinterval: 0
7513.446872350:main thread : action 17 queue:
queue.syncqueuefiles: 0
7513.446880609:main thread : action 17 queue:
queue.type: 1 [LinkedList]
7513.446889170:main thread : action 17 queue:
queue.workerthreads: 1
7513.446897182:main thread : action 17 queue:
queue.timeoutshutdown: 0
7513.446906089:main thread : action 17 queue:
queue.timeoutactioncompletion: 1000
7513.446914993:main thread : action 17 queue:
queue.timeoutenqueue: 2000
7513.446923139:main thread : action 17 queue:
queue.timeoutworkerthreadshutdown: 60000
7513.446931650:main thread : action 17 queue:
queue.workerthreadminimummessages: -1
7513.446940704:main thread : action 17 queue:
queue.maxfilesize: 104857600
7513.446949165:main thread : action 17 queue:
queue.saveonshutdown: 1
7513.446957648:main thread : action 17 queue:
queue.dequeueslowdown: 0
7513.446965620:main thread : action 17 queue:
queue.dequeuetimebegin: 0
7513.446974079:main thread : action 17 queue:
queue.dequeuetimeend: 25
[/code:b9f89l0m]
More information about the rsyslog-notify
mailing list