[rsyslog] Imkafka problem on topic subscription
luigi.tagliamonte86 at gmail.com
Wed Sep 20 21:40:32 CEST 2017
sorry if I'm re-posting here the GitHub issue
https://github.com/rsyslog/rsyslog/issues/1771 , I'm try to reach other
people that maybe had my same problem.
I'trying to build a centralized logging system using:
omkafka -> kafka -> imkafka -> omfile
The rsyslog sender configuration is:
$WorkDirectory /var/spool/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down
if $programname contains 'backend' then
action(type="omkafka" topic="logging" broker=["IP:9092"]
The sender is working properly i can see the messages in the kafka topic
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic
On the receiver part i configured rsyslog with the following:
input(type="imkafka" topic="logging" broker=["IP:9092"] ruleset="pRuleset"
I'm using for sender and receiver:
rsyslogd 8.29.0, compiled with:
PLATFORM (lsb_release -d):
GSSAPI Kerberos 5 support: No
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
Number of Bits in RainerScript integers: 64
and Kafka version : 0.11.0.0
I did a quick look at the code, and matching the logs with the code I think
the thread is having a problem on the subscription here:
The debug logs for the Kafka library are in the GitHub case and they don't
show any additional info that helps in solving the issue.
Does someone use imkafka? If so can please share the conf and the versions
that is using?
Looking forward to your reply.
More information about the rsyslog