[rsyslog] Imkafka problem on topic subscription

Luigi Tagliamonte luigi.tagliamonte86 at gmail.com
Wed Sep 20 21:40:32 CEST 2017


Dear community,
sorry if I'm re-posting here the GitHub issue
https://github.com/rsyslog/rsyslog/issues/1771  , I'm try to reach other
people that maybe  had my same problem.

I'trying to build a centralized logging system using:
omkafka -> kafka -> imkafka -> omfile
The rsyslog sender configuration is:

module(load="omkafka")
$WorkDirectory /var/spool/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList   # run asynchronously
$ActionResumeRetryCount -1    # infinite retries if host is down

if $programname contains 'backend'  then
action(type="omkafka" topic="logging" broker=["IP:9092"]
       confParam=["compression.codec=snappy",
                  "socket.timeout.ms=15",
                  "socket.keepalive.enable=true"]
      )
stop


The sender is working properly i can see the messages in the kafka topic
using:
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic
logging --from-beginning

On the receiver part i configured rsyslog with the following:


module(load="imkafka")
input(type="imkafka" topic="logging" broker=["IP:9092"] ruleset="pRuleset"
consumergroup="default"
        confParam=["compression.codec=snappy",
                  "socket.timeout.ms=15",
                  "socket.keepalive.enable=true"]
)

ruleset(name="pRuleset") {
      action(type="omfile" file="/var/log/backend")
}


I'm using for sender and receiver:
rsyslogd 8.29.0, compiled with:
PLATFORM: x86_64-redhat-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: No
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
Number of Bits in RainerScript integers: 64
and Kafka version : 0.11.0.0

I did a quick look at the code, and matching the logs with the code I think
the thread is having a problem on the subscription here:
https://github.com/rsyslog/rsyslog/blob/master/plugins/imkafka/imkafka.c#L459-L464
The debug logs for the Kafka library are in the GitHub case and they don't
show any additional info that helps in solving the issue.

Does someone use imkafka? If so can please share the conf and the versions
that is using?
Looking forward to your reply.
Regards
L.


More information about the rsyslog mailing list