[rsyslog] Stability of omkafka when broker DNS names disappear

Xavier Lange xrlange at tureus.com
Tue May 30 21:50:09 CEST 2017


I am using imrelp/omkafka to feed a kafka cluster and I find there are
some instabilities in my dynamic DNS environment. DNS is managed by
Kubernetes.

I list all 7 of my brokers in the omkafka configuration and if any one
of those hosts goes down and the DNS name disappears, the omkafka
plugin spews errors, in a tight loop, about the host being down.

Here's the config:

  action(
    Broker=["kafka-0.kafka-headless.default.svc.cluster.local:9092",
"kafka-1.kafka-headless.default.svc.cluster.local:9092",
"kafka-2.kafka-headless.default.svc.cluster.local:9092"]
    Type="omkafka"
    Topic="the-topic"
    confParam=["compression.codec=gzip",
"queue.buffering.max.messages=10000000"]
    Template="the-topic-json"
    errorfile="/dev/null"
    Partitions.auto="on"
    queue.type="linkedlist"
    queue.size="100000"
    queue.timeoutenqueue="0"
  )

So if kafka-0, kafka-1, or kafka-2 disappear, then rsyslog can
overflow my docker logs with error messages.

Is this type of behavior going to be addressed in the upcoming rsyslog
work? Is this something I can help with?


More information about the rsyslog mailing list