[rsyslog] Stability of omkafka when broker DNS names disappear

Rainer Gerhards rgerhards at hq.adiscon.com
Wed May 31 08:01:20 CEST 2017


I suggest that you open a github issue tracker with as many details as
possible (e.g. which error messages?) . That makes it easier to ensure
it is adressed as part of the current kafka work. We have a kafka
project defined on github and keep track of to-do items via it.

Rainer

2017-05-30 21:50 GMT+02:00 Xavier Lange <xrlange at tureus.com>:
> I am using imrelp/omkafka to feed a kafka cluster and I find there are
> some instabilities in my dynamic DNS environment. DNS is managed by
> Kubernetes.
>
> I list all 7 of my brokers in the omkafka configuration and if any one
> of those hosts goes down and the DNS name disappears, the omkafka
> plugin spews errors, in a tight loop, about the host being down.
>
> Here's the config:
>
>   action(
>     Broker=["kafka-0.kafka-headless.default.svc.cluster.local:9092",
> "kafka-1.kafka-headless.default.svc.cluster.local:9092",
> "kafka-2.kafka-headless.default.svc.cluster.local:9092"]
>     Type="omkafka"
>     Topic="the-topic"
>     confParam=["compression.codec=gzip",
> "queue.buffering.max.messages=10000000"]
>     Template="the-topic-json"
>     errorfile="/dev/null"
>     Partitions.auto="on"
>     queue.type="linkedlist"
>     queue.size="100000"
>     queue.timeoutenqueue="0"
>   )
>
> So if kafka-0, kafka-1, or kafka-2 disappear, then rsyslog can
> overflow my docker logs with error messages.
>
> Is this type of behavior going to be addressed in the upcoming rsyslog
> work? Is this something I can help with?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.


More information about the rsyslog mailing list