[rsyslog] working example of omkafka?

mostolog at gmail.com mostolog at gmail.com
Tue Jan 31 12:17:25 CET 2017


Finally managed to get it working, although not fully working :(

First: rsyslog wasn't adding topics to kafka, cause I was using "@" 
within topic names, and that's an unsupported character, and maybe 
because I omitted :9092

Second: logstash is still not able to connect with kafka, cause:

11:41:02.175 [[main]<kafka] WARN org.apache.kafka.clients.ClientUtils - 
Removing server from bootstrap.servers as DNS resolution failed: 
cluster_kafka:9092

Probably because "cluster_kafka" is the cluster alias, but no a valid 
broker name/IP, because I deployed kafka using "docker service".
Yet have to figure it out.

Thanks!

El 31/01/17 a las 10:22, mostolog at gmail.com escribió:
> El 30/01/17 a las 19:25, Andrew Griffin escribió:
>> I have a rsyslog -> kafka -> splunk stack working pretty well, I 
>> could probably answer a few of your questions -
>>
>> You can list topics (and a lot of other stuff) on the kafka brokers 
>> themselves using the kafka-topics.sh script included with kafka.  e.g.:
>>
>> bin/kafka-topics.sh —zookeeper=localhost:2181 —list
> only __consumer_offsets is shown, so probably I'm not adding topics 
> correctly
>
>> bin/kafka-topics.sh —zookeeper=locahost:2181 —topic “topic” —describe
> Topic: __consumer_offsets    Partition: 0    Leader: 9 Replicas: 
> 9,10,15    Isr: 10,9,15
> Topic: __consumer_offsets    Partition: 1    Leader: 10 Replicas: 
> 10,15,9    Isr: 10,9,15
> Topic: __consumer_offsets    Partition: 2    Leader: 15 Replicas: 
> 15,9,10    Isr: 10,9,15
> ...
> Topic: __consumer_offsets    Partition: 47    Leader: 15 Replicas: 
> 15,10,9    Isr: 10,9,15
> Topic: __consumer_offsets    Partition: 48    Leader: 9 Replicas: 
> 9,10,15    Isr: 10,9,15
> Topic: __consumer_offsets    Partition: 49    Leader: 10 Replicas: 
> 10,15,9    Isr: 10,9,15
>
> does this mean the cluster is properly formed?
>
>> I’d recommend using kafka-manager to manage your cluster.  It’ll give 
>> you a much quicker look in to your topics, your brokers, consumers, 
>> and throughput.  It also makes creating and deleting topics easy.
> It isn't able to show cluster list, so perhaps problems connecting to zk?
>
>> If you’re not seeing your topics get created the first place I’d look 
>> is in the kafka broker logs themselves - server.log and 
>> kafkaServer.out - then work your way back from there.  As you’ve 
>> found, omkafka isn’t terribly verbose when it comes to error reporting.
> plenty of:
> [2017-01-30 16:35:05,177] INFO [Group Metadata Manager on Broker 15]: 
> Removed 0 expired offsets in 0 milliseconds. 
> (kafka.coordinator.GroupMetadataManager)
> [2017-01-30 16:45:05,177] INFO [Group Metadata Manager on Broker 15]: 
> Removed 0 expired offsets in 0 milliseconds. 
> (kafka.coordinator.GroupMetadataManager)
> [2017-01-30 16:55:05,177] INFO [Group Metadata Manager on Broker 15]: 
> Removed 0 expired offsets in 0 milliseconds. 
> (kafka.coordinator.GroupMetadataManager)
>
>
>
>> For your timeout issues, the first place I’d look to is the local 
>> firewall configuration.  Also in your "broker=["cluster_kafka”]” 
>> portion are you including the port number for the broker (I’m 
>> assuming 9092)?
> I thought port was added by default. I'll try again.
>
> Thanks!



More information about the rsyslog mailing list