[rsyslog] working example of omkafka?

mostolog at gmail.com mostolog at gmail.com
Tue Jan 31 10:22:14 CET 2017


El 30/01/17 a las 19:25, Andrew Griffin escribió:
> I have a rsyslog -> kafka -> splunk stack working pretty well, I could 
> probably answer a few of your questions -
>
> You can list topics (and a lot of other stuff) on the kafka brokers 
> themselves using the kafka-topics.sh script included with kafka.  e.g.:
>
> bin/kafka-topics.sh —zookeeper=localhost:2181 —list
only __consumer_offsets is shown, so probably I'm not adding topics 
correctly

> bin/kafka-topics.sh —zookeeper=locahost:2181 —topic “topic” —describe
Topic: __consumer_offsets    Partition: 0    Leader: 9    Replicas: 
9,10,15    Isr: 10,9,15
Topic: __consumer_offsets    Partition: 1    Leader: 10    Replicas: 
10,15,9    Isr: 10,9,15
Topic: __consumer_offsets    Partition: 2    Leader: 15    Replicas: 
15,9,10    Isr: 10,9,15
...
Topic: __consumer_offsets    Partition: 47    Leader: 15 Replicas: 
15,10,9    Isr: 10,9,15
Topic: __consumer_offsets    Partition: 48    Leader: 9    Replicas: 
9,10,15    Isr: 10,9,15
Topic: __consumer_offsets    Partition: 49    Leader: 10 Replicas: 
10,15,9    Isr: 10,9,15

does this mean the cluster is properly formed?

> I’d recommend using kafka-manager to manage your cluster.  It’ll give 
> you a much quicker look in to your topics, your brokers, consumers, 
> and throughput.  It also makes creating and deleting topics easy.
It isn't able to show cluster list, so perhaps problems connecting to zk?

> If you’re not seeing your topics get created the first place I’d look 
> is in the kafka broker logs themselves - server.log and 
> kafkaServer.out - then work your way back from there.  As you’ve 
> found, omkafka isn’t terribly verbose when it comes to error reporting.
plenty of:
[2017-01-30 16:35:05,177] INFO [Group Metadata Manager on Broker 15]: 
Removed 0 expired offsets in 0 milliseconds. 
(kafka.coordinator.GroupMetadataManager)
[2017-01-30 16:45:05,177] INFO [Group Metadata Manager on Broker 15]: 
Removed 0 expired offsets in 0 milliseconds. 
(kafka.coordinator.GroupMetadataManager)
[2017-01-30 16:55:05,177] INFO [Group Metadata Manager on Broker 15]: 
Removed 0 expired offsets in 0 milliseconds. 
(kafka.coordinator.GroupMetadataManager)



> For your timeout issues, the first place I’d look to is the local 
> firewall configuration.  Also in your "broker=["cluster_kafka”]” 
> portion are you including the port number for the broker (I’m assuming 
> 9092)?
I thought port was added by default. I'll try again.

Thanks!


More information about the rsyslog mailing list