[rsyslog] working example of omkafka?

Andrew Griffin andrew_griffin at apple.com
Tue Jan 31 19:17:25 CET 2017


> On Jan 31, 2017, at 1:22 AM, mostolog--- via rsyslog <rsyslog at lists.adiscon.com> wrote:
> 
> 
>> bin/kafka-topics.sh —zookeeper=locahost:2181 —topic “topic” —describe
> Topic: __consumer_offsets    Partition: 0    Leader: 9    Replicas: 9,10,15    Isr: 10,9,15
> Topic: __consumer_offsets    Partition: 1    Leader: 10    Replicas: 10,15,9    Isr: 10,9,15
> Topic: __consumer_offsets    Partition: 2    Leader: 15    Replicas: 15,9,10    Isr: 10,9,15
> ...
> Topic: __consumer_offsets    Partition: 47    Leader: 15 Replicas: 15,10,9    Isr: 10,9,15
> Topic: __consumer_offsets    Partition: 48    Leader: 9    Replicas: 9,10,15    Isr: 10,9,15
> Topic: __consumer_offsets    Partition: 49    Leader: 10 Replicas: 10,15,9    Isr: 10,9,15

The __consumer_offsets topic is a built in topic (default is 50 partitions, 3 replicas).  “ISR” is In Sync Replicas - in your case all of your replicas are in sync.  Your cluster is working fine

>> 
>> I’d recommend using kafka-manager to manage your cluster.  It’ll give you a much quicker look in to your topics, your brokers, consumers, and throughput.  It also makes creating and deleting topics easy.
> It isn't able to show cluster list, so perhaps problems connecting to zk?

You’ll have to add the Kafka cluster to kafka-manager via the UI the first time you fire it up

> 
>> If you’re not seeing your topics get created the first place I’d look is in the kafka broker logs themselves - server.log and kafkaServer.out - then work your way back from there.  As you’ve found, omkafka isn’t terribly verbose when it comes to error reporting.
> plenty of:
> [2017-01-30 16:35:05,177] INFO [Group Metadata Manager on Broker 15]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
> [2017-01-30 16:45:05,177] INFO [Group Metadata Manager on Broker 15]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
> [2017-01-30 16:55:05,177] INFO [Group Metadata Manager on Broker 15]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)

Those are routine messages - basically just notifications on offsets (messages) that are deleted as they hit their expiration.  In Kafka data retention is based either on time or size on disk

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3391 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20170131/73c6ae9b/attachment.bin>


More information about the rsyslog mailing list