[rsyslog] working example of omkafka?

mostolog at gmail.com mostolog at gmail.com
Tue Jan 31 12:42:46 CET 2017


rsyslog 8.23 seems to be sucessfully writing on kafka 0.10

on the other hand, logstash doesn't use zk_connect anymore, but 
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html#plugins-inputs-kafka-bootstrap_servers

Seems that docker service (exposing cluster name) is not compatible with 
http://kafka.apache.org/documentation/#configuration 
advertised.listeners in logstash 5.1.1...or something else

El 31/01/17 a las 12:36, Radu Gheorghe escribió:
> Hi,
>
> Maybe this tutorial helps:
> https://sematext.com/blog/2015/10/06/recipe-rsyslog-apache-kafka-logstash/
>
> I think you'll need to take care of the version of Kafka (so that it
> works with rsyslog, it sounds like you figured that out) and also the
> version of Logstash. For the latter, you have a compatibility matrix
> in the Logstash Kafka Input docs:
> https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Tue, Jan 31, 2017 at 1:17 PM, mostolog--- via rsyslog
> <rsyslog at lists.adiscon.com> wrote:
>> Finally managed to get it working, although not fully working :(
>>
>> First: rsyslog wasn't adding topics to kafka, cause I was using "@" within
>> topic names, and that's an unsupported character, and maybe because I
>> omitted :9092
>>
>> Second: logstash is still not able to connect with kafka, cause:
>>
>> 11:41:02.175 [[main]<kafka] WARN org.apache.kafka.clients.ClientUtils -
>> Removing server from bootstrap.servers as DNS resolution failed:
>> cluster_kafka:9092
>>
>> Probably because "cluster_kafka" is the cluster alias, but no a valid broker
>> name/IP, because I deployed kafka using "docker service".
>> Yet have to figure it out.
>>
>> Thanks!
>>
>> El 31/01/17 a las 10:22, mostolog at gmail.com escribió:
>>
>>> El 30/01/17 a las 19:25, Andrew Griffin escribió:
>>>> I have a rsyslog -> kafka -> splunk stack working pretty well, I could
>>>> probably answer a few of your questions -
>>>>
>>>> You can list topics (and a lot of other stuff) on the kafka brokers
>>>> themselves using the kafka-topics.sh script included with kafka.  e.g.:
>>>>
>>>> bin/kafka-topics.sh —zookeeper=localhost:2181 —list
>>> only __consumer_offsets is shown, so probably I'm not adding topics
>>> correctly
>>>
>>>> bin/kafka-topics.sh —zookeeper=locahost:2181 —topic “topic” —describe
>>> Topic: __consumer_offsets    Partition: 0    Leader: 9 Replicas: 9,10,15
>>> Isr: 10,9,15
>>> Topic: __consumer_offsets    Partition: 1    Leader: 10 Replicas: 10,15,9
>>> Isr: 10,9,15
>>> Topic: __consumer_offsets    Partition: 2    Leader: 15 Replicas: 15,9,10
>>> Isr: 10,9,15
>>> ...
>>> Topic: __consumer_offsets    Partition: 47    Leader: 15 Replicas: 15,10,9
>>> Isr: 10,9,15
>>> Topic: __consumer_offsets    Partition: 48    Leader: 9 Replicas: 9,10,15
>>> Isr: 10,9,15
>>> Topic: __consumer_offsets    Partition: 49    Leader: 10 Replicas: 10,15,9
>>> Isr: 10,9,15
>>>
>>> does this mean the cluster is properly formed?
>>>
>>>> I’d recommend using kafka-manager to manage your cluster.  It’ll give you
>>>> a much quicker look in to your topics, your brokers, consumers, and
>>>> throughput.  It also makes creating and deleting topics easy.
>>> It isn't able to show cluster list, so perhaps problems connecting to zk?
>>>
>>>> If you’re not seeing your topics get created the first place I’d look is
>>>> in the kafka broker logs themselves - server.log and kafkaServer.out - then
>>>> work your way back from there.  As you’ve found, omkafka isn’t terribly
>>>> verbose when it comes to error reporting.
>>> plenty of:
>>> [2017-01-30 16:35:05,177] INFO [Group Metadata Manager on Broker 15]:
>>> Removed 0 expired offsets in 0 milliseconds.
>>> (kafka.coordinator.GroupMetadataManager)
>>> [2017-01-30 16:45:05,177] INFO [Group Metadata Manager on Broker 15]:
>>> Removed 0 expired offsets in 0 milliseconds.
>>> (kafka.coordinator.GroupMetadataManager)
>>> [2017-01-30 16:55:05,177] INFO [Group Metadata Manager on Broker 15]:
>>> Removed 0 expired offsets in 0 milliseconds.
>>> (kafka.coordinator.GroupMetadataManager)
>>>
>>>
>>>
>>>> For your timeout issues, the first place I’d look to is the local
>>>> firewall configuration.  Also in your "broker=["cluster_kafka”]” portion are
>>>> you including the port number for the broker (I’m assuming 9092)?
>>> I thought port was added by default. I'll try again.
>>>
>>> Thanks!
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.



More information about the rsyslog mailing list