[rsyslog] working example of omkafka?

mostolog at gmail.com mostolog at gmail.com
Tue Jan 31 19:32:25 CET 2017


I'll comment with team tomorrow.

Actually, all problems seems to be located on logstash side, cause I 
have been able to confirm zookeeper and kafka clusters are ok.

El 31/01/17 a las 19:28, Andrew Griffin escribió:
> Logstash works *ok* as a standalone consumer but it doesn’t scale well 
> to a multi-consumer setup.  If I were you I would consider using Kafka 
> Connect instead of logstash.  It has all the sources/sinks you’d ever 
> need and since it’s worker configuration is stored in a kafka topic, 
> separate from the workers, it’s a piece of cake to containerize
>
> Andrew Griffin
>  ETS / Integration Services
> ☏ 408-783-8348
>
>> On Jan 31, 2017, at 3:42 AM, mostolog--- via rsyslog 
>> <rsyslog at lists.adiscon.com <mailto:rsyslog at lists.adiscon.com>> wrote:
>>
>> rsyslog 8.23 seems to be sucessfully writing on kafka 0.10
>>
>> on the other hand, logstash doesn't use zk_connect anymore, but 
>> https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html#plugins-inputs-kafka-bootstrap_servers
>>
>> Seems that docker service (exposing cluster name) is not compatible 
>> with http://kafka.apache.org/documentation/#configuration 
>> advertised.listeners in logstash 5.1.1...or something else
>>
>> El 31/01/17 a las 12:36, Radu Gheorghe escribió:
>>> Hi,
>>>
>>> Maybe this tutorial helps:
>>> https://sematext.com/blog/2015/10/06/recipe-rsyslog-apache-kafka-logstash/
>>>
>>> I think you'll need to take care of the version of Kafka (so that it
>>> works with rsyslog, it sounds like you figured that out) and also the
>>> version of Logstash. For the latter, you have a compatibility matrix
>>> in the Logstash Kafka Input docs:
>>> https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html
>>> --
>>> Performance Monitoring * Log Analytics * Search Analytics
>>> Solr & Elasticsearch Support * http://sematext.com/
>>>
>>>
>>> On Tue, Jan 31, 2017 at 1:17 PM, mostolog--- via rsyslog
>>> <rsyslog at lists.adiscon.com> wrote:
>>>> Finally managed to get it working, although not fully working :(
>>>>
>>>> First: rsyslog wasn't adding topics to kafka, cause I was using "@" 
>>>> within
>>>> topic names, and that's an unsupported character, and maybe because I
>>>> omitted :9092
>>>>
>>>> Second: logstash is still not able to connect with kafka, cause:
>>>>
>>>> 11:41:02.175 [[main]<kafka] WARN org.apache.kafka.clients.ClientUtils -
>>>> Removing server from bootstrap.servers as DNS resolution failed:
>>>> cluster_kafka:9092
>>>>
>>>> Probably because "cluster_kafka" is the cluster alias, but no a 
>>>> valid broker
>>>> name/IP, because I deployed kafka using "docker service".
>>>> Yet have to figure it out.
>>>>
>>>> Thanks!
>>>>
>>>> El 31/01/17 a las 10:22, mostolog at gmail.com escribió:
>>>>
>>>>> El 30/01/17 a las 19:25, Andrew Griffin escribió:
>>>>>> I have a rsyslog -> kafka -> splunk stack working pretty well, I 
>>>>>> could
>>>>>> probably answer a few of your questions -
>>>>>>
>>>>>> You can list topics (and a lot of other stuff) on the kafka brokers
>>>>>> themselves using the kafka-topics.sh script included with kafka. 
>>>>>>  e.g.:
>>>>>>
>>>>>> bin/kafka-topics.sh —zookeeper=localhost:2181 —list
>>>>> only __consumer_offsets is shown, so probably I'm not adding topics
>>>>> correctly
>>>>>
>>>>>> bin/kafka-topics.sh —zookeeper=locahost:2181 —topic “topic” —describe
>>>>> Topic: __consumer_offsets    Partition: 0    Leader: 9 Replicas: 
>>>>> 9,10,15
>>>>> Isr: 10,9,15
>>>>> Topic: __consumer_offsets    Partition: 1    Leader: 10 Replicas: 
>>>>> 10,15,9
>>>>> Isr: 10,9,15
>>>>> Topic: __consumer_offsets    Partition: 2    Leader: 15 Replicas: 
>>>>> 15,9,10
>>>>> Isr: 10,9,15
>>>>> ...
>>>>> Topic: __consumer_offsets    Partition: 47    Leader: 15 Replicas: 
>>>>> 15,10,9
>>>>> Isr: 10,9,15
>>>>> Topic: __consumer_offsets    Partition: 48    Leader: 9 Replicas: 
>>>>> 9,10,15
>>>>> Isr: 10,9,15
>>>>> Topic: __consumer_offsets    Partition: 49    Leader: 10 Replicas: 
>>>>> 10,15,9
>>>>> Isr: 10,9,15
>>>>>
>>>>> does this mean the cluster is properly formed?
>>>>>
>>>>>> I’d recommend using kafka-manager to manage your cluster.  It’ll 
>>>>>> give you
>>>>>> a much quicker look in to your topics, your brokers, consumers, and
>>>>>> throughput.  It also makes creating and deleting topics easy.
>>>>> It isn't able to show cluster list, so perhaps problems connecting 
>>>>> to zk?
>>>>>
>>>>>> If you’re not seeing your topics get created the first place I’d 
>>>>>> look is
>>>>>> in the kafka broker logs themselves - server.log and 
>>>>>> kafkaServer.out - then
>>>>>> work your way back from there.  As you’ve found, omkafka isn’t 
>>>>>> terribly
>>>>>> verbose when it comes to error reporting.
>>>>> plenty of:
>>>>> [2017-01-30 16:35:05,177] INFO [Group Metadata Manager on Broker 15]:
>>>>> Removed 0 expired offsets in 0 milliseconds.
>>>>> (kafka.coordinator.GroupMetadataManager)
>>>>> [2017-01-30 16:45:05,177] INFO [Group Metadata Manager on Broker 15]:
>>>>> Removed 0 expired offsets in 0 milliseconds.
>>>>> (kafka.coordinator.GroupMetadataManager)
>>>>> [2017-01-30 16:55:05,177] INFO [Group Metadata Manager on Broker 15]:
>>>>> Removed 0 expired offsets in 0 milliseconds.
>>>>> (kafka.coordinator.GroupMetadataManager)
>>>>>
>>>>>
>>>>>
>>>>>> For your timeout issues, the first place I’d look to is the local
>>>>>> firewall configuration.  Also in your "broker=["cluster_kafka”]” 
>>>>>> portion are
>>>>>> you including the port number for the broker (I’m assuming 9092)?
>>>>> I thought port was added by default. I'll try again.
>>>>>
>>>>> Thanks!
>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
>>>> myriad of
>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you 
>>>> DON'T
>>>> LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT 
>>> POST if you DON'T LIKE THAT.
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT 
>> POST if you DON'T LIKE THAT.
>



More information about the rsyslog mailing list