[rsyslog-notify] Forum Thread: Rsyslog 7.4 forwarding to an Amazon ELB - (Mode 'post')

[noreply at adiscon.com]

User: dmh 
Forumlink: http://kb.monitorware.com/viewtopic.php?p=26109#p26109

Message: 
----------
Hi Team,

I'm trying to build a HA solution in Amazon for logs. Platform is
Centos/Rhel 7 and client is of course Rsyslog (version 7.4.7 is the current
version in the Centos 7 repo).

I have logstash sitting behind an amazon ELB accepting TCP 5140 from
clients ( note ELB cannot forward UDP ).

rsyslog has a simple forward '*.*   @@elbname:5140' into logstash which
then goes into elasticsearch.

this all works for a while but amazon elastic load balancers change their
IP addresses quite regularly - sometimes > once a day. For this reason
their TTL is usually quite short @ 60 seconds - refer this document
relating to java apps but same principle:
<!-- m --><a class="postlink"
href="http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/java-dg-jvm-ttl.html">http://docs.aws.amazon.com/AWSSdkDocsJa
... m-ttl.html</a><!-- m -->

rsyslog seems to cache the original IP and not re-query the DNS name when
the TTL expires. As it's TCP forwarding this can of course hang the stack
as rsyslog tries to send to a stale IP.

i've worked around this by doing a kill -HUP `cat /var/run/syslogd.pid`
which stabilises the system.

Is there any other solution anyone can think of?

Thanks for reading!