[rsyslog] "too many tcp sessions - dropping incoming request" - tried increasing max sessions to no avail

[Aaron Nichols]

I found the cause of my problem I believe - the $InputTCPMaxSessions value
needs to come after the imtcp module is loaded. This makes sense in
hindsight. Of course, the evidence was right in front of me had I looked in
the right place:

Oct  2 02:34:54 host rsyslogd-3003: invalid or yet-unknown config file
command - have you forgotten to load a module? [try
http://www.rsyslog.com/e/3003 ]
Oct  2 02:34:54 host rsyslogd: the last error occured in
/etc/rsyslog.d/rsyslog-server.conf, line 4

I have to thank you guys for making an effort to provide informative errors
with URL references.

Thanks,
Aaron

On Thu, Oct 1, 2009 at 5:52 PM, Aaron Nichols <anichols at trumped.org> wrote:

> Hi all,
>
> I am seeing the following in my logs running rsyslog 4.4.1:
>
> Oct  1 23:29:24 host rsyslogd-2079: too many tcp sessions - dropping
> incoming request [try http://www.rsyslog.com/e/2079 ]
> Oct  1 23:29:29 host rsyslogd-2079: too many tcp sessions - dropping
> incoming request [try http://www.rsyslog.com/e/2079 ]
> Oct  1 23:29:38 host rsyslogd-2079: too many tcp sessions - dropping
> incoming request [try http://www.rsyslog.com/e/2079 ]
>
> I went to the URL in the message and the answer makes sense so I added the
> following to my configuration:
>
> $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200)
>
> I added it before the imtcp module is loaded so the config looks like:
>
> # TCP Syslog Server:
> # provides TCP syslog reception and GSS-API (if compiled to support it)
> $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200)
> $ModLoad imtcp.so  # load module
> $InputTCPServerRun 514 # start up TCP listener at port 514
>
> I am still seeing the "too many tcp sessions" in the logs. When I take a
> look at netstat (assuming this is the relevant data) I see less than 1000
> *connections* - but I'm not sure if each of these is considered a "session"
>
> $ netstat -an | grep :514 | wc -l
> 269
>
> If I look for only ESTABLISHED connections I see less:
>
> [anichols at inlwr01p1 inlwr01p1]$ netstat -an | grep :514 | grep ESTABLISHED
> | wc -l
> 231
>
> Many of the connections show a SYN_RECV state on the server side - I assume
> these are cases where we aren't completing the tcp handshake due to too many
> connections.
>
> Are there other factors which may be impacting the max connections or is my
> configuration incorrect?
>
> Thanks,
> Aaron
>