Caused by: java.net.SocketException: No buffer space available (maximum connections reached?): connect
This is most likely due to the number of ephemeral TCP ports being exhausted, the server has too many active client connections due to which the ephemeral TCP ports are all used up and in this case no more ports can be allocated to a new client connection request resulting in the behaviour we see here that no new connections can be setup.
Quoting from Wikipedia entry:
“Ephemeral port is a transport protocol port for Internet Protocol (IP) communications allocated automatically from a predefined range by the TCP/IP Stack Software. It is typically used by the TCP, UDP or SCTP as port for the client end of a client-server communication when the application doesn’t bind the socket to a specific port number, or by a server application to free up service’s well known listening port and establish a service connection to the client host. The allocations are temporary and only valid for the duration of the connection. After completion of the communication session the ports become available for reuse, although most implementations simply increment the last used port number until the ephemeral port range is exhausted.”
This is a known issue on Windows boxes, a hotfix is available from Microsoft: “Kernel sockets leak on a multiprocessor computer that is running Windows Server 2008 R2 or Windows 7” – http://support.microsoft.com/kb/2577795