Our production systems encounter the dreadful ConnectException again:
2011/12/15 14:51:42:932 : http-10.1.200.34-8080-6 : [com.creamtec.ajaxswing.ServletHelper] Failed to process a request for session = org.apache.catalina.session.StandardSessionFacade@593bdac8 (Exception java.rmi.ConnectException, Connection refused to host: 172.25.161.113; nested exception is:
java.net.ConnectException: Connection refused: connect)
java.rmi.ConnectException: Connection refused to host: 172.25.161.113; nested exception is:
java.net.ConnectException: Connection refused: connect
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:110)
at com.creamtec.ajaxswing.ClientAgentImpl_Stub.processHttpRequest(Unknown Source)
at com.creamtec.ajaxswing.ServletHelper.delegateRequestToClientAgent(ServletHelper.java:549)
at com.creamtec.ajaxswing.ServletHelper.doPocessHttpRequest(ServletHelper.java:381)
at com.creamtec.ajaxswing.ServletHelper.processHttpRequest(ServletHelper.java:293)
at com.creamtec.ajaxswing.AjaxSwingRouter.doPost(AjaxSwingRouter.java:141)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at com.creamtec.ajaxswing.AjaxSwingRouter.service(AjaxSwingRouter.java:106)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:864)
at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:579)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1665)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at java.net.Socket.<init>(Socket.java:375)
at java.net.Socket.<init>(Socket.java:189)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
... 23 more
We checked all the previous causes for this, like:
- 2 network interfaces where one is blocked by firewall
- System.exit() that closes the whole JVM instead of only one session
- OutOfMemory issues
- Filesystem has no space left
- clientAgent.sh not executable
The interesting thing is, that these problems occur on both load balanced servers at nearly the same time. The machines are not at all on high load. The users as it seems get error pages when trying to get new sessions. Already running sessions seem not to be affected. This currently occurs with our own version of the 3.2.3 release. The problem only occured some days ago and previously worked flawlessly for some time.
The logs can be downloaded here:
https://www.zeta-uploader.com/558570766The download will expire in 30 days.
Do you have any other ideas why this could occur?