Leon
Reputation: 1181
hundreds of CLOSE_WAIT connections are eating away my ThreadPools
we have a very cruel situation. We have a site in germany which is an online shop. Multiple times a day we are having huge number of connections in CLOSE_WAIT states showing up in netstat from one and the same ip. It's different IPs but they are all in china. We do have chinese customers still. Checking the access logs, we see that the traffic from the ip in questions comes from one browser (useragent, session id), but it doesn't look like real traffic, for example requesting the / 500 times in a row, without requesting css,js,images behind it. So what we end up with is having like 1000 threads in socketWrite0, from which 820 would be tied to same ip:
"http--0.0.0.0-8443-1201" daemon prio=10 tid=0x00007f7435257800 nid=0x5361 runnable [0x00007f73e162a000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:724)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:449)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349)
at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:748)
at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:126)
at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:559)
at org.apache.coyote.Response.doWrite(Response.java:594)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:398)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:449)
at org.apache.catalina.connector.OutputBuffer.realWriteChars(OutputBuffer.java:473)
at org.apache.tomcat.util.buf.CharChunk.flushBuffer(CharChunk.java:469)
at org.apache.tomcat.util.buf.CharChunk.append(CharChunk.java:295)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:505)
at org.apache.catalina.connector.CoyoteWriter.write(CoyoteWriter.java:143)
at org.apache.catalina.connector.CoyoteWriter.write(CoyoteWriter.java:152)
at com.sun.faces.application.view.WriteBehindStateWriter.flushToWriter(WriteBehindStateWriter.java:240)
at com.sun.faces.application.view.FaceletViewHandlingStrategy.renderView(FaceletViewHandlingStrategy.java:419)
at com.sun.faces.application.view.MultiViewHandler.renderView(MultiViewHandler.java:125)
at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:288)
at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:288)
at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:121)
at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101)
at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:139)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:594)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:840)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:622)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:560)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:488)
at x.y.z.common.web.dispatch.StartPageDispatcherServlet.forward(StartPageDispatcherServlet.java:52)
at x.y.z.common.web.dispatch.StartPageDispatcherServlet.service(StartPageDispatcherServlet.java:37)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at org.jboss.weld.servlet.ConversationPropagationFilter.doFilter(ConversationPropagationFilter.java:62)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at net.anotheria.moskito.web.MoskitoFilter.doFilter(MoskitoFilter.java:110)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at net.anotheria.moskito.web.MoskitoFilter.doFilter(MoskitoFilter.java:110)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at net.anotheria.moskito.web.filters.JourneyFilter.doFilter(JourneyFilter.java:84)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at net.anotheria.moskito.web.filters.MoskitoCommandFilter.doFilter(MoskitoCommandFilter.java:26)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at x.y.z.common.web.useragent.TouchScreenDeviceFilter.doFilter(TouchScreenDeviceFilter.java:42)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at x.y.z.common.web.LandingPageFilter.doFilter(LandingPageFilter.java:44)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at x.y.z.common.web.CharsetFilter.doFilter(CharsetFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:397)
at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)
at java.lang.Thread.run(Thread.java:722)
grepping netstat output shows having 817 connections in CLOSE_WAIT and 3 in ESTABLISHED states for this ip.
accesslogs shows:
140.206.78.100 [13/Feb/2013:15:20:48 +0100] http--0.0.0.0-8443-364 GET o1uNdliDOQhJkDnbvXo4RIZ2.undefined 1276 HTTP/1.1 443 / 200 Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11
140.206.78.100 [13/Feb/2013:15:20:50 +0100] http--0.0.0.0-8443-364 GET o1uNdliDOQhJkDnbvXo4RIZ2.undefined 1259 HTTP/1.1 443 / 200 Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11
140.206.78.100 [13/Feb/2013:15:20:51 +0100] http--0.0.0.0-8443-477 GET o1uNdliDOQhJkDnbvXo4RIZ2.undefined 2991 HTTP/1.1 443 / 200 Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11
140.206.78.100 [13/Feb/2013:15:20:53 +0100] http--0.0.0.0-8443-428 GET o1uNdliDOQhJkDnbvXo4RIZ2.undefined 2456 HTTP/1.1 443 / 200 Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11
140.206.78.100 [13/Feb/2013:15:20:54 +0100] http--0.0.0.0-8443-639 GET o1uNdliDOQhJkDnbvXo4RIZ2.undefined 1305 HTTP/1.1 443 / 200 Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11
140.206.78.100 [13/Feb/2013:15:20:54 +0100] http--0.0.0.0-8443-491 GET o1uNdliDOQhJkDnbvXo4RIZ2.undefined 1326 HTTP/1.1 443 / 200 Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11
140.206.78.100 [13/Feb/2013:15:20:56 +0100] http--0.0.0.0-8443-491 GET o1uNdliDOQhJkDnbvXo4RIZ2.undefined 1293 HTTP/1.1 443 / 200 Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11
140.206.78.100 [13/Feb/2013:15:20:57 +0100] http--0.0.0.0-8443-663 GET o1uNdliDOQhJkDnbvXo4RIZ2.undefined 1315 HTTP/1.1 443 / 200 Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11
140.206.78.100 [13/Feb/2013:15:20:59 +0100] http--0.0.0.0-8443-663 GET o1uNdliDOQhJkDnbvXo4RIZ2.undefined 1277 HTTP/1.1 443 / 200 Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11
140.206.78.100 [13/Feb/2013:15:21:02 +0100] http--0.0.0.0-8443-225 GET o1uNdliDOQhJkDnbvXo4RIZ2.undefined 2427 HTTP/1.1 443 / 200 Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11
1
we use Jboss AS 7, java 6/7 (tried both), ubuntu on vm, ssl offloading and loadbalancing with alteon loadbalancer.
PS: added netstat sample, three of those:
tcp 0 0 my.public.ip:8443 140.206.78.100:14186 ESTABLISHED
tcp 0 35040 my.public.ip:8443 140.206.78.100:14620 ESTABLISHED
tcp 0 35040 my.public.ip:8443 140.206.78.100:13859 ESTABLISHED
and 817 of those:
tcp 1 35040 my.public.ip:8443 140.206.78.100:13233 CLOSE_WAIT
tcp 1 35040 my.public.ip:8443 140.206.78.100:11649 CLOSE_WAIT
tcp 1 35040 my.public.ip:8443 140.206.78.100:11605 CLOSE_WAIT
tcp 1 35040 my.public.ip:8443 140.206.78.100:11892 CLOSE_WAIT
tcp 1 35040 my.public.ip:8443 140.206.78.100:13692 CLOSE_WAIT
tcp 1 35040 my.public.ip:8443 140.206.78.100:11988 CLOSE_WAIT
tcp 1 35040 my.public.ip:8443 140.206.78.100:13055 CLOSE_WAIT
tcp 1 35040 my.public.ip:8443 140.206.78.100:13242 CLOSE_WAIT
tcp 1 35040 my.public.ip:8443 140.206.78.100:13073 CLOSE_WAIT
tcp 1 37960 my.public.ip:8443 140.206.78.100:10176 CLOSE_WAIT
tcp 1 35040 my.public.ip:8443 140.206.78.100:14557 CLOSE_WAIT
tcp 1 35040 my.public.ip:8443 140.206.78.100:12288 CLOSE_WAIT
tcp 1 35040 my.public.ip:8443 140.206.78.100:12509 CLOSE_WAIT
tcp 1 35040 my.public.ip:8443 140.206.78.100:11049 CLOSE_WAIT
tcp 1 35040 my.public.ip:8443 140.206.78.100:11839 CLOSE_WAIT
tcp 1 35040 my.public.ip:8443 140.206.78.100:14208 CLOSE_WAIT
tcp 1 35040 my.public.ip:8443 140.206.78.100:14662 CLOSE_WAIT
Upvotes: 2
Views: 2852
Answers (1)
user207421
Reputation: 310957
You are getting a denial of service attack. Blacklist that client IP address.
Upvotes: 2
Related Questions
- Troubleshooting connections stuck in CLOSE_WAIT status
- How do I fix 'too many close_wait connections are created' even though the connections have been closed?
- Why am i seeing lots of sockets in CLOSE_WAIT status when webservice stops working?
- Java: properly closing sockets for multi threaded servers
- Hundreds of JBoss 5.1 System Threads WAITING forever
- Reproduce tcp CLOSE_WAIT state with java client/server
- Multiple complete HTTP requests stuck in TCP CLOSE_WAIT state
- Java Multithreaded Socket Server hangs after getting ~ 50 simultaneous Connections
- Netstat shows numerous CLOSE_WAIT states for mysql connections
- Very high number of HTTP Threads in WAIT state