CODEWITHSUNDEEP
resteasywildfly-10
Matthew James
Matthew James

Reputation: 63

Wildfly Resteasy requests randomly taking a long time

We're using Wildfly 10.1.0 and Resteasy 3.1.1. At throughput in the hundreds of rps we see randomly long requests even though the average latency is very low.

We are looking at metrics generated from New Relic on our application. We notice that a small percentage of requests, ~1% get stuck inside "HttpServlet30Dispatcher.service()" which is a Resteasy lib function. For the requests that get stuck there, it looks like HttpServlet30Dispatcher (or some code underneath it) consumes 100% of the time of that request. We see requests that normally take 50ms go up to 20s+.

Does anyone know what could cause this? It feels like Wildfly could be blocking the request waiting for a free thread, and maybe thread starvation happens. Can anyone confirm, or any advice on how to profile this?

Thanks! Matt

Upvotes: 3

Views: 903

Answers (1)

Matthew James
Matthew James

Reputation: 63

The answer is that these are caused by slow network. I was able to prove this by using the chrome dev tools and setting the network to have very slow downloads.

What’s happening is: Wildfly is the place where responses are buffered, so a slow network client will cause this function to block.

New Relic is the tool we’re using and it isn’t able to filter out these requests, so they add to our overall APDEX score which is annoying.

This actually doesn’t indicate any problem in your code or app server configuration. Just note that slow network clients will chew up a thread while this happens. I have been able to reproduce the exact same issue in NodeJS and I imagine it’s the same in other app servers. I would’ve though Wildfly is using NIO and it would make this transaction async but from what I can tell that’s not happening.

There are plenty of solutions to IO buffering - you can use NGINX or some other proxy to do the buffering for you and your app server will not have to do it.

Upvotes: 2

Related Questions