CODEWITHSUNDEEP
mysqlscalaslickhikaricp
SergiGP
SergiGP

Reputation: 691

Slick with Hikari don't use more connections when needed

I'm trying to understand how Slick-Hikari works, I've read a lot of documentation but I've a use case whose behavior I don't understand.

I'm using Slick 3 with Hikari, with the default configuration. I already have a production app with ~1000 users connected concurrently. My app works with websockets and when I deploy a new release all clients are reconnected. (I know it's not the best way to handle a deploy but I don't have clustering at the moment.) When all these users reconnect, they all starts doing queries to get their user state (dog-pile effect). When it happens Slick starts to throw a lot of errors like:

java.util.concurrent.RejectedExecutionException: Task slick.backend.DatabaseComponent$DatabaseDef$$anon$2@4dbbd9d1 rejected from java.util.concurrent.ThreadPoolExecutor@a3b8495[Running, pool size = 20, active threads = 20, queued tasks = 1000, completed tasks = 23740]

What I think it's happening is that the slick queue for pending queries is full because it can't handle all the clients requesting information from the database. But if I see the metrics that Dropwizard provides me I see the following:

Example of Observed Dropwizard Metrics

Near 16:45 we se a deploy. Until old instance is terminated we can see that the number of connections goes from 20 to 40. I think that's normal, given how the deploy process is done.

But, if the query queue of Slick becomes full because of the dog-pile effect, why is it not using more than 3-5 connections if it has 20 connections available? The database is performing really well, so I think the bottleneck is in Slick.

Do you have any advice for improving this deploy process? I have only 1000 users now, but I'll have a lot more in few weeks.

Upvotes: 7

Views: 911

Answers (1)

qilinma
qilinma

Reputation: 1

Based on the "rejected" exception, I think many slick actions were submitted to slick concurrently, which exceeded the default size(1000) of the queue embedded in slick.

So I think you should:

  1. increase queue size(queueSize) to hold more unprocessed actions.
  2. increase number of thread(numThreads) in slick to process more actions concurrently. You can get more tips here

Upvotes: 0

Related Questions