CODEWITHSUNDEEP
scalaconcurrencyfuture
simao
simao

Reputation: 15529

Avoid deadlock with Future { blocking {} }

I am using the following code to get a JDBC connection, inside a blocking block, and pass that connection to a fn: Connection => Future[_]. After fn finishes I'd like to commit/rollback the transaction and close the connection.

  def withTransactionAsync[T](fn: Connection => Future[T]): Future[T] =
    Future {
      blocking {
        ds.getConnection
      }
    }.flatMap { conn =>
      fn(conn)
        .map { r => conn.commit(); conn.close(); r }
        .recoverWith {
          case e: Throwable =>
            conn.rollback()
            conn.close()
            throw e
        }
    }

I am using a separate execution context based on a ForkJoinPool.

With enough calls, this code goes into a deadlock. Intuitively, this makes sense. The first future, with the getConnection call, gets blocked while waiting for available connections, while available connections are waiting for available threads in the ExecutionContext to run the commit(); close() block to free the connection and free a thread in the execution context for getConnection to run. I verified this is the case with a thread dump.

The only way I found around this problem is to run everything on the same Future {} and therefore avoid switching the context:

  def withTransactionAsync[T](fn: Connection => Future[T]): Future[T] =
    Future {
      blocking {
        val conn = ds.getConnection

        try {
          conn.setAutoCommit(false)
          val r = Await.result(fn(conn), Duration.Inf)
          conn.commit()
          r
        } catch {
          case e: Throwable =>
            conn.rollback()
            throw e
        } finally
          conn.close()
      }
    }

But this way I am blocking on Await.result. I suppose this is not a big problem because I am blocking inside a blocking block, but I am afraid this would have unforeseen consequences and is not necessarily what the caller of this API expects.

Is there a way around this deadlock without using Await and just rely on Future composition?

I suppose a case could be made that this this function not be accepting Connection => Future[T] but only a Connection => T, but I'd like to keep that API.

If I increase the size of the ForkJoinPool enough, it works, but that size is difficult to calculate/predict for all workloads and I don't want to have a ForkJoinPool many times the size of my database pool.

Upvotes: 0

Views: 321

Answers (1)

Tim
Tim

Reputation: 27356

As mentioned in the comments, fn is blocking code. But it is not inside a blocking clause, so it will tie up one of the main threads in the pool. If this happens enough times, the pool will run out of threads and the system will deadlock.

So the call to fn and the code that follows needs to be inside a blocking clause so that a separate thread is created for it and the main threads remain available for non-blocking code.

Given the amount of blocking code, it is probably worth looking at a Task model with a thread per connection rather than a thread per pending operation, so that the number of threads is constrained. This is basically a work-around for the fact that getConnection is synchronous, which is a problem with HikariCP.

Upvotes: 1

Related Questions