How to chain Futures for async database I/O handling?

Question

I recently started developing an Application in Play Scala. Although I have used Play Java for several applications already, I am also new to Scala and Play Scala.

I use DAO pattern to abstract the database interaction. The DAO contains methods for insert, update delete. After reading async and thread-pool related documentation, I figured that making database interaction async was highly important, unless you tweak the Play default thread pool to have many threads.

To ensure that all database calls are handled asynchronously, I made all the call to return a Future instead of a value directly. I have created a separate execution context for the database interactions.

trait Dao[K, V] {
  def findById(id: K): Future[Option[V]]
  def update(v: V): Future[Boolean]
  [...]
}

This has lead to very complex and deeply nested code in actions.

trait UserDao extends Dao[Long, User] {
  def existsWithEmail(email: String): Future[Boolean]
  def insert(u: User) Future[Boolean]
}

object UserController extends Controller {
  def register = Action {
    [...]
    userDao.existsWithEmail(email).flatMap { exists =>
      exits match {

        case true =>
          userDao.insert(new User("foo", "bar")).map { created =>
            created match {
              case true => Ok("Created!")
              case false => BadRequest("Failed creation")
            }
          }

        case false =>
          Future(BadRequest("User exists with same email"))
      }
    }
  }
}

Above is a sample of simplest of actions. Level of nesting gets deeper as I have more database calls involved. Although I figured that some of the nesting can be reduced with the use of for comprehension, I am doubting if my approach itself is fundamentally wrong?

Consider a case where I need to create a user,

a. If none exists already with same email address.

b. If none exists already with same mobile number.

I can create two futures,

f(a) checking if user exists with email.

f(b) checking if user exists with mobile.

I cannot go and insert a new user unless I verify that both conditions evaluate false. I can actually have f(a) and f(b) running in parallel. The parallel execution maybe undesirable in case f(a) evaluates to true, and may work in favor otherwise. Step 3 of creating user depends on both these futures, so I wonder if following is equally good?

trait UserDao extends Dao[Long, User] {
  def existsWithEmail(email: String): Boolean
  def existsWithMobile(mobile: String): Boolean
  def insert(u: User): Unit
}

def register = Action {
  implicit val dbExecutionContext = myconcurrent.Context.dbExceutionContext

  Future {
    if (!userDao.existsWithEmail(email) && !userDao.existsWithMobile(mobile) {
      userDao.insert(new User("foo", "bar")
      Ok("Created!")
    } else {
      BadRequest("Already exists!")
    }
  }
}

Which one is a better approach? Does the approach of using a single Future with multiple calls to database have any downside?

Answer 1

I solved this problem using for comprehension in scala. I added a few implicit type converters to help with error handling.

Initially I did something like,

def someAction = Action.async {
  val result = 
    for {
      student <- studentDao.findById(studentId)
      if (student.isDefined)
      parent <- parentDao.findById(student.get.parentId)
      if (parent.isDefined)
      address <- addressDao.findById(parent.get.addressId)
      if (address.isDefined)
    } yield {
      // business logic
    }

  result fallbackTo Future.successful(BadRequest("Something went wrong"))
}

This is how the code was initially structured to counter the dependency between futures. Note that each subsequent future depends on the previous future. Also, each findById is returning a Future[Option[T]] so if within for comprehension is required to handle cases where the methods return None. I used fallbackTo method on the Future to fallback to a BadRequest result if any of the futures evaluated to None (In event of any if condition failing within for comprehension, it returns a failed future) Another issue with above approach was that it would suppress any kind of exception encountered (even exceptions as trivial as NPE) and simply fallback to BadRequest instead, which was very bad.

Above method was able to counter future of options and handle the failed cases, although it was not helpful to figure out exactly which of the futures in the for comprehension had failed. To overcome this limitation, I used implicit type converters.

object FutureUtils {
  class FutureProcessingException(msg: String) extends Exception(msg)
  class MissingOptionValueException(msg: String) extends FutureProcessingException(msg)

  protected final class OptionFutureToOptionValueFuture[T](f: Future[Option[T]]) {
    def whenUndefined(error: String)(implicit context: ExecutionContext): Future[T] = {
      f.map { value =>
        if (value.isDefined) value.get else throw new MissingOptionValueException(error)
      }
    }
  }

  import scala.language.implicitConversions

  implicit def optionFutureToValueFutureConverter[T](f: Future[Option[T]]) = new OptionFutureToOptionValueFuture(f)

}

The above implicit conversions allowed me to write readable for comprehensions chaining multiple futures.

import FutureUtils._

def someAction = Action.async {
  val result = 
    for {
      student <- studentDao.findById(studentId) whenUndefined "Invalid student id"
      parent <- parentDao.findById(student.get.parentId) whenUndefined "Invalid parent id"
      address <- addressDao.findById(parent.get.addressId) whenUndefined "Invalid address id"
    } yield {
      // business logic
    }

  result.recover {
    case fpe: FutureProcessingException => BadRequest(fpe.getMessage)
    case t: Throwable => InternalServerError
  } 
}

The above approach ensured that all exceptions caused by missing Option value are handled as a BadRequest with specific message about what exactly failed. All other failures are treated as InternalServerError. You can log the exact exception with stack trace in order to help debug.

Answer 2

You are correct when you say that a for comprehension can make for less nesting.

To solve the dual-future problem, consider:

existsWithEmail(email).zip(existsWithMobile(mobile)) map {
  case (false, false) => // create user
  case _              => // already exists
}

If you have a lot of these, you can use Future.sequence( Seq(future1, future2, ...) ) to turn a sequence of futures into a future sequence.

You may want to take a look at more functional idioms for DB access than DAO, e.g., Slick or Anorm. Usually those will compose better and end up being more flexible than DAO.

A side note: it is more efficient to use if/else for a simple true/false test than it is to use match/case, and is the preferred style.