CODEWITHSUNDEEP
scalaresthttpasynchronous
ethereumbrella
ethereumbrella

Reputation: 103

Async REST calls in scala

I'm using the scalaj library, which is currently synchronous, to make REST calls to the SAME endpoint repeatedly in a while loop (while updating parameters for the next call using data I get from the previous call, like an offset each time to get different batches of data each time). Each call currently takes about 6-7 seconds to return.

Is it possible to speed up this process using an async technique with or without scalaj? I'd just like a constant stream of data since I'm using Spark Streaming. But since I need data from the previous call for my next call, how do I make it work?

Additional details: The aforementioned while loop is in a thread which extends Runnable, and the thread is submitted via Executors.newFixedThreadPool()

Upvotes: 0

Views: 1128

Answers (1)

Dima
Dima

Reputation: 40510

Asynchronous does not make anything faster, it is just ... well ... asynchronous, so that while a long running task is in progress, you can do something else. But if what you need to do all depends on the result of the task, there is no point in parallelizing, because you'll have to wait for the task to complete anyway.

If your processing of the results takes some considerable time, you could do that in parallel with getting next batch with something like this:

def processBatch(batch: Batch): Future[Unit] {
  val nextBatch = Future(getNextBatch(batch))
  // do processing of the current batch
  nextBatch.flatMap { 
    case NoMoreResults => Future.successful(())
    case newBatch => processBatch(newBatch)
  }
}

Upvotes: 1

Related Questions