CODEWITHSUNDEEP
scalafunctional-programming
NiceOneMoney
NiceOneMoney

Reputation: 169

How to collect data into a single List with a while loop?

In my Play Scala app I am using a SOAP api library by way of a jar file. In this interface there is no way to fetch all domain items with on call; the docs suggest the developer to use a fetchDeltaSince with pagination parameters.

If I am to fetch any domain created between startTime and endTime I would have to utilize a method like this:

Account::fetchDeltaSince(startTime : Calendar, page : Int, pageSize : Int, endTime : Calendar) : Array[Account]

pageSize max that is allowed is 100 and there are about 30K+ accounts. I will have to get this data using some sort of while loop that monitors the number of items returned and incrementing page parameter (maybe?). I would like for the result to be all in one Array[Account] datatype.

What is the best way to do this in Scala and is there a way to do it asynchronously?

Upvotes: 0

Views: 365

Answers (1)

Mateusz Kubuszok
Mateusz Kubuszok

Reputation: 27535

Assuming that your fetching has more or less this signature:

def fetchDeltaSince(startTime: Calendar, page: Int, pageSize: Int, endTime: Calendar): Array[Account]

you could have an async version of it that is e.g. Future:

def fetchDeltaSinceAsync(startTime: Calendar, page: Int, pageSize: Int, endTime: Calendar): Future[Array[Account]]

and then you could combine it like e.g.

def fetchAll(startTime: Calendar,
             pageSize: Int,
             endTime: Calendar): Future[List[Account]] = {
  def fetchUntilAll(page: Int, listBuffer: ListBuffer[Account]): Future[List[Account]]
    fetchDeltaSinceAsync(startTime, page, pageSize, endTime).flatMap { newBatch =>
      if (newBatch.isEmpty) Future.successful(listBuffer.toList)
      else fetchUntilAll(page + 1, listBuffer ++= newBatch)
    }
  fetchUntilAll(0, new ListBuffer[Account]())
}

I used ListBuffer it avoid reallocating List again and again. In your code you would have to adjust e.g. stop condition (maybe it is not empty array but exception, or sth else?). You might also prefer to use something more functional e.g. replace Future with Monix's Task and ListBuffer with Cats' Chain.

Upvotes: 2

Related Questions