CODEWITHSUNDEEP
apiscalaiteratorpagination
pr1001
pr1001

Reputation: 21962

Making a Scala Iterator for a Paginated API

I'm writing a Scala library to make querying an paginated JSON API easier. Each API call returns an object that looks something like this:

{
  "count": 100,
  "current_page": 1,
  "total_pages": 2,
  "records": [
    ...
  ]
}

I'd like to have a function that returned some sort of iterator like MyIterator[Record]. Are there any standard ways to do this in the Scala world, perhaps even constructs in the standard library that could help me?

I'm using lift-json generally for my JSON parsing, if that's helpful.

Thanks.

Upvotes: 4

Views: 2588

Answers (1)

David Winslow
David Winslow

Reputation: 8590

If you have an Iterator[Iterator[A]] you can use flatten to produce an Iterator[A] which chains all the nested Iterators together:

scala> Iterator(Iterator(1, 2), Iterator(3, 4)).flatten.toList
res0: List[Int] = List(1, 2, 3, 4)

Combined with one of the factory methods on the Iterator companion object, you could probably turn this into a one-liner:

Iterator.range(1, pageCount).map(i: Int => fetchPage(i)).flatten

It might need to be a bit more complex if you can only get the page count by retrieving a page... something like:

val (firstPage, pageCount) = getFirstPageWithCount()
firstPage ++ (Iterator.range(2, pageCount).map(fetchPage).flatten)

Upvotes: 6

Related Questions