CODEWITHSUNDEEP
rustrust-tokio
xy2_
xy2_

Reputation: 7162

Rust Multi-thread chaining HTTP requests and returning results in Iterator

I have an response that returns a list of items backed by cursor-based pagination, that goes like this:

{
  "object": "collection",
  "pages": {
    "per_page": 500,
    "next_url": "https://api.com/v2/assignments?page_after_id=80469434",
    "previous_url": null
  },
  "total_count": 523,
  "data_updated_at": "2017-09-21T11:45:01.691388Z",
  "data": [
    {
      "id": 80469400,
      "value": "owo",
    }
    // ...
  ]
}

Here I have a next_url set here, and there are 500 results per page (the total count is 523) so I can get 500 results now and make a second request to that url for the remaining results.

But I'd like to get the first results now, and not wait for all the requests to finish.

My idea was something like:

Would this work? Are there other easier ways to do this? I'm not sure on the last step in particular.

Upvotes: 0

Views: 389

Answers (1)

nee
nee

Reputation: 3293

Yes this would work just fine, however I'd personally implement this with async rather than with multiple threads. No need to spin up a whole thread whose whole job is just waiting around most of the time, and you can still keep your queue setup.

As far the last step, I think the iterator approach would be fine, there's many ways of approaching it and that's a perfectly valid way.

Upvotes: 1

Related Questions