CODEWITHSUNDEEP
apitwitterpaginationazure-data-factory
Mikael Burban
Mikael Burban

Reputation: 21

Implement the Rest API pagination rule with Twitter API

I'm trying to get tweets data in a json file with Microsoft Azure Data Factory. I've created a copy data activity which takes the URL for the get tweets API with the needed parameters (those parameters are included in the dataset definition with a parameter that represents the keyword I'm looking for) as the source, and a json file dataset as a sink.

This works pretty well, the file is created with the rest API response. Now I want to handle the pagination rule for Twitter as I will get more that the number of tweets contained in one response. The twitter way to handle this is to have a "meta" element in the API response containing various information and particularly a "next_token" element containing the value to add to a parameter in the next get query that has to be submitted, up until the next_token element is absent from the response.

so : first get example :

tweets/search/recent?query=textToSearch&max_results=100

You receive in the response

"meta": {
     "next_token": "MyNextToken"
   }

So the 2nd get should be:

tweets/search/recent?query=textToSearch&max_results=100&pagination_token=MyNextToken

I've checked the help for pagination rule but can't quite figured out what to do with that, I've seen examples with the complete URL but not with a new parameter which is not included in the first get query. I'm guessing you'll have to create a variable which will contain the new parameter but I don't know how to add the variable to the URL.

I've tried this but it did not work, I've only gotten the first response in my json file and not the other pages: pagination_rule

Thanks in advance for any tips on how to implement that!

Upvotes: 1

Views: 498

Answers (1)

Mikael Burban
Mikael Burban

Reputation: 21

So, after a few tries and some research, it doesn't seem possible for the moment in Data Factory to use the pagination rules in that way. The solution I found is to implement the pagination manually using a "until" activity where the URL is a variable string containing the next_token value and the loop finishes when there is no next_token value in the result.

Upvotes: 1

Related Questions