CODEWITHSUNDEEP
rtwitter
Sun Bee
Sun Bee

Reputation: 1820

R TwitteR and Rate-Limit Issues

I have experienced what appears to be vastly inconsistent behaviour with TwitteR package in R. I fire queries as follows, each in its 15-min window for compliance with rate-limiting feature of REST API 1.1. The first three work. The 4th fails. Repeatedly.

monsanto.tweets = searchTwitter('@monsanto', n = 1500);  
monsanto.tweets.20131207 = searchTwitter('@monsanto', n = 6000, until = '2013-12-07'); 
monsanto.tweets.20131205 = searchTwitter('@monsanto', n = 6000, until = '2013-12-05');
monsanto.tweets.20131202 = searchTwitter('@monsanto', n = 6000, until = '2013-12-02');

Error is: [1] "Client Error (429)" Error in twInterfaceObj$doAPICall(cmd, params, "GET", ...) : Error: Client Error (429)

How is that? I have tried experimenting with the count, stepping it down to 1000 in steps of 1000. No success. I have also tried variations upon the theme.

monsanto.tweets.20131202 = searchTwitter('@monsanto', 
    since = '2013-12-02', until = '2013-12-02', lang = 'en');
monsanto.tweets.20131130 = searchTwitter('@monsanto', 
    n = 1000, until = '2013-11-30');

None of these work. Under the hood, each instance of 'searchTwitter' likely generates multiple GET requests to the REST API 1.1. Per documentation, a single GET request returns upto 100 tweets, defaulting to 25. Fulfilling a request for 6000 tweets should require 60 requests at best and 240 requests at the default setting. Both are within the permissible limit of 480/application in a 15 min window.

What gives?

PS - I use OAuth. I am beginning to think I am better off hand-rolling my own interface to the REST API 1.1 and parsing JSON.

Upvotes: 0

Views: 1288

Answers (1)

Vishal
Vishal

Reputation: 1263

I don't use 'R' but my recommendation from experience with a very good python twitter library is that for advanced use cases you're much better off with your own interfaces using any standard OAuth library. You'll have to build good error handling and re-fetch logic since transient failures abound with twitter or any service at that scale.

The default count from twitter seems to be 15, does the R library use 25 as the default count? Is there configuration you can use to bump it to 100?

Upvotes: 1

Related Questions