CODEWITHSUNDEEP
rtwitterweb-scraping
Naomi Peer
Naomi Peer

Reputation: 367

R : Create a loop for Twitter scraping with userTimeline

I would like to create a loop to get all the tweets from 2 Twitter users (@Better_Together and @YesScotland). Even when I select 3200 tweets, which is the limit according to this error message: 

In statusBase(cmd, params, n, 3200, ...) : statuses/user_timeline has a cap of 3200 statuses, clipping )

I can't seem to get more than 40 tweets or so.

I would like to get all the +- 5000 tweets for those two accounts. I think creating a loop would be the solution. Anyone can help?

 UK_Together.list <- userTimeline('UK_Together', n=3200)  
 UK_Together.df = twListToDF(UK_Together.list) 
 UK_Together.df$text <- sapply(UK_Together.df$text,function(row) iconv(row, "latin1", "ASCII", sub=""))
 write.csv(UK_Together.df, file='UK_TogetherTweets.csv', row.names=F)

 YesScotland.list <- userTimeline('YesScotland', n=3200)  
 YesScotland.df = twListToDF(YesScotland.list) 
 YesScotland.df$text <- sapply(YesScotland.df$text,function(row) iconv(row, "latin1", "ASCII", sub=""))
 write.csv(YesScotland.df, file='YesScotlandTweets.csv', row.names=F)

Upvotes: 2

Views: 599

Answers (2)

schlumpf
schlumpf

Reputation: 1

I would write a loop that uses the maxID Parameter in User Timeline:

code

userTimeline("user", n=3200, maxID = min(as.numeric(user_timeline_df$id)))

Then put this into some kind of loop and iterate until it returns nothing. Rate Limit shouldn't be an issue when you are downloading just a few thousand tweets.

Upvotes: 0

Chris Bail
Chris Bail

Reputation: 307

I would probably write a for loop and add a Sys.sleep line that pauses your loop for a sufficient amount of time to prevent rate limiting

Upvotes: 1

Related Questions