CODEWITHSUNDEEP
rfor-loopjsonlite
Bilal Hafeez
Bilal Hafeez

Reputation: 11

Jsonlite and For loop issue

I am downloading data from google trends. Sometimes the connection drops and I have to restart the loop from the beginning (1-726). Suppose error is on 721th attempt still, I have to restart it from 1 again.

library(jsonlite)
baseurl <- "https://www.googleapis.com/trends/v1beta/graph?&terms="
for(i in 1:726){
mydata <- fromJSON(paste0(baseurl, file.list$query[i]), flatten=TRUE)
message("Retrieving query ", i)
pages[[i]] <- mydata$lines
}

The error looklikes this 

Retrieving query 1
Retrieving query 2
Retrieving query 3
Error in open.connection(con, "rb") : HTTP error 503.

Then I have to restart the whole process repeatedly to get the desired data. Can there be a method that it retries itself without losing the progress?

Upvotes: 1

Views: 358

Answers (1)

Alex Knorre
Alex Knorre

Reputation: 629

This answer on SO describes useful snippet to retry iteration in case of error/exception. Your code (it's impossible to reproduce BTW, as you didn't assign anything to file.list) with such a snippet would look like this:

library(jsonlite)
baseurl <- "https://www.googleapis.com/trends/v1beta/graph?&terms="
for(i in 1:726){
  mydata<-NULL
  attempt <- 1
  while( is.null(mydata) && attempt <= 3 ) {
    attempt <- attempt + 1
    try(
      mydata <- fromJSON(paste0(baseurl, file.list$query[i]), flatten=TRUE)
    )
  } 
    message("Retrieving query ", i)
  pages[[i]] <- mydata$lines
}

Feel free to change number of attempts in while condition. Also, the reason can be the limit for number of requests per second - as digEmAll suggested, try adding Sys.sleep().

Upvotes: 2

Related Questions