CODEWITHSUNDEEP
rapihttr
Ed Cunningham
Ed Cunningham

Reputation: 179

Rate limiting by count per minute with httr

I'm working with the Discogs API, trying to extract community information (haves and wants, lowest price etc) for my collection.

Unfortunately, it has a rate limit of 25 per minute and I can't work out a way to work that limit into my current code (see below). I could use sys.sleep(), but I'm not sure where that would go within the code.

communityData <- lapply(as.list(collection$release_id), function(obj){
  url <- httr::GET(paste0("https://api.discogs.com/releases/", obj))
  url <- rjson::fromJSON(rawToChar(url$content))
  data.frame(release_id = obj, 
             label = url$label[[1]]$name %||% NA,
             year = url$year %||% NA, 
             title = url$title %||% NA, 
             artist_name = url$artist[[1]]$name %||% NA, 
             styles = url$styles[[1]] %||% NA,
             genre = url$genre[[1]] %||% NA,
             average_note = url$community$rating$average %||% NA, 
             votes = url$community$rating$count %||% NA, 
             want = url$community$want %||% NA, 
             have = url$community$have %||% NA, 
             lowest_price = url$lowest_price %||% NA, 
             country = url$country %||% NA)
}) %>% do.call(rbind, .) %>% 
  unique()

Any help here would be appreciated!

Upvotes: 0

Views: 287

Answers (1)

Daniel Molitor
Daniel Molitor

Reputation: 657

It should work fine to just insert the sleep command directly before you return your value. This will give you a pattern of query the url, extract information, sleep, return value, repeat. Aka, edit the above code chunk like so:

communityData <- lapply(as.list(collection$release_id), function(obj){
  url <- httr::GET(paste0("https://api.discogs.com/releases/", obj))
  url <- rjson::fromJSON(rawToChar(url$content))

  # 1 minute for 25 requests -- ~ 2.4 seconds of sleep between each request
  Sys.sleep(2.4)
  
  data.frame(release_id = obj, 
             label = url$label[[1]]$name %||% NA,
             year = url$year %||% NA, 
             title = url$title %||% NA, 
             artist_name = url$artist[[1]]$name %||% NA, 
             styles = url$styles[[1]] %||% NA,
             genre = url$genre[[1]] %||% NA,
             average_note = url$community$rating$average %||% NA, 
             votes = url$community$rating$count %||% NA, 
             want = url$community$want %||% NA, 
             have = url$community$have %||% NA, 
             lowest_price = url$lowest_price %||% NA, 
             country = url$country %||% NA)
}) %>% do.call(rbind, .) %>% 
  unique()

Upvotes: 1

Related Questions