CODEWITHSUNDEEP
rapitidyversepurrrhttr
Mark Druffel
Mark Druffel

Reputation: 886

API Data Extraction with Tidyverse

My end goal is to build some reporting for my company. We have a few SaaS tools we use including Harvest for time tracking (example below). I was to write wrappers around the API's to pull a few data sets from each, but I would like to do it in a way that's easy to read/maintain (i.e. I would like to stick to the tidyverse where possible).

The code below extracts the timesheet data I need and converts it into a data frame. However, the API is paginated to 100 rows per return. The API returns my current page and the total number of pages so it would be simple enough to write a while loop.

My question: is there a way to perform the while loop in the piped statement below (e.g. while 'page' <= 'total_pages' rbind('time_entries')). I cannot post the reproducible code since it's on our API and Harvest does not have a generic test account that I can find so I realize this question might be difficult to answer... Just wondering if anyone has any insight from a similar problem.

httr::modify_url(url="https://api.harvestapp.com",path="v2/time_entries") %>% 
      purrr::map(~httr::GET(.,httr::add_headers("Harvest-Account-ID" = user,Authorization = my_key,"User-Agent" = my_email))) %>% 
      purrr::map(~httr::content(., as="text", encoding = "UTF-8")) %>% 
      purrr::map(~jsonlite::fromJSON(., flatten = T)) %>% 
      purrr::map('time_entries')

Any advice or references is much appreciated, thanks!

Upvotes: 1

Views: 491

Answers (1)

Mark Druffel
Mark Druffel

Reputation: 886

Thanks to @alistaire I was able to get to an answer for anyone interested in the future. This could probably be greatly simplified so please feel free to add on.

report <- 'v2/time_entries'
httr::modify_url(url="https://api.harvestapp.com",path=report) %>% 
  purrr::map(~httr::GET(.,httr::add_headers("Harvest-Account-ID" = user,Authorization = my_key,"User-Agent" = my_email))) %>% 
  purrr::map(~httr::content(., as="text", encoding = "UTF-8")) %>% 
  purrr::map(~jsonlite::fromJSON(., flatten = T)) %>% 
  purrr::map("total_pages") %>% 
  unlist %>% 
  seq(.) %>% 
  purrr::map(~httr::modify_url(url="https://api.harvestapp.com",path=paste0("v2/time_entries?page=",.))) %>% 
  unlist %>% 
  purrr::map(~httr::GET(.,httr::add_headers("Harvest-Account-ID" = user,Authorization = my_key,"User-Agent" = my_email))) %>% 
  purrr::map(~httr::content(., as="text", encoding = "UTF-8")) %>% 
  purrr::map(~jsonlite::fromJSON(., flatten = T)) %>% 
  purrr::map("time_entries") %>% 
  do.call("rbind", .)

Upvotes: 2

Related Questions