CODEWITHSUNDEEP
rdataframeloopsfor-loopweb-scraping
Ken Lee
Ken Lee

Reputation: 209

How to automate webscraping with for loop

I have a df with two columns: id and url. id contains project ids, and url contains website links which I would like to use for scraping ids of parent projects. Here is a sample of df that I have:

Here is a sample df:

df <- structure(list(id = c("P173165", "P175875", "P175841", "P175730"
), url = c("https://search.worldbank.org/api/v2/projects?format=json&fl=*&id=P173165&apilang=en", 
"https://search.worldbank.org/api/v2/projects?format=json&fl=*&id=P175875&apilang=en", 
"https://search.worldbank.org/api/v2/projects?format=json&fl=*&id=P175841&apilang=en", 
"https://search.worldbank.org/api/v2/projects?format=json&fl=*&id=P175730&apilang=en"
)), row.names = c(NA, -4L), class = c("data.table", "data.frame"))

> df
        id                                                                                 url
1: P173165 https://search.worldbank.org/api/v2/projects?format=json&fl=*&id=P173165&apilang=en
2: P175875 https://search.worldbank.org/api/v2/projects?format=json&fl=*&id=P175875&apilang=en
3: P175841 https://search.worldbank.org/api/v2/projects?format=json&fl=*&id=P175841&apilang=en
4: P175730 https://search.worldbank.org/api/v2/projects?format=json&fl=*&id=P175730&apilang=en

I was suggested by @Sirius that I can scrape parent project ids by using the following code:

library(jsonlite)

#let's do an example for row 1

json_data <- fromJSON("https://search.worldbank.org/api/v2/projects?format=json&fl=*&id=P173165&apilang=en")
json_data$projects[["P173165"]]$parentprojid

As you see, I input the url from the first row; and then I input the id from the first row. This code outputs a parent project id:

[1] "P147665"

I want to write a code that would automatise this process, and would create a variable that would contain the parent projects' ids. This is want I want to achieve:

        id                                                                                 url par_proj_id
1: P173165 https://search.worldbank.org/api/v2/projects?format=json&fl=*&id=P173165&apilang=en     P147665
2: P175875 https://search.worldbank.org/api/v2/projects?format=json&fl=*&id=P175875&apilang=en     P173883
3: P175841 https://search.worldbank.org/api/v2/projects?format=json&fl=*&id=P175841&apilang=en     P170267
4: P175730 https://search.worldbank.org/api/v2/projects?format=json&fl=*&id=P175730&apilang=en     P173799

I guess I should be using a for loop here, but I'm not sure. Any ideas? I'd appreciate any help a lot.

Upvotes: 1

Views: 76

Answers (1)

Sirius
Sirius

Reputation: 5429

This is pretty simple, but I'd go with async so you don't have to wait for each one.


ids <- c("P173165", "P175875", "P175841", "P175730")

df <- data.table(
    id=ids,
    url = sprintf(
        "https://search.worldbank.org/api/v2/projects?format=json&fl=*&id=%s&apilang=en",
        ids
    )
)

library(remotes)
remotes::install_github("r-lib/async")
library(async)

async_get <- async(function(url,project) {
    http_get(url)$
        then(function(x) { rawToChar(x$content)})$
        then(function(x) { fromJSON(x)})$
        then(function(x) { x$projects[[1]]$parentprojid } )
})
parent.ids <- df$synchronise(async_map(df$url, async_get, .limit=5)) ## 5 is a nice limit not to bombard the site

df$par_proj_id <- parent.ids

See this page for more info on async.

Upvotes: 1

Related Questions