CODEWITHSUNDEEP
rrvesthttr
Scott
Scott

Reputation: 131

Setting cookies/submitting forms with rvest/httr in R: problems setting local store for web scraping homedepot.com

I am setting up an R script to scrape data from homedepot.com. It is going fine, except that I would like to scrape the stock levels for products, which requires setting the local store. I have tried a few ways to do this using rvest without success. How can I set the local store on homedepot.com? I have found these related questions that have not led me to a solution: (R language ) How to make a click on webpage using rvest or rcurl

Submit form with no submit button in rvest

How to properly set cookies to get URL content using httr

More info: - the store location code seems to be stored in a cookie called THD-LOC-STORE, with a 4-digit store ID. I have been unsuccessful in setting this cookie:

library("rvest")
library("httr")
# try to set cookie in site with store ID:
session <- html_session("http://www.homedepot.com", set_cookies('THD-LOC-STORE'='2679'))
# if this worked, it would show the store name instead of "Select a Store":
storefinder <- session %>% read_html() %>% html_nodes(".headerStoreFinder") %>% html_text() %>% gsub("\\t","",.)
storefinder
cookies(session)

I also thought about using submit_form() in rvest, but the buttons to select a store are run by javascript and there are no SUBMIT buttons to choose.

Upvotes: 0

Views: 2050

Answers (1)

Tripartio
Tripartio

Reputation: 2189

Concerning your possible option "I also thought about using submit_form() in rvest, but the buttons to select a store are run by javascript and there are no SUBMIT buttons to choose", I posted an answer to the question "Submit form with no submit button in rvest" which might provide this solution for your.

In brief, you can inject a submit button into your version of the code and then submit that. Details of how to do that are in the linked post.

Upvotes: 2

Related Questions