CODEWITHSUNDEEP
htmlrweb-scrapingrvest
Ramiro Bentes
Ramiro Bentes

Reputation: 348

rvest html_nodes returns {xml_nodeset (0)}

I've been trying to scrape this page using rvest and selectorGadge. I am able to scrape the product description, but when I try to get the values as shown in the picture:

Text

However, when I run the code:

library(dplyr)
library(rvest)

read_html("https://www.dicasanet.com.br/material-de-construcao") %>%
  html_nodes(".product-payment")

I keep getting the result "{xml_nodeset (0)}". I noticed that, unlike other values (like the name of the product), this is not a div.a, but a div.div instead. Is there another way to get these values? How should I proceed? Thanks in advance!

Upvotes: 0

Views: 425

Answers (2)

QHarr
QHarr

Reputation: 84465

Data is dynamically loaded from a JavaScript object within a script tag. You can regex that out from the response text, parse with jsonlite into a json object and then extract what you want for the products

library(magrittr)
library(rvest)
library(stringr)
library(jsonlite)

page <- read_html('https://www.dicasanet.com.br/loja/catalogo.php?loja=790930&categoria=1')

data <- page %>% 
  toString() %>% 
  stringr::str_match('dataLayer = (\\[.*\\])') %>% 
  .[2] %>% 
  jsonlite::parse_json()

print(data[[1]]$listProducts)

Upvotes: 1

Ran K
Ran K

Reputation: 162

I could not manage to scrape the prices with Rvest, however it is possible to do using RSelenium:

library(RSelenium)
# I use RSelenium in combination with docker.
remDr <- remoteDriver(
  remoteServerAddr = "localhost",
  port = 4445L,
  browserName = "chrome"
)
remDr$open()
remDr$navigate("https://www.dicasanet.com.br/material-de-construcao")

page <- read_html(remDr$getPageSource()[[1]])

page %>% html_nodes("product-price") %>% html_text()

price <- remDr$findElements(using = "class","product-price")
price <- sapply(price,function(x){x$getElementText()[[1]]})
price

output is :

price
 [1] ""                     ""                     ""                     ""                    
 [5] ""                     ""                     "R$ 40,50"             "R$ 40,50"            
 [9] "R$ 194,90"            "R$ 194,90"            "R$ 171,00\nR$ 122,90" "R$ 171,00\nR$ 122,90"
[13] "R$ 393,00"            "R$ 393,00"            "R$ 357,50"            "R$ 357,50"           
[17] "R$ 433,20"            "R$ 433,20"            "R$ 120,60"            "R$ 120,60"           
[21] "R$ 89,50"             "R$ 89,50"             "R$ 56,20"             "R$ 56,20"

Upvotes: 0

Related Questions