CODEWITHSUNDEEP
rweb-scrapingrvest
Maxl Gemeinderat
Maxl Gemeinderat

Reputation: 555

R Web Scraping with RVEST: DT and DD Tag

i am scraping data of an car website (Example link). Since now everything worked fine but i got problems with getting one specific value. Below in the image, you see a list, where i want to get the value of "Anzahl Türen" (represents the number of doors in one car).

List I want to scrape

In HTML, "Anzahl" looks like this:

<dt>Anzahl Türen</dt>
<dd>3</dd>

How do I get the value "3" here?

For getting other values from this list, I have already bypassed the way with the HTML tag "dt" and "dd", with following function for example which gets me the value of "Außenfarbe" (color):

get_color = function(links) {
     car_page = read_html(links)
     col = car_page %>% html_nodes("dd a") %>%
        html_text()
     colors = c("Blau","Rot","Schwarz","Silber", "Beige", "Braun","Bronze","Gelb","Grau", 
                "Grün","Violett","Weiß","Orange","Gold")
     intersect(col, colors)
}

But now the value I want is a number and no text anymore, so this function doesn't work for another row.

Thanks for your help! :)

Upvotes: 0

Views: 277

Answers (2)

QHarr
QHarr

Reputation: 84465

You could also use a function which returns for a given search term (note I had to use iconv to handle some mojibake)

library(rvest)
library(magrittr)

page <- read_html('https://www.autoscout24.de/angebote/volkswagen-lupo-1-2-tdi-3l-diesel-gruen-b5ee4316-b672-490a-917d-6732bb6065a6?&cldtidx=20&cldtsrc=listPage&searchId=603681428')

get_value <-function(html, search_term){
  page %>% 
    html_node(xpath = sprintf("//dt[text()[contains(.,'%s')]]/following-sibling::dd[1]",  
                              iconv(c(search_term), from = 'iso_8859-15', to = 'utf-8'))) %>% 
    html_text(trim = T) -> value
    return(value)
}

get_value(page, 'Anzahl Türen') %>% as.integer()
get_value(page, 'Außenfarbe')

The above is for contains match. An exact match would use:

get_value <-function(html, search_term){
  page %>% 
    html_node(xpath = sprintf("//dt[text()='%s']/following-sibling::dd[1]",  
                              iconv(c(search_term), from = 'iso_8859-15', to = 'utf-8'))) %>% 
    html_text(trim = T) -> value
    return(value)
}

Upvotes: 2

Ronak Shah
Ronak Shah

Reputation: 389135

You can actually extract all the values in a dataframe and then select the ones which you want.

library(rvest)
url <- 'https://www.autoscout24.de/angebote/volkswagen-lupo-1-2-tdi-3l-diesel-gruen-b5ee4316-b672-490a-917d-6732bb6065a6?&cldtidx=20&cldtsrc=listPage&searchId=603681428'
car_page = read_html(url)

data <- tibble::tibble(name = car_page %>% html_nodes("dt") %>% html_text(), 
               value = car_page %>% html_nodes("dd") %>% html_text() %>% trimws)

# A tibble: 22 x 2
#   name                            value      
#   <chr>                           <chr>      
# 1 Zustand                         Gebraucht  
# 2 HU Prüfung                      04/2023    
# 3 Letzter Kundendienst            03/2021    
# 4 Letzter Wechsel des Zahnriemens 05/2015    
# 5 Marke                           Volkswagen 
# 6 Modell                          Lupo       
# 7 Erstzulassung                   2000       
# 8 Außenfarbe                      Grün       
# 9 Innenausstattung                Stoff, Grün
#10 Karosserieform                  Kleinwagen 
# … with 12 more rows

data$value[data$name == "Anzahl Türen"]
#[1] "3"

Testing on another url -

url <- 'https://www.autoscout24.de/angebote/peugeot-206-110-tendance-tuev-neu-1-6-109ps-4-5tuetig-benzin-blau-cd721b01-494f-47cc-aead-c3c41d728935?ipc=recommendation&ipl=detailpage-engine-itemBased'

data$value[data$name == "Anzahl Türen"]
#[1] "5"

Upvotes: 1

Related Questions