CODEWITHSUNDEEP
htmlcssrweb-scraping
Владимир Кузовкин
Владимир Кузовкин

Reputation: 622

Get values from children class while web scraping

I want to get list of Phase state from site. I make such code:

library("rvest") 
library("magrittr")

url <- 'https://energybase.ru/en/oil-gas-field/index'

read_html(url) %>% 
        html_nodes(".info")%>% 
             html_children()%>% 
                  html_children()

and I got:

 [1] <small>City</small>
 [2] <div class="value">Игарка</div>
 [3] <small>Phase state</small>
 [4] <div class="value">нефтегазовое</div>
 [5] <small>Извлекаемые запасы A+B1+B2+C1</small>
 [6] <div class="value">479.10 mln. tons</div>
 [7] <small>City</small>
 [8] <div class="value">Тазовский</div>
 [9] <small>Phase state</small>
[10] <div class="value">газонефтяное</div>
[11] <small>Извлекаемые запасы A+B1+B2+C1</small>
[12] <div class="value">422.00 mln. tons</div>
[13] <small>City</small>
[14] <div class="value">Лянтор</div>
[15] <small>Phase state</small>
[16] <div class="value">нефтегазоконденсатное</div>
[17] <small>Извлекаемые запасы A+B1+B2+C1</small>
[18] <div class="value">380.00 mln. tons</div>
[19] <small>City</small>
[20] <div class="value">Тобольск</div>

I want to get all notes after 

<div class="value">

the result should be:

нефтегазовое
газонефтяное
нефтегазоконденсатное

and so on. What function should I use to solve my problem?

Upvotes: 1

Views: 152

Answers (2)

QHarr
QHarr

Reputation: 84465

You could just pull from the dropdown options then you get the unique list without repeats. Depends if you want full list with repeats or not.

library(rvest)
library(magrittr)

phases <- (read_html('https://energybase.ru/en/oil-gas-field/index') %>%
          html_nodes('#fieldsearch-phase option') %>%
          html_text())[-1]

Upvotes: 1

Rentrop
Rentrop

Reputation: 21507

You can use

read_html(url) %>% 
  html_nodes(".col-md-8:nth-child(2) .value") %>% 
  html_text

to get

 [1] "нефтегазовое"          "газонефтяное"          "нефтегазоконденсатное" "нефтяное"             
 [5] "нефтяное"              "нефтегазовое"          "нефтяное"              "нефтяное"             
 [9] "нефтяное"              "нефтегазоконденсатное" "нефтегазоконденсатное" "нефтяное"             
[13] "нефтегазоконденсатное" "нефтегазоконденсатное" "нефтяное"              "нефтяное"             
[17] "газонефтяное"          "нефтегазоконденсатное" "нефтяное"              "нефтегазовое"

A very good tool to get the right css-selector (.col-md-8:nth-child(2) .value) is https://selectorgadget.com/ - here the screenshot for your example:

enter image description here

Upvotes: 1

Related Questions