CODEWITHSUNDEEP
htmlrweb-scraping
Daniel Ordoñez
Daniel Ordoñez

Reputation: 158

Trying to find hyperlinks by scraping

So I am fairly new to the topic of webscraping. I am trying to find all the hyperlinks that the html code of the following page contains: https://www.exito.com/mercado/lacteos-huevos-y-refrigerados/leches

So this is what I tried:

url <- "https://www.exito.com/mercado/lacteos-huevos-y-refrigerados/leches"
webpage <- read_html(url)
html_attr(html_nodes(webpage, "a"), "href")

The result only contains like 6 links but just by viewing the page you can see that there are a lot more of hyperlinks.

For example the code behind the first image has something like: <a href="/leche-entera-sixpack-en-bolsa-x-11-litros-cu-807650/p" class="vtex-product-summary-2-x-clearLink h-100 flex flex-column"> ...

What am I doing wrong?

Upvotes: 0

Views: 916

Answers (2)

QHarr
QHarr

Reputation: 84465

The data, including the urls, are returned dynamically from a GraphQL query you can observe in the network tab when clicking Mostrar más on the page. This is why the content is not present in your initial query - it has not yet been requested.

XHR for the product info

The relevant XHR in the network tab of dev tools:

enter image description here

The actual query params of the url query string:

enter image description here

You can do away with most of the request info. What you do need is the extensions param. More specifically, you need to provide the sha256Hash and the base64 encoded string value associated with the variables key in the persistedQuery.

The SHA256 Hash

The appropriate hash can be extracted from at least one of the js files which essentially governs the set up. An example file you can use is:

https://exitocol.vtexassets.com/_v/public/assets/v1/published/bundle/public/react/asset.min.js?v=1&[email protected],OrderFormContext,Mutations,Queries,PWAContext&[email protected],common,11,3,SearchBar&[email protected],common,useResponsiveValues&[email protected],common,0,Dots,Slide,Slider,SliderContainer&[email protected],common,0,1,3,4&workspace=master.

The query hash can be regex'd from the response text of an xhr request to this uri. The regex is explained here and the first match is sufficient:

enter image description here

To apply in R, with stringr, you will need some extra escapes in e.g. \s becomes \\s.

The Base64 encoded product query

The base64 encoded string you can generate yourself with the appropriate library e.g. it seems there is a base64encode R function in caTools package.

The encoded string looks like (depending on page/result batch):

eyJ3aXRoRmFjZXRzIjpmYWxzZSwiaGlkZVVuYXZhaWxhYmxlSXRlbXMiOmZhbHNlLCJza3VzRmlsdGVyIjoiQUxMX0FWQUlMQUJMRSIsInF1ZXJ5IjoiMTQ4IiwibWFwIjoicHJvZHVjdENsdXN0ZXJJZHMiLCJvcmRlckJ5IjoiT3JkZXJCeVRvcFNhbGVERVNDIiwiZnJvbSI6MjAsInRvIjozOX0=

Decoded:

{"withFacets":false,"hideUnavailableItems":false,"skusFilter":"ALL_AVAILABLE","query":"148","map":"productClusterIds","orderBy":"OrderByTopSaleDESC","from":20,"to":39}

The from and to params are the offsets for the results batches of products which come in batches of twenty. So, you can write functions which return the appropriate sha256 hash and send a subsequent request for product info where you base64 encode, with the appropriate library, the string above and alter the from and to params as required. Potentially others as well (have a play!).

The xhr response:

The response is json so you might need a json library (e.g. jsonlite) to handle the result (UPDATE: Seems you don't with R and httr). You can extract the links from a list of dictionaries nested within result['data']['products'], as per Python example, where result is the json object retrieved from the xhr with from and to params.

Examples:

Examples using R and Python are shown below (N.B. I am less familiar with R). The above has been kept fairly language agnostic.

Bear in mind, whilst I am extracting the urls, the json returned has a lot more info including product title, price, image info etc.

Example output:

enter image description here

TODO:

  1. Add in error handling
  2. Use Session objects to benefit from re-use of underlying tcp connection especially if making multiple requests to get all products
  3. Add in functionality to return total product number and loop structure to retrieve all (Python example might benefit from decorator)

R (a quick first go):

library(purrr)
library(stringr)
library(caTools)
library(httr)

get_links <- function(sha, start, end){
  string = paste0('{"withFacets":false,"hideUnavailableItems":false,"skusFilter":"ALL_AVAILABLE","query":"148","map":"productClusterIds","orderBy":"OrderByTopSaleDESC","from":' , start , ',"to":' , end , '}')
  base64encoded <- caTools::base64encode(string)
  params = list(
    'extensions' = paste0('{"persistedQuery":{"version":1,"sha256Hash":"' , sha , '","sender":"[email protected]","provider":"[email protected]"},"variables":"' , base64encoded , '"}')
  )

  product_info <- content(httr::GET(url = 'https://www.exito.com/_v/segment/graphql/v1', query = params))$data$products
  links <- map(product_info, ~{
     .x %>% .$link
  })
  return(links)
}


start <- '0'
end <- '19'     
sha <- httr::GET('https://exitocol.vtexassets.com/_v/public/assets/v1/published/bundle/public/react/asset.min.js?v=1&[email protected],OrderFormContext,Mutations,Queries,PWAContext&[email protected],common,11,3,SearchBar&[email protected],common,useResponsiveValues&[email protected],common,0,Dots,Slide,Slider,SliderContainer&[email protected],common,0,1,3,4&workspace=master') %>%
  content(., as = "text")%>% str_match(.,'query\\s+productSearch.*?hash:\\s+"(.*?)"')%>% .[[2]] 
links <- get_links(sha, start, end)
print(links)

Py:

import requests, base64, re, json

def get_sha():
    r = requests.get('https://exitocol.vtexassets.com/_v/public/assets/v1/published/bundle/public/react/asset.min.js?v=1&[email protected],OrderFormContext,Mutations,Queries,PWAContext&[email protected],common,11,3,SearchBar&[email protected],common,useResponsiveValues&[email protected],common,0,Dots,Slide,Slider,SliderContainer&[email protected],common,0,1,3,4&workspace=master')
    p = re.compile(r'query\s+productSearch.*?hash:\s+"(.*?)"') #https://regex101.com/r/VdC27H/5
    sha = p.findall(r.text)[0]
    return sha

def get_json(sha, start, end):
    #these 'from' and 'to' values correspond with page # as pages cover batches of 20 e.g. start 20 end 39

    string = '{"withFacets":false,"hideUnavailableItems":false,"skusFilter":"ALL_AVAILABLE","query":"148","map":"productClusterIds","orderBy":"OrderByTopSaleDESC","from":' + start + ',"to":' + end + '}' 
    base64encoded = base64.b64encode(string.encode('utf-8')).decode()
    params = (('extensions', '{"persistedQuery":{"sha256Hash":"' + sha + '","sender":"[email protected]","provider":"[email protected]"},"variables":"' + base64encoded  + '"}'),)
    r = requests.get('https://www.exito.com/_v/segment/graphql/v1',params=params)
    return r.json()

def get_links(sha, start, end):
    result = get_json(sha, start, end)
    links = [i['link'] for i in result['data']['products']]
    return links

sha = get_sha()
links = get_links(sha, '0', '19')
#print(len(links))
print(links)

Upvotes: 2

cimentadaj
cimentadaj

Reputation: 1488

You won't be able to get the a tags you're after because that part of the website is not visible to html/xml parsers. This is because it's a dynamic part of the website that changes if you choose another part of the website; the only 'static' part of the website is the top header, which is why you only got 6 a tags: the six a tags from the header.

For this, we need to mimic the behavior of a browser (firefox, chrome, etc...), go into the website (note that we're not entering the website as an html/xml parser but as a 'user' through a browser) and read the html/xml source code from there.

For this we'll need the R package RSelenium. Make sure you install it correctly together with docker, as none of the code below can work without it.

After you install RSelenium and docker, run docker run -d -p 4445:4444 selenium/standalone-firefox:2.53.1 from your terminal (if on Linux, you can run this the terminal; if on Windows you'll have to download a docker terminal, run it there). After that you're all set to reproduce the code below.

Why you're approach didn't work

We need to access the 5th div tag from the image below:

div_args

As you can see, this 5th div tag has three dots (...) inside, denoting that there's code inside: this is precisely where all of the bottom part of the website is (including the a tags that you're after). If we tried to access this 5th tag using rvest or xml2, we won't find anything:

library(xml2)
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union

lnk <- "https://www.exito.com/mercado/lacteos-huevos-y-refrigerados/leches?page=2"

# Note how the 5th div element is empty and it should contain the lower
# part of the website
lnk %>%
  read_html() %>%
  xml_find_all("//div[@class='flex flex-grow-1 w-100 flex-column']") %>%
  xml_children()
#> {xml_nodeset (6)}
#> [1] <div class=""></div>\n
#> [2] <div class=""></div>\n
#> [3] <div class=""></div>\n
#> [4] <div class=""></div>\n
#> [5] <div class=""></div>\n
#> [6] <div class=""></div>

Note how the 5th div tag doesn't have any code inside. A simple html/xml parser won't catch it.

How it can work

We need to use RSelenium. After you've installed everything correctly, we need to setup a 'remote driver', open it and navigate to the website. All of these steps are just to make sure that we're coming into the website as a 'normal' user from a browser. This will make sure that we can access the rendered code that we actually see when we enter the website. Below are the detailed steps from entering the website and constructing the links.


# Make sure you install docker correctly: https://docs.ropensci.org/RSelenium/articles/docker.html
library(RSelenium)

# After installing docker and before running the code, make sure you run
# the rselenium docker image: docker run -d -p 4445:4444 selenium/standalone-firefox:2.53.1

# Now, set up your remote driver
remDr <- remoteDriver(
  remoteServerAddr = "localhost",
  port = 4445L,
  browserName = "firefox"
)

# Initiate the driver
remDr$open(silent = TRUE)

# Navigate to the exito.com website
remDr$navigate(lnk)

prod_links <-
  # Get the html source code
  remDr$getPageSource()[[1]] %>%
  read_html() %>%
  # Find all a tags which have a certain class
  # I searched for this tag manually on the website code and saw that all products
  # had an a tag that shared the same class
  xml_find_all("//a[@class='vtex-product-summary-2-x-clearLink h-100 flex flex-column']") %>%
  # Extract the href attribute
  xml_attr("href") %>%
  paste0("https://www.exito.com", .)

prod_links
#>  [1] "https://www.exito.com/leche-semidescremada-deslactosada-en-bolsa-x-900-ml-145711/p"
#>  [2] "https://www.exito.com/leche-entera-en-bolsa-x-900-ml-145704/p"                     
#>  [3] "https://www.exito.com/leche-entera-sixpack-x-1300-ml-cu-987433/p"                  
#>  [4] "https://www.exito.com/leche-deslactosada-en-caja-x-1-litro-878473/p"               
#>  [5] "https://www.exito.com/leche-polvo-deslactos-semidesc-764522/p"                     
#>  [6] "https://www.exito.com/leche-slight-sixpack-en-caja-x-1050-ml-cu-663528/p"          
#>  [7] "https://www.exito.com/leche-semidescremada-sixpack-en-caja-x-1050-ml-cu-663526/p"  
#>  [8] "https://www.exito.com/leche-descremada-sixpack-x-1300-ml-cu-563046/p"              
#>  [9] "https://www.exito.com/of-leche-deslact-pag-5-lleve-6-439057/p"                     
#> [10] "https://www.exito.com/sixpack-de-leche-descremada-x-1100-ml-cu-414454/p"           
#> [11] "https://www.exito.com/leche-en-polvo-klim-fortificada-360g-239085/p"               
#> [12] "https://www.exito.com/leche-deslactosada-descremada-en-caja-x-1-litro-238291/p"    
#> [13] "https://www.exito.com/leche-deslactosada-en-caja-x-1-litro-157334/p"               
#> [14] "https://www.exito.com/leche-entera-larga-vida-en-caja-x-1-litro-157332/p"          
#> [15] "https://www.exito.com/leche-en-polvo-klim-fortificada-780g-138121/p"               
#> [16] "https://www.exito.com/leche-entera-en-bolsa-x-1-litro-125079/p"                    
#> [17] "https://www.exito.com/leche-entera-en-bolsa-sixpack-x-11-litros-cu-59651/p"        
#> [18] "https://www.exito.com/leche-deslactosada-descremada-sixpack-x-11-litros-cu-22049/p"
#> [19] "https://www.exito.com/leche-entera-en-polvo-instantanea-x-760-gr-835923/p"         
#> [20] "https://www.exito.com/of-alpin-cja-cho-pag9-llev12/p"

Hope this answers your questions

Upvotes: 1

Related Questions