CODEWITHSUNDEEP
rrvest
Hanish Kiran Sanghrajka
Hanish Kiran Sanghrajka

Reputation: 61

Error in open.connection(x, "rb") : HTTP error 404 in R

I am trying to scrape data using R to obtain details about some of the listings in the below website but I am getting an error that I am unsure how to resolve: Error in open.connection(x, "rb") : HTTP error 404 I tried using httr package and try function as seen in similar posts but cant resolve it. Am I doing something wrong?

library(XML)
library(RCurl)
library(curl) 
library(rvest)
library(tidyverse)
library(dplyr)
library(httr) 

url <- "https://www.sgcarmart.com/new_cars/index.php"
cardetails <- read_html(url)

listing <- html_nodes(cardetails, "#nc_popular_car")
popularcars <- html_nodes(listing,".link")
count<-length(popularcars)


info <- data.frame(CarName=NA, Distributer=NA, Hotline= NA, CountryBuilt= NA, Predecessor= NA, stringsAsFactors = F )

for(i in 1:count)
{
  h <- popularcars[[i]]

details_url <- paste0("https://www.sgcarmart.com/new_cars",html_attr(h,"href"))

details <- read_html(details_url)

info[i,]$CarName <- html_node(details,".link_redbanner")

}

info

Upvotes: 0

Views: 3488

Answers (1)

r2evans
r2evans

Reputation: 160417

TL;DR

Add a slash:

  details_url <- paste0("https://www.sgcarmart.com/new_cars/",html_attr(h,"href"))
  # --->          --->          --->          --->         ^

The Journey

  1. I ran your source far enough to get the popularcars, and looked at the first one:

    h <- popularcars[[1]]
h
# {html_node}
# <a href="newcars_overview.php?CarCode=12618" class="link">
# [1] <div style="position:relative; padding-bottom:6px;">\r\n                                < ...
# [2] <div style="padding-bottom:3px;" class="limittwolines">Toyota Corolla Altis</div>
details_url <- paste0("https://www.sgcarmart.com/new_cars",html_attr(h,"href"))
details_url
# [1] "https://www.sgcarmart.com/new_carsnewcars_overview.php?CarCode=12618"

    Like you, for me that URL returned 404.

  2. I navigated (in a boring, normal, browser) to the main URL, looked at the source for the page, and searched for 12618:

    <div style="padding:10px 10px 5px 10px;" id="nc_popular_car">
                                        <div class="floatleft" style="text-align:center;width:136px;padding-right:22px;">
                        <a href="newcars_overview.php?CarCode=12618" class="link">
                            <div style="position:relative; padding-bottom:6px;">
                                <div style="position:absolute; border:1px solid #B9B9B9; width:134px; height:88px;"><img src="https://i.i-sgcm.com/images/spacer.gif" width="1" height="1" alt="spacer" /></div>
                                <img src="https://i.i-sgcm.com/new_cars/cars/12618/12618_m.jpg" width="136" height="90" border="0" alt="Toyota Corolla Altis" />
                            </div>
                            <div style="padding-bottom:3px;" class="limittwolines">Toyota Corolla Altis</div>
                        </a>
                        <div style="padding-bottom:14px;" class="font_black">$91,888</div>
                    </div>

  3. I right-clicked on the <a href="newcars_overview.php?CarCode=12618" class="link"> portion and copied the "link location. I found that it was:

    https://www.sgcarmart.com/new_cars/newcars_overview.php?CarCode=12618 <-- from the source
https://www.sgcarmart.com/new_carsnewcars_overview.php?CarCode=12618  <-- from your code

BTW, you might find this a little easier to manage than a for loop. Iteratively building a frame is horribly inefficient, and while it may not be bad for the 18 entries I found it, it's not good in the long run (if you can avoid it).

info <- lapply(popularcars, function(h) {
    details_url <- paste0("https://www.sgcarmart.com/new_cars/", html_attr(h,"href"))
    details <- read_html(details_url)
    html_text(html_node(details,".link_redbanner"))
  })

str(info)
# List of 18
#  $ : chr "Toyota Corolla Altis"
#  $ : chr "Hyundai Venue"
#  $ : chr "Hyundai Avante"
#  $ : chr "SKODA Octavia"
#  $ : chr "Honda Civic"
#  $ : chr "Mazda 3 Sedan Mild Hybrid"
#  $ : chr "Honda Jazz"
#  $ : chr "Kia Cerato"
#  $ : chr "Mazda CX-5"
#  $ : chr "Mercedes-Benz GLA-Class(Parallel Imported)"
#  $ : chr "Toyota Raize(Parallel Imported)"
#  $ : chr "Toyota Camry Hybrid(Parallel Imported)"
#  $ : chr "Mercedes-Benz A-Class Hatchback(Parallel Imported)"
#  $ : chr "Mercedes-Benz A-Class Saloon(Parallel Imported)"
#  $ : chr "Honda Fit(Parallel Imported)"
#  $ : chr "Mercedes-Benz C-Class Saloon(Parallel Imported)"
#  $ : chr "Mercedes-Benz CLA-Class(Parallel Imported)"
#  $ : chr "Honda Freed Hybrid(Parallel Imported)"

Last point: while this is a worthwhile learning endeavor, that website's Terms of Service are clear to state: "You agree that you will not: ... engage in mass automated, systematic or any form of extraction of the material ("the Content") on our Website". I assume your efforts are under that limit.

Upvotes: 2

Related Questions