CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsouphtml-parsing
Giangio
Giangio

Reputation: 35

Scraping: cannot access information from web

I am scraping some information from this url: https://www.rockethub.com/projects/34210-lunar-lion-the-first-ever-university-led-mission-to-the-moon#description-tab

Everything was fine till I scraped the description. I tried and tried to scrape, but I failed so far. It seems like I can't reach that information. Here is my code:

html = urllib.urlopen("https://www.rockethub.com/projects/34210-lunar-lion-the-first-ever-university-led-mission-to-the-moon")
tree=BeautifulSoup(html, "lxml")
description=tree.find('div',{'id':'description_section','class':'description-section'})

Any of you has any suggestion?

Upvotes: 1

Views: 463

Answers (3)

John Smith
John Smith

Reputation: 1698

I found out how to scrap with R: 

library("rvest")

url="https://www.rockethub.com/projects/34210-lunar-lion-the-first-ever-university-led-mission-to-the-moon/description"

url %>% 
  html() %>% 
  html_nodes(xpath='//div[@id="description_section"]', xmlValue) %>%
  html_text()

Upvotes: 0

alecxe
alecxe

Reputation: 473763

You would need to make an additional request to get the description. Here is a complete working example using requests + BeautifulSoup:

import requests
from bs4 import BeautifulSoup

url = "https://www.rockethub.com/projects/34210-lunar-lion-the-first-ever-university-led-mission-to-the-moon/"
with requests.Session() as session:
    session.headers = {
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36"
    }

    # get the token
    response = session.get(url)
    soup = BeautifulSoup(response.content, "html.parser")
    token = soup.find("meta", {"name": "csrf-token"})["content"]

    # get the description
    description_url = url + "description"
    response = session.get(description_url, headers={"X-CSRF-Token": token, "X-Requested-With": "XMLHttpRequest"})

    soup = BeautifulSoup(response.content, "html.parser")
    description = soup.find('div', {'id':'description_section', 'class': 'description-section'})
    print(description.get_text(strip=True))

Upvotes: 1

John Smith
John Smith

Reputation: 1698

I use XML package to web scraping, and I can't get the description section as you described with BeautifulSoup.

However if you just want to scrap this page only, you can download the page. Then:

page = htmlTreeParse("Lunar Lion - the first ever university-led mission to the Moon _ RocketHub.html", useInternal = TRUE,encoding="utf8")

unlist(xpathApply(page, '//div[@id="description_section"]', xmlValue))

I tried the R code to download, and I can't find the description_section either.

url="https://www.rockethub.com/projects/34210-lunar-lion-the-first-ever-university-led-mission-to-the-moon"

download.file(url,"page.html",mode="w")

Maybe we have to add some options in the function download.file. I hope that some html experts could help.

Upvotes: 0

Related Questions