CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsouprequest
royal
royal

Reputation: 51

Web Scraping: How do I get 'href' links and scrape table from them

I am trying to scrape table from link. So that need to scrape 'href' links from it and then try to scrape the table from it . I try following code but couldn't find:

from bs4 import BeautifulSoup
import requests


url = 'http://www.stats.gov.cn/was5/web/search?channelid=288041&andsen=%E6%B5%81%E9%80%9A%E9%A2%86%E5%9F%9F%E9%87%8D%E8%A6%81%E7%94%9F%E4%BA%A7%E8%B5%84%E6%96%99%E5%B8%82%E5%9C%BA%E4%BB%B7%E6%A0%BC%E5%8F%98%E5%8A%A8%E6%83%85%E5%86%B5'
page = requests.get(url)

soup = BeautifulSoup(page.text, 'html.parser')
#table = soup.find("table")
#print(table)
# links = []
# for href in soup.find_all(class_='searchresulttitle'):
#     print(href)
#     links.append(href.find('a').get('href'))
#     print(links)
link = soup.find(attr={"class":"searchresulttitle"})
print(link)

So please guide me how to find href and scrape table from them

Upvotes: 1

Views: 4294

Answers (1)

Martin Evans
Martin Evans

Reputation: 46759

The URLs are stored in the HTML as variables inside Javascript. BeautifulSoup can be used to grab all the <script> elements and then a regular expression can be used to extract the value for urlstr.

Assuming Python 3.6 is being used, a dictionary can be used to create a unique ordrered list of the URLs displayed:

from bs4 import BeautifulSoup
import requests
import re

headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36'}
url = 'http://www.stats.gov.cn/was5/web/search?channelid=288041&andsen=%E6%B5%81%E9%80%9A%E9%A2%86%E5%9F%9F%E9%87%8D%E8%A6%81%E7%94%9F%E4%BA%A7%E8%B5%84%E6%96%99%E5%B8%82%E5%9C%BA%E4%BB%B7%E6%A0%BC%E5%8F%98%E5%8A%A8%E6%83%85%E5%86%B5'
r = requests.get(url, headers=headers)
soup = BeautifulSoup(r.content, 'html.parser')
urls = {}   # Use a dictionary to create unique, ordered URLs (Assuming Python >=3.6)

for script in soup.find_all('script'):
    for m in re.findall(r"var urlstr = '(.*?)';", script.text):
        urls[m] = None

urls = list(urls.keys())
print(urls)

This would display URLS starting as:

['http://www.stats.gov.cn/tjsj/zxfb/201811/t20181105_1631364.html',  
'http://www.stats.gov.cn/tjsj/zxfb/201810/t20181024_1629464.html', 
'http://www.stats.gov.cn/tjsj/zxfb/201810/t20181015_1627579.html', 
...]

Upvotes: 1

Related Questions