CODEWITHSUNDEEP
pythonrequest
user2525951
user2525951

Reputation: 30

List links in web page with python

I am trying to write a python script that lists all the links in a webpage that contain some substring. The problem that I am running into is that the webpage has multiple "pages" so that it doesn't clutter all the screen. Take a look at https://www.go-hero.net/jam/17/solutions/1/1/C++ for an example.

This is what I have so far:

import requests
from bs4 import BeautifulSoup
url = "https://www.go-hero.net/jam/17/solutions/1/1/C++"
response = requests.get(url)
soup = BeautifulSoup(response.content, "html5lib")
links = soup.find_all('a')

for tag in links:
  link = tag.get('href', None)
  if link is not None and 'GetSource' in link:
    print(link)

Any suggestions on how I might get this to work? Thanks in advance.

Upvotes: 1

Views: 4218

Answers (1)

Brendan Goggin
Brendan Goggin

Reputation: 2361

Edit/Update: Using Selenium, you could click the page links before scraping the html to collect all the content into the html. Many/most websites with pagination don't collect all the text in the html when you click through the pages, but I noticed that the example you provided does. Take a look at this SO question for a quick example of making Selenium work with BeautifulSoup. Here is how you could use it in your code:

import requests
from bs4 import BeautifulSoup
from selenium import webdriver

driver = webdriver.Firefox()
original_url = "https://www.go-hero.net/jam/17/solutions/1/1/C++"
driver.get(original_url)

# click the links for pages 1-29
for i in range(1, 30):
    path_string = '/jam/17/solutions/1/1/C++#page-' + str(i)
    driver.find_element_by_xpath('//a[@href=' + path_string + ']').click()

# scrape from the accumulated html
html = driver.page_source
soup = BeautifulSoup(html)
links = soup.find_all('a')

# proceed as normal from here
for tag in links:
    link = tag.get('href', None)
    if link is not None and 'GetSource' in link:
        print(link)

Original Answer: For the link you provided above, you could simply loop through possible urls and run your scraping code in the loop:

import requests
from bs4 import BeautifulSoup
original_url = "https://www.go-hero.net/jam/17/solutions/1/1/C++"

# scrape from the original page (has no page number)
response = requests.get(original_url)
soup = BeautifulSoup(response.content, "html5lib")
links = soup.find_all('a')

# prepare to scrape from the pages numbered 1-29
# (note that the original page is not numbered, and the next page is "#page-1")
url_suffix = '#page-'

for i in range(1, 30):
    # add page number to the url
    paginated_url = original_url + url_suffix + str(i)
    response = requests.get(paginated_url)
    soup = BeautifulSoup(response.content, "html5lib")
    # append resulting list to 'links' list
    links += soup.find_all('a')

# proceed as normal from here
for tag in links:
    link = tag.get('href', None)
    if link is not None and 'GetSource' in link:
        print(link)

I don't know if you mind that you'll get duplicates in your results. You will get duplicate results in your link list as the code currently stands, but you could add the links to a Set or something instead to easily remedy that.

Upvotes: 1

Related Questions