CODEWITHSUNDEEP
pythonbeautifulsoupscreen-scraping
Alexis
Alexis

Reputation: 89

Scraper not extracting url link:

Hi I am trying to scrape the Amazon url address on this site linked under "View item on Amazon".

My code is below, I get zero response. Appreciate any assistance. Thanks

import requests
url = "https://app.jumpsend.com/deals/230513"

response = requests.get(url)
data = response.text

soup = BeautifulSoup(data, 'lxml')

tags = soup.find_all('a')

for tag in tags:
    print(tag.get('href'))

Upvotes: 0

Views: 47

Answers (1)

James Dellinger
James Dellinger

Reputation: 1261

The Amazon link (https://www.amazon.com/dp/B07MH9DK5B) isn't in the html page source. You need to use Selenium in order to read-in the html of all the elements that are set by Java script:

from bs4 import BeautifulSoup
from selenium import webdriver

url = "https://app.jumpsend.com/deals/230513"
driver = webdriver.Firefox()
driver.get(url)
html = driver.page_source
soup = BeautifulSoup(html, 'html.parser')
soup.find('a', attrs={'class': 'deal-modal-link'})['href']

The above code prints out the Amazon link:

'https://www.amazon.com/dp/B07MH9DK5B'

Upvotes: 2

Related Questions