CODEWITHSUNDEEP
pythonbeautifulsoupscreen-scraping
Dawid Olejnik
Dawid Olejnik

Reputation: 67

trying to scrape url

So im trying to get all the url from the free games site on steam however it keeps returning empty. I dont know what I'm doing wrong here, image below shows the path 

result = requests.get("https://steamdb.info/upcoming/free/")
src = result.content
soup = BeautifulSoup(src, 'lxml')

urls = []
for td_tag in soup.find_all('td'):
    a_tag = td_tag.find('a')
    urls.append(a_tag.attrs['href'])

print(urls)

enter image description here

Upvotes: 1

Views: 605

Answers (1)

furas
furas

Reputation: 142651

You have to use header User-Agent and it can't be short Mozilla/5.0 but full string from real web browser

import requests
from bs4 import BeautifulSoup

headers = {
    "User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0",
}

result = requests.get("https://steamdb.info/upcoming/free/", headers=headers)
soup = BeautifulSoup(result.content, 'lxml')

#print(result.content)
urls = []
for td_tag in soup.find_all('td'):
    a_tag = td_tag.find('a')
    if a_tag:
        urls.append(a_tag.attrs['href'])

print(urls)

Upvotes: 4

Related Questions