CODEWITHSUNDEEP
python-3.xweb-scrapingurllib
Marco Sousa-Poza
Marco Sousa-Poza

Reputation: 53

urllib.request.urlopen is not working

I am trying to fetch some data from a webpage using bs4, but I am having troubles opening the link. So here is the code I am using:

    from bs4 import BeautifulSoup as soup
    from urllib.request import urlopen

    my_url = "https://www.transfermarkt.com/wettbewerbe/europa/"
    client = urlopen(my_url)
    page_html = client.read()
    client.close()

The curious thing is that only this particular link won't work. Others work completely fine. So what is so special about this link? And how can I open it?

Upvotes: 0

Views: 2087

Answers (1)

Wright
Wright

Reputation: 3424

The problem is from the User-Agent. Use urllib.request.Request to set/change the header.

from urllib.request import urlopen, Request
from bs4 import BeautifulSoup as soup

my_url = "https://www.transfermarkt.com/wettbewerbe/europa/"
client = Request(my_url, headers={"User-Agent" : "Mozilla/5.0"})

page = urlopen(client).read()
print(page)

Upvotes: 2

Related Questions