hassan abbas
Reputation: 72
How to successfully request this endpoint?
I am trying to scrape this website, I have checked the robots.txt and according to my understanding, they allow the scraping of the website.
No matter how much I try, fiddle with the headers, simulate browser behaviour, everything, but I could not get a <200> status code.
The website is https://www.mastersportal.com/universities/80/tilburg-university.html the network tab shows trivial headers, there are no cookies passed according to my understanding, but still the servers deny the request.
I would appreciate if someone could tell me the correct way to scrape the site, I do not want to use selenium as it defeats the key requirement which is speed.
Upvotes: 0
Views: 27
Answers (0)
Related Questions
- How to find the index for a given item in a list?
- How do I make a flat list out of a list of lists?
- How to use a global variable in a function?
- How slicing in Python works
- How do I check whether a file exists without exceptions?
- How do I merge two dictionaries in a single expression in Python?
- How to access the index value in a 'for' loop?
- How do I execute a program or call a system command?
- How to catch multiple exceptions in one line? (in the "except" block)
- How do I create a directory, and any missing parent directories?