CODEWITHSUNDEEP
webweb-scraping
Maxii Paguera
Maxii Paguera

Reputation: 33

Webscraping Endpoint

Hello I was wondering how i can find a hidden Endpoint of a website that stores product urls.

I tried getting the sitemap of the website but the website either does not have one or it is hidden and I cant find it. Also searched for

I hope someone could help me or point me in the right direction

Upvotes: 1

Views: 37

Answers (1)

user11553043
user11553043

Reputation:

A helpful resource to request from the server you are crawling is /robots.txt. Most hosts will serve this file, and it tells your crawler where it is allowed to go.

Upvotes: 1

Related Questions