CODEWITHSUNDEEP
pythonxpathscrapy
Mark Genesis Romantigue
Mark Genesis Romantigue

Reputation: 15

Get xpath of a link in a webpage containing "sometext"

I'm using scrapy (web crawling framework). Is there any way I can get the xpath of a certain element (containing "sometext") in a web page so I can extract elements with similar xpaths? I don't want my xpaths to be hardcoded because I will be crawling multiple websites.

I'm new to scrapy and I have been searching for days and I can't find anything :(

Upvotes: 0

Views: 86

Answers (1)

salmanwahed
salmanwahed

Reputation: 9647

You have to explicitly specify which element you want to scrape either you use xpath or regular expression or some library like beautifulsoup. One way of not explicitly specifying the xpath is to traverse the DOM and extracting the elements you need. But even in this case you need some kind of mechanism for identifying the elements you want to scrape. Also you should write different spiders for scraping different websites. Scraping multiple website with a single spider will make your task much harder and its not a good practice either.

For deploying and running spiders you can look into scrapyd

Upvotes: 1

Related Questions