CODEWITHSUNDEEP
pythonscrapyweb-crawler
omargamal8
omargamal8

Reputation: 601

How to extract the url my crawler is currently standing on?

I am making a web crawler using scrapy framework in python. The main idea is that the crawler extracts some data from the page, if the data matches some criteria the crawler should extract the URL its currently standing on. Is there some method/function in scrappy that gets the current URL?

Thanks.

Upvotes: 0

Views: 105

Answers (1)

Will
Will

Reputation: 5490

The 'response' variable that's passed to parse() has the info you want. You shouldn't need to override anything.

eg.

def parse(self, response):
    print "URL: " + response.url

Upvotes: 2

Related Questions