CODEWITHSUNDEEP
scrapy
wrongusername
wrongusername

Reputation: 18918

Having one spider use items returned from another spider?

So I've written a spider that extracts certain desired links from a webpage and puts the URL, link text, and other information not necessarily contained in the <a> tag itself, into an item for each link.

How should I pass this item onto another spider which scrapes the URL provided in that item?

Upvotes: 2

Views: 2150

Answers (1)

zoonosis
zoonosis

Reputation: 799

This question has been asked many times. Below are some links on this site that answer your question. Some answer it directly ie passing items to another function but you may realise that you do not need to do it that way, so other methods are linked to show whats possible.

Using multiple spiders at in the project in Scrapy

Scrapy - parse a page to extract items - then follow and store item url contents

Scrapy: Follow link to get additional Item data?

Upvotes: 6

Related Questions