CODEWITHSUNDEEP
pythonscrapyweb-crawleroutput
Yevhenii
Yevhenii

Reputation: 11

How to pass several parameters in FEED_URI (Scrapy) Python

My question is simple - is there any way to pass several output sources for Scrapy? Like I want to save duplicate of scraped result into ftp. Should I create additional pipeline for this purpose? Thank You!

Upvotes: 0

Views: 120

Answers (1)

Gallaecio
Gallaecio

Reputation: 3857

This is not possible at the moment. There is a feature request for it, and a proposed implementation that may become available in the upcoming Scrapy 2.0 version. Maybe you could try the proposal out and review it.

In the meantime, yes, item pipelines are one way to do it. But if you don’t want to slow down your crawl you need to use deferreds. The code in open pull request #3961 may help, but it is far from trivial, specially if you are not familiar with Twisted.

There’s also the option of uploading the file to FTP after the spider has finished generating the local file, which is probably the most straightforward solution until multiple-feed support is implemented.

Upvotes: 1

Related Questions