CODEWITHSUNDEEP
pythonweb-scrapingscrapy
Joseph
Joseph

Reputation: 3959

downloading dynamic webcontent through Scrapy (python 2.7)

I've came back to this project (web scraping dynamic content) and I followed through with this tutorial, however I cannot complete it.

I am using python 2.7 with the scrapy framework 0.14

With my project, getting data is obviously the most important point, I am coming to understand the technique to scraping sites that pull down dynamic data with ajax, however in this example I am unable to actually download the images.

The scraper runs with the command (in cmd.exe on windows 7):

scrapy runspider nasa.py

but it does not actually download anything (it only parses the data)

Am I missing something? This is the first real programming project where I have ventured out into non-beginner territory and am certainly getting lost.

This is the tutorial:

Scraping ajax with Scrapy

Any advice on the correct code to allow the script to download images to local storage?

Thanks :)

Upvotes: 0

Views: 862

Answers (1)

akhter wahab
akhter wahab

Reputation: 4085

in your setting.py

add

IMAGES_STORE = 'you dir path where you want to download images'
ITEM_PIPELINES = ['scrapy.contrib.pipeline.images.ImagesPipeline'] #enable image pipline

in your item define image_urls

add images urls your in image_urls # should be treated as a list in your program

suggested study

Upvotes: 1

Related Questions