CODEWITHSUNDEEP
djangopython-2.7scrapy
Brian Williams
Brian Williams

Reputation: 125

Using Scrapy DjangoItem with Django best way

I am am new to Django / Scrapy and well to programming in general. I am trying to make a Django site to help me learn.

What I want to do is Scrape product information from different sites and store them in my postgres database using DjangoItem from Scrapy.

I have read all the docs from both Scrapy and Django. I have searched here and other sites for a couple days and just couldn't find exactly what I was looking for that made the light bulb go off.

Anyway, my question is, what is the standard for deploying Scrapy and Django together. Ideally I would like to scrape 5-10 different sites and store their information in my database.

Scrapy's docs are a little short on information on the best way to implement DjangoItem.
1) Should the Scrapy project be inside my Django app, at the root level of my Django project or outside all together. 2) Other than setting DjangoItem to my Django model, do I need to change any other settings?

Thanks Brian

Upvotes: 3

Views: 924

Answers (1)

dpn
dpn

Reputation: 612

I generally put my scrapy project somewhere inside my Django project root folder. Just remember you will need to make sure both projects are in the python path. This is easy to do if you are using virtualenv properly.

Aside from that as long as you can import your Django models from Scrapy i think everything else in the Scrapy docs is very clear. When you import your Django model, the Django settings are set up at that point, this means your database connection etc should all be working fine as long as they are already working in Django.

The only real trick is getting the python path set up properly (which is probably a topic for another questions).

Upvotes: 2

Related Questions