CODEWITHSUNDEEP
pythonimportscrapy
Mark
Mark

Reputation: 229

Python include Scrapy from subdirectory

I would like to know if there's a way that I can put Scrapy into a subdirectory and import it. I did this with BeautifulSoup, rather than installing it, I just drop the bs4 directory into the directory of my app, and import it:

from bs4 import BeautifulSoup

In the source that I downloaded from scrapy.org there is no scrapy.py so I tried importing

from scrapy import *

This returned a bunch of errors.

Traceback (most recent call last):
File "C:\Users\Kat\Desktop\linkscrape\cookie.py", line 1, in <module> from scrapy import *
File "C:\Users\Kat\Desktop\linkscrape\scrapy\__init__.py", line 27, in <module>
from . import _monkeypatches
File "C:\Users\Kat\Desktop\linkscrape\scrapy\_monkeypatches.py", line 2, in <module>
from six.moves import copyreg
ImportError: No module named six.moves

Is there any way that I can simply include this to make it easy to migrate the app from computer to computer, or does this have to be installed? Thanks.

Upvotes: 1

Views: 305

Answers (1)

alexanderlukanin13
alexanderlukanin13

Reputation: 4715

Don't do that. You will hurt yourself.

To achieve easy portability, use virtualenv - it's a golden standard of Python development.

Create a file named requirements.txt in a root directory of your project, and write all necessary dependencies, like this:

python-dateutil==2.4.2
Scrapy==1.0.4
XlsxWriter==0.7.7

When you are setting up the environment, create a fresh virtualenv and then simply:

pip install -r requirements.txt

Voilà! You have a working environment.

In case of Scrapy, it's even more important because in production Scrapy is typically deployed using scrapyd, where having a correct package with a version number and fixed requirements is absolutely necessary.

Upvotes: 4

Related Questions