How to install portia, a python application from Github (Mac)

Question

I am attempting to install Portia, a python app from Github: https://github.com/scrapinghub/portia

I use the following steps at the command line:

1. set up new virtualenv 'portia' in Mac terminal
2. git clone https://github.com/scrapinghub/portia.git

3. follow readme instructions:

   cd slyd

   pip install -r requirements.txt

4. run Portia

   cd slyd

   twistd -n slyd

But every time I attempt the last step to run the program, I get the following error:

ImportError: No module named scrapy

Any idea why this error is occurring? All previous steps seem to install correctly. Is it an error earlier in my install process?

Thanks!

Answer 1

Another Installation Method For Portia: Using Vagrant

Here is the method that made me install Portia with ease. Works with Mac, Windows and Linux. With a few commands and clicks, you'll get a fully functional web scraper.

Things Needed:

1. VirtualBox
2. Vagrant
3. Clone the repo for Portia or download the zip file.

Additional Steps To Take:

1. Install VirtualBox.
2. Install Vagrant

3. Open your terminal and navigate to where you cloned the Portia repo or where you've extracted it (in case of a zip file).

4. Then make a command vagrant up - This will download and setup a VirtualBox Guest VM for you + will install all the necessary requirements for Portia and will install Portia from start to finished.

After the above process, you may now open your browser and navigate to

http://the-virtualbox-ip:8000/static/main.html

And you're setup.

Answer 2

I don't have the rep to upvote Alagappan's answer but he's correct. Also, if you're as inexperienced as I am, you may need further clarity on this.

You have to create, activate and navigate into the virtualenv before installing anything (including cloning portia from github). Here's the whole thing working from start to finish:

1: cd to wherever you’d like to store your project... and Install virtualenv:

$ pip install virtualenv

2: Create the virtual environment. (I called mine “portia” but this can be anything.):

$ virtualenv portia

3: Activate the virtual environment you created (change the path to reflect the name you used here if not “portia”.):

$ source portia/bin/activate

At this point your terminal should have display the virtualenv name in parenthesis before the standard directory path prompt:
 (name-of-virtualenv) [your-machine]:[current-directory]: [user]$ ...and if you list the files within your pwd you’ll see the name of you virtualenv there.

4: cd into your virtualenv (“portia” for me):

$ cd portia

5: Now you can clone portia from github into your virtualenv...

$ git clone https://github.com/scrapinghub/portia

6: cd into the cloned portia/slyd...

$ cd portia/slyd

7/8: pip install twisted and Scrapy...

$ pip install twisted
$ pip install Scrapy

You’re virtualenv should still be activated and you should still be in [virtualenv-name]/portia/slyd

9: Install the requirements.txt:

$ pip install -r requirements.txt

10: Run slyd:

$ twistd -n slyd

--- No more scrapy error! ---

Answer 3

I suppose the issue you are facing is because of the virtualenv. Once you setup a new virtual environment you need to run the activate script in order to start using it. In your case you'll have to run the following command:

$ source portia/bin/activate

On successful activation, your prompt will look like:

(portia) $

Can you check if you activated your virtual environment before you installed the packages using pip? I believe doing so will fix your issue.

Answer 4

It's quite simple, you just need to install the python module scrapy in the same way that the Twitter API requires setuptools

pip install scrapy