CODEWITHSUNDEEP
scrapy
Domingo Oh
Domingo Oh

Reputation: 99

How to install Scrapy on Amazon Linux AMI

I want to use Scrapy which is a crawler framework, and install it to my Server.

This is my Server spec.

========================================================

NAME="Amazon Linux AMI"

VERSION="2015.09"

ID="amzn"

ID_LIKE="rhel fedora"

VERSION_ID="2015.09"

PRETTY_NAME="Amazon Linux AMI 2015.09"

ANSI_COLOR="0;33"

CPE_NAME="cpe:/o:amazon:linux:2015.09:ga"

HOME_URL="http://aws.amazon.com/amazon-linux-ami/"

Amazon Linux AMI release 2015.09

========================================================

I have read a lot of web pages about this and followed the steps. Still I always get an error.

I've used the command sudo pip install scrapy and I get this log.

enter image description here

This is red logs.

-> Failed building wheel for lxml

-> Command "/usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-lT29Ha/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-WTCmji-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-lT29Ha/lxml

I've spent 2 days on the Scrapy install. Please help me.

Upvotes: 0

Views: 1618

Answers (1)

dataisbeautiful
dataisbeautiful

Reputation: 546

I just did this yesterday on AWS. I used a nano instance so needed to add some swap first.

sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo echo "/swapfile swap swap sw 0 0" >> /etc/fstab
sudo swapon /swapfile

Then install the prerequisites and scrapy:

sudo su
yum update -y
yum install python-pip -y
yum install python-devel -y
yum install gcc gcc-devel -y
yum install libxml2 libxml2-devel -y
yum install libxslt libxslt-devel -y
yum install openssl openssl-devel -y
yum install libffi libffi-devel -y
CFLAGS="-O0" pip install lxml
pip install scrapy
scrapy -v

Upvotes: 4

Related Questions