Python + scrapy + web scraping : page is not being scraped

Question

I want to scrape movie titles from this page: https://www.imdb.com/list/ls055386972/ . I wrote the following code:

import scrapy
from scrapy import Spider
from scrapy.http import Request
import re
import pymysql
import sys
import hashlib
from datetime import *
#import time
import csv
import os
import requests


class MoviesSpider(scrapy.Spider):

  name = 'movies' #name of the spider
  allowed_domains = ['imdb.com/list/ls055386972/']
  start_urls = ['http://imdb.com/list/ls055386972//']


  def parse(self, response):
      #events = response.xpath('//*[@property="url"]/@href').extract()
      links = response.xpath('//h3[@class]/a/@href').extract()

      final_links = []

      for link in links:
          final_link = 'http://www.imdb.com' + link 
          final_links.append(final_link)     

      for final_link in final_links:
          absolute_url = response.urljoin(final_link)
          yield Request(absolute_url, callback = self.parse_movies)

          #process next page url
          #next_page_url = response.xpath('//a[text() = "Next"]/@href').extract_first()
          #absolute_next_page_url = response.urljoin(next_page_url)
          #yield Request(absolute_next_page_url)    

  def parse_movies(self, response):

      title  = response.xpath('//div[@class = "title_wrapper"]/h1[@class]/text()').extract_first()

      yield{
                'title': title,
      } 

But it's not scraping anything. I am getting this error message:

2019-03-04 18:08:37 [scrapy.core.engine] DEBUG: Crawled (404) <GET https://www.imdb.com/list/ls055386972//> (referer: None)
2019-03-04 18:08:37 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.imdb.com/list/ls055386972//>: HTTP status code is not handled or not allowed
2019-03-04 18:08:37 [scrapy.core.engine] INFO: Closing spider (finished)

print('final_links') produces the correct links to individual movie pages:

[u'https://www.imdb.com/title/tt0068646/?ref_=ttls_li_tt', u'https://www.imdb.com/title/tt0108052/?ref_=ttls_li_tt', u'https://www.imdb.com/title/tt0050083/?ref_=ttls_li_tt', u'https://www.imdb.com/title/tt0118799/?ref_=ttls_li_tt', u'https://www.imdb.com/title/tt0060196/?ref_=ttls_li_tt',..........]

Answer 1

You haven't parsed start url to the parse function, below is working code.

class MoviesSpider(BaseSpider):

  name = 'movies' #name of the spider
  allowed_domains = ['imdb.com']
  start_url = 'http://imdb.com/list/ls055386972/'

  def __init__(self):
      super(MoviesSpider, self).__init__()

  def start_requests(self):
      yield Request(self.start_url, callback=self.parse, headers=self.headers)

  def parse(self, response):
      #events = response.xpath('//*[@property="url"]/@href').extract()
      links = response.xpath('//h3[@class]/a/@href').extract()

      final_links = []

      for link in links:
          final_link = 'http://www.imdb.com' + link
          final_links.append(final_link)

      for final_link in final_links:
          absolute_url = response.urljoin(final_link)
          yield Request(absolute_url, callback = self.parse_movies)

          #process next page url
          #next_page_url = response.xpath('//a[text() = "Next"]/@href').extract_first()
          #absolute_next_page_url = response.urljoin(next_page_url)
          #yield Request(absolute_next_page_url)

  def parse_movies(self, response):

      title  = response.xpath('//div[@class = "title_wrapper"]/h1[@class]/text()').extract_first()

      yield{
                'title': title,
      }

Answer 2

Not sure about Scrapy, but if you use below code you get desired output.

import requests
import lxml
from lxml import html

url = "https://www.imdb.com/list/ls055386972/"

r = requests.get(url)
tree = html.fromstring(r.content)
domain = "https://www.imdb.com/"

movies_list = tree.xpath('//div[@class="lister-item-content"]/h3/a')
for i, movie in enumerate(movies_list, 1):
  print(i, movie.text, domain+movie.get('href'))


1 The Godfather https://www.imdb.com//title/tt0068646/?ref_=ttls_li_tt
2 Schindler's List https://www.imdb.com//title/tt0108052/?ref_=ttls_li_tt
3 12 Angry Men https://www.imdb.com//title/tt0050083/?ref_=ttls_li_tt
4 Life Is Beautiful https://www.imdb.com//title/tt0118799/?ref_=ttls_li_tt
5 The Good, the Bad and the Ugly https://www.imdb.com//title/tt0060196/?ref_=ttls_li_tt
6 The Shawshank Redemption https://www.imdb.com//title/tt0111161/?ref_=ttls_li_tt
7 The Pursuit of Happyness https://www.imdb.com//title/tt0454921/?ref_=ttls_li_tt
8 Seven Samurai https://www.imdb.com//title/tt0047478/?ref_=ttls_li_tt
9 The Intouchables https://www.imdb.com//title/tt1675434/?ref_=ttls_li_tt
10 Central Station https://www.imdb.com//title/tt0140888/?ref_=ttls_li_tt
11 Requiem for a Dream https://www.imdb.com//title/tt0180093/?ref_=ttls_li_tt
12 A Beautiful Mind https://www.imdb.com//title/tt0268978/?ref_=ttls_li_tt
13 Hachi: A Dog's Tale https://www.imdb.com//title/tt1028532/?ref_=ttls_li_tt
14 Taken https://www.imdb.com//title/tt0936501/?ref_=ttls_li_tt
15 My Sassy Girl https://www.imdb.com//title/tt0293715/?ref_=ttls_li_tt
16 Amores Perros https://www.imdb.com//title/tt0245712/?ref_=ttls_li_tt
17 The Shining https://www.imdb.com//title/tt0081505/?ref_=ttls_li_tt
18 Apocalypto https://www.imdb.com//title/tt0472043/?ref_=ttls_li_tt
19 Gladiator https://www.imdb.com//title/tt0172495/?ref_=ttls_li_tt
20 Cast Away https://www.imdb.com//title/tt0162222/?ref_=ttls_li_tt
21 The Dark Knight https://www.imdb.com//title/tt0468569/?ref_=ttls_li_tt
22 The Pianist https://www.imdb.com//title/tt0253474/?ref_=ttls_li_tt
23 Titanic https://www.imdb.com//title/tt0120338/?ref_=ttls_li_tt
24 3-Iron https://www.imdb.com//title/tt0423866/?ref_=ttls_li_tt
25 Braveheart https://www.imdb.com//title/tt0112573/?ref_=ttls_li_tt
26 It's a Wonderful Life https://www.imdb.com//title/tt0038650/?ref_=ttls_li_tt
27 Spring, Summer, Fall, Winter... and Spring https://www.imdb.com//title/tt0374546/?ref_=ttls_li_tt
28 Alien https://www.imdb.com//title/tt0078748/?ref_=ttls_li_tt
29 Memories of Murder https://www.imdb.com//title/tt0353969/?ref_=ttls_li_tt
30 The Return https://www.imdb.com//title/tt0376968/?ref_=ttls_li_tt
31 I Saw the Devil https://www.imdb.com//title/tt1588170/?ref_=ttls_li_tt
32 Children of Heaven https://www.imdb.com//title/tt0118849/?ref_=ttls_li_tt
33 A Separation https://www.imdb.com//title/tt1832382/?ref_=ttls_li_tt
34 The Sixth Sense https://www.imdb.com//title/tt0167404/?ref_=ttls_li_tt
35 A Moment to Remember https://www.imdb.com//title/tt0428870/?ref_=ttls_li_tt
36 Departures https://www.imdb.com//title/tt1069238/?ref_=ttls_li_tt
37 Wo de fu qin mu qin https://www.imdb.com//title/tt0235060/?ref_=ttls_li_tt
38 Saving Private Ryan https://www.imdb.com//title/tt0120815/?ref_=ttls_li_tt
39 The Bridge on the River Kwai https://www.imdb.com//title/tt0050212/?ref_=ttls_li_tt
40 Ben-Hur https://www.imdb.com//title/tt0052618/?ref_=ttls_li_tt
41 The Exorcist https://www.imdb.com//title/tt0070047/?ref_=ttls_li_tt
42 The Secret in Their Eyes https://www.imdb.com//title/tt1305806/?ref_=ttls_li_tt
43 Léon: The Professional https://www.imdb.com//title/tt0110413/?ref_=ttls_li_tt
44 The Green Mile https://www.imdb.com//title/tt0120689/?ref_=ttls_li_tt
45 Gran Torino https://www.imdb.com//title/tt1205489/?ref_=ttls_li_tt
46 Kill Bill: Vol. 1 https://www.imdb.com//title/tt0266697/?ref_=ttls_li_tt
47 Jurassic Park https://www.imdb.com//title/tt0107290/?ref_=ttls_li_tt
48 Terminator 2: Judgment Day https://www.imdb.com//title/tt0103064/?ref_=ttls_li_tt
49 Back to the Future https://www.imdb.com//title/tt0088763/?ref_=ttls_li_tt
50 Finding Nemo https://www.imdb.com//title/tt0266543/?ref_=ttls_li_tt

Answer 3

You are getting a 404 because your starting URL is incorrect. You need to remove the trailing forward slash in start_urls:

start_urls = ['https://www.imdb.com/list/ls055386972/']

Also, your allowed_domains is incorrect. It should contain only domains, not partial URLs:

allowed_domains = ['imdb.com']

Refer to the documentation.