CODEWITHSUNDEEP
pythonjsonweb-scrapingscrapy
ch1maera
ch1maera

Reputation: 1447

Empty .json File in Scrapy

I've written this very short spider to go to a U.S. News link and take the names of the colleges listed there.

#!/usr/bin/python
# -*- coding: utf-8 -*-

import scrapy

class CollegesSpider(scrapy.Spider):
    name = "colleges"
    start_urls = [
        'http://colleges.usnews.rankingsandreviews.com/best-colleges/rankings/national-universities?_mode=list&acceptance-rate-max=20'
    ]

    def parse(self, response):
        for school in response.css('div.items'):
            yield {
                'name': school.xpath('//*[@id="view-1c4ddd8a-8b04-4c93-8b68-9b7b4e5d8969"]/div/div[1]/div[1]/h3/a').extract_first(),
            }

However, when I run this spider and ask for the names to be stored in a file named schools.json, the file comes out blank. What am I doing wrong?

Upvotes: 0

Views: 662

Answers (3)

Bobby
Bobby

Reputation: 1571

Got it! It is because the robot detection.

Encode

>>> r = requests.get('http://colleges.usnews.rankingsandreviews.com/best-colleges/rankings/national-universities?_mode=list&acceptance-rate-max=20', headers={'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36'})
>>> r.status_code
200

Then you will have all the content you need. Do whatever parsing or extraction you need. The procedure to encode a header should be very similar in Scrapy.

scrapy doc for request with headers

User agent for Chrome

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36

Upvotes: 1

mizhgun
mizhgun

Reputation: 1887

The page you're referring to as start url doesn't contain any element with id view-1c4ddd8a-8b04-4c93-8b68-9b7b4e5d8969- it looks like quite unique and doesn't seem to be the good choice for pretty universal XPath expression. I'd recommend to use something like school.xpath('.//div[@data-view="colleges-search-results-card"]//h3/a/text()').extract()

Upvotes: 0

Umair Ayub
Umair Ayub

Reputation: 21261

I am on my mobile so don't remember exact variable name, but it should be robots_follow

Set it to False

Upvotes: 0

Related Questions