Scrapy didn't produce output correctly for a different Base URL?

Question

I am Still Beginner and learning Scrapy

So I am making Scrapy script to scrape an amount of links in rumah123.com, exactly at https://www.rumah123.com/en/sale/surabaya/surabaya-kota/all-residential/, and it turns out a success! it produces csv of links

But when I changed link at https://www.rumah123.com/en/rent/surabaya/surabaya-kota/all-residential/, My Scrapy Script didn't produce anything

When I run the script, Scrapy Log exactly says:

2019-10-18 13:02:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.rumah123.com/en/rent/surabaya/surabaya-kota/all-residential/> (referer: None)
2019-10-18 13:02:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.rumah123.com/en/rent/surabaya/surabaya-kota/all-residential/?page=2> (referer: None)
2019-10-18 13:02:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.rumah123.com/en/rent/surabaya/surabaya-kota/all-residential/?page=6> (referer: None)
2019-10-18 13:02:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.rumah123.com/en/rent/surabaya/surabaya-kota/all-residential/?page=5> (referer: None)
2019-10-18 13:02:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.rumah123.com/en/rent/surabaya/surabaya-kota/all-residential/?page=7> (referer: None)
2019-10-18 13:02:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.rumah123.com/en/rent/surabaya/surabaya-kota/all-residential/?page=4> (referer: None)
2019-10-18 13:02:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.rumah123.com/en/rent/surabaya/surabaya-kota/all-residential/?page=8> (referer: None)
2019-10-18 13:02:06 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.rumah123.com/en/rent/surabaya/surabaya-kota/all-residential/?page=9> (referer: None)
2019-10-18 13:02:06 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.rumah123.com/en/rent/surabaya/surabaya-kota/all-residential/?page=10> (referer: None)
2019-10-18 13:02:16 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.rumah123.com/en/rent/surabaya/surabaya-kota/all-residential/?page=3> (referer: None)
2019-10-18 13:02:16 [scrapy.core.engine] INFO: Closing spider (finished)

But when I check the real csv, it contains nothing inside!

This is the Scripts whole code:

class Rumah123_Spyder(scrapy.Spider):
    name = "Home_Rent"
    url_list = []
    page = 1
    def start_requests(self):
        headers = {
            'accept-encoding': 'gzip, deflate, sdch, br',
            'accept-language': 'en-US,en;q=0.8,zh-CN;q=0.6,zh;q=0.4',
            'upgrade-insecure-requests': '1',
            'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36',
            'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
            'cache-control': 'max-age=0',
        }
        #base = 'https://www.rumah123.com/en/sale/surabaya/surabaya-kota/all-residential/'
        base = 'https://www.rumah123.com/en/rent/surabaya/surabaya-kota/all-residential/'
        for x in range(10): #depends on number of page in search results
            if x==0:
                yield scrapy.Request(url=base, headers=headers, callback=self.parse)
                self.page += 1
            else:
                yield scrapy.Request(url=base + "?page=" + str(self.page), headers=headers, callback=self.parse)
                self.page += 1

        #Filter a not valid URL 
        self.url_list = [rum for rum in self.url_list if "/property/" in rum]
        for x in range(len(self.url_list)):
            self.url_list[x] = "rumah123.com" + self.url_list[x]

        url_df = pd.DataFrame(self.url_list, columns=["Sub URL"])
        #url_df.to_csv("home_sale_link.csv", encoding="utf_8_sig")
        url_df.to_csv("home_rent_link.csv", encoding="utf_8_sig")

    def parse(self, response):
        for rumah in response.xpath('//a/@href'):
            if rumah.get() not in self.url_list:
                self.url_list.append(rumah.get())

from scrapy import cmdline
cmdline.execute("scrapy runspider Rumah123_url.py".split())

The Expected Results is like when in the First try of URL, here is a screenshot of the links:

https://i.sstatic.net/jWxSr.jpg

The Current Result for the "rent" URL is empty, here is a screenshot:

https://i.sstatic.net/ppX16.jpg

Extra Note: I tested to run using scrapy shell https://www.rumah123.com/en/sale/surabaya/surabaya-kota/all-residential/, if i run the code manually, it can produces the CSV directly, but it will be very tiring by running code one on one :(

Can Anyone point me why this can happen? Thankyou :)

Answer 1

Extracting url in our spider import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"    
    start_urls = ['https://www.rumah123.com/en/rent/surabaya/surabaya-kota/all-residential/?page=' + str(i) for i in range(1, 10)]

    def parse(self, response):
        for quote in response.xpath('//*[@class="sc-bRbqnn iRnfmd"]'):
            yield {
                'url1': quote.xpath('a/@href').extract(),
            }

The simplest way to store the scraped data is by using Feed exports, with the following command:

scrapy crawl quotes -o 1.csv

Answer 2

I have found out the issues! By changing the loops from 10 to something bigger like 30, My CSV is now filled with the URL list! although I don't really know why this works

for x in range(30): #depends on number of page in search results
    if x==0:
        yield scrapy.Request(url=base, headers=headers, callback=self.parse)
        self.page += 1
    else:
        yield scrapy.Request(url=base + "?page=" + str(self.page), headers=headers, callback=self.parse)
    self.page += 1

Answer 3

The issue is your spider is not yielding anything.

You can try the following parse method

def parse(self, response):
        for rumah in response.xpath('//a/@href'):
            if rumah.get() not in self.url_list:
                self.url_list.append(rumah.get())
        yield {'result': self.url_list}