CODEWITHSUNDEEP
pythonscrapyweb-crawler
fatemeh zamanipour
fatemeh zamanipour

Reputation: 85

Scrapy doesn't write file in my directory so I can't see that

Hey I'm trying to crawl over a website and get its titles but it won't write the text file that I need. it wont give me any error and I got everything done by creating new file from cmd and etc.

Here is my code:

import scrapy

class My_spider(scrapy.Spider):
    name= "book_crawler"

    
    def start_request(self):
        url_list=[]
        for i in range(2 , 5):
            url_list.append("https://books.toscrape.com/catalogue/page-" + str(i) + ".html")
        urls=[]
        urls.append("https://books.toscrape.com/")
        for i in range(0 , len(url_list)):
            urls.append(url_list[i])

        for url in urls:
            yield scrapy.request(url= url, callback= self.parse)

     
    
    
    def parse(self,response):
        title_list=response.xpath("article[@class='product_pod']/h3/a/text()").extract()
        
        
        with open('book_titel.txt' , 'a+') as f:
            for i in range(0, len(title_list)) :
                f.write(str(i)+ " : " + title_list[i] + "\n")

Upvotes: 0

Views: 353

Answers (1)

Samsul Islam
Samsul Islam

Reputation: 2619

I found 3 three typo in your code. use start_requests instead of start_request, use scrapy.Request instead of scrapy.Request and last in your xpath.

class My_spider(scrapy.Spider):
    name= "book_crawler"

    def start_requests(self):
        url_list=[]
        for i in range(2 , 5):
            url_list.append("https://books.toscrape.com/catalogue/page-" + str(i) + ".html")
        urls=[]
        urls.append("https://books.toscrape.com/")
        for i in range(0 , len(url_list)):
            urls.append(url_list[i])

        for url in urls:
            yield scrapy.Request(url= url, callback= self.parse)


    def parse(self,response):
        title_list = response.css("ol.row").xpath('//h3/a/text()').extract()
        print(title_list)
        with open('book_titel.txt' , 'a+') as f:
            for i in range(0, len(title_list)) :
                f.write(str(i)+ " : " + title_list[i] + "\n")

Upvotes: 1

Related Questions