CODEWITHSUNDEEP
python-3.xweb-scrapingscrapyweb-crawler
Karthik Bhandary
Karthik Bhandary

Reputation: 1651

I am trying to Scrape Data Using Scrapy

I am trying to get all the video links of from the pewdiepie channel. I wrote the following code, it is not showing any errors, but it is not scraping the links.

Here is the code:

import scrapy
from scrapy.crawler import CrawlerProcess

class PewSpider(scrapy.Spider):
   name = "pew_spider"
   def start_request(self):
      urls = ['https://www.youtube.com/user/PewDiePie/videos'] 
      for url in urls:
        yield scrapy.Request(url=url, callback=self.parser)

   def parser(self, response):
      links = response.css('div#contents > a#thumbnail::attr(href)')
      filepath = "./Desktop/pew.csv"
      with open(filepath, 'w') as f:
         f.writelines( [link + '/n' for link in links])

process = CrawlerProcess()
process.crawl(PewSpider)
process.start()

Upvotes: 0

Views: 78

Answers (1)

Yall
Yall

Reputation: 261

I guess you should take a look at YouTube API before trying to scrape it from website. https://developers.google.com/youtube/v3

Upvotes: 1

Related Questions