CODEWITHSUNDEEP
pythonbeautifulsouppython-requestsweb-crawler
blackscratch22
blackscratch22

Reputation: 90

Python Web Crawler No Results

I am making a basic Web Crawler/Spider with Python. I am trying to crawl through a YouTube channel and print all the titles of the videos on it but it never returns anything.

Here is my code so far:

import requests
from bs4 import BeautifulSoup

url = 'https://www.youtube.com/c/DanTDM/videos'

response = requests.get(url)

soup = BeautifulSoup(response.text, 'html.parser')

x = soup.select(".yt-simple-endpoint style-scope ytd-grid-video-renderer")

print(x)

And the output is always: []. An empty list (which means it didn't find anything). I need to know what I'm doing wrong.

Upvotes: 0

Views: 166

Answers (2)

blackscratch22
blackscratch22

Reputation: 90

Browser Automation with Selenium

When I send a request to YouTube, I receive the following page:

(A 'Before you continue to Youtube' page).

So...

We should use Selenium instead as we need to click one of the buttons. I don't think we can interact with the website using the requests module.

Selenium allows you to have control over your browser. Read the documentation!

Upvotes: 1

Jonathan Ciapetti
Jonathan Ciapetti

Reputation: 1285

The code seems correct.

Call print(response.text) and see if YouTube is returning you a blocking page.

Anti scraping measures can be in action, as checking your user agent, etc.

Upvotes: 1

Related Questions