CODEWITHSUNDEEP
pythonpython-3.xbeautifulsoupyoutubepython-requests
Faizan Shah
Faizan Shah

Reputation: 161

problems getting links from youtube channel with beautifulsoup

I am trying to scrape a youtube channel and return all of the links for each video of this channel, however when I try to print out these links, I only get a few links that have nothing to do with the videos. I am suspecting the videos may be loaded by Javascript, so would there we a way to even do this with beautifulsoup? Will I have to use selenium? Can somebody please help me and do some testing. Here is my code so far:

import requests
from bs4 import BeautifulSoup

print('scanning page...')

youtuber = 'memeulous'
result = requests.get('https://www.youtube.com/c/' + youtuber + '/videos')
status = result.status_code
src = result.content
soup = BeautifulSoup(src, 'lxml')
links = soup.find_all('a')

if status == 200:
    print('valid URL, grabbing uploads...')
else:
    print('invalid URL, status code: ' + str(status))
    quit()

print(links)

and here is my output:

scanning page...
valid URL, grabbing uploads...
[<a href="https://www.youtube.com/about/" slot="guide-links-primary" style="display: none;">About</a>, <a href="https://www.youtube.com/about/press/" slot="guide-links-primary" style="display: none;">Press</a>, <a href="https://www.youtube.com/about/copyright/" slot="guide-links-primary" style="display: none;">Copyright</a>, <a href="/t/contact_us" slot="guide-links-primary" style="display: none;">Contact us</a>, <a href="https://www.youtube.com/creators/" slot="guide-links-primary" style="display: none;">Creators</a>, <a href="https://www.youtube.com/ads/" slot="guide-links-primary" style="display: none;">Advertise</a>, <a href="https://developers.google.com/youtube" slot="guide-links-primary" style="display: none;">Developers</a>, <a href="/t/terms" slot="guide-links-secondary" style="display: none;">Terms</a>, <a href="https://www.google.co.uk/intl/en-GB/policies/privacy/" slot="guide-links-secondary" style="display: none;">Privacy</a>, <a href="https://www.youtube.com/about/policies/" slot="guide-links-secondary" style="display: none;">Policy and Safety</a>, <a href="https://www.youtube.com/howyoutubeworks?utm_campaign=ytgen&amp;utm_source=ythp&amp;utm_medium=LeftNav&amp;utm_content=txt&amp;u=https%3A%2F%2Fwww.youtube.com%2Fhowyoutubeworks%3Futm_source%3Dythp%26utm_medium%3DLeftNav%26utm_campaign%3Dytgen" slot="guide-links-secondary" style="display: none;">How YouTube works</a>, <a href="/new" slot="guide-links-secondary" style="display: none;">Test new features</a>]
[Finished in 4.0s]

as you can see, no video links.

Upvotes: 1

Views: 425

Answers (1)

baduker
baduker

Reputation: 20022

One way of doing this would be with the following code:

import requests

api_key = "PASTE_YOUR_API_KEY_HERE!"
yt_user = "memeulous"
api_url = f"https://www.googleapis.com/youtube/v3/channels?part=contentDetails&forUsername={yt_user}&key={api_key}"

response = requests.get(api_url).json()

playlist_id = response["items"][0]["contentDetails"]["relatedPlaylists"]["uploads"]

channel_url = f"https://www.googleapis.com/youtube/v3/playlistItems?" \
              f"part=snippet%2CcontentDetails&maxResults=50&playlistId={playlist_id}&key={api_key}"


def get_video_ids(vid_data: dict) -> list:
    return [_id["contentDetails"]["videoId"] for _id in vid_data["items"]]


def build_links(vid_ids: list) -> list:
    return [f"https://www.youtube.com/watch?v={_id}" for _id in vid_ids]


def get_all_links() -> list:
    all_links = []
    url = channel_url
    while True:
        res = requests.get(url).json()
        all_links.extend(build_links(get_video_ids(res)))
        try:
            paging_token = res["nextPageToken"]
            url = f"{channel_url}&pageToken={paging_token}"
        except KeyError:
            break
    return all_links


print(get_all_links())

This gets you all the video links (469) for the memeulous user.

['https://www.youtube.com/watch?v=4L8_isnyGfg', 'https://www.youtube.com/watch?v=ogpaiD2e-ss', 'https://www.youtube.com/watch?v=oH-nJe9XMN0', 'https://www.youtube.com/watch?v=kUcbKl4qe5g', ...

You can get the total video count from the videos_data object likes this:

print(f"Total videos: {videos_data['pageInfo']['totalResults']}")

I hope this helps and will get you started. All you need to do, is get the API key for the YouTube Data API.

Upvotes: 1

Related Questions