CODEWITHSUNDEEP
pythonweb-scrapingyoutubeyoutube-data-apiurllib
Prashant Ranjan
Prashant Ranjan

Reputation: 3

How can I get more than 20 youtube search results using urllib?

I am trying to make a python script that fetches youtube channel URLs with a keyword input. I'm using urlib to request the html of the search result page and then filtering out the channel IDs using RE. I can't seem to find a way to get the script to fetch more than 20 results. Can anyone help me out here? Here's the code so far

import urllib.request
import re

search_keyword = input("Search Keyword \n")
html = urllib.request.urlopen("https://www.youtube.com/results?search_query=" + search_keyword + "&sp=EgIQAg%253D%253D")
regex = r"\"channelId\": (\S{24})"
#print(html.read().decode())
ids = re.findall(r"\"channelId\"\:(\S{25})", html.read().decode())
idsLen = len(ids)
for i in range(idsLen):
    ids[i] = ids[i][1:]
    ids[i] = "https://www.youtube.com/channel/" + ids[i]
    print(ids[i])

I have looked up the urllib APIs to find something to get the job done, but I can't find anything related to it. I'm expecting someone can tell me how this can be achieved, with or without urllib.

Upvotes: 0

Views: 367

Answers (1)

Benjamin Loison
Benjamin Loison

Reputation: 5612

For more results you need to implement in some way pagination, as you are working with the YouTube UI, as I did in my open-source API.

Note that you are anyway limited to 500 results, if you use YouTube Data API v3 Search: list endpoint or the YouTube UI.

Upvotes: 1

Related Questions