goldenwallnut
Reputation: 13
How to select all data under a certain class in beautiful soup
When I use findAll, with my class and my tag the code does not find anything
This is my try:
import requests
from bs4 import BeautifulSoup
# get the data
data = requests.get('https://www.twitch.tv/directory/game/Counter-Strike%3A%20Global%20Offensive/clips?range=24hr/')
# load data into bs4
soup = BeautifulSoup(data.text, 'html.parser')
clip = soup.find_all("a", ["tw-full-width tw-interactive tw-link tw-link--hover-underline-none tw-link--inherit"])
print(clip)
What's the problem?
Upvotes: 0
Views: 114
Answers (2)
Sushil
Reputation: 5531
Try using selenium instead of requests to get the website's html code. Here is how you do it:
from bs4 import BeautifulSoup
from selenium import webdriver
import time
driver = webdriver.Chrome()
driver.get('https://www.twitch.tv/directory/game/Counter-Strike%3A%20Global%20Offensive/clips?range=24hr/')
time.sleep(5)
data = driver.page_source
driver.close()
soup = BeautifulSoup(data, 'html.parser')
clip = soup.find_all("a", class_ = "tw-full-width tw-interactive tw-link tw-link--hover-underline-none tw-link--inherit")
print(clip)
Output:
[<a class="tw-full-width tw-interactive tw-link tw-link--hover-underline-none tw-link--inherit" href="/natosaphix/clip/GoldenBlitheBillKeepo"><div class="tw-align-items-start tw-flex"><h3 class="tw-ellipsis tw-font-size-5" title="12 y/o ace">12 y/o ace</h3></div></a>, <a class="tw-full-width tw-interactive tw-link tw-link--hover-underline-none tw-link--inherit" href="/dreamhackcs/clip/TenderBlatantCattleWoofer"><div class="tw-align-items-start tw-flex"><h3 class="tw-ellipsis tw-font-size-5" title="Dupreeh feeling himself">Dupreeh feeling himself</h3></div></a>, <a class="tw-full-width tw-interactive tw-link tw-link--hover-underline-none tw-link--inherit" href="/dreamhackcs3/clip/FurryWiseLemurLitty"><div class="tw-align-items-start tw-flex"><h3 class="tw-ellipsis tw-font-size-5" title="ZyWoo Deagle Ace">ZyWoo Deagle Ace</h3></div></a>, <a class="tw-full-width tw-interactive tw-link tw-link--hover-underline-none tw-link--inherit" href="/artcsgo/clip/FamousDeliciousCasetteSMOrc"><div class="tw-align-items-start tw-flex"><h3 class="tw-ellipsis tw-font-size-5" title="ART + GAU JOGO DA TEAM ONE">ART + GAU JOGO DA TEAM ONE</h3></div></a>, <a class="tw-full-width tw-interactive tw-link tw-link--hover-underline-none tw-link--inherit" href="/dreamhackcs/clip/NaiveSavoryLadiesOneHand"><div class="tw-align-items-start tw-flex"><h3 class="tw-ellipsis tw-font-size-5" title="flusha insane 1v4">flusha insane 1v4</h3></div></a>, <a class="tw-full-width tw-interactive tw-link tw-link--hover-underline-none tw-link--inherit" href="/pago3/clip/AveragePoisedDelicataCmonBruh"><div class="tw-align-items-start tw-flex"><h3 class="tw-ellipsis tw-font-size-5" title="UNLUCKY XDD">UNLUCKY XDD</h3></div></a>, <a class="tw-full-width tw-interactive tw-link tw-link--hover-underline-none tw-link--inherit" href="/deercheerup/clip/SmokyIronicMangetoutDatBoi"><div class="tw-align-items-start tw-flex"><h3 class="tw-ellipsis tw-font-size-5" title="lindinho clutchzero 1x3">lindinho clutchzero 1x3</h3></div></a>, <a class="tw-full-width tw-interactive tw-link tw-link--hover-underline-none tw-link--inherit" href="/esl_csgo/clip/TallViscousSlothBloodTrail"><div class="tw-align-items-start tw-flex"><h3 class="tw-ellipsis tw-font-size-5" title="Stewie2k Selfless Play">Stewie2k Selfless Play</h3></div></a>, <a class="tw-full-width tw-interactive tw-link tw-link--hover-underline-none tw-link--inherit" href="/pago3/clip/TangibleAverageDugongNononoCat"><div class="tw-align-items-start tw-flex"><h3 class="tw-ellipsis tw-font-size-5" title="bedzie magik flesh">bedzie magik flesh</h3></div></a>, <a class="tw-full-width tw-interactive tw-link tw-link--hover-underline-none tw-link--inherit" href="/tarik/clip/CrowdedYummyNigiriPastaThat"><div class="tw-align-items-start tw-flex"><h3 class="tw-ellipsis tw-font-size-5" title="moe vs autimatic on stream">moe vs autimatic on stream</h3></div></a>, <a class="tw-full-width tw-interactive tw-link tw-link--hover-underline-none tw-link--inherit" href="/tarik/clip/HappyTangibleChowderRiPepperonis"><div class="tw-align-items-start tw-flex"><h3 class="tw-ellipsis tw-font-size-5" title="Tarik insane 1v5 his first EU FPL game">Tarik insane 1v5 his first EU FPL game</h3></div></a>, <a class="tw-full-width tw-interactive tw-link tw-link--hover-underline-none tw-link--inherit" href="/tebtv/clip/SpinelessFragileEchidnaImGlitch"><div class="tw-align-items-start tw-flex"><h3 class="tw-ellipsis tw-font-size-5" title="Snax zaskakuje przeciwnika i wygrywa przegraną rundę!">Snax zaskakuje przeciwnika i wygrywa przegraną rundę!</h3></div></a>, <a class="tw-full-width tw-interactive tw-link tw-link--hover-underline-none tw-link--inherit" href="/snax/clip/UninterestedPoisedBadgerNononoCat"><div class="tw-align-items-start tw-flex"><h3 class="tw-ellipsis tw-font-size-5" title="snax o neo i tazie">snax o neo i tazie</h3></div></a>, <a class="tw-full-width tw-interactive tw-link tw-link--hover-underline-none tw-link--inherit" href="/niko/clip/StormySavageDotterelYee"><div class="tw-align-items-start tw-flex"><h3 class="tw-ellipsis tw-font-size-5" title="Welcome to EU Tarik">Welcome to EU Tarik</h3></div></a>, <a class="tw-full-width tw-interactive tw-link tw-link--hover-underline-none tw-link--inherit" href="/esl_csgo/clip/EmpathicAwkwardOtterHeyGuys"><div class="tw-align-items-start tw-flex"><h3 class="tw-ellipsis tw-font-size-5" title="csgo'd">csgo'd</h3></div></a>, <a class="tw-full-width tw-interactive tw-link tw-link--hover-underline-none tw-link--inherit" href="/dextersjourney/clip/AttractiveRacyPhoneYouWHY"><div class="tw-align-items-start tw-flex"><h3 class="tw-ellipsis tw-font-size-5" title="epic sauce">epic sauce</h3></div></a>, <a class="tw-full-width tw-interactive tw-link tw-link--hover-underline-none tw-link--inherit" href="/dreamhackcs2/clip/SuspiciousJoyousSalsifyPogChamp"><div class="tw-align-items-start tw-flex"><h3 class="tw-ellipsis tw-font-size-5" title="NiKo 1v2 2x 1taps">NiKo 1v2 2x 1taps</h3></div></a>, <a class="tw-full-width tw-interactive tw-link tw-link--hover-underline-none tw-link--inherit" href="/innocenttt/clip/PuzzledLightMonkeyMingLee"><div class="tw-align-items-start tw-flex"><h3 class="tw-ellipsis tw-font-size-5" title="morda">morda</h3></div></a>, <a class="tw-full-width tw-interactive tw-link tw-link--hover-underline-none tw-link--inherit" href="/dreamhackcs2/clip/TastySavoryKangarooPRChase"><div class="tw-align-items-start tw-flex"><h3 class="tw-ellipsis tw-font-size-5" title="RAIN 3k vs sAw ">RAIN 3k vs sAw </h3></div></a>, <a class="tw-full-width tw-interactive tw-link tw-link--hover-underline-none tw-link--inherit" href="/csgomc_ru/clip/TransparentFrailCoffeePipeHype"><div class="tw-align-items-start tw-flex"><h3 class="tw-ellipsis tw-font-size-5" title="Synchronous grenade throwing">Synchronous grenade throwing</h3></div></a>]
I think that you are trying to scrape the video link and the title from these anchor tags. In order to do this, add these lines to your code:
for a in clip:
print('-'*100)
print(f"Title = {a.div.h3['title']}")
print(f"Link = https://www.twitch.tv/{a['href']}")
Output:
----------------------------------------------------------------------------------------------------
Title = 12 y/o ace
Link = https://www.twitch.tv//natosaphix/clip/GoldenBlitheBillKeepo
----------------------------------------------------------------------------------------------------
Title = Dupreeh feeling himself
Link = https://www.twitch.tv//dreamhackcs/clip/TenderBlatantCattleWoofer
----------------------------------------------------------------------------------------------------
Title = ZyWoo Deagle Ace
Link = https://www.twitch.tv//dreamhackcs3/clip/FurryWiseLemurLitty
----------------------------------------------------------------------------------------------------
Title = ART + GAU JOGO DA TEAM ONE
Link = https://www.twitch.tv//artcsgo/clip/FamousDeliciousCasetteSMOrc
----------------------------------------------------------------------------------------------------
Title = flusha insane 1v4
Link = https://www.twitch.tv//dreamhackcs/clip/NaiveSavoryLadiesOneHand
----------------------------------------------------------------------------------------------------
Title = UNLUCKY XDD
Link = https://www.twitch.tv//pago3/clip/AveragePoisedDelicataCmonBruh
----------------------------------------------------------------------------------------------------
Title = lindinho clutchzero 1x3
Link = https://www.twitch.tv//deercheerup/clip/SmokyIronicMangetoutDatBoi
----------------------------------------------------------------------------------------------------
Title = Stewie2k Selfless Play
Link = https://www.twitch.tv//esl_csgo/clip/TallViscousSlothBloodTrail
----------------------------------------------------------------------------------------------------
Title = bedzie magik flesh
Link = https://www.twitch.tv//pago3/clip/TangibleAverageDugongNononoCat
----------------------------------------------------------------------------------------------------
Title = moe vs autimatic on stream
Link = https://www.twitch.tv//tarik/clip/CrowdedYummyNigiriPastaThat
----------------------------------------------------------------------------------------------------
Title = Tarik insane 1v5 his first EU FPL game
Link = https://www.twitch.tv//tarik/clip/HappyTangibleChowderRiPepperonis
----------------------------------------------------------------------------------------------------
Title = Snax zaskakuje przeciwnika i wygrywa przegraną rundę!
Link = https://www.twitch.tv//tebtv/clip/SpinelessFragileEchidnaImGlitch
----------------------------------------------------------------------------------------------------
Title = Welcome to EU Tarik
Link = https://www.twitch.tv//niko/clip/StormySavageDotterelYee
----------------------------------------------------------------------------------------------------
Title = csgo'd
Link = https://www.twitch.tv//esl_csgo/clip/EmpathicAwkwardOtterHeyGuys
----------------------------------------------------------------------------------------------------
Title = epic sauce
Link = https://www.twitch.tv//dextersjourney/clip/AttractiveRacyPhoneYouWHY
----------------------------------------------------------------------------------------------------
Title = NiKo 1v2 2x 1taps
Link = https://www.twitch.tv//dreamhackcs2/clip/SuspiciousJoyousSalsifyPogChamp
----------------------------------------------------------------------------------------------------
Title = morda
Link = https://www.twitch.tv//innocenttt/clip/PuzzledLightMonkeyMingLee
----------------------------------------------------------------------------------------------------
Title = RAIN 3k vs sAw
Link = https://www.twitch.tv//dreamhackcs2/clip/TastySavoryKangarooPRChase
----------------------------------------------------------------------------------------------------
Title = Synchronous grenade throwing
Link = https://www.twitch.tv//csgomc_ru/clip/TransparentFrailCoffeePipeHype
----------------------------------------------------------------------------------------------------
Title = sneaky fail ;p
Link = https://www.twitch.tv//pashabiceps/clip/SquareRockyBunnyCclamChamp
Upvotes: 0
Andrej Kesely
Reputation: 195438
The data is loaded dynamically via Ajax, but you can use requests/json modules to load it:
import json
import requests
url = 'https://gql.twitch.tv/gql#origin=twilight'
params = [{"operationName":"ClipsCards__Game","variables":{"gameName":"Counter-Strike: Global Offensive","limit":20,"criteria":{"languages":[],"filter":"LAST_WEEK"}},"extensions":{"persistedQuery":{"version":1,"sha256Hash":"0d8d0eba9fc7ef77de54a7d933998e21ad7a1274c867ec565ac14ffdce77b1f9"}}}]
headers = {'Client-Id': 'kimne78kx3ncx6brgo4mv6wki5h1ko'}
data = requests.post(url, headers=headers, json=params).json()
# uncomment this to print all data:
# print(json.dumps(data, indent=4))
for g in data[0]['data']['game']['clips']['edges']:
print('{:<60} {:<10} {}'.format(g['node']['title'], g['node']['viewCount'], g['node']['url']))
Prints:
12 y/o ace 121409 https://clips.twitch.tv/GoldenBlitheBillKeepo
Dupreeh feeling himself 51384 https://clips.twitch.tv/TenderBlatantCattleWoofer
ZyWoo Deagle Ace 49288 https://clips.twitch.tv/FurryWiseLemurLitty
ART + GAU JOGO DA TEAM ONE 34676 https://clips.twitch.tv/FamousDeliciousCasetteSMOrc
flusha insane 1v4 32522 https://clips.twitch.tv/NaiveSavoryLadiesOneHand
UNLUCKY XDD 31698 https://clips.twitch.tv/AveragePoisedDelicataCmonBruh
lindinho clutchzero 1x3 24808 https://clips.twitch.tv/SmokyIronicMangetoutDatBoi
Stewie2k Selfless Play 24693 https://clips.twitch.tv/TallViscousSlothBloodTrail
bedzie magik flesh 20393 https://clips.twitch.tv/TangibleAverageDugongNononoCat
moe vs autimatic on stream 20311 https://clips.twitch.tv/CrowdedYummyNigiriPastaThat
Tarik insane 1v5 his first EU FPL game 20104 https://clips.twitch.tv/HappyTangibleChowderRiPepperonis
Snax zaskakuje przeciwnika i wygrywa przegraną rundę! 19614 https://clips.twitch.tv/SpinelessFragileEchidnaImGlitch
Welcome to EU Tarik 18400 https://clips.twitch.tv/StormySavageDotterelYee
csgo'd 17318 https://clips.twitch.tv/EmpathicAwkwardOtterHeyGuys
epic sauce 15501 https://clips.twitch.tv/AttractiveRacyPhoneYouWHY
NiKo 1v2 2x 1taps 15088 https://clips.twitch.tv/SuspiciousJoyousSalsifyPogChamp
morda 14394 https://clips.twitch.tv/PuzzledLightMonkeyMingLee
RAIN 3k vs sAw 14387 https://clips.twitch.tv/TastySavoryKangarooPRChase
Synchronous grenade throwing 13881 https://clips.twitch.tv/TransparentFrailCoffeePipeHype
sneaky fail ;p 13668 https://clips.twitch.tv/SquareRockyBunnyCclamChamp
Upvotes: 1
Related Questions
- How to extract value of all classes in beautiful Soup
- extract an element from the class with Beautiful Soup
- Python How i can extract data with same class name in BeautifulSoup
- Beautiful soup - select all elements that have a class, but also separate them or ignore elements if they have also other clases
- Beautiful Soup find element with multiple classes
- select specific rows with class names
- How can I select an element with multiple classes in BeautifulSoup4 - Python?
- Filtering into specific class in BeautifulSoup
- Python and beautifulsoup - How to select only a specific slection of a class?
- How to select a class of div inside of a div with beautiful soup?