CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsouppython-requestspython-3.8
Manas bafna
Manas bafna

Reputation: 1

How to get more than 8 items in beautiful soup

This is my current code and it would only output 8 titles. I was wondering how can I make it work to output more than 8 and all the titles in the past 24 hours? Thank you in advance.

import requests
from bs4 import BeautifulSoup

url = "https://www.reddit.com/r/wallstreetbets/new/"
headers = {'User-Agent' : 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.2 Safari/605.1.15'}


r=requests.get(url,headers=headers)
soup = BeautifulSoup(r.content,"html.parser")

articles = soup.find_all('h3', class_ = '_eYtD2XCVieq6emjKBH3m')


cleaned_tittles = []
for item in articles:
    name = item.text
    cleaned_tittles.append(str(name))

for i in cleaned_tittles:
    print(i)

Upvotes: 0

Views: 75

Answers (1)

Andrej Kesely
Andrej Kesely

Reputation: 195553

You can append .json to Reddit URL to obtain data in Json form (it's easier to parse):

import json
import requests

url = "https://old.reddit.com/r/wallstreetbets/hot/.json"

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0"
}
data = requests.get(url, headers=headers).json()

# uncomment to print all data
# print(json.dumps(data, indent=4))

for c in data["data"]["children"]:
    print(c["data"]["title"])

Prints:

Daily Discussion Thread for June 08, 2021
Daily Popular Ticker Thread for June 08, 2021 - BB | AMC | CLOV
How it felt buying my first share of BB stock - Based on a true story
We Need Separate Daily Megathreads For GME / AMC / BB / CLOV
Crayon-eating retail investors deciding today is the day GameStop will go back up above $300...
Holy shit guys I am shaking. THANK YOU SO MUCH. LETS GO $CLOV TO THE MOON 🚀
$WISH 100k YOLO on margin with my student loans 🦍🦍🦍
$WISH 250K YOLO. 51% Short Float according to Bloomberg Terminal. Short Squeeze season lets gooo 🚀🚀🚀
Speed dating with an ape
$CLOV pre market volume already 25 million and up +35%.
$CLNE to the mooo-n! 🐄💨🚀🌙
Cow Farts Go BRRRRRR, how CLNE will bring ALL THE TENDIES 🚀🚀🚀
$CLOV YOLO BAG HOLD FINALLY PAYS OFF
$CLNE Primed for Takeoff🚀🌕 Moon Imminent🐄💎
$BB 🚀🚀 I LIKE THE STOCK 🚀🚀 ANYTHING BELOW $50 IS BUYING THE DIP 🐒
Spotted! | Big dick Choombas this morning | GME
My current $3.7 mil portfolio, powered entirely by meme power, ie AMC, GME, PLTR and TSLA
CLOV Short Seller Numbers And Gamma Squeeze Info (6/7 - 6/8)
$BB The Boomer Theory
$WISH it to the MOON!
GME to the moon🚀🚀🚀
Wish is just getting started!
Remember: Nothing has changed about CLNE
Because I’m a BBelieber 🍇🚀🌝
I WISH I never would have met you all. Why do I do this to myself. 🦍 5000 shares @ 8.71
$CLOV - Your Lucky Day
WISH DD Part 2 (more pieces of puzzle)

Upvotes: 3

Related Questions