CODEWITHSUNDEEP
pythonjsonweb-scrapingquora
linacarrillo
linacarrillo

Reputation: 79

Scraping answers from Quora question only returns 10 answers

I am trying to scrape all the answers from a Quora question like this

from urllib.request import urlopen, Request
from bs4 import BeautifulSoup
import json

url = 'https://www.quora.com/What-is-it-like-to-live-in-London'
r = requests.get(url)
soup = BeautifulSoup(r.content, 'html.parser')
answers = soup.find("script", {"type": "application/ld+json"})
data = json.loads(answers.string)
len(data["mainEntity"]["suggestedAnswer"])

However when I check the length of the suggestedAnswer, I am only getting 10 items. It appears that these 10 items are filtered by using the soup.find("script", {"type": "application/ld+json"}) . Is there a way to get all the answers?

Upvotes: 1

Views: 374

Answers (0)

Related Questions