CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsouppython-requestsquora
nikolai_ye
nikolai_ye

Reputation: 51

How to load more answers (equivalent to scrolling down) when parsing answers on Quora using Python requests?

I would like to collect the text, date and some other attributes of every answer under a question on Quora. My plan is to make requests on Python and parse the returned HTML using BeautifulSoup. Nevertheless, when I submitted a GET request on Python, the returned object only contains 10 answers, while the actual number of answers is way over 10. I know that on a browser, you scroll down to load more answers. What seems to happen under the hood is that you submit a POST request that would return more answers in the form of json object. See the screenshot below.

POST request made

However, when I submitted a POST request using the same URL as in the screenshot, I got Bad Request (status code 400). It seems like it's quite difficult to implement a viable solution that could retrieve all the answers (rather than only the top 10) iteratively.

Any help or suggestions will be greatly appreciated.

Upvotes: 2

Views: 181

Answers (0)

Related Questions