CODEWITHSUNDEEP
pythonhtmlweb-scrapingforum
Zubin JAIN
Zubin JAIN

Reputation: 49

Scrape Forum posts from a specific user using Beautiful soup

I am a complete python newbie and for my first project, I am trying to scrape posts from a specific user at a forum thread, using a python scrip and then pasting it onto a raw file.

I am using the python coding language and the beautiful soup coding library but have a hit a roadblock in filtering for a specific user on a thread,

How should I filter for my script to save only posts by a specific user in python? this is the forum from which I will be scrapping

https://forums.spacebattles.com/threads/the-wizard-of-woah-and-irrational-methods-of-irrationality.337233/

Upvotes: 1

Views: 1060

Answers (1)

Sathish Kumar VG
Sathish Kumar VG

Reputation: 2172

use Xpath like this to get the user name posted in the forum

from bs4 import BeautifulSoup 
import requests 
page = requests.get('https://forums.spacebattles.com/threads/the-wizard-of-woah-and-irrational-methods-of-irrationality.337233/page-2')
page_source = page.content
soup = BeautifulSoup(page_source)
post = soup.body.find('div', 'messageContent')
user_name = post.find('div', 'attribution type')
if 'Harry Leferts' in user_name:
    '''save the post '''

I have done to get single post from the forum, you can get all post by using find_all

Upvotes: 1

Related Questions