CODEWITHSUNDEEP
python-3.xweb-scraping
Jay Pratap Pandey
Jay Pratap Pandey

Reputation: 352

Scraping the stackoverflow user data

import requests
from bs4 import BeautifulSoup
import csv

response = requests.get('https://stackoverflow.com/users?page=3&tab=reputation&filter=week').text
soup = BeautifulSoup(response, 'lxml')
for items in soup.select('.user-details'):
    name = items.select("a")[0].text
    location = items.select(".user-location")[0].text
    reputation = items.select(".reputation-score")[0].text
    print(name,location,reputation)

    with open('stackdata.csv','a',newline='') as csv_file:
        writer = csv.writer(csv_file)
        writer.writerow([name,location,reputation])

When we change the url of this code the output remains same.

Upvotes: 1

Views: 478

Answers (1)

MaNKuR
MaNKuR

Reputation: 2704

I came across a similar problem. The solution that works for me is using selenium. Though I used headless browser i.e phantomjs I assume it should work for other browsers too.

driver = webdriver.PhantomJS('/home/practice/selenium/webdriver/phantomjs/bin/phantomjs')
users = []
page_num = 1
driver.get('https://stackoverflow.com/users?page={page_num}&tab=reputation&filter=week'.format(page_num=page_num))
content = driver.find_element_by_id('content')

for details in content.find_elements_by_class_name('user-details'):
    users.append(details.text)

print(users)

Change the page_num to get the desired result.

Hope this will help!

Upvotes: 1

Related Questions