CODEWITHSUNDEEP
pythonhtmlpython-3.xweb-scrapingnavigation
Robert O'Brien
Robert O'Brien

Reputation: 183

Suitable Python modules for navigating a website

I am looking for a python module that will let me navigate searchbars, links etc of a website. For context I am looking to do a little webscraping of this website [https://www.realclearpolitics.com/] I simply want to take information on each state (polling data etc) in relation to the 2020 election and organize it all in a collection of a database. Obviously there are a lot of states to go through and each is on a seperate webpage. So im looking for a method in python in which i could quickly navigate the site and take the data of each page etc aswell as update and add to existing data. So finding a method of quickly navigating links and search bars with my inputted data would be very helpful. Any suggestions would be greatly appreciated.

# a simple list that contains the names of each state
states = ["Alabama", "Alaska" ,"Arizona", "....."] 
for state in states:
    #code to look up the state in the searchbar of website
    #figures being taken from website etc
    break

Here is the rough idea i have

Upvotes: 2

Views: 1252

Answers (1)

TJB
TJB

Reputation: 3846

There are many options to accomplish this with Python. As @LD mentioned, you can use Selenium. Selenium is a good option if you need to interact with a websites UI via a headless browser. E.g clicking a button, entering text into a search bar, etc. If your needs aren't that complex, for instance if you just need to quickly scrape all the raw content from a web page and process it, than you should use the requests module from Python's standard library.

For processing raw content from a crawl, I would recommend beautiful soup.

Hope that helps!

Upvotes: 3

Related Questions