How do I redirect to different webpages while web scraping within a python program?

Question

I want to build a web scraping program that takes user input, and feeds that input into the search bar of a website. I then want to redirect to the webpage that would result from the search of that term, and scrape that page's HTML.

What is the best Python module to do this? Some suggestions and resources to learn would be greatly appreciated, thank you.

Answer 1

You need to learn these modules.

• requests - https://docs.python-requests.org/en/master/
• beautifulsoup - https://www.crummy.com/software/BeautifulSoup/bs4/doc/
• Selenium - https://selenium-python.readthedocs.io/