CODEWITHSUNDEEP
pythonbeautifulsouppython-requestsanaconda
Kyle Reynolds
Kyle Reynolds

Reputation: 21

Python web scrape when url does not change

I am working on a web scraping projecting using python. I am supposed to scrape this website(https://www.fec.gov/data/receipts/?committee_id=C00703975&two_year_transaction_period=2020&data_type=processed). The issue I am having is that when you go through the data, the url never changes. I am unsure of how to proceed. From my experience when scraping a website the url changes when switching pages but this one doesn't change pages it just goes through the data on the same page. Thank you!

Upvotes: 2

Views: 166

Answers (1)

Lalith Kumar Nimmala
Lalith Kumar Nimmala

Reputation: 11

This website deals with a lot of data and it is dynamic. Within the same layout, the data displayed will keep changing. Another way to approach your problem (may not be the best) is to export the data and save it to a staging area and work on that data. You can regularly refresh your staging area from https://www.fec.gov/data/browse-data/?tab=bulk-data

This significantly reduces the need for a scraping program and lets you focus on actual data analysis.

Upvotes: 1

Related Questions