CODEWITHSUNDEEP
pythonweb-scraping
DavidGetter
DavidGetter

Reputation: 359

Scrape dynamic Website where content is loaded as you scroll

I dont know the right term but I think "dynamic website" might do the trick.
By that I mean that as I scroll, resources are being loaded. I searched for solutions and I came across webdrivers, personally I dont like a whole browser being loaded, just for the purpose of scrolling down.
A different approach would be looking at the network tab and scraping the url, that I find there.

https://www.immowelt.de/liste/hamburg/wohnungen/mieten?prima=700&sort=relevanz&cp=1

However the content really gets loaded as I scroll.

the link that gets shown when I open the network tab and scroll down: https://www.immowelt.de/liste/getlistitems

Im new to web development so I dont get how these links that I see in the network tab can be named exactly identical but hold different values.

Upvotes: 0

Views: 1673

Answers (2)

user12541086
user12541086

Reputation:

If you look at the network packet more closely. You will see that it is a POST request and sends form data to that link. Looking at the form data more closely:

query: geoid=108020&etype=1&esr=2&prima=700&sort=relevanz&cp=1
offset: 12
pageSize: 4

You see that it sends an offset. That is what paging the next results.

Upvotes: 1

shongyang low
shongyang low

Reputation: 77

Just looked-up an interesting question (with answers) for a similar problem:
How can I scroll a web page using selenium webdriver in python?

The answers point to the code:

driver.execute_script("window.scrollTo(0, Y)")

There are also infinite scrolling options in the linked Q&A


You might want to establish a parameter of how many entries you want to scrape by setting-up the "scroll" script and printing the output once the scrolling is done.


Hope this helps, cheers!

Upvotes: 0

Related Questions