CODEWITHSUNDEEP
pythonseleniumweb-scraping
José Guedes
José Guedes

Reputation: 400

How can I extract this information with Python without simulating clicks?

I'm scraping Ali Express products such as this one using Python. It has multiple variations, each with its own price. When one is clicked on, the price is updated to reflect this choice.

In a similar fashion, there are multiple buttons to choose where you want the item to be shipped from, which updates the shipping cost accordingly.

I want to scrape each variation's price as sent from each country. How can I do that without simulating clicks to change the prices so that I can scrape them? Where is the underlying logic that governs these price changes laid out? I couldn't find it when inspecting elements. Is it easily decipherable?

Or do I just need to give up and simulate clicks? If so, would that be done with Selenium? The reason I would prefer to extract it without clicking is that, for products such as the one I linked to, for example, there are 49 variations and 5 places from which the product is shipped so it would be a lot of clicking and a rather inelegant approach.

Thanks a lot!

Upvotes: 0

Views: 270

Answers (2)

Freez
Freez

Reputation: 63

I know that ecommerce companies applies this kind of logic in their backend apis. And to protect the apis from normal users. They use consul which is used to resolve the ips recieved from front end.

Now coming to your question. There can be two cases.

  1. Frontend recieves the data from backend and applies their own logic. So i can tell you that the front end has already recieved all the data related to variants and its price. So they are storing it at their end in some data structure. And they update the values on the view only when you click the item.( You can find if this is the case if after clicking there is no delay and result is shown instantly). Though you can check the response fetched from the backend, it is bound to have all data which frontend is recieving and storing. You can check in chrome-debug tools->network->gql to filter

  2. Second case in which it is fetching data each time from backend when you click. In that case it is changing some parameters on the link. If you can find out some kind of logic behind how parameters are being changed for similar variants maybe you can fetch the information then.(There will be delay in showing results after clicking)

I think its a good idea to use selenium or cypress. I know it will take time. But its the best option you got.

Upvotes: 1

flaxon
flaxon

Reputation: 942

take a look in the browser, all the data is in the dom

type window.runParams.data.skuModule.skuPriceList in you console you will see

Upvotes: 1

Related Questions