CODEWITHSUNDEEP
pythonweb-scrapingscrapy
Zaky
Zaky

Reputation: 409

Get all products from specific amazon store using scrapy

Is there is a way to get all items of specific seller on amazon? When I try to submit requests using different forms of urls to the store (the basic is ("https://www.amazon.com/shops/"), I'm getting 301 with no additional info.

even before the spider itself, from the scrapy shell (some random shop from amazon)

scrapy shell "https://www.amazon.com/shops/A3TJVJMBQL014A"

There is 301 response code:

 request    <GET https://www.amazon.com/shops/A3TJVJMBQL014A>
 response   <301 https://www.amazon.com/shops/A3TJVJMBQL014A>

In the browser it will be redirected to https://www.amazon.com/s?marketplaceID=ATVPDKIKX0DER&me=A3TJVJMBQL014A&merchant=A3TJVJMBQL014A&redirect=true

using resulting URL also leads to 301 response.

Upvotes: 1

Views: 974

Answers (2)

Zaky
Zaky

Reputation: 409

I was using scrapy shell, while as answered by @PadraicCunningham it doesn't support location header. Running code from spider resolved the issue.

Upvotes: 1

Acepcs
Acepcs

Reputation: 274

Since you want a list of all goods sold by one specific seller, you can analyze the page of that seller specifically.

Here, I am going to take Kindle E-readers Seller as an example.

  1. Open the console in your browser and select the max page count element on the seller's page, you can see the number of max pages of this seller is inside a tag <span class="pagnLink"> </span>, so you can find this tag and extract the max page count from it.

  2. you can see there is a slight change in the url when you move to next page of this seller's goods list (from page=1 to page=2), so you can easily construct a new url when you wanna move to next page.

  3. set a loop whose limitation is the number of max page count you got in the first step.

  4. analyze the specific data you wanna get on that page, analyze what html tags they are inside and use some text analyze libraries to help you extract the data. (re, BeautifulSoup .etc)

Briefly, you have to analyze the page before writing codes. When you start coding, you should first making requests, then get response from your request, then extracting useful data from the response(according to the rules you analyzed before writing codes).

Upvotes: 0

Related Questions