Why can't I scrape all data from ecommerce websites?

Question

Actually I'm working on a project where I have to scrape data from e-commerce websites. But I can't access my desired data from these sites. For example, when I want to scrape all list from https://evaly.com.bd/search-results?query=remax%20610d site, I only get

as output. Besides, when I print HTML code of the site using print(soup.prettify()) The full code is not in the output. Here is my code for all list items :

from bs4 import BeautifulSoup
import requests
link = "https://evaly.com.bd/search-results?query=remax%20610"

source = requests.get(
       link).text

soup = BeautifulSoup(source, 'lxml')
#print(soup.prettify())
li = soup.find_all("li")
print(li)

And here is the output when I run print(soup.prettify()) :



 
  
  
  
   E-valy Limited | Online Shopping Mall
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
 
 
  
   
    
     
      
       
        
         
          
          Merchant zone
         
        
        
         
          
          News Feed
         
        
        
         
          
          Download App
         
        
       
       
        
         
          
          
          Help
         
        
        
         
          
          
          Follow us


How to solve these problems?
EDIT : using Selenium and Chrome Driver will be more time consuming for my project

Vin · Accepted Answer

Try the below approach using requests and json. I have created the script with the API URL which is fetched by inspecting the network calls in chrome which are triggering on page load and then creating a dynamic form data to traverse on each and every page to get the data.

What exactly script is doing:

First script will create a form data to query the the API call where page_no, query string and max values per facet(numbers of results to show) are dynamic where parameter page_no will increment by 1 upon completion of each traversal.
Requests will get the data from the created form data and URL using POST method which will then pass to JSON to parse it and load in json format.
Then from the parsed data script will traverse on the json object where data is actually present.
Finally looping on all the batch of each and every page data one by one and printing.

Right now script is displaying few information you can access more information form the json object like i have done below.

import json
import requests
from urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
from bs4 import BeautifulSoup as bs

def scrap_evaly_data():

QUERY = 'remax%20610' #query string can be changed to fetch another product data
MAX_VALUES_PER_FACET = 10 #no. of result show per page
page_no = 0 # default page no.
URL = 'https://eza2j926q5-3.algolianet.com/1/indexes/*/queries?x-algolia-agent=Algolia%20for%20JavaScript%20(3.35.1)%3B%20Browser%20(lite)%3B%20react%20(16.13.1)%3B%20react-instantsearch%20(5.7.0)%3B%20JS%20Helper%20(2.28.1)&x-algolia-application-id=EZA2J926Q5&x-algolia-api-key=ca9abeea06c16b7d531694d6783a8f04' # API URL for querying

while True:
    print('Hold on creating new form data...')
    form_data = {
    "requests":[{"indexName":"products","params":"query=" + QUERY + "&maxValuesPerFacet=" + str(MAX_VALUES_PER_FACET) + "&page=" + str(page_no) + "&highlightPreTag=%3Cais-highlight-0000000000%3E&highlightPostTag=%3C%2Fais-highlight-0000000000%3E&facets=%5B%22price%22%2C%22category_name%22%2C%22brand_name%22%2C%22shop_name%22%2C%22color%22%5D&tagFilters="}]
    } # form_data which is dynamic and creates new set of results and send back
    response = requests.post(URL,json = form_data,verify = False) #requests for data using POST and JSON form data
    print('Created new form data going to fetch data...')

    result = json.loads(response.text) #load json data result
    if len(result) == 0: #condition to check whether result has length or not if not then break and come out from the while loop.
        break
    else:
        for item in result['results'][0]['hits']: #loop on the product information JSON object
            print('-' * 100)
            print('Brand Name: ', item['brand_name'])
            print('Category Name: ' , item['category_name'])
            print('Discount Price: ' , item['discounted_price'])
            print('Max Price: ' , item['max_price'])
            print('Min Price: ' , item['min_price'])
            print('Product Name: ' , item['name'])
            print('Product Image: ' , item['product_image'])
            print('Shop Item ID: ' , item['shop_item_id'])
            print('Shop Name: ' , item['shop_name'])
            print('Slug Info: ' , item['slug'])
            print('-' * 100)

        page_no +=1 #Increment the page number by 1 after each traversal


   scrap_evaly_data()

Why can't I scrape all data from ecommerce websites?

Answers (1)

Related Questions

Why can&#39;t I scrape all data from ecommerce websites?

Answers (1)

Related Questions

Why can't I scrape all data from ecommerce websites?