CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
Ghazaleh Alizadegan
Ghazaleh Alizadegan

Reputation: 15

beautifulsoup not returning complete html of the page in python

I have taken help to extract data from the package BeautifulSoup, but unfortunately, it does not give me the output of the complete HTM file.

   from bs4 import BeautifulSoup
   import requests

   url = 'https://www.digikala.com/search/category-mobile-phone/?page=1'
   response = requests.get(url)
   print(response.text)
   soup = BeautifulSoup(response.content,"lxml")
   m= soup.find_all("h2", {'class':'ellipsis-2 text-body2-strong 
    color-700'})
   print(m)

Upvotes: 1

Views: 887

Answers (1)

Barry the Platipus
Barry the Platipus

Reputation: 10460

This page is loading data from an API. You can see this by inspecting Network tab in Chrome Dev Tools, reloading the page.

The following code will return a json with all items from page 1:

from bs4 import BeautifulSoup
import requests

url = 'https://api.digikala.com/v1/categories/mobile-phone/search/?page=1'
response = requests.get(url).json()
print(response)

Upvotes: 1

Related Questions