CODEWITHSUNDEEP
pythonpython-requestsrequest
HASAN HÜSEYİN YÜCEL
HASAN HÜSEYİN YÜCEL

Reputation: 373

How to use Request with query parameter in Python?

I would like to get data from an e-commerce website. However, I can get products but no more than 24. The site is using page index query parameter. I am sending the query parameter but still, the script gets 24 products.

The code, result and the website --> Screenshots

Code:

import requests
import sqlite3
from bs4 import BeautifulSoup

db = sqlite3.connect('veritabani.sqlite')
cursor = db.cursor()
cursor.execute("CREATE TABLE products (id, product, price)")
url = 'https://www.trendyol.com/cep-telefonu-x-c103498'
html_text = requests.get(url,params={'q': 'pi:5'}).text
soup = BeautifulSoup(html_text, 'lxml')
print(soup.contents)
products = soup.find_all("div", {"class": "p-card-wrppr"})
for product in products:
    product_id = product['data-id']
    product_name = product.find("div", {"class": "prdct-desc-cntnr-ttl-w two-line-text"}).find("span",{"class": "prdct-desc-cntnr-name"})["title"]
    price = product.find_all("div", {"class": "prc-box-sllng"})[0].text
    cursor.execute("INSERT INTO products VALUES (?,?,?)", (product_id,product_name,price))
    print(product_id,product_name,price)
db.commit()
db.close()

Upvotes: 1

Views: 6161

Answers (1)

Muslimbek Abduganiev
Muslimbek Abduganiev

Reputation: 941

The website you have mentioned uses scroll pagination. But you still can get the data you want.

First of all, you are passing params wrong. Try to change this line

html_text = requests.get(url,params={'q': 'pi:5'}).text

with this:

html_text = requests.get(url,params={'pi':'5'}).text

And you will get 24 products on page number 5. So basically, you can go like this:

for i in range(10):
    html_text = requests.get(url,params={'pi': str(i)}).text
    soup = BeautifulSoup(html_text, 'lxml')
    print(soup.contents)
    products = soup.find_all("div", {"class": "p-card-wrppr"})
    for product in products:
        product_id = product['data-id']
        product_name = product.find("div", {"class": "prdct-desc-cntnr-ttl-w two-line-text"}).find("span",{"class": "prdct-desc-cntnr-name"})["title"]
        price = product.find_all("div", {"class": "prc-box-sllng"})[0].text
        cursor.execute("INSERT INTO products VALUES (?,?,?)", (product_id,product_name,price))
        print(product_id,product_name,price)
    db.commit()
    db.close()

This should get you products on 10 pages.

Upvotes: 5

Related Questions