CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsouppython-requestsscrapinghub
user15215612
user15215612

Reputation:

How to specify css selector in beautiful soup and python?

I am trying to scrape the titles of the cards from this link : https://www.axisbank.com/retail/cards/credit-card

Using the below code

from urllib.request import urlopen
from bs4 import BeautifulSoup
import json, requests, re

axis_url = ["https://www.axisbank.com/retail/cards/credit-card"]

html = requests.get(axis_url[0])
soup = BeautifulSoup(html.content, 'lxml')

print(soup.select("#ulCreditCard h3"))

Output is as follows :

[]

My main concern is how shall I specify a css selector tag that I got using the selector gadget when it is in this form #ulCreditCard h3 in the soup.select() function.

Upvotes: 0

Views: 739

Answers (1)

Martin Evans
Martin Evans

Reputation: 46759

As mentioned, the information you are looking for is obtained via Javascript. It uses a slightly different URL to get the JSON data containing all of the card details. If you use this instead, you can easily list all of the card names without needing to use BeautifulSoup. For example:

import requests
import json

axis_url = "https://www.axisbank.com/AjaxService/GetCreditCardsProducts"
data = {"strcategory" : "[]", "strrewardtypes" :"[]"}
r = requests.post(axis_url, data=data)

for entry in json.loads(r.json()[0]):
    print(entry['Name'])

Would give you the following cards:

Axis Bank ACE Credit Card
Axis Bank AURA Credit Card
Privilege Easy Credit Card
Axis Bank Reserve Credit Card
Axis Bank Freecharge Plus Credit Card
IndianOil Axis Bank Credit Card
Axis Bank Magnus Credit Card
Flipkart Axis Bank Credit Card
Axis Bank Freecharge Credit Card
Axis Bank MY Zone Credit Card
Axis Bank Neo Credit Card
Axis Bank Vistara Credit Card
Axis Bank Vistara Signature Credit Card
Axis Bank Vistara Infinite Credit Card
Axis Bank Privilege Credit Card
Miles and More Axis Bank Credit Card
Axis Bank Select Credit Card
Axis Bank Pride Platinum Credit Card
Axis Bank Pride Signature Credit Card
Axis Bank MY Zone Easy Credit Card
Axis Bank Insta Easy Credit Card
Axis Bank Signature Credit Card with Lifestyle Benefits
Platinum Credit Card
Titanium Smart Traveler Credit Card
Axis Bank My Wings Credit Card

Upvotes: 1

Related Questions