CODEWITHSUNDEEP
python-3.xweb-scraping
Ashish
Ashish

Reputation: 9

Nested tag while scrapping the data from URL in python

I have a url from which I need to do scrapping but the issue is that data is in the nested tag.

import pandas as pd
import requests
from bs4 import BeautifulSoup

page=requests.get('https://www.christies.com/en/auction/modern-british-and-irish-art-evening-sale-29987/').text

soup=BeautifulSoup(webpage,'html.parser')

I need to fetch the name of images with their date of birth and death. PFA the image in which I want to access li tag inside main

enter image description here

                                          A

Upvotes: 0

Views: 25

Answers (1)

Andrej Kesely
Andrej Kesely

Reputation: 195573

The data you see on the page is stored inside <script> tag, so beautifulsoup doesn't see it. To parse the data you can use next example:

import re
import json
import requests


url = "https://www.christies.com/en/auction/modern-british-and-irish-art-evening-sale-29987/"

html_doc = requests.get(url).text
data = re.search(r"window\.chrComponents\.lots = (.*);", html_doc).group(1)
data = json.loads(data)

# uncomment this to print all data:
# print(json.dumps(data, indent=4))

for lot in data["data"]["lots"]:
    print(lot["title_primary_txt"])

Prints:

PAULINE BOTY (1938-1966)
DAVID HOCKNEY, O.M., C.H., R.A. (B. 1937)
DAVID HOCKNEY, O.M., C.H., R.A. (B. 1937)
DAVID HOCKNEY, O.M., C.H., R.A. (B. 1937)
DAVID HOCKNEY, O.M., C.H., R.A. (B. 1937)
DAME BARBARA HEPWORTH (1903-1975)
WILLIAM SCOTT, R.A. (1913-1989)
BEN NICHOLSON, O.M. (1894-1982)
DAME BARBARA HEPWORTH (1903-1975)
LAURENCE STEPHEN LOWRY, R.A. (1887-1976)
BARRY FLANAGAN, R.A. (1941-2009)
EUAN UGLOW (1932-2000)
HENRY MOORE, O.M., C.H. (1889-1986)
LAURENCE STEPHEN LOWRY, R.A. (1887-1976)
LAURENCE STEPHEN LOWRY, R.A. (1887-1976)
DAME BARBARA HEPWORTH (1903-1975)
BEN NICHOLSON, O.M. (1894-1982)
HENRI GAUDIER-BRZESKA (1891-1915)
WYNDHAM LEWIS (1882-1957)
DAVID BOMBERG (1890-1957)
WYNDHAM LEWIS (1882-1957)
DAVID BOMBERG (1890-1957)
HENRI GAUDIER-BRZESKA (1891-1915)
EDWARD WADSWORTH, A.R.A. (1889-1949)
LAWRENCE ATKINSON (1873-1931)
WILLIAM ROBERTS, R.A. (1895-1980)
CHRISTOPHER RICHARD WYNNE NEVINSON, A.R.A. (1889-1946)
JOHN DUNCAN FERGUSSON (1874-1961)
EDWARD WADSWORTH, A.R.A. (1889-1949)
DAME BARBARA HEPWORTH (1903-1975)
GERALD LAING (1936-2011)
BARRY FLANAGAN, R.A. (1941-2009)
SIR FRANK BOWLING, R.A. (B. 1934)
LYNN CHADWICK, R.A. (1914-2003)
PATRICK HERON (1920-1999)
WILLIAM SCOTT, R.A. (1913-1989)
PETER LANYON (1918-1964)
CHRISTOPHER WOOD (1901-1930)
HENRY MOORE, O.M., C.H. (1898-1986)
HENRY MOORE (1898-1986)
SIR WILLIAM NICHOLSON (1872-1949)
FRANCIS CAMPBELL BOILEAU CADELL, R.S.A. (1883-1937)
HENRY MOORE, O.M., C.H. (1898-1986)
LAURENCE STEPHEN LOWRY, R.A. (1887-1976)
LAURENCE STEPHEN LOWRY, R.A. (1887-1976)

Upvotes: 1

Related Questions