can't find correct 'select' HTML tag value, and trying to wait for a select option to load, playwright Python

Question

I have an issue where I use a url that ends such as T-shirts page

I am trying to scrape the product links off the pages. I have been trying for some time now, nothing is working yet. This is my current attempt after some Googling and reading the Playwright docs:

Website html:

def playwright_get_soup(url, wait_after_page_load=None):
    with sync_playwright() as this_playwright:
        browser = this_playwright.chromium.launch()
        page = browser.new_page()
        start = time.perf_counter()
        page.goto(url)
        try:
            page.wait_for_load_state("load")
            if wait_after_page_load:
                time.sleep(wait_after_page_load
            products_on_page = page.querySelector('#prodPerPageSelTop').innerText()
            page.waitForFunction("document.querySelector('#prodPerPageSelTop').innerText() !== '" + products_on_page + "'")

            # attempt 1
            page.click('#prodperpageselect').select_option('96')

            # attempt 2
            # products_on_page = page.querySelector('#prodperpageselect ').innerText()
            # page.waitForFunction("document.querySelector('#prodPerPageSelTop').innerText() !== '" + products_on_page + "'")

            # attempt 3
            # new_selector = 'id=prodPerPageSelTop'
            # page.waitForSelector(new_selector)
            # handle = page.querySelector(new_selector)
            # handle.selectOption({"value": "96"})
   
            # attempt 4
            # page.select_option('select#prodperpageselect', value='96')

            time.sleep(15) 

            # try to wait 
            page.wait_for_selector('select#prodperpageselect option[value="96"]')
        except:
            pass

        soup = BeautifulSoup(page.content(), "html.parser")
        browser.close()
        return soup

soup = playwright_get_soup("https://www.alphabroder.com/category/t-shirts")

def get_links(page_soup):
    these_links = []
    all_product_thumbnails = page_soup.find_all("div", class_="thumbnail")
    for thumbnail in all_product_thumbnails:
        a_tag = thumbnail.find("a")
        link = a_tag["href"]
        these_links.append(link)
        return these_links

page_links = get_links(soup)

assert(len(page_links) == 96

As the page loads, it starts on 24 items, continues loading for 4-5 seconds, then flickers and the select option then changes from say 24 items to 96 items.

I was expecting wait_for_selector to work. I also wait 15 seconds after the page loads, yet returns 24 items, not 96.

So far, I've also tried clicking the select option tag 4 different ways myself, and nothing has worked yet.

I did review similar questions that use Playwright. I'm trying to be more respectful on this site than I was when I was younger.

Any help appreciated, thank you

HedgeHog · Accepted Answer

Even if your focus is to get the information with playwright - Therefore, I would just like point out additionally that scraping the information can also be implemented quite simply using requests and the endpoint via which the information is loaded:

import requests

page_num = 1

data = []

while True:
    json_data  = requests.get(f'https://www.alphabroder.com/cgi-bin/livewamus/wam_tmpl/catalog_browse.p?action=getProduct&content=json&page=catalog_browse&startpath=1017&getNumProd=true&sort=pl&sortdir=asc&pageNum={page_num}&prodPerPage=96&site=ABLive&layout=Responsive&nocache=62059').json()
    data.extend(json_data.get('browseProd'))
    
    if page_num < json_data.get('paging')[0].get('pgTotal'):
        page_num = page_num+1
    else:
        break

data

[{'productID': 'G500', 'colorCode': '93', 'description': 'Gildan Adult Heavy Cotton\x99 T-Shirt', 'division': 'AB', 'prodCat': '130', 'mill': '07', 'prodImg': '', 'prodURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html', 'regPriceDisp': '$0.00', 'onSale': True, 'salePrice': 2.32512, 'salePriceDisp': '$2.33', 'colorCount': 75, 'colorCountLabel': 'Colors', 'sizeCount': 8, 'primePlus': 'primeplus', 'primePlusLogo': True, 'sizeLabel': ' S - 5XL', 'sizeLabelDesc': 'Sizes:', 'msrpPriceDesp': 'Starting At: Pricing upon request', 'salesRank': 1, 'primePlusHTML': "", 'showPriceHTML': " $2.33", 'gaMktgMill': 'Gildan', 'gaMktgCategory': 'T-Shirts', 'gaCurrency': 'USD', 'gaList': 'Results from Search List', 'sustainLogo': True, 'sustainLogoHTML': "", 'colorURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html?color=93', 'colorSwatch': [{'productID': 'G500', 'colorCode': '00', 'colorXref': 'White ', 'description': 'WHITE', 'hexColor': 'FFFFFF', 'image': '', 'imageURL': 'https://www.alphabroder.com//prodimg/small/g500_00_g.jpg', 'sortOrder': 1, 'productURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html?color=00', 'mainProdURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html'}, {'productID': 'G500', 'colorCode': '01', 'colorXref': 'Pink', 'description': 'AZALEA', 'hexColor': 'FF76A0', 'image': '', 'imageURL': 'https://www.alphabroder.com//prodimg/small/g500_01_g.jpg', 'sortOrder': 2, 'productURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html?color=01', 'mainProdURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html'}, {'productID': 'G500', 'colorCode': '05', 'colorXref': 'Yellow', 'description': 'YELLOW HAZE', 'hexColor': 'EEE8A0', 'image': '', 'imageURL': 'https://www.alphabroder.com//prodimg/small/g500_05_g.jpg', 'sortOrder': 3, 'productURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html?color=05', 'mainProdURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html'}, {'productID': 'G500', 'colorCode': '08', 'colorXref': 'Light Blue', 'description': 'INDIGO BLUE', 'hexColor': '34657f', 'image': '', 'imageURL': 'https://www.alphabroder.com//prodimg/small/g500_08_g.jpg', 'sortOrder': 4, 'productURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html?color=08', 'mainProdURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html'}, {'productID': 'G500', 'colorCode': '11', 'colorXref': 'Pink', 'description': 'LIGHT PINK', 'hexColor': 'FFE4E4', 'image': '', 'imageURL': 'https://www.alphabroder.com//prodimg/small/g500_11_g.jpg', 'sortOrder': 5, 'productURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html?color=11', 'mainProdURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html'}, {'productID': 'G500', 'colorCode': '12', 'colorXref': 'Orange', 'description': 'TANGERINE', 'hexColor': 'FF8A3D', 'image': '', 'imageURL': 'https://www.alphabroder.com//prodimg/small/g500_12_g.jpg', 'sortOrder': 6, 'productURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html?color=12', 'mainProdURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html'}, {'productID': 'G500', 'colorCode': '18', 'colorXref': 'Tan', 'description': 'SAND', 'hexColor': 'c5b9ac', 'image': '', 'imageURL': 'https://www.alphabroder.com//prodimg/small/g500_18_g.jpg', 'sortOrder': 7, 'productURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html?color=18', 'mainProdURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html'}, {'productID': 'G500', 'colorCode': '20', 'colorXref': 'Tan', 'description': 'NATURAL', 'hexColor': 'F3E4C4', 'image': '', 'imageURL': 'https://www.alphabroder.com//prodimg/small/g500_20_g.jpg', 'sortOrder': 8, 'productURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html?color=20', 'mainProdURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html'}, {'productID': 'G500', 'colorCode': '21', 'colorXref': 'Yellow', 'description': 'DAISY', 'hexColor': 'F9F46F', 'image': '', 'imageURL': 'https://www.alphabroder.com//prodimg/small/g500_21_g.jpg', 'sortOrder': 9, 'productURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html?color=21', 'mainProdURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html'}, {'productID': 'G500', 'colorCode': '25', 'colorXref': 'Orange', 'description': 'TEXAS ORANGE', 'hexColor': 'af5c37', 'image': '', 'imageURL': 'https://www.alphabroder.com//prodimg/small/g500_25_g.jpg', 'sortOrder': 10, 'productURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html?color=25', 'mainProdURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html'}, {'productID': 'G500', 'colorCode': '27', 'colorXref': 'Red', 'description': 'GARNET', 'hexColor': '8B0000', 'image': '', 'imageURL': 'https://www.alphabroder.com//prodimg/small/g500_27_g.jpg', 'sortOrder': 11, 'productURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html?color=27', 'mainProdURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html'}, {'productID': 'G500', 'colorCode': '29', 'colorXref': 'Yellow', 'description': 'OLD GOLD', 'hexColor': 'e0b06e', 'image': '', 'imageURL': 'https://www.alphabroder.com//prodimg/small/g500_29_g.jpg', 'sortOrder': 12, 'productURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html?color=29', 'mainProdURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html'}, {'productID': 'G500', 'colorCode': '30', 'colorXref': 'Pink', 'description': 'HELICONIA', 'hexColor': 'FF00FF', 'image': '', 'imageURL': 'https://www.alphabroder.com//prodimg/small/g500_30_g.jpg', 'sortOrder': 13, 'productURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html?color=30', 'mainProdURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html'}, {'productID': 'G500', 'colorCode': '31', 'colorXref': 'Orange', 'description': 'TENNESSEE ORANGE', 'hexColor': 'EB9501', 'image': '', 'imageURL': 'https://www.alphabroder.com//prodimg/small/g500_31_g.jpg', 'showMoreColors': True, 'sortOrder': 14, 'productURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html?color=31', 'mainProdURL': 'https://www.alphabroder.com/product/g500/gildan-adult-heavy-cotton-t-shirt.html'}]},...]

can't find correct 'select' HTML tag value, and trying to wait for a select option to load, playwright Python

Answers (1)

Related Questions

can&#39;t find correct &#39;select&#39; HTML tag value, and trying to wait for a select option to load, playwright Python

Answers (1)

Related Questions

can't find correct 'select' HTML tag value, and trying to wait for a select option to load, playwright Python