CODEWITHSUNDEEP
javascriptnode.jsweb-scrapingplaywright
Dan
Dan

Reputation: 2098

Getting a list of list elements using Playwright

I am trying to write an application where it will go to amazon and get a list of books on the page. I am using Playwright as the tool. I can get to the right section but I can't get the list of books. Looking online the examples seem to use page.$$(selector) but when I try that, I get an empty array back. Found this information here and here. Reading the docs on $$, this seems like the right call as all the list elements have the same class name. I have no idea what I am doing wrong, any advice on this?

Here is my code so far;

const AMAZON_KINDLE_EBOOK_STORE_URL = 'https://www.amazon.com/Best-Sellers-Kindle-Store-eBooks/zgbs/digital-text/154606011/ref=zg_bs_nav_kstore_1_kstore/';
(async () => {
    const browser = await chromium.launch();
    try {
        const amazonPage = await browser.newPage();
        await amazonPage.goto(AMAZON_KINDLE_EBOOK_STORE_URL);

        await amazonPage.waitForSelector('"Best Sellers in"');
        await amazonPage.click('"Self-Help"');
        await amazonPage.click('"Creativity"')

        const books = await amazonPage.$$('li[class="zg-item-immersion"]');
        console.log(books);
    } finally {
        await browser.close();
    }
})();

For the selector I have tried it numerous ways as well;

Upvotes: 3

Views: 22474

Answers (1)

pavelsaman
pavelsaman

Reputation: 8352

It seems the only problem is that Plawright is too fast and you don't wait for those elements li[class="zg-item-immersion"].

I debugged the script and the selector is fine, so with this line, it returns 50 element handles:

const { chromium } = require('playwright');

const AMAZON_KINDLE_EBOOK_STORE_URL = 'https://www.amazon.com/Best-Sellers-Kindle-Store-eBooks/zgbs/digital-text/154606011/ref=zg_bs_nav_kstore_1_kstore/';
(async () => {
    const browser = await chromium.launch({ headless: false});
    try {
        const amazonPage = await browser.newPage();
        await amazonPage.goto(AMAZON_KINDLE_EBOOK_STORE_URL);

        await amazonPage.waitForSelector('"Best Sellers in"');
        await amazonPage.click('"Self-Help"');

        await Promise.all([
            amazonPage.waitForNavigation(),
            amazonPage.click('"Creativity"')
        ]);
        
        const books = await amazonPage.$$('li[class="zg-item-immersion"]');
        console.log(books);
    } finally {
        await browser.close();
    }
})();

you can perhaps do what you did a few lines above and for a selector:

const { chromium } = require('playwright');

const AMAZON_KINDLE_EBOOK_STORE_URL = 'https://www.amazon.com/Best-Sellers-Kindle-Store-eBooks/zgbs/digital-text/154606011/ref=zg_bs_nav_kstore_1_kstore/';
(async () => {
    const browser = await chromium.launch({ headless: false});
    try {
        const amazonPage = await browser.newPage();
        await amazonPage.goto(AMAZON_KINDLE_EBOOK_STORE_URL);

        await amazonPage.waitForSelector('"Best Sellers in"');
        await amazonPage.click('"Self-Help"');
        await amazonPage.click('"Creativity"')

        await amazonPage.waitForSelector('li[class="zg-item-immersion"]');
        const books = await amazonPage.$$('li[class="zg-item-immersion"]');
        console.log(books);
    } finally {
        await browser.close();
    }
})();

It does work like this as well.

Upvotes: 5

Related Questions