CODEWITHSUNDEEP
javascriptnode.jsjavascript-objectspuppeteer
Pwntastic
Pwntastic

Reputation: 481

Puppeteer looping and converting to object

I'm new to the puppetteer library and am attempting to loop over amazon reviews and save each comment as an object.

It seems to be working but it will only grab the first comment and be done. 

async function scrapeProduct(urls) {
    urls.map(async function(url, index) {
        const browser = await puppeteer.launch({ headless: true });
        const page = await browser.newPage();
        await page.goto(url);
        await page.waitFor(5000);

        const result = await page.$$eval('.filterable-reviews-content', rows => {
            return rows.map(review => {
                const properties = {};
                const titleElement = review.querySelector(".review-title-content span");
                properties.title = titleElement.innerText;
                properties.ranking = review.querySelector(".review-title-content span");
                // return properties;
            })
        })
})

my other attempt was this... and it grabs everything but pushes it all to a string. 

let content = await page.evaluate(() => {
    let commentWrapper = [...document.querySelectorAll('.filterable-reviews-content')];
    return commentWrapper.map(item => {
        return item.textContent.replace(/(\r\n|\n|\r)/gm,"")
    } );
  });

I really appreciate the help. thank you!

Upvotes: 0

Views: 376

Answers (1)

hardkoded
hardkoded

Reputation: 21597

I think you need to improve your selector. This might work

const result = await page.$$eval('.filterable-reviews-content .a-section.review', rows => {
    return rows.map(review => {
        const properties = {};
        const titleElement = review.querySelector(".review-title-content span");
        properties.title = titleElement.innerText;
        properties.ranking = review.querySelector(".review-title-content span");
        return properties;
    })
})

Upvotes: 2

Related Questions