CODEWITHSUNDEEP
javascriptpuppeteer
Aessandro
Aessandro

Reputation: 5761

Puppeteer takes too many seconds to load the data

I have the following:

const puppeteer = require('puppeteer');
var express = require('express');
var app  = express();
var port = process.env.PORT || 3001;

app.listen(port);

app.get('/', (req, res) => {
    res.setHeader('Access-Control-Allow-Origin', '*');
    (async () => {
        const browser = await puppeteer.launch();
        const page = await browser.newPage();
        await page.goto(`url`);

        const data = await page.evaluate(() => {
            const imgs = Array.from(document.querySelectorAll('.dp-gallery__list-item img'));
            return imgs.map(td => {
                var txt = td.src;
                return txt;
            });
        });

        res.send({ data });
        await browser.close();
    })();
});

the content that I need .dp-gallery__list-item img is loaded dynamically by js (I have tested it by disabling js) hence the use of puppeteer.

The issue I am facing is that every time I hit '/' it takes a good 3/4 seconds to receive the response in the ui:

const About = ({data}) => (
  <div>
    <Header />
    {data &&
        data.map(((img, i) => <img key={`${i} '-' ${img}`} src={img} />))
    }
    <p>Hello Next.js</p>
  </div>
);

About.getInitialProps = async ({query: {id}}) => {
    const res = await fetch('http://localhost:3001/');
    const data = await res.json();
    console.log('id: ', id);
    return data;
}

export default About;

Is it because the content I am looking for is loaded dynamically or am I missing something the the Puppeteer configuration?

Upvotes: 1

Views: 864

Answers (1)

Md. Abu Taher
Md. Abu Taher

Reputation: 18826

When you hit "/", it does a lot of stuff including the following:

  • It opens creates a new temporary chrome profile.
  • Launches the chrome with certain flags and options.
  • Creates a new tab.
  • Loads the target url.
  • Wait for some events to fire.
  • Evaluates your code.
  • Closes the browser.
  • Sends your data.

Basically it does as it's instructed. The speed of creating a new chrome tab depends on your computer resources, navigating to the website depends on both your computer and the target website. If it's dynamic website, then there is added cost with various page load events. Also there is the overhead of your UI as well.

Thus it takes 3-4 seconds to go through all those steps.

Upvotes: 3

Related Questions