CODEWITHSUNDEEP
node.jsweb-scrapingpuppeteer
BillTaha
BillTaha

Reputation: 105

Puppeteer times out when headless is true on waitForNavigation and waitForSelector

I was successfully able to navigate through a website, log in, and get the information I need. However when I switch to using the same code with headless being true, it times out on each line where I am trying to wait.

const self = {
  browser: null,
  page: null,

  initialize: async () => {
    self.browser = await puppeteer.launch({
      headless: true,
    });

    self.page = await self.browser.newPage();

    /* Go to homepage */
    await self.page.goto(SHIPT_URL, { waitUntil: "networkidle0" });
  },

getResults: async () => {
    await self.page.type("#email", email, { delay: 100 });
    await self.page.type("#password", password, { delay: 100 });
    let loginButton = await self.page.$("button.sc-fzplWN");
    if (loginButton) {
      await loginButton.click();
      await self.page.waitForNavigation({ waitUntil: "networkidle0" }); // Hangs here

      await self.page.waitForSelector("a.pointer.link.gray900"); // Hangs here if I remove above waitForNavigation

I've tried researching this and tried a few different approaches like setting my user agent, putting the .click and waitForNavigation in a promise.all, removing "waitUntil" in waitForNavigation, and removing await on .click but nothing seems to be working.

Edit. So it looks like the login isn't working, it says wrong username or email which makes no sense since it works when I am not on headless. Going to need to look into this more

Upvotes: 1

Views: 1650

Answers (1)

deepfriedmars
deepfriedmars

Reputation: 55

Try and screenshot the output before running your commands. It could be that the default size of the page is different and the selectors don't exist.

Puppeteer sets an initial page size to 800×600px, which defines the screenshot size.

getResults: async () => {
await page.screenshot({path: 'example.png'});
await self.page.type("#email", email, { delay: 100 });
await self.page.type("#password", password, { delay: 100 });
let loginButton = await self.page.$("button.sc-fzplWN");

Upvotes: 1

Related Questions