CODEWITHSUNDEEP
javascriptnode.jsautomationweb-scrapingnightmare
nlln
nlln

Reputation: 11

Automating ajax generated websites with nightmare

I am using Nightmare to automate a website. It's been great so far, but I noticed that it has some problems when I want to interact with content that is loaded dynamically. There is even a method, that waits for an element to appear on a page .wait(#elementId), but it does not work for content that is generated on the fly.

Have somebody encountered this problem before, or could you maybe recommend some other technology? What I like about nightmare is, that it is not in fact headless and via its integration with Electron it also has a GUI that shows everything that is done. I would prefer if this would be possible.

EDIT

To better illustrate my concern, here is the code I am using, but abstracted:

Let's say that I want to search on https://www.google.com, but the form for searching is generated dynamically via a library. My code would look like this

vo(function* () {
var nightmare = Nightmare({ show: true });

var search = yield nightmare
    .goto('https://google.com')
    .wait('input[name="search"]')
    .type('input[name="search"]', ‘the term I am searching for’)
    .click('#submitButton')
    .wait(2000)
    .evaluate(function () {
        return $('input[name="search"]').val();
    });

yield nightmare.end();
return search;

})(function (err, result) {
    if (err) return console.log(err);
    console.log(result);
});

But since the input[name="search"] is not written in html, but generated after the page has loaded, even though I can see it in the GUI, the scraper does not recognize it and will wait forever. My guess it is only working with the static code. Is there a way to update the html after some time, or something like that?

Upvotes: 1

Views: 490

Answers (1)

Aditya Jhunjhunwala
Aditya Jhunjhunwala

Reputation: 635

Update your code as:

.wait("input[type='text'][title='Search']")
.type("input[type='text'][title='Search']", 'the term I am searching for')

This works perfectly fine. The problem was that the component was waiting for input[name="search"] , which never gets attached to the google search's input field.

Changing to the above will resolve your issue. input[name="search"] this is no available on google's search bar even after the page has fully loaded.

Also, use DEBUG=nightmare:actions node --harmony test.js while running the test script as it will help you in determining the action your code gets stuck at.

Hope this helps.

Upvotes: 1

Related Questions