CODEWITHSUNDEEP
javajsoup
Romas Augustinavičius
Romas Augustinavičius

Reputation: 443

Get full HTML using Jsoup

I'am scraping web-page using Jsoup library by selecting class attributes which contains "nav" string in them.

This is the code which fetch HTML of the site:

var bodyString = Jsoup.connect(url)
                .ignoreContentType(true)
                .userAgent("Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0) Gecko/20100101 Firefox/25.0")
                .timeout(12000)
                .followRedirects(true)
                .execute()
                .body();

Example of Html which is selected by Jsoup CSS selector: enter image description here

Yet in browser same website Html looks like this: enter image description here

As you can see ul element with id="varPreviewMenu" contains li elements which HTML retrieved by Jsoup does not contains.

How can I get those elements?

Upvotes: 2

Views: 422

Answers (1)

Code-Apprentice
Code-Apprentice

Reputation: 83527

Most likely the elements you see are dynamically added to the DOM by some JavaScript code. That means they are not available in the body of the request when you use Jsoup.

Upvotes: 2

Related Questions