Java HtmlUnit Ajax call never get rendered in the HtmlPage

Question

I am attempting to scan a page using HtmlUnit 2.31. I am simply getting an HtmlPage by URL. However, there seems to be some AJAX calls kicked off by the page (no user interaction). I need to wait for the calls to finish and be "rendered" so I can see the values. Here is my code:

    public static void main( String[] args ) throws IOException, InterruptedException {
        WebClient webClient = new WebClient(BrowserVersion.CHROME);
        webClient.getOptions().setJavaScriptEnabled(true);
        webClient.getOptions().setThrowExceptionOnScriptError(false);
        webClient.getOptions().setCssEnabled(false);
        webClient.setAjaxController(new NicelyResynchronizingAjaxController());
        webClient.setJavaScriptTimeout(10000);

        String itemsForSalePageURL = "https://miami-dade.realtaxdeed.com/index.cfm?zaction=AUCTION&Zmethod=PREVIEW&AUCTIONDATE=07/24/2018";
        HtmlPage previewItemsForSalePage = webClient.getPage(itemsForSalePageURL);

        final JavaScriptJobManager tmpJobManager = previewItemsForSalePage.getEnclosingWindow().getJobManager();
        while(tmpJobManager.getJobCount()>0){
            System.out.println("javascript is still running...");
        }

        boolean doneLoading = false;
        while(!doneLoading){
            System.out.println("checking if page is done loading...");
            List loadingElements = previewItemsForSalePage.getByXPath("//div[@class = 'Loading']");
            if(loadingElements.isEmpty()){
                doneLoading = true;
            }
        }

        String contents = previewItemsForSalePage.asXml();

        System.out.println( "Run complete." );
    }

The problem is: the while(!doneLoading) loop becomes an infinite loop. The classes that my xpath is searching for never evaluates to anything besides "loading".

One area I am interested in is:

In Chrome, this evaluates to


    
This item has complimentary photos, valuation reports and title information.
Auction Status
Canceled per Bankruptcy
 
 Auction Type: FORECLOSURE
Case #:  2008-069895-CA-01
Final Judgment Amount: $567,016.30
Parcel ID:  04-2036-007-0830
Property Address: 618 W 65 DR
HIALEAH, FL- 33012
 Assessed Value: $131,793.00
Plaintiff max bid: $279,450.00
 
 
Auction Sold
07/24/2018 09:07 AM ET
Amount
 $198,200.00
Sold To
3rd Party Bidder
Auction Type: FORECLOSURE
Case #:  2012-007563-CA-01
Final Judgment Amount: $123,600.66
Parcel ID:  30-6935-001-0170
Property Address: 26711 SW 133 CT
MIAMI, FL- 33032
 Assessed Value: $104,378.00
Plaintiff max bid: $190,539.81
 
 
Auction Status
Canceled per County
 
Auction Type: FORECLOSURE
Case #:  2013-006996-CA-01
Final Judgment Amount: $1,060,190.67
Parcel ID:  33-5033-055-0030
Property Address: 17321 SW 93 AVE
PALMETTO BAY, FL- 33157
 Assessed Value: $276,458.00
Plaintiff max bid: Hidden
 
 
Auction Status
Canceled per Order
 
Auction Type: FORECLOSURE
Case #:  2013-009405-CA-01
Final Judgment Amount: $161,175.75
Parcel ID:  08-2128-014-0050
Property Address: 13247 ASWAN RD
OPA-LOCKA, FL- 33054
 Assessed Value: $88,602.00
Plaintiff max bid: Hidden
 
 
Auction Sold
07/24/2018 09:14 AM ET
Amount
 $390,200.00
Sold To
3rd Party Bidder
Auction Type: FORECLOSURE
Case #:  2014-012973-CA-01
Final Judgment Amount: $255,004.82
Parcel ID:  01-4121-007-1790
Property Address: 3321 FROW AVE
MIAMI, FL- 33133
 Assessed Value: $199,959.00
Plaintiff max bid: $287,550.00
 
 
Auction Status
Canceled per Bankruptcy
 
Auction Type: FORECLOSURE
Case #:  2014-020812-CA-01
Final Judgment Amount: $294,642.58
Parcel ID:  30-5029-019-0190
Property Address: 15900 SW 105 AVE
MIAMI, FL- 33157
 Assessed Value: $153,877.00
Plaintiff max bid: Hidden
 
 
This item has complimentary photos, valuation reports and title information.
Auction Status
Canceled per Order
 
 Auction Type: FORECLOSURE
Case #:  2015-002717-CA-01
Final Judgment Amount: $500,378.48
Parcel ID:  30-2003-038-1200
Property Address: 19522 NW 77 CT
HIALEAH , FL- 33015
 Assessed Value: $273,509.00
Plaintiff max bid: Hidden
 
 
Auction Sold
07/24/2018 09:20 AM ET
Amount
 $373,100.00
Sold To
3rd Party Bidder
Auction Type: FORECLOSURE
Case #:  2015-004160-CA-01
Final Judgment Amount: $650,255.11
Parcel ID:  30-4909-004-0080
Property Address: 15495 SW 13 TER
MIAMI, FL- 33194
 Assessed Value: $267,671.00
Plaintiff max bid: $339,375.00
 
 
Auction Status
Canceled per Order
 
Auction Type: FORECLOSURE
Case #:  2015-005595-CA-01
Final Judgment Amount: $217,737.00
Parcel ID:  01-4001-021-0019
Property Address: 6261 TAMIAMI CANAL RD
MIAMI, FL- 33126
 Assessed Value: $153,080.00
Plaintiff max bid: Hidden
 
 
Auction Status
Canceled per Order
 
Auction Type: FORECLOSURE
Case #:  2015-022277-CA-01
Final Judgment Amount: $374,062.43
Parcel ID:  30-4028-007-0030
Property Address: 6020 SW 92 AVE
MIAMI, FL- 33173
 Assessed Value: $208,500.00
Plaintiff max bid: Hidden

I am not sure what else to try at this point. I have tried changing the BrowserVersion. I have tried increasing the JavaScriptTimeout. I tried using the NicelyResynchronizingAjaxController. I even wait until all javascript is done processing. I think I am missing some fundamental piece of understanding. Can someone please fill me in.

RBRi · Accepted Answer

Here comes the workaround

String url = "https://miami-dade.realtaxdeed.com/index.cfm?zaction=AUCTION&Zmethod=PREVIEW&AUCTIONDATE=07/24/2018";

try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_60)) {
    webClient.setWebConnection(new FalsifyingWebConnection(webClient) {
        @Override
        public WebResponse getResponse(WebRequest webRequest) throws IOException {
            if (webRequest.getUrl().getPath().endsWith("jquery-1.6.1.min.js")) {
                String jQuery = FileUtils.readFileToString(new File("jquery-1.6.1.js"));
                return createWebResponse(webRequest, jQuery, "application/javascript");
            }
            return super.getResponse(webRequest);
        }
    });

    HtmlPage previewItemsForSalePage = webClient.getPage(url);
    webClient.waitForBackgroundJavaScript(1000);

    System.out.println("----------------");
    System.out.println(previewItemsForSalePage.asText());
}

And here some notes:

by using FalsifyingWebConnection you can intercept and manipulate the request and the response if you like
be careful with this while statements without a timeout, many pages are using some kind of heartbeat; you while will never stop

At least here I got the same content as in real browsers.

Java HtmlUnit Ajax call never get rendered in the HtmlPage

Answers (2)

Related Questions