CODEWITHSUNDEEP
javamemory-leakshtmlunit
Schippert
Schippert

Reputation: 75

Java HtmlUnit Memory Leak

I got the following code:

for (int i = 0; i < list.size(); i++) {
    navigate(list.get(i));
}

And for the navigate method:

HtmlTextInput txtInput = page.getElementByName("input");
txtInput.setValueAttribute("random");


HtmlSubmitInput btnNext = page.getHtmlElementById("btn_next");
page = btnNext.click();


HtmlSubmitInput btnConfirm = page.getElementByName("submit");
page = btnConfirm.click();


System.out.println("before: " + webClient.getWebWindows().size());
page.cleanUp();
page.deregisterFramesIfNeeded(); 
System.out.println("after: " + webClient.getWebWindows().size());

return true;

HtmlPage and WebClient have been declared globally.

To put it in perspective, I fill in a form, press OK, press Confirm and I got redirected to my initial page so I repeat the process. The problem is that my application is eating up all the memory. And later on this will throw a heap exception.

If I count the web windows I see that number growing and never decreasing. A lot of other topic about this memory leak suggests that I have to call the .CloseAllWindows() method but this feature has been removed.

Currently I am using HtmlUnit 2.25.

Did I make a mistake while redirecting so that older windows stay in the background?

UPDATED VERSION:

Creation of a new WebClient method:

private WebClient createNewWebClient(){
        WebClient webClient = new WebClient(BrowserVersion.FIREFOX_45);

        // no exceptions
        webClient.getOptions().setThrowExceptionOnScriptError(false);
        webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
        // other settings 
        webClient.getOptions().setJavaScriptEnabled(true);
        webClient.getOptions().setCssEnabled(false);
        webClient.getOptions().setRedirectEnabled(true);
        webClient.getOptions().setThrowExceptionOnScriptError(false);
        webClient.getOptions().setTimeout(300000);

        // enable sessions
        webClient.getCookieManager().setCookiesEnabled(true);

        // Set session if any
        if (cookieManager == null){
            cookieManager = new CookieManager();
        } else {
            webClient.setCookieManager(cookieManager);
        }
        return webClient;
    }

Loop stayed the same. Navigate method:

     try (WebClient webClient = createNewWebClient()) {
                HtmlPage page = webClient.getPage("URL");
                HtmlTextInput txtInput = page.getElementByName("input");
                txtInput.setValueAttribute("random");


                HtmlSubmitInput btnNext = page.getHtmlElementById("btn_next");
                page = btnNext.click();


                HtmlSubmitInput btnConfirm = page.getElementByName("submit");
                page = btnConfirm.click();

             ...
       }

Memory is still increasing with every method call.

Upvotes: 1

Views: 1498

Answers (3)

RBRi
RBRi

Reputation: 2879

maimArt is right, in some cases the page.cleanUp(); was not completely processed from webClient.close();. This is fixed now and will be part of the upcoming 2.44 release.

In cases like this it will be great if you report the issues to the GitHub project, otherwise we have no chance to fix that.

Upvotes: 1

maimArt
maimArt

Reputation: 399

I struggled with the same problem. After many hours of debugging and memory analysis, i found out that it´s not enough to just close the webclient. You also need to cleanup the loaded page to release all nested references.

I wrapped the WebClient in another Closable class to do this.

So in your case this sould fix your memory leak:

WebClient webclient = null;
HtmlPage page = hull;
try {
            webClient = createNewWebClient()
            page = webClient.getPage("URL");
            HtmlTextInput txtInput = page.getElementByName("input");
            txtInput.setValueAttribute("random");


            HtmlSubmitInput btnNext = page.getHtmlElementById("btn_next");
            page = btnNext.click();


            HtmlSubmitInput btnConfirm = page.getElementByName("submit");
            page = btnConfirm.click();

         ...
   } finally {
            page.cleanUp();
            webClient.close();
   }

Upvotes: 0

Ahmed Ashour
Ahmed Ashour

Reputation: 5549

You need to call webClient.close(), or simply put it in try-with-resources, for example:

try (WebClient webClient = new WebClient()) {
    String url = "http://localhost:8080";
    HtmlPage page = webClient.getPage(url);

    // do something
}

Update :

You can store the CookieManager of webClient.getCookieManager() and use it for other WebClients.

Upvotes: 1

Related Questions