CODEWITHSUNDEEP
javahttp
Jirka Hubáček
Jirka Hubáček

Reputation: 187

Download a webpage's source which uses a loading spinner

I need to download a source code of this webpage: https://app.zonky.cz/#/marketplace/ so I could have the code checking if there is a new loan available. Unfortunate for me, the web page uses a loading spinner for the time the page is being loaded in the background. When I try to download the page's source using:

String url = "https://app.zonky.cz/#/marketplace/";
    StringBuilder text = new StringBuilder();
    try
    {
        URL pageURL = new URL(url);            
        Scanner scanner = new Scanner(pageURL.openStream(), "utf-8");
        try {
            while (scanner.hasNextLine()){
                text.append(scanner.nextLine() + "\n");
            }
        }
        finally{
            scanner.close();
        }

    }
    catch(Exception ex)
    {
        //
    }        
    System.out.println(text.toString());

I get the page's source from the moment the spinner is being shown. Do you know of a better approach?

Solution:

public static String getSource() {
    WebDriver driver = new FirefoxDriver();
    driver.get("https://app.zonky.cz/#/marketplace/");
    String output = driver.getPageSource();
    driver.close();
    return output;
}

Upvotes: 0

Views: 52

Answers (1)

user2649305
user2649305

Reputation: 149

You could always wait until the page has finished loading by checking if an element exists(one that is only loaded after the spinner disappears)

Also have you looked into using selenium ? it can be really useful for interacting with websites and handling tricky procedures such as waiting for elements :P

Edit: a pretty simple tutorial for Selenium waiting can be found here - http://docs.seleniumhq.org/docs/04_webdriver_advanced.jsp#explicit-and-implicit-waits

Upvotes: 1

Related Questions