CODEWITHSUNDEEP
javaseleniumhtml-parsing
Charles
Charles

Reputation: 65

Java - Parsing HTML elements generated dinamically

I would like to parse an HTML page after all dynamic elements have been generated (such as the "Inspect" button in Chrome).

I did some research and came across the Selenium API, but I still can't get what I want.

In the rorgente there is the following element "div":

<div id="tab-match-history" style="display: none;">
    <div id="match-history-preload" class="preload-panel">
        <div class="preload">
            <span>Loading...</span>
        </div>
    </div>
    <div id="match-history-content"></div>
</div>

As you can see it contains the "match-history-content" element which is empty. This element is dynamically created and becomes:

<div id="match-history-content">
    <div class="color-px-spacer submenu">&nbsp;</div>
    <div class="lines-bookmark">
        <ul class="ifmenu">
            <li class="divider"/>
            <li id="mhistory-1-history" class="li0 selected">
                <span>
                    <a onclick="detail_tab(['match-history', '1-history']);">Set 1</a>
                </span>
            </li>
            <li class="divider"/>
            <li id="mhistory-2-history" class="li1">
                <span>
                    <a onclick="detail_tab(['match-history', '2-history']);">Set 2</a>
                </span>
            </li>
            <li class="divider"/>
        </ul>
    </div>
    <!-- . . . -->
    <div id="tab-mhistory-1-history" style="display: block;">
        <table id="parts" class="parts-first">
            <tbody>
                <!-- . . . -->
            </tbody>
        </table>
        <div class="spacer-block"/>
    </div>
    <div id="tab-mhistory-2-history" style="display: block;">
        <table id="parts" class="parts-first">
            <tbody>
                <!-- . . . -->
            </tbody>
        </table>
        <div class="spacer-block"/>
    </div>
</div>

I would need to analyze the 2 elements "tab-mhistory-1-history" and "tab-mhistory-2-history".

The problem is that the WebDriver.getPageSource() method gives me the "match-history-content" div without the dynamically generated elements.

<div id="match-history-content"></div>

Below is the Java code I use:

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;

public class Main {

    public static void main( String[] args ) {
        final String URL = "https://www.flashscore.it/partita/GIecNDAM/#cronologia-dell-incontro;1";
                
        System.setProperty("webdriver.chrome.driver","C:\\dev_prog\\chromedriver_win32\\chromedriver.exe");
        WebDriver driver = new ChromeDriver();
        driver.get(URL);
        
        System.out.println(driver.getPageSource());
        System.out.println(driver.findElement(By.id("match-history-content")));
        System.out.println(driver.findElement(By.id("mhistory-1-history"))); // This command generates the error: no such element: Unable to locate element: {"method":"css selector","selector":"#mhistory\-1\-history"}
        driver.close();
        driver.quit();
    }
}

Thanks in advance

Upvotes: 0

Views: 208

Answers (1)

pburgr
pburgr

Reputation: 1778

put:

driver.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);

before:

driver.get(URL);

or instead of:

driver.findElement(By.id("mhistory-1-history"))

use:

new WebDriverWait(driver, 30).until(ExpectedConditions.elementToBeClickable(By.id("mhistory-1-history")))

Upvotes: 1

Related Questions