CODEWITHSUNDEEP
pythonhtmlseleniumweb-scraping
devcayed
devcayed

Reputation: 37

Element has ID in inspection mode, but not in raw HTML

I am currently working on a small web scraping script with Python and Selenium.

I am trying to get some information from a table, which has in inspection mode a certain ID. However, when I open the page as raw HTML (which I did after not being able to locate that table using neither xpath or css_selector), the table does not have the mentioned ID.

How is that possible?

For better explanation:

This is what it looks like in inspection mode in my browser

<table id='ext-gen1076' class='bats-table bats-table--center'>
[...]
</table>

And this is what it looks like when I open the page as raw HTML file

<table class='bats-table bats-table--center'>
[...]
</table>

How is it possible, that the ID is simply disappearing? (JFI, this is my first question so apologies for the bad formatting!).

Thanks in advance!

Upvotes: 0

Views: 205

Answers (2)

undetected Selenium
undetected Selenium

Reputation: 193208

The value of the id attribute i.e. ext-gen1076 contains a number and is clearly dynamically generated. The prefix of the value of the id attribute i.e. ext-gen indicates that the id was generated runtime using Ext JS.

Ext JS

Ext JS is a JavaScript framework for building data-intensive, cross-platform web and mobile applications for any modern device.

This usecase

Possibly you have identified the <table> element even before the JavaScript have rendered the complete DOM Tree. Hence the id attribute was missing.

Identifying Ext JS elements

As the value of the id attribute changes i.e. dynamic in nature you won't be able to use the complete value of the id attribute and can use only partial value which is static. As per the HTML you have provided:

<table id='ext-gen1076' class='bats-table bats-table--center'>
[...]
</table>

To identify the <table> node you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:

  • Using CSS_SELECTOR:

    WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "table[id^='ext-gen']")))

  • Using XPATH:

    WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//table[starts-with(@id,'ext-gen')]")))

  • Note : You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

However, there would be a lot other elements with id attribute starting with ext-gen. So to uniquely identify the <table> element you need to club up the class attribute as follows:

  • Using CSS_SELECTOR:

    WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "table.bats-table.bats-table--center[id^='ext-gen']")))

  • Using XPATH:

    WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//table[@class='bats-table bats-table--center' and starts-with(@id,'ext-gen')]")))

Reference

You can find a relevant detailed discussion in:

Upvotes: 0

Archer
Archer

Reputation: 1124

The reason is, the ID was added during the runtime.

Upvotes: 2

Related Questions