CODEWITHSUNDEEP
pythonseleniumselenium-webdriver
Ronron
Ronron

Reputation: 89

How to handle selenium python script when element is already present on page?

I am scraping a small site wherein I loop to send_keys to a textbox then click on the search button, the page loads some results, I check for presence_of_element and finally I get text of those results. But the issue is when the site opens it already has a few results Present on the page, so when the loop for the search starts and the search button is clicked, the page takes a few secs to load the New Results but the script continues and selenium see the presence of initial results and captures them again and the loop continues with the same result. I tried adding time.sleep but still runs into some issues. Below is the workflow and code

URL Opens
Result 0 already present on page
Change Dropdown
Loop starts>>
Text sent to searchBox >> Search button Clicked
Result 0 still present on Site
Page is loading >> But Selenium sees presence of Result0 and gets text
Loop continues to send new key and click search button
Page is still loading with Result 1>> selenium again checks presence and this continues.

self.driver.get(self.url)
self.waitForPresenceOfElement(locator=self.radius_drop_down, locatorType='id')
self.dropByType(data='100', locator=self.radius_drop_down, locatorType='id', type='value')
time.sleep(6)
for state in self.states:
    self.sendKeysWhenReady(data=state, locator=self.search_box, locatorType='id')
    time.sleep(1)
    self.elementClick(locator=self.search_button, locatorType='xpath')
    time.sleep(3)
    if self.getElementList(self.storesXpath, locatorType='xpath'):  # to ignore Empty states
        stores = self.waitForPresenceOfAllElements(locator=self.storesXpath, locatorType='xpath')
        for store in stores:  
            self.full_list.append(self.getText(element=store).lower())

Upvotes: 0

Views: 140

Answers (1)

JeffC
JeffC

Reputation: 25621

The way you fix this is to:

  1. Start your loop.

  2. Find an existing search result on the page and get a reference to it.

    result = driver.find_element(...)

  3. Send the search terms and click Search.

  4. Wait for the result reference to be stale, that tells you that the page is reloading.

     wait = WebDriverWait(driver, 10)
 wait.until(EC.staleness_of(result))

  5. Wait for results to be visible and continue looping.

Upvotes: 1

Related Questions