CODEWITHSUNDEEP
pythonpython-3.xseleniumselenium-webdriver
Jeong In Kim
Jeong In Kim

Reputation: 373

How can I find elements including certain strings in id with selenium?

I was working on some simple crawler to scrape retweet counts on Twitter. And I'm stuck with this:

<span class="ProfileTweet-actionCountForAria" id="profile-tweet-action-retweet-count-aria-123456789123456789">리트윗 0개</span>

That's the targeted tags I want to collect. And you can see the id of the tag has some varying id number for each user. So I was trying to collect those with find_elements_by_xpath like this:

retweets = driver.find_elements_by_xpath("//span[@id='profile-tweet-action-retweet-count-area-*'].text")

I thought * worked in some places in selenium, but it doesn't work in that code.

So, in short, how can I find elements with id including 'profile-tweet-action-retweet-count-area'?

Thanks for your attention. I couldn't find the questions like this (maybe I didn't search it with the right question, hmm), but good references or other links are also found with me!

Upvotes: 1

Views: 2760

Answers (3)

S A
S A

Reputation: 1910

You can use the contains() or starts-with() method in xpath or css selector.

Also, to get the text from the element, you have to use the .text outside of the find_element method

XPath:

retweets = driver.find_elements_by_xpath("//span[starts-with(@id,'profile-tweet-action-retweet-count-area-')]")

Or,

retweets = driver.find_elements_by_xpath("//span[contains(@id,'profile-tweet-action-retweet-count-area-')]")

CSS Selector:

retweets = driver.find_elements_by_css_selector("span[@id^='profile-tweet-action-retweet-count-area-']")

Or,

retweets = driver.find_elements_by_css_selector("span[@id*='profile-tweet-action-retweet-count-area-']")

You have to iterate the list to get all the element and then you can get the text of the element using .text 

for retweet in retweets:
    print(retweet.text)

Edit: As cruisepanday mentioned find_elements_ returns a list and .text is not applicable. Also, the CSS selector shouldn't have the //. I've changed the code accordingly.

Upvotes: 5

cruisepandey
cruisepandey

Reputation: 29362

Css selector would be : 

span[id*="profile-tweet-action-retweet-count-aria"]

or a way better css selector would be : 

span[id^='profile-tweet-action-retweet-count-aria']

If you have multiple entries with that, you can use find_elements method which will give you list of web element.

If you don't want css selector and would want to stick with xpath : 

//span[contains(@id,"profile-tweet-action-retweet-count-aria")]

Code : 

list_retweet = driver.find_elements_by_xpath("//span[contains(@id,"profile-tweet-action-retweet-count-aria")]")

for retweet in list_retweet:
  print(retweet.text)

Upvotes: 5

KunduK
KunduK

Reputation: 33384

Use WebdriverWait to handle dynamic elements Try following Xpath Or Css Selector

element=WebDriverWait(driver,30).until(EC.element_to_be_clickable((By.XPATH,'//span[@class="ProfileTweet-actionCountForAria"][starts-with(@id,"profile-tweet-action-retweet-count-aria-")]')))
print(element.text)

OR

element1=WebDriverWait(driver,30).until(EC.element_to_be_clickable((By.CSS_SELECTOR,'span.ProfileTweet-actionCountForAria[id^="profile-tweet-action-retweet-count-aria-"]')))
    print(element1.text)

Please note that you need to use following imports.

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

Upvotes: 1

Related Questions