Extracting Hyperlink Text from <a> tag using selenium on Python

Question

I am currently trying to pull all the, elements using tag name in selenium, but then when I try to iterate through the list of all elements and use get_attribute('text') or ('innerHTML') to get the text attribute, I am left with a list full of "None".

I am checked that the list of elements wasn't also full of none since I verified to see if it had any value using print statements but I'm unsure what could be the problem now. I tried multiple different operators I saw being used by others such as only extracting the active tags and using .text() instead of get_attribute to extract the text, but looking further into it, that also wouldnt work since once selenium finds the element, its stored as a hash or something. I am going to put my code and output underneath.

    from selenium import webdriver
    from selenium.webdriver import Firefox
    from selenium.webdriver.firefox.service import Service
    from selenium.webdriver.common.keys import Keys
    from selenium.webdriver.support.select import Select
    from selenium.webdriver.common.by import By
    import time

    # User-defined list of items
    items = ['water', 'orange juice', 'soda']

    # Start the browser session
    driver_path = r"C:\\VSCode\geckodriver.exe"
    service = Service(executable_path='C:\\VSCode\\geckodriver.exe')
    driver = Firefox(service=service).driver.get("https://bangyourbuck.com/")

    # Iterate through each item in the list
    for item in items:
    # Find the search bar and enter the item name
        search_bar = driver.find_element("xpath", '//div[@class="col-12 col-md-12 mb-2 mb-md-0"]//input[@placeholder="Search for anything (e.g. rice)..."]')
        search_bar.send_keys(str(item))

    # Sort by price per fl oz
    sort_button = driver.find_element(By.NAME, 'desired_units')
    select = Select(sort_button)
    sort_by_ppf = driver.find_element(By.CSS_SELECTOR, 'option[value=Fluid_Ounce]')
    sort_by_ppf.click()
    calculateButton = driver.find_element(By.NAME, 'submitButton')
    calculateButton.click()
    time.sleep(1)

    # Get the top 5 result
    linkPath = driver.find_elements(By.TAG_NAME, 'a')[10:15]
    namePath = driver.find_elements(By.TAG_NAME, "a")
    pricePath = driver.find_elements(By.CLASS_NAME, 'card-text')
    Links = []
    Names = []

    Links = [Links.append(link.get_attribute('href')) for link in linkPath]
    Names = [Names.append(name.get_attribute('text')for name in namePath]
    print(namePath)
    print(Names)

    driver.get('https://bangyourbuck.com/')
    search_bar.clear()

    # Close the browser session
    driver.quit()

Output:

`[<selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="eafb8305-0f91-48b6-84c7-5ffab4a91499")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="3ff0fee1-007f-4ba1-a1ae-f7ab943206cc")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="8ad761e1-50de-4c00-9165-3ea8cf6f9650")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="4996414f-1c03-48ed-bb27-22eb01f0b01a")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="1dfa5113-9c3a-40b7-b11f-d671f43b381d")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="905b423f-8684-48d3-b8e8-2b6da169b6f8")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="058fb95e-c0ea-4e83-91ba-38e303149793")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="adc37497-6a3b-4419-a3bf-56bfe5ea8665")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="d468ce14-519c-4d0b-be42-649d22cb528f")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="700e553e-3a53-483a-adf2-b64fa1fb373b")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="083da717-e308-4b57-b5c6-e4a5f360380d")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="457327cb-202a-4f86-8eec-5a0d3dd5924e")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="0164df36-f6cb-4aae-908f-e199452b28f2")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="d412262c-776d-421d-adad-3f4d18772efe")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="e4191100-89d6-4da5-9e8d-36423379c9d5")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="eca73863-ac7d-48ae-832d-383d11d4d58c")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="85f9f826-fdcc-49a8-aacc-cf4c779bb836")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="c4007323-63be-499e-b908-4b191056ff99")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="786e2833-aea0-4994-9c97-d1548692296b")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="2cb107e6-abac-47e8-8815-2b4348d94878")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="eaa00e76-f6a1-4367-81bd-4f738560deb7")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="93e415ea-a7eb-4596-90af-915cc311d157")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="2cb66695-73a1-4852-9126-9ebb58e063f6")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="120b5e06-f51a-4ceb-8833-908e9b26eb7b")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="d24ad6d4-1209-4d16-9b8c-e677f4a9a318")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="f692589c-1975-4abc-9566-20305c9e9033")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="f55360a0-45bd-4979-b174-e9836e780ef1")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="60e7f725-7bb7-48b0-9aac-cc193cf2907b")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="0118032f-e96b-42a4-9401-187bbd4a80bf")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="dde2e741-9a70-487a-8300-5ad113649d1d")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="c3002259-2084-4a9c-85dc-9ff9d795b6cf")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="2f99f800-32b6-4486-bb32-84149f3cd498")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="62f8e7cf-57ca-447f-9618-193537517a60")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="cdaebb2c-a1e4-4cec-841f-836ca0f3818b")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="6efdafb2-2d55-4fa2-9cb7-8b35e0ce5c73")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="4e1aac45-3536-4f0d-a4e7-abffb0d4c692")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="29b57288-879b-4647-8121-c85c2f4f7bac")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="10bb6ed3-e2d8-471b-b3c4-58667a01b564")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="d33c74a2-1dc2-441d-bad6-0a24dc517f32")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="18ea9912-dc05-46dd-ae1a-c8af27a5c9d6")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="42e86568-59e0-4e63-8f2c-2f18e4573930")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="a46dd8b9-8042-42a2-bae9-71ba15d192ef")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="f5233783-503c-4ff1-a00e-cfa92a6af3b0")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="141e3d97-b484-496e-a15f-9276aa1893d3")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="5f1e5b00-cc14-48a4-9d83-4ba7c859c966")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="d5f8a1b6-b141-4f83-9029-ccb73029797b")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="d2286e2e-5f59-452f-b537-d8d3f0474a20")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="b0cf8160-a8f8-408c-8171-3400f076237e")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="9a349ff0-7eb9-45f7-8f14-06c1a6ffb8eb")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="19064dbc-c549-4efc-90c4-5a0f316360c8")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="f6908940-f65b-4a63-aeb8-6693bbc6ac17")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="38a22ac4-48f6-44cb-9b09-289d53e4d855")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="01998626-cedc-448b-a633-4fc7ccd7cd7e")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="88d9dc52-c004-433c-bc0c-b927b506e4ed")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="3b6ce269-789d-4e99-81c8-dfc7e18e8cea")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="88de2123-4659-46e6-a3db-8a50114d1cb6")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="3a1216e5-0856-49b8-89d9-13ff9cecfde5")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="f5ebcff6-08dc-410d-9fcd-53514b95f83b")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="85ea8ce9-a902-4f7d-977f-f20c207fb6dd")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="889f5af9-1f04-4801-8a4d-e4c77a1299fa")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="ba61649a-d7f9-4830-974e-25bbf96fae03")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="1da03f83-729e-4b42-8ec1-68fa9eeb0145")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="1884200b-7ecc-41e2-981b-daf26251a64f")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="6ea6ef65-8f0b-4a8b-a9a7-b5b17678ec58")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="78404bb1-6ce2-4f97-8518-154e4995f8cd")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="558077dc-86da-4212-b7c5-2559b11f1b90")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="7c32580e-6dc7-4fba-9158-9cca7285833d")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="c1fd875b-c352-461b-980e-22eaea077351")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="18682551-2d31-483d-89bf-fb6588bb0583")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="27be5406-20ba-4488-bfbe-582f1db21cbc")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="79dbf5d1-6e9b-499b-ba19-ccb56efa9973")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="f8d0e8dc-4529-4474-8346-26fb9c235409")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="ed582186-68f5-4caa-99ba-53be1ab059bd")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="c8af9dfb-5156-45f6-9bc4-1fc6d2241590")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="851e9e61-fa35-4fe1-a309-fa7c103d103e")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="c22002aa-6da1-4ef4-8672-338007ce3971")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="98ec497e-1a4c-49b1-915b-d34b89545a28")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="2a1e4595-594f-4305-9b6d-062503694b69")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="86e47638-bf6f-4b61-b5a3-fec84b5630be")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="7f38f201-1be6-42d2-b0c4-a6887b113430")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="4a488d20-6480-418b-b1e5-3368d378b8d5")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="2926f797-f928-45c2-9328-e6cc02cb62b8")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="1473983c-1cec-4964-9fb2-70a0e5c4f8af")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="68fb8bb4-d181-42d7-9505-e10b7bfa4527")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="54424d99-8857-4c0e-8a44-263d852910f7")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="31e9b4c4-37ae-42da-8033-d6b1b7744697")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="d903eb35-f41d-4c9a-a5c3-5db6af75900c")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="a6c27c94-76e3-494c-8261-2c47fc45eab6")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="c8c103c7-da51-4bdb-8711-00b1dbc3f7c5")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="2e985e74-fc03-4e5c-8ab2-d2fdade38227")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="9248059a-ce49-4e23-a0bf-ae9e46448d4c")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="6a9cf622-dfd1-468f-a5eb-6aadf7d69a67")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="d6a4a1de-2705-4195-8a47-747c382b3d6f")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="1670002e-cb9c-428e-8439-0c5e88a2baab")>, <selenium.webdriver.remote.webelement.WebElement (session="01696fd6-d458-4960-9feb-25c558d5cb59", element="a617a387-d626-4085-b3a7-97c7d041365b")>]
[None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]`

Answer 1

Probably you don't wait for element to be rendered and as a name you print namePath, which is in your case is WebElement.

Keep it simple. You products have unique class card-body.

To get first 5 products, you need just to wait for presence of all cards, loop through first one and get it inner elements that corresponding needed card fields.

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

#your code 

cards = wait.until(EC.presence_of_all_elements_located((By.CLASS_NAME, 'card-body')))
Links = []
Names = []
Prices = []

for i in range(5):
    link = cards[i].find_element(By.TAG_NAME, 'a').get_attribute('href')
    name = cards[i].find_element(By.CLASS_NAME, 'card-title').get_attribute('innerText')
    price = cards[i].find_element(By.CLASS_NAME, 'black-link').get_attribute('innerText')
    Links.append(link)
    Names.append(name)
    Prices.append(price)