Selenium AttributeError: 'WebDriver' object has no attribute 'find_element_by_css_selector'

Question

I am following this build of a scraper for LinkedIn job data.

Here is my code:

from selenium import webdriver
import time
import pandas as pd

url = 'https://www.linkedin.com/jobs/search?keywords=&location=San%20Francisco%2C%20California%2C%20United%20States&locationId=&geoId=102277331&f_TPR=&distance=100&position=1&pageNum=0'

wd = webdriver.Chrome(executable_path=r'/Users/voi/chromedriver')
wd.get(url)

no_of_jobs = int(wd.driver.find_element_by_css_selector('h1>span').get_attribute('innerText'))

I have seen this, and attempted the solution, but received a similar error, except with regards to the WebDriver object not having a driver attribute.

Here is the full error message:

cd /Users/voi ; /usr/bin/env /usr/local/bin/python3 /Users/voi/.vscode/extensions/ms-python.python-2
022.8.1/pythonFiles/lib/python/debugpy/launcher 59402 -- /Users/voi/jobscrape.py 
/Users/voi/jobscrape.py:7: DeprecationWarning: executable_path has been deprecated, please pass in a Service object
  wd = webdriver.Chrome(executable_path=r'/Users/voi/chromedriver')
Traceback (most recent call last):
  File "/Users/voi/jobscrape.py", line 10, in <module>
    no_of_jobs = int(wd.find_element_by_css_selector('h1>span').get_attribute('innerText'))
AttributeError: 'WebDriver' object has no attribute 'find_element_by_css_selector'

Answer 1

Okay, I answered my own question. The individual methods find_element_by_* have been replaced by find_element, e.g.

no_of_jobs = int(wd.find_element(By.CSS_SELECTOR, 'h1>span'))

More info is here

Answer 2

Starting from Selenium version 4.0, the methods find_element_by_* and find_elements_by_* are no longer recommended to be used and have been replaced by the methods find_element() and find_elements().

These new methods allow you to locate elements on a web page by specifying a search strategy, such as by CSS selector, ID, name, etc.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

url = 'https://www.linkedin.com/jobs/search?keywords=&location=San%20Francisco%2C%20California%2C%20United%20States&locationId=&geoId=102277331&f_TPR=&distance=100&position=1&pageNum=0'

wd = webdriver.Chrome(executable_path=r'/Users/voi/chromedriver')
wd.get(url)
element = WebDriverWait(wd, 20).until(
    EC.visibility_of_element_located((By.CSS_SELECTOR, "h1>span"))
)

Please checkout details here

Answer 3

To add on to the answer from @m.lekk ( https://stackoverflow.com/a/72854301/7733418 ), I also tried to use dir() to get all the attributes from the object and find the text attribute that contains the information that I need.

Answer 4

Selenium recently removed the 16 deprecated find_element(s)_by_x functions in favor of a general find_element and find_elements function that take the "by" part as their first argument.

To update your code, you can use your IDE's find-and-replace-all feature to replace these 16 search terms:

.find_element_by_class_name(
.find_element(By.CLASS_NAME, 

.find_element_by_css_selector(
.find_element(By.CSS_SELECTOR, 

.find_element_by_id(
.find_element(By.ID, 

.find_element_by_link_text(
.find_element(By.LINK_TEXT, 

.find_element_by_name(
.find_element(By.NAME, 

.find_element_by_partial_link_text(
.find_element(By.PARTIAL_LINK_TEXT, 

.find_element_by_tag_name(
.find_element(By.TAG_NAME, 

.find_element_by_xpath(
.find_element(By.XPATH, 

.find_elements_by_class_name(
.find_elements(By.CLASS_NAME, 

.find_elements_by_css_selector(
.find_elements(By.CSS_SELECTOR, 

.find_elements_by_id(
.find_elements(By.ID, 

.find_elements_by_link_text(
.find_elements(By.LINK_TEXT, 

.find_elements_by_name(
.find_elements(By.NAME, 

.find_elements_by_partial_link_text(
.find_elements(By.PARTIAL_LINK_TEXT, 

.find_elements_by_tag_name(
.find_elements(By.TAG_NAME, 

.find_elements_by_xpath(
.find_elements(By.XPATH, 

You'll also need to import By if you haven't already done so:

from selenium.webdriver.common.by import By

Answer 5

Selenium just removed that method in version 4.3.0. See the CHANGES: https://github.com/SeleniumHQ/selenium/blob/a4995e2c096239b42c373f26498a6c9bb4f2b3e7/py/CHANGES

Selenium 4.3.0
* Deprecated find_element_by_* and find_elements_by_* are now removed (#10712)
* Deprecated Opera support has been removed (#10630)
* Fully upgraded from python 2x to 3.7 syntax and features (#10647)
* Added a devtools version fallback mechanism to look for an older version when mismatch occurs (#10749)
* Better support for co-operative multi inheritance by utilising super() throughout
* Improved type hints throughout

You now need to use:

driver.find_element("css selector", SELECTOR)

In your example, you would use:

no_of_jobs = int(wd.find_element("css selector", "h1 > span").get_attribute("innerText"))

For improved reliability, you should consider using WebDriverWait in combination with visibility_of_element_located.

Here's how that might look:

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

# ...

element = WebDriverWait(wd, 10).until(
    EC.visibility_of_element_located(("css selector", "h1 > span"))
)
no_of_jobs = int(element.get_attribute("innerText"))