Python :selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element:

Question

I am trying to scrape data from Youtube using selenium python. The data which I am scraping has fields like - subscribers, location, joined on and views. Despite giving the correct xpath I get an error like this

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="subscriber-count"]"}
  (Session info: chrome=90.0.4430.93)

Things that I've tried are- working with css_selectors, full xpath, id, classname but none of them really work. All return the same error like mentioned above.

Here's how I have written it in my python script:

        youtube_subscribers = browser.find_element_by_xpath('//*[@id="subscriber-count"]').text
        youtube_location = browser.find_element_by_xpath('//*[@id="details-container"]/table/tbody/tr[2]/td[2]/yt-formatted-string').text
        youtube_joined_on = browser.find_element_by_xpath('//*[@id="right-column"]/yt-formatted-string[2]/span[2]').text
        youtube_views = browser.find_element_by_xpath('//*[@id="right-column"]/yt-formatted-string[3]').text

        print('Youtube Subscribers:', youtube_subscribers)
        print('Youtube Location:', youtube_location)
        print('Youtube Joined on:', youtube_joined_on)
        print('Youtube views:', youtube_views)

I am scraping from here https://www.youtube.com/c/adidas/about. Where exactly am I going wrong? Please help!

EDIT: Here's the full code for the same.

website = ['https://www.pinterest.com/adidas/', 'https://www.pinterest.com/nike/', 'https://www.pinterest.com/puma/']
    options = webdriver.ChromeOptions()
    options.add_argument('start-maximized')
    options.add_experimental_option("excludeSwitches", ["enable-automation"])
    options.add_experimental_option("useAutomationExtension", False)

    browser = webdriver.Chrome(ChromeDriverManager().install(), options=options)
    delays = [7, 4, 6, 2, 10, 19]
    delay = np.random.choice(delays)
    for crawler in website:
        browser.get(crawler)
        time.sleep(2)
        time.sleep(delay)
        pinterest_brand_name = browser.find_element_by_xpath('/html/body/div[1]/div[1]/div/div/div[1]/div[1]/div[2]/div/div/div/div[2]/div/div/h1').text
        pinterest_followers = browser.find_element_by_xpath('/html/body/div[1]/div[1]/div/div/div[1]/div[1]/div[2]/div/div/div/div[2]/div/div/div[2]/div/span[1]').text
        pinterest_following = browser.find_element_by_xpath('/html/body/div[1]/div[1]/div/div/div[1]/div[1]/div[2]/div/div/div/div[2]/div/div/div[2]/div/div[1]/span[1]').text
        youtube_subscribers = browser.find_element_by_xpath('//*[@id="subscriber-count"]').text
        youtube_location = browser.find_element_by_xpath('//*[@id="details-container"]/table/tbody/tr[2]/td[2]/yt-formatted-string').text
        youtube_joined_on = browser.find_element_by_xpath('//*[@id="right-column"]/yt-formatted-string[2]/span[2]').text
        youtube_views = browser.find_element_by_xpath('//*[@id="right-column"]/yt-formatted-string[3]').text

        print('Pinterest Brand Name:', pinterest_brand_name)
        print('Pinterest Followers:', pinterest_followers)
        print('Pinterest Following:', pinterest_following)
        print('Youtube Subscribers:', youtube_subscribers)
        print('Youtube Location:', youtube_location)
        print('Youtube Joined on:', youtube_joined_on)
        print('Youtube views:', youtube_views)

Answer 1

I can't see the rest of your code but assuming all those elements are on the same page, and it's failing on the first one, you probably just need to add a wait command in there for that first element to be visible.

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

youtube_subscribers = WebDriverWait(browser, 10).until(EC.presence_of_element_located((By.XPATH,'//*[@id="subscriber-count"]')))

Assuming all the elements you want are on that page, you should only need the wait on that one above. You can play around with it though to see what works

Updated code

The below should work, split the loop into 2 parts so it won't fail trying to find YouTube web elements on Pinterest sites, and vice versa...

website = ['https://www.pinterest.com/adidas/', 'https://www.pinterest.com/nike/', 'https://www.pinterest.com/puma/',
'https://www.youtube.com/c/adidas/about']

for crawler in website:
    if "pinterest" in crawler:
        browser.get(crawler)
        sleep(3)
        #time.sleep(delay)
        pinterest_brand_name = browser.find_element_by_xpath('/html/body/div[1]/div[1]/div/div/div[1]/div[1]/div[2]/div/div/div/div[2]/div/div/h1').text
        pinterest_followers = browser.find_element_by_xpath('/html/body/div[1]/div[1]/div/div/div[1]/div[1]/div[2]/div/div/div/div[2]/div/div/div[2]/div/span[1]').text
        pinterest_following = browser.find_element_by_xpath('/html/body/div[1]/div[1]/div/div/div[1]/div[1]/div[2]/div/div/div/div[2]/div/div/div[2]/div/div[1]/span[1]').text
        print('Pinterest Brand Name:', pinterest_brand_name)
        print('Pinterest Followers:', pinterest_followers)
        print('Pinterest Following:', pinterest_following)
    elif "youtube" in crawler:
        browser.get(crawler)
        sleep(3)
        #time.sleep(delay)
        youtube_subscribers = browser.find_element_by_xpath('//*[@id="subscriber-count"]').text
        youtube_location = browser.find_element_by_xpath('//*[@id="details-container"]/table/tbody/tr[2]/td[2]/yt-formatted-string').text
        youtube_joined_on = browser.find_element_by_xpath('//*[@id="right-column"]/yt-formatted-string[2]/span[2]').text
        youtube_views = browser.find_element_by_xpath('//*[@id="right-column"]/yt-formatted-string[3]').text
        print('Youtube Subscribers:', youtube_subscribers)
        print('Youtube Location:', youtube_location)
        print('Youtube Joined on:', youtube_joined_on)
        print('Youtube views:', youtube_views)

Results:

Pinterest Brand Name: adidas
Pinterest Followers: 623,732
Pinterest Following: 9
Pinterest Brand Name: Nike
Pinterest Followers: 765,138
Pinterest Following: 3
Pinterest Brand Name: PUMA
Pinterest Followers: 88,759
Pinterest Following: 280
Youtube Subscribers: 925K subscribers
Youtube Location: United States
Youtube Joined on: Oct 29, 2005
Youtube views: 168,725,975 views