Reputation: 521
I am trying to scrape data from Youtube using selenium python. The data which I am scraping has fields like - subscribers, location, joined on and views. Despite giving the correct xpath I get an error like this
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="subscriber-count"]"}
(Session info: chrome=90.0.4430.93)
Things that I've tried are- working with css_selectors, full xpath, id, classname but none of them really work. All return the same error like mentioned above.
Here's how I have written it in my python script:
youtube_subscribers = browser.find_element_by_xpath('//*[@id="subscriber-count"]').text
youtube_location = browser.find_element_by_xpath('//*[@id="details-container"]/table/tbody/tr[2]/td[2]/yt-formatted-string').text
youtube_joined_on = browser.find_element_by_xpath('//*[@id="right-column"]/yt-formatted-string[2]/span[2]').text
youtube_views = browser.find_element_by_xpath('//*[@id="right-column"]/yt-formatted-string[3]').text
print('Youtube Subscribers:', youtube_subscribers)
print('Youtube Location:', youtube_location)
print('Youtube Joined on:', youtube_joined_on)
print('Youtube views:', youtube_views)
I am scraping from here https://www.youtube.com/c/adidas/about
. Where exactly am I going wrong?
Please help!
EDIT: Here's the full code for the same.
website = ['https://www.pinterest.com/adidas/', 'https://www.pinterest.com/nike/', 'https://www.pinterest.com/puma/']
options = webdriver.ChromeOptions()
options.add_argument('start-maximized')
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option("useAutomationExtension", False)
browser = webdriver.Chrome(ChromeDriverManager().install(), options=options)
delays = [7, 4, 6, 2, 10, 19]
delay = np.random.choice(delays)
for crawler in website:
browser.get(crawler)
time.sleep(2)
time.sleep(delay)
pinterest_brand_name = browser.find_element_by_xpath('/html/body/div[1]/div[1]/div/div/div[1]/div[1]/div[2]/div/div/div/div[2]/div/div/h1').text
pinterest_followers = browser.find_element_by_xpath('/html/body/div[1]/div[1]/div/div/div[1]/div[1]/div[2]/div/div/div/div[2]/div/div/div[2]/div/span[1]').text
pinterest_following = browser.find_element_by_xpath('/html/body/div[1]/div[1]/div/div/div[1]/div[1]/div[2]/div/div/div/div[2]/div/div/div[2]/div/div[1]/span[1]').text
youtube_subscribers = browser.find_element_by_xpath('//*[@id="subscriber-count"]').text
youtube_location = browser.find_element_by_xpath('//*[@id="details-container"]/table/tbody/tr[2]/td[2]/yt-formatted-string').text
youtube_joined_on = browser.find_element_by_xpath('//*[@id="right-column"]/yt-formatted-string[2]/span[2]').text
youtube_views = browser.find_element_by_xpath('//*[@id="right-column"]/yt-formatted-string[3]').text
print('Pinterest Brand Name:', pinterest_brand_name)
print('Pinterest Followers:', pinterest_followers)
print('Pinterest Following:', pinterest_following)
print('Youtube Subscribers:', youtube_subscribers)
print('Youtube Location:', youtube_location)
print('Youtube Joined on:', youtube_joined_on)
print('Youtube views:', youtube_views)
Upvotes: 0
Views: 351
Reputation: 3801
I can't see the rest of your code but assuming all those elements are on the same page, and it's failing on the first one, you probably just need to add a wait command in there for that first element to be visible.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
youtube_subscribers = WebDriverWait(browser, 10).until(EC.presence_of_element_located((By.XPATH,'//*[@id="subscriber-count"]')))
Assuming all the elements you want are on that page, you should only need the wait on that one above. You can play around with it though to see what works
Updated code
The below should work, split the loop into 2 parts so it won't fail trying to find YouTube web elements on Pinterest sites, and vice versa...
website = ['https://www.pinterest.com/adidas/', 'https://www.pinterest.com/nike/', 'https://www.pinterest.com/puma/',
'https://www.youtube.com/c/adidas/about']
for crawler in website:
if "pinterest" in crawler:
browser.get(crawler)
sleep(3)
#time.sleep(delay)
pinterest_brand_name = browser.find_element_by_xpath('/html/body/div[1]/div[1]/div/div/div[1]/div[1]/div[2]/div/div/div/div[2]/div/div/h1').text
pinterest_followers = browser.find_element_by_xpath('/html/body/div[1]/div[1]/div/div/div[1]/div[1]/div[2]/div/div/div/div[2]/div/div/div[2]/div/span[1]').text
pinterest_following = browser.find_element_by_xpath('/html/body/div[1]/div[1]/div/div/div[1]/div[1]/div[2]/div/div/div/div[2]/div/div/div[2]/div/div[1]/span[1]').text
print('Pinterest Brand Name:', pinterest_brand_name)
print('Pinterest Followers:', pinterest_followers)
print('Pinterest Following:', pinterest_following)
elif "youtube" in crawler:
browser.get(crawler)
sleep(3)
#time.sleep(delay)
youtube_subscribers = browser.find_element_by_xpath('//*[@id="subscriber-count"]').text
youtube_location = browser.find_element_by_xpath('//*[@id="details-container"]/table/tbody/tr[2]/td[2]/yt-formatted-string').text
youtube_joined_on = browser.find_element_by_xpath('//*[@id="right-column"]/yt-formatted-string[2]/span[2]').text
youtube_views = browser.find_element_by_xpath('//*[@id="right-column"]/yt-formatted-string[3]').text
print('Youtube Subscribers:', youtube_subscribers)
print('Youtube Location:', youtube_location)
print('Youtube Joined on:', youtube_joined_on)
print('Youtube views:', youtube_views)
Results:
Pinterest Brand Name: adidas
Pinterest Followers: 623,732
Pinterest Following: 9
Pinterest Brand Name: Nike
Pinterest Followers: 765,138
Pinterest Following: 3
Pinterest Brand Name: PUMA
Pinterest Followers: 88,759
Pinterest Following: 280
Youtube Subscribers: 925K subscribers
Youtube Location: United States
Youtube Joined on: Oct 29, 2005
Youtube views: 168,725,975 views
Upvotes: 1