CODEWITHSUNDEEP
pythonseleniumweb-scraping
utsav
utsav

Reputation: 7

Cannot scrape certain DIV tags using python selenium

I am trying to extract the articles (their titles/links) on a certain topic (for instance Machine learning) from this website. https://www.semanticscholar.org/search?q=machine%20learning&sort=relevance

The div tag that I need to access is which is nested under several other div tags.

This is what I have tried so far. I getting empty lists. Any help is appreciated.

import time
from selenium import webdriver

# Get all the paper url in the search result
def paper_crawler():
    driver = webdriver.Firefox('path')
    driver.get ('https://www.semanticscholar.org/search?q=machine%20learning&sort=relevance&fos=chemistry')
    result_counts = driver.find_elements_by_xpath('//*[@class="result-count"]')
    print(result_counts)
    for item in result_counts:
        count = item.text
        print(count)
    #search_result_urls = driver.find_elements_by_xpath('.//div[contains(@class,"result-page")]/article/header/div/a')
    search_result_urls = driver.find_elements_by_xpath('//*[@class="result-page"]/article/header/div/a')
    print(search_result_urls)
    for item in search_result_urls:
        paper_url =  item.get_attribute('href')
        print(paper_url)
    search_result_titles = driver.find_elements_by_xpath('//*[@class="result-page"]/article/header/div/a/span')
    for item in search_result_titles:
        paper_title = item.text
        print(paper_title)
    time.sleep(2)

if __name__ == '__main__':
    paper_crawler ()

Upvotes: 0

Views: 204

Answers (3)

αԋɱҽԃ αмєяιcαη
αԋɱҽԃ αмєяιcαη

Reputation: 11515

Use API better and make your life easier. parse whatever you want.

import requests


data = {
    "queryString": "machine learning",
    "page": 1,
    "pageSize": 10,
    "sort": "relevance",
    "authors": [],
    "coAuthors": [],
    "venues": [],
    "yearFilter": None,
    "requireViewablePdf": False,
    "publicationTypes": [],
    "externalContentTypes": []
}
r = requests.post(
    'https://www.semanticscholar.org/api/1/search', json=data).json()

print(r)

Upvotes: 1

undetected Selenium
undetected Selenium

Reputation: 193178

To extract the Title and HREF attributes of the articles you have to induce WebDriverWait for the visibility_of_all_elements_located() and you can use the following Locator Strategies:

  • Code Block:

    from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://www.semanticscholar.org/search?q=machine%20learning&sort=relevance&fos=chemistry')
my_titles = [my_elem.text for my_elem in WebDriverWait(driver, 5).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a[data-selenium-selector='title-link']>span")))]
my_hrefs = [my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a[data-selenium-selector='title-link']")))]
for i,j in zip(my_titles, my_hrefs):
    print("{} link is {}".format(i, j))
driver.quit()

  • Console Output:

    UCI Repository of Machine Learning Databases link is https://www.semanticscholar.org/paper/UCI-Repository-of-Machine-Learning-Databases-Blake/e068be31ded63600aea068eacd12931efd2a1029
Energy landscapes for machine learning. link is https://www.semanticscholar.org/paper/Energy-landscapes-for-machine-learning.-Ballard-Das/735d4099d3be0d919ddedb054043e6763205e0f7
Finding Nature′s Missing Ternary Oxide Compounds Using Machine Learning and Density Functional Theory. link is https://www.semanticscholar.org/paper/Finding-Nature%E2%80%B2s-Missing-Ternary-Oxide-Compounds-Hautier-Fischer/e3ab9e1162fc8f63d215dfdb21801ef5e1fde7b5
Distributed secure quantum machine learning link is https://www.semanticscholar.org/paper/Distributed-secure-quantum-machine-learning-Sheng-Zhou/ef944614bfc82b1dedfea19ff249a97ceea5ad90
Neural-Symbolic Machine Learning for Retrosynthesis and Reaction Prediction. link is https://www.semanticscholar.org/paper/Neural-Symbolic-Machine-Learning-for-Retrosynthesis-Segler-Waller/71cc9eefb17d7c4d1062162523b5fdad7ca66a2a
Transferable Machine-Learning Model of the Electron Density link is https://www.semanticscholar.org/paper/Transferable-Machine-Learning-Model-of-the-Electron-Grisafi-Fabrizio/f809258b65a00a06f9584e76620e6c6395cf81eb
Crystal structure representations for machine learning models of formation energies link is https://www.semanticscholar.org/paper/Crystal-structure-representations-for-machine-of-Faber-Lindmaa/1bdca98dc8c730ee92d5b19d2973a5bf461a500a
Machine learning for quantum mechanics in a nutshell link is https://www.semanticscholar.org/paper/Machine-learning-for-quantum-mechanics-in-a-Rupp/29b9ff8f4a26acc90e6182e1e749f15f688bc7cf
Machine-Learning-Augmented Chemisorption Model for CO2 Electroreduction Catalyst Screening. link is https://www.semanticscholar.org/paper/Machine-Learning-Augmented-Chemisorption-Model-for-Ma-Li/d6f30032c8fac43a8eabf2b67d2e84db6d3d0409
Adaptive machine learning framework to accelerate ab initio molecular dynamics link is https://www.semanticscholar.org/paper/Adaptive-machine-learning-framework-to-accelerate-Botu-Ramprasad/c9934d684fcc0b8ac6ed25b34d96e726cf2d7b99

Upvotes: 0

Hubert
Hubert

Reputation: 446

The page is loaded but not completely rendered when you start to search for the elements. A "time.sleep(5)" after

driver.get ('https://www.semanticscholar.org/search?q=machine%20learning&sort=relevance&fos=chemistry') should help as a quick workaround.

For a better, more robust solution you should wait for result_counts to be greater than 0 for a few seconds or the page being the error page (https://www.semanticscholar.org/search?q=learning333&sort=relevance&fos=chemistry).

Upvotes: 0

Related Questions