CODEWITHSUNDEEP
pythonpython-3.xseleniumselenium-webdriverweb-scraping
SMTH
SMTH

Reputation: 95

Script fails to produce descriptions from a webpage when hardcoded delay is not in place

I'm trying to fetch titles and descriptions of results from a webpage. The descriptions are revealed when the titles are clicked. The script below works only when I define a hardcoded delay after the click. However, I wish to get rid of the hardcoded delay from the script.

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

link = 'https://innovation.ised-isde.canada.ca/s/list-liste?language=en_CA&token=a0B5W000000WsFSUA0'

driver = webdriver.Chrome()
wait = WebDriverWait(driver,10)
driver.get(link)

for i,item in enumerate(wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,".advanced-results .h4")))):
    driver.execute_script("arguments[0].click();",item)
    title = item.text
    time.sleep(2)
    desc = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,"p.program-dov-description")))[i].text
    print(title,desc)
    print()

driver.quit()

How to fetch titles and descriptions from that webpage without using hardcoded delay?

Upvotes: 0

Views: 57

Answers (2)

Arundeep Chohan
Arundeep Chohan

Reputation: 9969

wait=WebDriverWait(driver,30)                                 
driver.get('https://innovation.ised-isde.canada.ca/s/list-liste?language=en_CA&token=a0B5W000000WsFSUA0')
elements=wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"div.cIC_DraggableList.cIC_Dovs > div")))
for x in elements:
    x.click()
    time.sleep(1)
title=[x.get_attribute("data-dovtitle") for x in wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"div.row.mh-0 button:nth-child(2)")))]
desc=[x.text for x in wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"p.program-dov-description")))]
    
print(list(zip(title,desc)))

So just click and open each one and collect their title and desc.

Imports:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC

Outputs:

[('Canada Recovery Sickness Benefit', 'Get up to $500 per week for six weeks if you are employed or self-employed and need to take time off because you’re sick or need to self-isolate due to COVID-19 or have an underlying medical condition that puts you at higher risks of getting COVID-19.'), ('Canada Recovery Caregiver Benefit', 'Get up to $500 per week for up to 44 weeks, per household, if you haven’t been able to work for at least 50% of your normal work week because you are caring for children under 12 years old or other dependent family members because of COVID-19.'), ('Work-Sharing program — COVID-19', 'If your employees are eligible for Employment Insurance (EI) benefits, and your business is suffering a downturn due to COVID-19, you can apply for a work-sharing agreement. The agreement would allow your employees to work a temporarily reduced work week while receiving EI benefits.'), ('Workplace screening of employees', 'Get free rapid tests to help identify employees who may be infected with COVID-19 but have no symptoms. Rapid tests are safe, easy to administer and provide results in 15 minutes.'), ('Canada Emergency Rent Subsidy (CERS)', 'If your business has been impacted by COVID-19, you could get a retroactive rent subsidy that covers up to 65% of eligible expenses to a maximum of $75,000 per business location, and $300,000 in total for all locations, with additional amounts for temporary closures.'), ('BDC Co-Lending Program (BCAP)', 'If your business has been affected by COVID-19, you could get up to $12.5 million to support your operational cash flow, co-financed by BDC and your primary lending institution.'), ('Essential Services Contingency Reserve', 'If you’re an essential business and can’t get personal protective equipment for your workers through other means, you can purchase from Canada’s reserve on a cost-recovery basis.'), ('Accelerate HER PEI', "Take part in a 12-week pilot program (virtually or in person) that includes one-on-one mentorship, peer support, market research guidance and more. Learn how to grow your business whether you're in the early stage of development or taking your business in a new direction."), ('Indigenous Community Business Fund — Stream 2', 'Get a non-repayable contribution to cover eligible operating costs if your Indigenous business has experienced a significant reduction in revenues due to COVID-19.'), ('COVID-19: Waiving tariffs on certain medical goods', 'If your business imported certain medical items into Canada on or after May 5, 2020, including PPE (masks, gloves, garments, sterilization products), you will not have to pay tariffs on those items.'), ('Mortgage Payment Deferral', 'If you have experienced financial hardship due to COVID-19, you may be eligible for deferred payments on your insured home mortgage, for a specified amount of time, through your lender.'), ('COVID-19 Support: Futurpreneur Canada', 'If you received a start-up loan from Futurpreneur prior to April 1, 2020, your payments will be covered from March 25 through to September 24, 2020. If you do not qualify for the Canadian Emergency Business Account, you can top up your loan to a maximum of $10,000, interest-free, with one year of deferred repayment, to help cover operating costs during this challenging period.'), ('COVID-19: ExploreIP', 'Search this database for available patents related to COVID-19 for licensing or collaboration opportunities with public sector patent holders. Filter your search results by technology, expertise, or location.'), ('Community Futures Network of Canada', 'If your business is located in a rural community, and doesn’t qualify for the federal wage subsidy or other supports, you may be eligible for financial assistance through your Community Futures Network Canada.'), ('Regional development agencies: COVID-19 support', 'If the COVID-19 pandemic is affecting your operations, contact your local Regional Development Agency office to get a review of your situation and guidance on getting funding, support and/or flexible arrangements, on a case-by-case basis.'), ('Smart Renewables and Electrification Pathways Program', 'Get a repayable contribution of up to $50 million for a project that involves renewable energy, an emerging technology, or grid modernization, and covers up to 10%, 30% and 50% of the project costs for each of these 3 streams, respectively.'), ('COVID-19 and Your Business newsletter', 'Subscribe to receive timely information from the federal government on keeping your workers safe, receiving your vaccine and finding the support you need.'), ('PPE Supply Hub', 'Whether you’re selling personal protective equipment (PPE) during the COVID-19 pandemic, or looking for PPE to protect your workers, you can use this hub that connects buyers and sellers.'), ('BDC Capital Bridge Financing Program', 'If you operate a high-growth company that has been specifically impacted by COVID-19, you could get matched financing of at least $500,000 in the form of a convertible note if you are already backed by a qualified venture capital firm and have raised at least $500,000 in external capital.'), ('Temporary Wage Subsidy for Employers', 'Get a wage subsidy of up to $25,000 to cover 10% of the remuneration you paid from March 18, 2020 to June 19, 2020, up to $1,375 per eligible employee.'), ('Innovation Canada — Innovation Advisors', "If you're developing a new product, technology or process, an Innovation Advisor can provide you with free, one-on-one assistance to help you find the government supports that are right for your business. Our advisors are former entrepreneurs with established networks in both private and public sectors, and can help put you in touch with specialists in finance, technical advice and foreign market expertise, to help your business grow from here. Innovation Advisors help companies that are incorporated for profit in Canada. Is this you?")]

Upvotes: 0

Anand Gautam
Anand Gautam

Reputation: 2101

This should work. The thing is I couldn't figure out how to get multiple elements with css_selector, so I trusted my old friend, the xpath, and so I fetched the result, without using time.sleep as required. I get an error with this line of yours, stating that the element is not subscriptable because of [i], but my xpath works.

desc = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,"p.program-dov-description")))[i].text
    print(title,desc)

Here is the refactored code:

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from webdriver_manager.chrome import ChromeDriverManager

link = 'https://innovation.ised-isde.canada.ca/s/list-liste?language=en_CA&token=a0B5W000000WsFSUA0'

driver = webdriver.Chrome(ChromeDriverManager().install())
wait = WebDriverWait(driver,10)
driver.get(link)

for i,item in enumerate(wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,".advanced-results .h4")))):
    # print(i, item)
    driver.execute_script("arguments[0].click();",item)
    title = item.text
    desc_text = "//*[@class='program-dov-description']"
    desc = wait.until(EC.visibility_of_element_located((By.XPATH,'(' + desc_text + ')' + '[' + str(i+1) + ']' ))).text
    print(title,desc)
    print()

driver.quit()

Output (pasting partial output to save space)

Income support for sick or self-isolating workers Get up to $500 per week for six weeks if you are employed or self-employed and need to take time off because you’re sick or need to self-isolate due to COVID-19 or have an underlying medical condition that puts you at higher risks of getting COVID-19.

Income support to care for dependent family members Get up to $500 per week for up to 44 weeks, per household, if you haven’t been able to work for at least 50% of your normal work week because you are caring for children under 12 years old or other dependent family members because of COVID-19.

Support for businesses to avoid layoffs If your employees are eligible for Employment Insurance (EI) benefits, and your business is suffering a downturn due to COVID-19, you can apply for a work-sharing agreement. The agreement would allow your employees to work a temporarily reduced work week while receiving EI benefits.

Upvotes: 1

Related Questions