CODEWITHSUNDEEP
pythonseleniumweb-scrapingbeautifulsoup
Martyn Bell
Martyn Bell

Reputation: 97

issue with scraping using beautiful soup and selenium using python

Im trying to practice using selenium / BS4 and ive ran into an issue.

my code is here link to code

basically im trying to get a url from:

`'class':'td_text_highlight_marker_green td_text_highlight_marker'`

but whenever i run the script the returned number of links is always different.

everytime it runs it should return 18. I even tried to click the link to follow it using selenium and for the life of me i cannot figure it out.

please help :)

Upvotes: 0

Views: 54

Answers (1)

0buz
0buz

Reputation: 3503

The webpage DOM does not load fully in time for your search. So to ensure the span is actually there before you start looking for it with BeautifulSoup, add a WebDriverWait.

for coupon_url in coupon_url_list:
    driver.get(coupon_url)
    WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//span[@class='td_text_highlight_marker_green td_text_highlight_marker']")))
    content = driver.page_source
    # rest of your code here

Upvotes: 1

Related Questions