CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsouppython-requests
Josh Murphy
Josh Murphy

Reputation: 11

I can't seem to scrape hrefs or text from web

I've tried various methods including searching through full xpath and finding via class then search for the href via the a tag. I've been at it for hours and I just cant seem to get it. Any help would be greatly appreciated.

from selenium import webdriver 
import time
namelist=[]
driver=webdriver.Chrome()
driver.get("https://waxpeer.com/")
time.sleep(15)

when i use .text it just doesn't work

search=driver.find_elements_by_xpath('//*[@id="container"]/div/div/a')
print(search)
namelist.append(search)

Upvotes: 0

Views: 65

Answers (1)

Bhavya Parikh
Bhavya Parikh

Reputation: 3400

Hey you can use time module give delay for loading the elements so it can find the associate tag

As you mention from this code you can extract href links from div tag container

Code:

from selenium import webdriver 
import time
path="C:\Program Files (x86)\chromedriver.exe"
driver=webdriver.Chrome(path)
driver.get("https://waxpeer.com/")
time.sleep(10)
main_div=driver.find_elements_by_xpath('//*[@id="container"]')
for divs in main_div:
    links=div.find_elements_by_tag_name("a")
    for link in links:
        print(link.get_attribute('href'))

Output:

https://waxpeer.com/sport-gloves-vice-field-tested/item/21642893513 ...

Upvotes: 1

Related Questions