CODEWITHSUNDEEP
javascriptpythonweb-scraping
abc
abc

Reputation: 13

Fail to scrap the text located in such case but others work?

I was building scrapper to get the information of products.

I would like the get the description text about delivery as shwon in this screenshot (Sorry reputation is not high enough to add pictures yet). The text highlighted in blue is the desired information to obtain. The code I am using is

reg = wait(driver, 2).until(EC.presence_of_element_located((By.TAG_NAME, "pns-product-pickup")))
methods = wait(reg, 1).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "div.delivery-options")))
delivery=''
    
for method in methods:
    method1 = method.text
    info = wait(method, 1).until(EC.presence_of_element_located((By.CSS_SELECTOR, "div.option-tips"))).text
    delivery=delivery+method1+info+'\n'

print(delivery)

The result I get is

Home Delivery
Click & Collect

Which means all the 'info' I add to 'delivery' is empty text, only 'method1' did return something.

I was wondering if it is the case that, the text I want is not shown on web unless I hold the mouse on the icon next to 'home delivery' like I was doing in the screenshot. Yet it is just a random guess and probably make no sense. Any idea of what might be wrong? Thanks in advance!

Upvotes: 0

Views: 54

Answers (1)

Barry the Platipus
Barry the Platipus

Reputation: 10460

You will be able to access the delivery info, if you navigate to that element first (hover over it). THe following will get the home delivery info:

### chromedriver setup for linux
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
import time as t

chrome_options = Options()
chrome_options.add_argument("--no-sandbox")


webdriver_service = Service("chromedriver/chromedriver") ## path to where you saved chromedriver binary
browser = webdriver.Chrome(service=webdriver_service, options=chrome_options)

actions = ActionChains(browser)

url='https://parknshop.com/en/full-cream-evaporated-milk/p/BP_491777'
browser.get(url)

t.sleep(5) 
pns_more_info = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.TAG_NAME, "pns-more-information")))
actions.move_to_element(pns_more_info)
actions.perform()
tips = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, "//div[@class='option-tips']")))
print(tips.text)

This will print out:

Free delivery on orders above $500. For orders below $500, delivery fee of $50 shall apply

Do not forget to import ActionChains.

Upvotes: 1

Related Questions