CODEWITHSUNDEEP
pythonpython-3.xseleniumselenium-webdriver
reggie jones
reggie jones

Reputation: 99

Have Selenium Do the same process for multiple URLS

I created a script that goes to a property auction site that monitors the price and alerts me of any of the changes. I originally made it to monitor 1 url at a time. I now need it to monitor more then 1 url and do the same exact process at the same time.

I've tried looping the urls in browser.get but that doesn't seem to work for me(idk if i wrote it wrong, probably did). But i need it to either, open a tun of new tabs and go to the urls, or another way for it to monitor everything.

The Original Script for 1 property url, works perfectly, And the only difference between that and the code below is browser.get.

PS: If script is run, and auction urls are expired (meaning theres no price it can look at) there will be a error

import time
import requests
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import NoSuchElementException
import ssl
from twilio.rest import Client
from twilio.rest import TwilioRestClient
from bs4 import BeautifulSoup as soup

urls = ['https://www.hubzu.com/property/90089436968-730-Trillium-Ln-Lilburn-GA-30047',
'https://www.hubzu.com/property/90016097522-225-Marriott-Ave-Schenectady-NY-12304',
'https://www.hubzu.com/property/90016098285-17-Spring-Meadows-Dr-Ormond-Beach-FL-32174'
]

browser = webdriver.Chrome()

# Start URL switch tryouts.

while True:
    for url in urls:
        browser.get((url))


       #This is the process i need all urls to do.
        time.sleep(2)
        address = soup(browser.page_source, 'html.parser').find('span', {'class':'h1'}).text
        propertyprice = browser.find_element_by_css_selector('span.current-bid')
        currentBidText = propertyprice.text
        try:                                
            WebDriverWait(browser, 90000).until_not(
                EC.text_to_be_present_in_element((By.CSS_SELECTOR, 'span.current-bid'), currentBidText)
                )
        finally:
            print("+++ Send notifications.")
            account_sid = "***"
            auth_token = "***"
            client = Client(account_sid, auth_token)

            PhoneNumber1 = "+***"
            PhoneNumber2 = "+***"
            print("+ Send notifications to: ", PhoneNumber1, " and ", PhoneNumber2)

            sendTo1 = "{\"binding_type\":\"sms\",\"address\":\"" + PhoneNumber1 + "\"}"
            print("+ sendTo1: ", sendTo1)
            sendTo2 = "{\"binding_type\":\"sms\",\"address\":\"" + PhoneNumber2 + "\"}"
            print("+ sendTo2: ", sendTo2)

            notify_service_sid = "***"
            notification = client.notify.services(notify_service_sid).notifications.create(
                    body='There has been a change at: '+address,
                    to_binding=[sendTo1, sendTo2]
                )

            print("+ Notification SID: ", notification.sid)

            print("+++ Exit.")
    continue

Upvotes: 0

Views: 1160

Answers (1)

Reedinationer
Reedinationer

Reputation: 5774

Your code seems to be mostly fine and cycles through them for me. Unfortunately all the properties are not for sale, so they all give me errors and I cannot produce the expected behaviour. I think you should implement a try: and except:, the finally: runs regardless of what happened, and I wouldn't imagine you want an update every 2 seconds XD. You also don't need continue at the end, it does nothing. The code

import time
import requests
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import NoSuchElementException
import ssl
from twilio.rest import Client
from twilio.rest import TwilioRestClient
from bs4 import BeautifulSoup as soup

urls = ['https://www.hubzu.com/property/90089436968-730-Trillium-Ln-Lilburn-GA-30047',
        'https://www.hubzu.com/property/90016097522-225-Marriott-Ave-Schenectady-NY-12304',
        'https://www.hubzu.com/property/90016098285-17-Spring-Meadows-Dr-Ormond-Beach-FL-32174']

browser = webdriver.Chrome()

# Start URL switch tryouts.

while True:
    for url in urls:
        browser.get(url)

        # This is the process i need all urls to do.
        time.sleep(2)
        address = soup(browser.page_source, 'html.parser').find('span', {'class': 'h1'}).text
        try:
            propertyprice = browser.find_element_by_css_selector('span.current-bid')
            currentBidText = propertyprice.text
            WebDriverWait(browser, 90000).until(
                EC.text_to_be_present_in_element((By.CSS_SELECTOR, 'span.current-bid'), currentBidText)
                )
        except Exception as e:
            print("an error occurred on property {}:\n{}".format(address.strip(), e))
        finally:
            pass
            # print("+++ Send notifications.")
            # account_sid = "***"
            # auth_token = "***"
            # client = Client(account_sid, auth_token)
            #
            # PhoneNumber1 = "+***"
            # PhoneNumber2 = "+***"
            # print("+ Send notifications to: ", PhoneNumber1, " and ", PhoneNumber2)
            #
            # sendTo1 = "{\"binding_type\":\"sms\",\"address\":\"" + PhoneNumber1 + "\"}"
            # print("+ sendTo1: ", sendTo1)
            # sendTo2 = "{\"binding_type\":\"sms\",\"address\":\"" + PhoneNumber2 + "\"}"
            # print("+ sendTo2: ", sendTo2)
            #
            # notify_service_sid = "***"
            # notification = client.notify.services(notify_service_sid).notifications.create(
            #         body='There has been a change at: '+address,
            #         to_binding=[sendTo1, sendTo2]
            #     )
            #
            # print("+ Notification SID: ", notification.sid)
            #
            # print("+++ Exit.")

Does cycle through all the properties, and a sample output for me is:

an error occurred on property 730
Trillium Ln

 Lilburn, GA 30047:
Message: no such element: Unable to locate element: {"method":"css selector","selector":"span.current-bid"}
  (Session info: chrome=74.0.3729.131)
  (Driver info: chromedriver=2.40.565498 (ea082db3280dd6843ebfb08a625e3eb905c4f5ab),platform=Windows NT 10.0.14393 x86_64)

an error occurred on property 225 Marriott Ave

 Schenectady, NY 12304:
Message: no such element: Unable to locate element: {"method":"css selector","selector":"span.current-bid"}
  (Session info: chrome=74.0.3729.131)
  (Driver info: chromedriver=2.40.565498 (ea082db3280dd6843ebfb08a625e3eb905c4f5ab),platform=Windows NT 10.0.14393 x86_64)

an error occurred on property 17 Spring Meadows Dr

 Ormond Beach, FL 32174:
Message: no such element: Unable to locate element: {"method":"css selector","selector":"span.current-bid"}
  (Session info: chrome=74.0.3729.131)
  (Driver info: chromedriver=2.40.565498 (ea082db3280dd6843ebfb08a625e3eb905c4f5ab),platform=Windows NT 10.0.14393 x86_64)

an error occurred on property 730
Trillium Ln

 Lilburn, GA 30047:
Message: no such element: Unable to locate element: {"method":"css selector","selector":"span.current-bid"}
  (Session info: chrome=74.0.3729.131)
  (Driver info: chromedriver=2.40.565498 (ea082db3280dd6843ebfb08a625e3eb905c4f5ab),platform=Windows NT 10.0.14393 x86_64)

an error occurred on property 225 Marriott Ave

 Schenectady, NY 12304:
Message: no such element: Unable to locate element: {"method":"css selector","selector":"span.current-bid"}
  (Session info: chrome=74.0.3729.131)
  (Driver info: chromedriver=2.40.565498 (ea082db3280dd6843ebfb08a625e3eb905c4f5ab),platform=Windows NT 10.0.14393 x86_64)

and so on in an infinite loop...

Upvotes: 1

Related Questions