CODEWITHSUNDEEP
pythonselenium
Teacup
Teacup

Reputation: 65

Selenium run very slow

from selenium import webdriver
import sys, time

def main():
    url = sys.argv[1]
    sku_t = 'skuId='
    sku = url[url.index(sku_t)+len(sku_t):]

    chrome_options = webdriver.ChromeOptions()
    chrome_options.headless = True
    chrome_options.add_argument('--window-size=1920,1080')
    prefs = {"profile.managed_default_content_settings.images": 2}
    chrome_options.add_experimental_option("prefs", prefs)
    driver = webdriver.Chrome(options=chrome_options)
    start_time = time.time()
    driver.get(url)
    print("--- %s seconds ---" %(time.time() - start_time))

    css_param = 'button.add-to-cart-button[data-sku-id="'+sku+'"]'
    add_to_cart_btn = driver.find_element_by_css_selector(css_param)
    print(add_to_cart_btn.text)
    driver.quit()

if __name__ == '__main__':
    main()

I am trying to write a python script to check BestBuy inventory (to checkout later if possible). It takes around 90s to finish. I am not sure if there is something wrong with how I use selenium or is there anything I can do to improve the performance. Below is the log. Thank you in advance

DevTools listening on ws://127.0.0.1:62866/devtools/browser/fb0dfd00-0aee-4bec-9e6e-8411e18593d6
[0130/193519.329:INFO:CONSOLE(127)] "bby_privacy: getServerComponent", source: https://nexus.ensighten.com/bestbuy/privacy_init/Bootstrap.js (127)
[0130/193519.660:INFO:CONSOLE(143)] "getServerComponent suppressed", source: https://nexus.ensighten.com/bestbuy/privacy_prod/Bootstrap.js (143)
[0130/193529.699:INFO:CONSOLE(0)] "Access to script at 'https://assets.bbystatic.com/shop/master-ad/dist/client/client-978b7e491e0889e92a05356af57308d2.mjs' from origin 'https://www.bestbuy.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.", source: https://www.bestbuy.com/site/xfx-amd-radeon-rx-6800-16gb-gddr6-pci-express-4-0-gaming-graphics-card-black/6442077.p?skuId=6442077 (0)
[0130/193535.454:INFO:CONSOLE(1)] "Uncaught (in promise) t: request to /button-state/api/v5/button-state?skus=6442077&conditions=&storeId=&destinationZipCode=&context=pdp&consolidated=false&source=buttonView&xboxAllAccess=false timed out after 5000ms", source: https://www.bestbuy.com/~assets/bby/_com/fulfillment/fulfillment-summary/dist/client/client-51a75235b1a89815b2bb615b099f1639.js (1)
[0130/193539.609:INFO:CONSOLE(0)] "Access to script at 'https://assets.bbystatic.com/shop/display-ad/dist/client/client-72a4443f23ae7927233dfaa3239ddc89.mjs' from origin 'https://www.bestbuy.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.", source: https://www.bestbuy.com/site/xfx-amd-radeon-rx-6800-16gb-gddr6-pci-express-4-0-gaming-graphics-card-black/6442077.p?skuId=6442077 (0)
[0130/193539.791:INFO:CONSOLE(718)] "LibEmbeddedComponentRenderer: Embedded component queries can only contain strings, numbers, booleans or CsiVar objects with non-empty name strings.
The query param object key "locale" contains unsupported value:  undefined", source: https://www.bestbuy.com/~assets/bby/_com/gvp-287c65223d1c49e3d4181939b6c98ef4.min.js (718)
[0130/193540.756:INFO:CONSOLE(8)] "firing initData from parent", source: https://www.bestbuy.com/~assets/bby/_com/analytics-dotcom/chunks/tagIframe-6b64f0d171878bd514b6.js (8)
[0130/193540.758:INFO:CONSOLE(6)] "starting gate fired!", source: https://nexus.ensighten.com/bestbuy/tagframe_dev/code/d0f9a4f369cc4aace811197f11e3a9b9.js?conditionId0=1195706 (6)
[0130/193541.920:INFO:CONSOLE(14)] "canary version not set", source: https://www.bestbuy.com/~assets/bby/_com/@ds/unified-chat/dist/javascript/unified-chat-init-f92615d5.js (14)
[0130/193558.181:INFO:CONSOLE(1)] "Failed to pass the CSI variable to account libTypeError: Cannot read property 'innerText' of undefined", source: https://www.bestbuy.com/~assets/bby/_com/shop/header/dist/client/client-current-6040b166d0f69756a5b4d62f14255057.mjs (1)
[0130/193558.593:INFO:CONSOLE(14)] "Dispatching button is sold out for  - 233--null", source: https://assets.bbystatic.com/fulfillment/add-to-cart-button/dist/client/client-807bf7e58728fff64dda829e243a6d4e.js (14)
[0130/193558.627:INFO:CONSOLE(14)] "Dispatching button is sold out for  - 233-77001-null", source: https://assets.bbystatic.com/fulfillment/add-to-cart-button/dist/client/client-807bf7e58728fff64dda829e243a6d4e.js (14)
[0130/193603.621:INFO:CONSOLE(1)] "Uncaught (in promise) t: request to /fulfillment/ispu/api/ispu/v2 timed out after 5000ms", source: https://www.bestbuy.com/~assets/bby/_com/fulfillment/fulfillment-summary/dist/client/client-51a75235b1a89815b2bb615b099f1639.js (1)
[0130/193603.655:INFO:CONSOLE(1)] "Uncaught (in promise) t: request to /fulfillment/shipping/api/v1/fulfillment/sku;skuId=6442077;postalCode=77001;deliveryDateOption=EARLIEST_AVAILABLE_DATE timed out after 5000ms", source: https://www.bestbuy.com/~assets/bby/_com/fulfillment/fulfillment-summary/dist/client/client-51a75235b1a89815b2bb615b099f1639.js (1)

Upvotes: 0

Views: 753

Answers (1)

Mitchell Olislagers
Mitchell Olislagers

Reputation: 1817

There are API's available specifically for Best Buy. If you don't want to use an API, try including these chrome_options

chrome_options.add_argument('--no-proxy-server')
chrome_options.add_argument("--proxy-server='direct://'")
chrome_options.add_argument("--proxy-bypass-list=*")
chrome_options.add_argument('--blink-settings=imagesEnabled=false')

If these don't work, try to run the script without headless mode. Additionally, you don't actually need selenium to scrape the availability and prices, as these are stored in the page source. So you can alternatively use requests to check for the availability of an item and only invoke selenium if an item is available and you want to add it to your cart.

Upvotes: 1

Related Questions