CODEWITHSUNDEEP
pythonseleniumweb-scraping
Snyder Fox
Snyder Fox

Reputation: 174

Web scraping with selenium - 0 records returned

I am working with this code for a long time to solve my problem and cannot find a solution for this. Using Selenium, I have first logged on to the site and doing scraping. But its returning 0 values, which is not expected.

import pandas as pd
import requests
from bs4 import BeautifulSoup
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
# Github credentials
username = "[email protected]"
password = "xxxxxxxxxxxxxxxxx"
# initialize the Chrome driver
driver = webdriver.Chrome(executable_path=r'C:\Users\snyder\seaborn-data\chromedriver.exe')
driver.maximize_window()
driver.implicitly_wait(5)
driver.get("https://www.finq.com/en/login")
wait = WebDriverWait(driver, 10)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, "iframe[id='login']")))
# find username/email field and send the username itself to the input field
driver.find_element_by_id("login_username").send_keys(username)
# find password input field and insert password as well
driver.find_element_by_id("login_password").send_keys(password)
# click login button
#driver.find_element_by_name("submit").click()
driver.find_element_by_css_selector("button[id='submit_login']").click()
# wait the ready state to be complete
WebDriverWait(driver=driver, timeout=10).until(
    lambda x: x.execute_script("return document.readyState === 'complete'")
)
error_message = "Incorrect username or password."
# get the errors (if there are)
errors = driver.find_elements_by_class_name("flash-error")
# print the errors optionally
# for e in errors:
#     print(e.text)
# if we find that error message within errors, then login is failed
if any(error_message in e.text for e in errors):
    print("[!] Login failed")
else:
    print("[+] Login successful")

    
url = "https://live-cosmos.finq.com/trading-platform/#trading/Shares/Global/USA/All/FACEBOOK"
data  = requests.get(url).text
soup = BeautifulSoup(data, 'html5lib')
df = pd.DataFrame(columns=["Instrument", "Sell", "Buy", "Change"])
for row in soup.find_all('tr'):
    col = row.find_all("td")
    Instrument = col[0].text
    Sell = col[1].text
    Buy = col[2].text
    Change = col[3].text
    df = df.append({"Instrument":Instrument,"Sell":Sell,"Buy":Buy,"Change":Change}, ignore_index=True)
print(df)

After logged to the site, I am getting an another message which need to be closed. I gave the below condition for that too.

driver.find_element_by_css_selector("button[id='popup-actions fn-close-popup btn autotests__popup-btn-close']").click()

But it says

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"button[id='popup-actions fn-close-popup btn autotests__popup-btn-close']"}
  (Session info: chrome=92.0.4515.159)

Upvotes: 1

Views: 127

Answers (2)

Prophet
Prophet

Reputation: 33361

You are missing

driver.switch_to.default_content()

I see you switched to some iframe on login page to access some lements there, but after that to access elements out of that iframe you have to switch back to the default content.
UPD
To close the pop-up appearing after logging in you can use this command:

wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.fn-close-popup.popup-actions"))).click()

Again, you have to switch to the default content after clicking the login button with

driver.switch_to.default_content()

Upvotes: 1

gangabass
gangabass

Reputation: 10666

If you have "Login successful" then you need to replace:

data  = requests.get(url).text

with

driver.get(url)
time.sleep(3) # import time
data = driver.page_source

Or you need to pass cookies from your Selenium driver to requests headers.

Upvotes: 1

Related Questions