CODEWITHSUNDEEP
pythonseleniumweb-scraping
Henil Shah
Henil Shah

Reputation: 137

Fetching table data from webpage using selenium in python

I am very new to web scraping. I have the following url:

https://www.bloomberg.com/markets/symbolsearch

So, I use Selenium to enter the Symbol Textbox and press Find Symbols to get the details. This is the code:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

driver = webdriver.Firefox()
driver.get("https://www.bloomberg.com/markets/symbolsearch/")
element = driver.find_element_by_id("query")
element.send_keys("WMT:US")
driver.find_element_by_name("commit").click()

It returns the table. How can I retrieve that? I am pretty clueless.

Second question, Can I do this without Selenium as it is slowing down things? Is there a way to find an API which returns a JSON?

Upvotes: 0

Views: 1110

Answers (1)

Exprator
Exprator

Reputation: 27503

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
from bs4 import BeautifulSoup
import requests
driver = webdriver.Firefox()
driver.get("https://www.bloomberg.com/markets/symbolsearch/")
element = driver.find_element_by_id("query")
element.send_keys("WMT:US")
driver.find_element_by_name("commit").click()
time.sleep(5)
url = driver.current_url
time.sleep(5)

parsed = requests.get(url)


soup = BeautifulSoup(parsed.content,'html.parser')
a = soup.findAll("table", { "class" : "dual_border_data_table" })

print(a)

here is the total code by which you can get the table you are looking for. now do what you need to do after getting the table. hope it helps

Upvotes: 1

Related Questions