CODEWITHSUNDEEP
selenium-webdriverpython-3.6
dwebb
dwebb

Reputation: 63

Get complete data Table scraping using selenium

I want to scrape data from the king of the hill table and return accurate data. My code returns inaccurate data. I need to get (e.g. for row 1). I only want to use Selenium not Json.

1. los angeles lebrons 2 0 0

1.000

2

from selenium import webdriver
 driver=webdriver.Chrome(executable_path="D:\Programs\Programs\chromedriver\chromedriver.exe")
 driver.get("https://fantasy.espn.com/basketball/league/standings?leagueId=1878319")
rows=len(driver.find_elements_by_xpath('//*[@id="espn-analytics"]/div/div[5]/div[2]/div[1]/div/div/div[3]/div/section/table/tbody/tr/td/div/div/div[2]/table/tbody/tr/td/div/table/tbody/tr'))
>>> print(rows)
10
>>> cols=len(driver.find_elements_by_xpath('//*[@id="espn-analytics"]/div/div[5]/div[2]/div[1]/div/div/div[3]/div/section/table/tbody/tr/td/div/div/div[2]/table/tbody/tr/td/div/table/thead/tr/th'))
>>> print(cols)
7
>>> for r in range(2,rows+1):
    for c i range(1,cols+1):
value=driver.find_element_by_xpath('//*[@id="espn-analytics"]/div/div[5]/div[2]/div[1]/div/div/div[3]/div/section/table/tbody/tr/td/div/div/div[2]/table/tbody/tr/td/div/table/tbody/tr["+str(r)+"]/td["+str(c)+"]').text
        print(value,end='')
    print()

what i get is: 1111111 1111111 1111111 1111111 1111111 1111111 1111111 1111111 1111111

Upvotes: 0

Views: 321

Answers (1)

Sureshmani Kalirajan
Sureshmani Kalirajan

Reputation: 1938

The for loop is incorrect. You can follow the below example and it prints all the rows correctly.

from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait

driver=webdriver.Chrome(executable_path="D:\Programs\Programs\chromedriver\chromedriver.exe")
driver.get("https://fantasy.espn.com/basketball/league/standings?leagueId=1878319")
# wait until my page table is loaded
WebDriverWait(driver, 15).until(EC.element_to_be_clickable((By.CLASS_NAME, 'Table2__tbody')))

rows=len(driver.find_elements_by_xpath('//*[@id="espn-analytics"]/div/div[5]/div[2]/div[1]/div/div/div[3]/div/section/table/tbody/tr/td/div/div/div[2]/table/tbody/tr/td/div/table/tbody/tr'))
print(rows)
cols=len(driver.find_elements_by_xpath('//*[@id="espn-analytics"]/div/div[5]/div[2]/div[1]/div/div/div[3]/div/section/table/tbody/tr/td/div/div/div[2]/table/tbody/tr/td/div/table/thead/tr/th'))
print(cols)
for r in range(rows):
    values = driver.find_elements_by_xpath("(//*[@class='Table2__tbody'])[1]/tr["+str(r+1)+"]/td")
    print("Printing Row value#: "+str(r+1))
    for value in values:
        print(value.text)

Output:

Printing Row value#: 1
1
los angeles lebrons
3
0
0
1.000

--

Upvotes: 1

Related Questions