CODEWITHSUNDEEP
pythonseleniumselenium-webdriverweb-scrapinggeckodriver
Neil RT
Neil RT

Reputation: 3

How to scrape an updating HTML table using Selenium?

I am looking to scrape the coin table from link and create a CSV file datewise. For every new coin update, a new entry at the top should be created in the existing data file.

Desired output

Coin,Pings,...Datetime

BTC,25,...07:17:05 03/18/21

I haven't reached far but below is my attempt at it

from selenium import webdriver
import numpy as np
import pandas as pd

firefox = webdriver.Firefox(executable_path="/usr/local/bin/geckodriver")
firefox.get('https://agile-cliffs-23967.herokuapp.com/binance/')

rows = len(firefox.find_elements_by_xpath("/html/body/div/section[2]/div/div/div/div/table/tr"))
columns = len(firefox.find_elements_by_xpath("/html/body/div/section[2]/div/div/div/div/table/tr[1]/th"))

df = pd.DataFrame(columns=['Coin','Pings','Net Vol BTC','Net Vol per','Recent Total Vol BTC', 'Recent Vol per', 'Recent Net Vol', 'Datetime'])

for r in range(1, rows+1):
    for c in range(1, columns+1): 
        value = firefox.find_element_by_xpath("/html/body/div/section[2]/div/div/div/div/table/tr["+str(r)+"]/th["+str(c)+"]").text
        print(value)
        
#         df.loc[i, ['Coin']] =

Upvotes: 0

Views: 211

Answers (2)

RJ Adriaansen
RJ Adriaansen

Reputation: 9639

Since the data is loaded dynamically you can retrieve it directly from the source, no Selenium needed. It will return json with rows with |-delimited values that need to be split and can be appended to the DataFrame. Since the site updates once per minute, you can wrap everything in a while True that makes the code run every 60 seconds:

import requests
import time
import json
import pandas as pd

headers = ['Coin','Pings','Net Vol BTC','Net Vol %','Recent Total Vol BTC', 'Recent Vol %', 'Recent Net Vol', 'Datetime (UTC)']
df = pd.DataFrame(columns=headers)

s = requests.Session()
starttime = time.time()

while True:
    response = s.get('https://agile-cliffs-23967.herokuapp.com/ok', headers={'Connection': 'keep-alive'})
    d = json.loads(response.text)
    rows = [str(i).split('|') for i in d['resu'][:-1]]
    if rows:
        data = [dict(zip(headers, l)) for l in rows]
        df = df.append(data, ignore_index=True)
        df.to_csv('filename.csv', index=False)
    time.sleep(60.0 - ((time.time() - starttime) % 60.0))

Upvotes: 1

joao
joao

Reputation: 2293

You can append row data to a DataFrame by putting it into a dictionary:

# We reuse the headers when building dicts below
headers = ['Coin','Pings','Net Vol BTC','Net Vol per','Recent Total Vol BTC', 'Recent Vol per', 'Recent Net Vol', 'Datetime']
df = pd.DataFrame(columns=headers)

for r in range(1, rows+1):
    data = [firefox.find_element_by_xpath("/html/body/div/section[2]/div/div/div/div/table/tr["+str(r)+"]/th["+str(c)+"]").text \
                for c in range(1, columns+1)]
    row_dict = dict(zip(headers, data))
    df = df.append(row_dict, ignore_index=True)

Upvotes: 0

Related Questions