Download data from Yahoo! Finance

Question

Hi I recently took up a project about bitcoin analysis and need to download financial data from Yahoo! Finance via Python. I tried fix_yahoo_finance and pandas datareader but there seems to be a bug on the website when downloading files. It always miss some days. So I decided to use beautiful soup, the code is as follow:

import requests
import time
import pandas as pd
from bs4 import BeautifulSoup

def time_convert(dt):
    time.strptime(dt,'%Y-%m-%d %H:%M:%S')
    s = time.mktime(time.strptime(dt,'%Y-%m-%d %H:%M:%S'))
    return str(int(s))

s = requests.Session()
start = time_convert("2016-02-15 00:00:00")
end   = time_convert("2018-02-15 00:00:00")

r = s.get("https://uk.finance.yahoo.com/quote/BTC-USD/history?period1="+start+"&period2="+end+"&interval=1d&filter=history&frequency=1d"

soup = BeautifulSoup(r.text, 'lxml')
tables = soup.select('table')

df_list = []
for table in tables:
    df_list.append(pd.concat(pd.read_html(table.prettify())))
    df = pd.concat(df_list)
    df.to_excel("E:\PythonData\price_"+'.xlsx')

It works but the data is not complete because the website loads the data when your mouse scroll s down to the end of the page, but codes doesn't do that. How can I fix this?

Answer 1

Have you tries using Yahoo Financials? It's really well built and doesn;t scrap the webpages. It hashes out the data you want from the ["context"]["dispatcher"]["stores"] object. It's pretty fast and really well built.

$ pip install yahoofinancials

Usage Examples:

from yahoofinancials import YahooFinancials

tech_stocks = ['AAPL', 'MSFT', 'INTC']
bank_stocks = ['WFC', 'BAC', 'C']

yahoo_financials_tech = YahooFinancials(tech_stocks)
yahoo_financials_banks = YahooFinancials(bank_stocks)

tech_cash_flow_data_an = yahoo_financials_tech.get_financial_stmts('annual', 'cash')
bank_cash_flow_data_an = yahoo_financials_banks.get_financial_stmts('annual', 'cash')

banks_net_ebit = yahoo_financials_banks.get_ebit()
tech_stock_price_data = tech_cash_flow_data.get_stock_price_data()
daily_bank_stock_prices = yahoo_financials_banks.get_historical_stock_data('2008-09-15', '2017-09-15', 'daily')

Output Example:

yahoo_financials = YahooFinancials('WFC')
print(yahoo_financials.get_historical_stock_data("2017-09-10", "2017-10-10", "monthly"))

returns

{
    "WFC": {
        "prices": [
            {
                "volume": 260271600,
                "formatted_date": "2017-09-30",
                "high": 55.77000045776367,
                "adjclose": 54.91999816894531,
                "low": 52.84000015258789,
                "date": 1506830400,
                "close": 54.91999816894531,
                "open": 55.15999984741211
            }
        ],
        "eventsData": [],
        "firstTradeDate": {
            "date": 76233600,
            "formatted_date": "1972-06-01"
        },
        "isPending": false,
        "timeZone": {
            "gmtOffset": -14400
        },
        "id": "1mo15050196001507611600"
    }
}

Answer 2

Yahoo used to have a financial api, they've terminated it since, there is a workaround though.

I've used this with success before, you might want to take a look at it.