CODEWITHSUNDEEP
pythonccxt
Ufx
Ufx

Reputation: 2695

How to get historical data with one minute interval?

I need to get historical trading data with one minute interval.
I'm trying to get it using ccxt. But I'm getting several cycling values.
What did I do wrong?

import ccxt
import pandas as pd
import numpy as np
import time

np.set_printoptions(threshold=np.inf)

hitbtc = ccxt.hitbtc({'verbose': True})
bitmex = ccxt.bitmex()
huobi = ccxt.huobipro()
exchange = ccxt.exmo({
    'apiKey': 'K-...',
    'secret': 'S-...',
})

symbol = 'BTC/USD'
tf = '1m'
from_timestamp = exchange.parse8601('2019-01-10 00:00:00')
end = exchange.parse8601('2019-01-10 03:00:00')

# set timeframe in msecs
tf_multi = 60 * 1000
hold = 30

# make list to hold data
data = []

candle_no = (int(end) - int(from_timestamp)) / tf_multi + 1
print('downloading...')
while from_timestamp < end:
    try:
        ohlcvs = exchange.fetch_ohlcv(symbol, tf, from_timestamp)
        from_timestamp += len(ohlcvs) * tf_multi
        print(from_timestamp)
        data += ohlcvs
        print(str(len(data)) + ' of ' + str(int(candle_no)) + ' candles loaded...')
    except (ccxt.ExchangeError, ccxt.AuthenticationError, ccxt.ExchangeNotAvailable, ccxt.RequestTimeout) as error:
        print('Got an error', type(error).__name__, error.args, ', retrying in', hold, 'seconds...')
        time.sleep(hold)

header = ['t', 'o', 'h', 'l', 'c', 'v']
df = pd.DataFrame(data, columns=header)
open('btcusd.txt', 'w')
np.savetxt('btcusd.txt', df.o, fmt='%.8f')

// https://pastebin.com/xy1Ddb5z - btcusd.txt

enter image description here

Upvotes: 3

Views: 8443

Answers (2)

MG3
MG3

Reputation: 53

I think the problem is coming from the fact that fetch_ohlcv returns duplicate values in your while-loop. Once you have the df, just try with

# Keep only needed rows
df = df[df.Timestamp <= end]
# Delete duplicate rows
df = df.drop_duplicates()

Then you can plot (against index), e.g.

df['Close'].plot()

Or, if you convert the timestamps to a more readable format (for example using exchange.iso8601(),

plt.plot(df['Date']._values, df['Close']._values)

Upvotes: 0

Igor Kroitor
Igor Kroitor

Reputation: 1646

This is because in CCXT exmo.has['fetchOHLCV'] == 'emulated' as explained here:

See the description of the trades method in the EXMO API, it does not accept any time-range params whatsoever, so the since argument to fetch_ohlcv has no effect and is ignored in case of EXMO.

import ccxt
import pandas as pd
import numpy as np
import time
import sys  # ←---------------- ADDED

np.set_printoptions(threshold=np.inf)

hitbtc = ccxt.hitbtc({'verbose': True})
bitmex = ccxt.bitmex()
huobi = ccxt.huobipro()
exchange = ccxt.exmo({
    'apiKey': 'K-...',
    'secret': 'S-...',
})

symbol = 'BTC/USD'
tf = '1m'
from_timestamp = exchange.parse8601('2019-01-10 00:00:00')
end = exchange.parse8601('2019-01-10 03:00:00')

# set timeframe in msecs
tf_multi = 60 * 1000
hold = 30

# make list to hold data
data = []

# -----------------------------------------------------------------------------
# ADDED:
if exchange.has['fetchOHLCV'] == 'emulated':
    print(exchange.id, " cannot fetch old historical OHLCVs, because it has['fetchOHLCV'] =", exchange.has['fetchOHLCV'])
    sys.exit ()
# -----------------------------------------------------------------------------

candle_no = (int(end) - int(from_timestamp)) / tf_multi + 1
print('downloading...')
while from_timestamp < end:
    try:
        ohlcvs = exchange.fetch_ohlcv(symbol, tf, from_timestamp)
        # --------------------------------------------------------------------
        # ADDED:
        # check if returned ohlcvs are actually
        # within the from_timestamp > ohlcvs > end range
        if (ohlcvs[0][0] > end) or (ohlcvs[-1][0] > end):
            print(exchange.id, "got a candle out of range! has['fetchOHLCV'] =", exchange.has['fetchOHLCV'])
            break
        # ---------------------------------------------------------------------
        from_timestamp += len(ohlcvs) * tf_multi
        print(from_timestamp)
        data += ohlcvs
        print(str(len(data)) + ' of ' + str(int(candle_no)) + ' candles loaded...')
    except (ccxt.ExchangeError, ccxt.AuthenticationError, ccxt.ExchangeNotAvailable, ccxt.RequestTimeout) as error:
        print('Got an error', type(error).__name__, error.args, ', retrying in', hold, 'seconds...')
        time.sleep(hold)

header = ['t', 'o', 'h', 'l', 'c', 'v']
df = pd.DataFrame(data, columns=header)
open('btcusd.txt', 'w')
np.savetxt('btcusd.txt', df.o, fmt='%.8f')

// https://pastebin.com/xy1Ddb5z - btcusd.txt

Upvotes: 2

Related Questions