CODEWITHSUNDEEP
pythonamazon-web-servicesubuntuamazon-ec2kraken.com
Charles1
Charles1

Reputation: 57

Why does Python-Process on ec2 stop

I have a python script running on a t2.micro EC2 instance on AWS.

I start the program with ./myscript > /dev/null 2>&1 & and than I check if my script is running with ps -elf | grep python. The problem is, after a certain and rather arbitrary time the program stops and I can't see it anymore with ps -elf | grep python. I cannot really see the reason why this happens (is it Out of Memory, etc.?).

My code became a bit ugly as I tried all kind of stuff. Basically it takes all kind of Crypto-Symbols (like BTC) from an excel, creates a websocket to kraken.com, collects the data and saves it to a .pkl every hour. Has anybody an advice how to figure out why this works perfectly fine for a couple of hours or even days and then stops??

My program runs without problems if I run it locally on jupyter/anaconda on Windows.

#!/usr/bin/python3

import json
from websocket import create_connection
import pandas as pd
import time
import requests
from datetime import datetime, timedelta

df = pd.read_excel('Kraken_All_pairs.xlsx')
kraken_pairs = ''
for i in df['Symbol']:
    kraken_pairs = kraken_pairs + '"' + str(i) + '/USD",'
kraken_pairs = kraken_pairs[0:-1]
kraken_pairs = '{"event":"subscribe", "subscription":{"name":"trade"}, "pair":[' + kraken_pairs + ']}'
lasttime = datetime.utcfromtimestamp(time.time()).strftime('%H')
while True:
    try:
        data_list = list()
        try:
            ws = create_connection("wss://ws.kraken.com/")
            ws.send(kraken_pairs)
        except Exception as e:
            time.sleep(1)
            break;
        while True:
            try:
                data_list.append(json.loads(ws.recv()))
            except Exception as e:
                timestamp_exception = datetime.utcfromtimestamp(time.time()).strftime('%Y-%m-%d %H:%M:%S.%f')
                try:
                    ws = create_connection("wss://ws.kraken.com/")
                    ws.send(kraken_pairs)
                except Exception as e:
                    time.sleep(1)
                    break;
            now = datetime.utcfromtimestamp(time.time()).strftime('%H')

            if lasttime != now:
                lasttime = now
                break
                
        data_list = [value for value in data_list if value != {'event': 'heartbeat'}]
        data_list = [value for value in data_list if type(value) != dict]
        Pair = [value[3] for value in data_list]
        Price = [value[1][0][0] for value in data_list]
        Date = [value[1][0][2] for value in data_list]
        df_all = pd.DataFrame(list(zip(Pair, Price, Date)), columns =['Pair', 'Price', 'Date'])
        df_all['Date'] = df_all['Date'].astype(float)
        df_all['Date'] = df_all['Date']*1000
        df_all['Date'] = pd.to_datetime(df_all['Date'], unit='ms')
        df_all['Date'] = df_all['Date'].dt.strftime("%m.%d.%Y, %H:%M:%S")
        list_all_df_pairs = []
        for value in df_all['Pair'].unique():
            list_all_df_pairs.append(df_all.loc[df_all['Pair'] == value])
        for index in range(0,len(list_all_df_pairs)):
            path = './pkl/' + str((datetime.utcfromtimestamp(time.time()) - timedelta(hours=1)).strftime('%H')) + '_' + list_all_df_pairs[index]['Pair'].iloc[0].replace('/','_') + '.pkl'
            list_all_df_pairs[index].to_pickle(path)
    except Exception as e:
        timestamp_exception = datetime.utcfromtimestamp(time.time()).strftime('%Y-%m-%d %H:%M:%S.%f')

Upvotes: 0

Views: 105

Answers (1)

AKX
AKX

Reputation: 168824

Well, I'd recommend not just piping the output to /dev/null if you want to see what's wrong.

Run with

./myscript >> myscript.log 2>&1 &

so you can actually debug it.

Upvotes: 1

Related Questions