CODEWITHSUNDEEP
pythonpython-3.xwebsocketcryptocurrency
JayK23
JayK23

Reputation: 263

Can websockets "lag" when a lot of messages are received?

I'm trying to create a script using asyncio and websocket that should connect to 4-5 cryptocurrency exchange websockets and receive trades in real time from those exchanges. My code works and it's very simple, it looks like this:

import asyncio
import websockets
import json

subscriptions = ['btcusdt@trade', 'ethusdt@trade', 'bchusdt@trade', 'xrpusdt@trade', 'eosusdt@trade', 'ltcusdt@trade', 'trxusdt@trade', 'etcusdt@trade', 'linkusdt@trade', 'xlmusdt@trade', 'adausdt@trade', 'xmrusdt@trade', 'dashusdt@trade', 'zecusdt@trade', 'xtzusdt@trade', 'bnbusdt@trade', 'atomusdt@trade', 'ontusdt@trade', 'iotausdt@trade', 'batusdt@trade', 'vetusdt@trade', 'neousdt@trade', 'qtumusdt@trade', 'iostusdt@trade', 'thetausdt@trade', 'algousdt@trade', 'zilusdt@trade', 'kncusdt@trade', 'zrxusdt@trade', 'compusdt@trade', 'omgusdt@trade', 'dogeusdt@trade', 'sxpusdt@trade', 'kavausdt@trade', 'bandusdt@trade', 'rlcusdt@trade', 'wavesusdt@trade', 'mkrusdt@trade', 'snxusdt@trade', 'dotusdt@trade', 'defiusdt@trade', 'yfiusdt@trade', 'balusdt@trade', 'crvusdt@trade', 'trbusdt@trade', 'yfiiusdt@trade', 'runeusdt@trade', 'sushiusdt@trade', 'srmusdt@trade', 'bzrxusdt@trade', 'egldusdt@trade', 'solusdt@trade', 'icxusdt@trade', 'storjusdt@trade', 'blzusdt@trade', 'uniusdt@trade', 'avaxusdt@trade', 'ftmusdt@trade', 'hntusdt@trade', 'enjusdt@trade', 'flmusdt@trade', 'tomousdt@trade', 'renusdt@trade', 'ksmusdt@trade', 'nearusdt@trade', 'aaveusdt@trade', 'filusdt@trade', 'rsrusdt@trade', 'lrcusdt@trade', 'maticusdt@trade', 'oceanusdt@trade', 'cvcusdt@trade', 'belusdt@trade', 'ctkusdt@trade', 'axsusdt@trade', 'alphausdt@trade', 'zenusdt@trade', 'sklusdt@trade']

async def connect():
    while True:
        async with websockets.client.connect('wss://fstream.binance.com/ws/trade') as ws:
        
                tradeStr = {"method": "SUBSCRIBE", "params": subscriptions, 'id': 1}
                await ws.send(json.dumps(tradeStr))

                while True:
                    try:
                        msg = await asyncio.wait_for(ws.recv(), 5)
                        message = json.loads(msg)

                        try:    
                            print(message)
                        except Exception as e:
                            print(e)

                    except asyncio.TimeoutError:
                        break


asyncio.get_event_loop().run_until_complete(connect())

In the example above, i'm connecting to Binance and i'm receiving trades for all the markets available. I do this for more exchanges at once, but the problem will happen with one too as long as i'm receiving a lot of messages per second.

Each message looks like this {"rate": "xx", "market": "xx", "amount": "xx", "side": "xx"}, so very small.

The big problem i'm noticing is that after a while the script is running, i start receiving less messages, a lot of them will come after a lot of seconds and i don't even receive a lot others, as if they get lost or as if the connection is freezing.

Now, i know that it's not a very specific question, but what could be the problem here?

Is it possible that when websockets receive a lot of messages per second there could be problems of this kind? I tried to test this system from my local and from a vps, and in both cases i encountered the same issues. Is it possible that this is a resource problem? Or is it most likely related to the server, and not the client which is me? I tried to be as specific as possible, i can be more detailed if needed. I read that websockets stores received messages in a buffer. Is it possible that the problem is with the buffer getting filled? Any kind of advice is appreciated!

Upvotes: 1

Views: 3598

Answers (1)

r4cc00n
r4cc00n

Reputation: 2137

From what you explained and in my experience seems to be related to resources management, yes the WebSockets are affected if you receive a good amount of messages per second and yes this causes issues in your server. Why? because the buffer of course is limited and also the amount of memory available to process all those messages at the same time, as they mention in the official docs from the WebSocket library (version 8.1), I think that your issue here is that you are opening a lot number of connections at the same time and this causes memory exhaustion of course these depends on the size of the messages and the resources of your server, this can be easily tested you can try it with 2 VPS with different amount of resources if the 2 servers have different times to run into that state definitely one of the issues are resources (which I think should be the expected result fewer resources should run into the issue first and by resources means memory). Here are some links to the official docs for the WebSockets library where they make reference to these issues caused by memory and an approach to optimize the use of the memory. Hope this helps you 👍.

Memory Usage Optimizations

Upvotes: 1

Related Questions