socket connection receive bytes python

Question

I'm trying to send a stream of data via socket in Python. So far I manage to create a dummy_data_gen.py which sends a line containing 4 floats to the server.py. However, I'm still having issues in the stability of the all setup.

server.py:

import sys
import time
import socket
import numpy as np

# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# Bind the socket to the port
server_address = ('localhost', 5002)
print >>sys.stderr, 'starting up on %s port %s' % server_address
sock.bind(server_address)

# Listen for incoming connections
sock.listen(1)

# Create a list to store the incoming data
data = []

while True:

    # Wait for a connection
    print >>sys.stderr, 'waiting for a connection'
    connection, client_address = sock.accept()

    try:
        print >>sys.stderr, 'connection from', client_address

        while True:

            incoming_data = connection.recv(48).split(',')
            print incoming_data

            event = float(incoming_data[0]), float(incoming_data[1]), float(incoming_data[2]), float(incoming_data[3])

            data += [event]

            time.sleep(0.01)

    finally:
        # Clean up the connection
        connection.close()

dummy_data_gen.py

import sys
import time
import socket

# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# Connect the socket to the port where the server is listening
server_address = ('localhost', 5002)
sock.connect(server_address)

file = '../data/myfile.txt'

# Simulating a real-time data stream at 100 Hz
try:
    with open(file) as f:
        for line in f:
            sock.sendall(line)
            time.sleep(0.01)

finally:
    print >>sys.stderr, 'closing socket'
    sock.close()

My problem is that sometimes the communication is working properly, however, I have situations where I receive more data per line than I should. In the following output example the first 7 lines are correct, however the following lines are incorrect and therefore problematic:

['391910745379', '24.134277', '-1.9487305', '-117.373535', '\n']
['391920745379', '24.434082', '-1.3491211', '-117.373535', '\n']
['391930745379', '23.68457', '-0.5996094', '-116.62402', '\n']
['391940745379', '24.434082', '-1.0493164', '-115.57471', '\n']
['391950745379', '24.134277', '-1.0493164', '-116.47412', '\n']
['391960745379', '23.234863', '-1.0493164', '-116.47412', '\n']
['391970745379', '24.583984', '-0.89941406', '-116.92383', '\n']
['391980745379', '23.384766', '-0.2998047', '-116.62402', '\n39']
['1990745379', '23.68457', '-0.5996094', '-115.72461', '\n39200']
['0745379', '23.834473', '-0.44970703', '-115.87451', '\n392010']
['745379', '23.534668', '-1.0493164', '-114.9751', '\n392020745']
['379', '23.384766', '-1.7988281', '-115.72461', '\n39203074537']
['9', '22.935059', '-0.44970703', '-114.9751', '\n392040745379', '']

I tried to play with the connection.recv bytes but I'm still facing this issue.

EDIT1: Following some suggestions I modified the server.py as follows:

del_message = '\n'
del_stream = ','

while True:
        _buffer += connection.recv(1)
        if del_message in _buffer:
            incoming_data = _buffer.split(del_stream)
            event = float(incoming_data[0]), \
                        float(incoming_data[1]), \
                            float(incoming_data[2]), \
                                float(incoming_data[3])

This approach seems to solve my issue, however the performance is extremely slow. My files contains approximately 6300 lines that were actually sent in 70 seconds (time interval at which the socket was closed on my dummy data generator). However, I took almost 10 minutes to receive all of the 6300 lines. It seems also that I receive more samples per second on the beginning rather that on the end of the stream.

Answer 1

If you have a message protocol that terminates messages with a newline, you need to write some code to implement that protocol. It won't work by magic.

You need a "receive a message" function, where "message" is defined as "a sequence of bytes delimited by a newline". You've never written any such function, so you're not receiving messages but just the chunks of bytes you're sending.