where to add encode and decode in TCP/IP python

Question

I need to send/receive data from one laptop to another, I'm using python 3.6 and it seems like this version of python needs the .encode() and .decode() functions, I was using python 2.7 and I had no troubles before, but now and with python 3.6 I've to use these functions, but I'm not quit sure where exactly I should use the .decode() command You'll find below my server file and my client file Server file :

import socket
import threading
import os

def RetrFile(name, sock):
    filename = sock.recv(1024)
    filename.decode()
    if os.path.isfile(filename):
        sock.send("EXISTS " + str(os.path.getsize(filename)))
        userResponse = sock.recv(1024)
        if userResponse[:2] == 'OK':
            with open(filename, 'rb') as f:
                bytesToSend = f.read(1024)
                sock.send(bytesToSend)
                while bytesToSend != "":
                    bytesToSend = f.read(1024)
                    sock.send(bytesToSend)
    else:
        sock.send("ERR ")

    sock.close()

def MainServer():
    host = '172.16.1.2' #my IP adress
    port = 5000

    s = socket.socket()
    s.bind((host,port))

    s.listen(5)

    print ("Server Started.")
    while True:
        c, addr = s.accept()
        print ("client connected ip:<" + str(addr) + ">")
        t = threading.Thread(target=RetrFile, args=("RetrThread", c))
        t.start()

    s.close()

if __name__ == '__main__':
    MainServer()

and this is my client file :

import socket
def MainClient():
    host = '172.16.1.2'
    port = 5000

    s = socket.socket()
    s.connect((host, port))

    filename = input("Filename? -> ")
    byt=filename.encode()
    if byt != 'q':
        s.send(byt)
        data = s.recv(1024)
        if data[:6] == 'EXISTS':
            filesize = len(data[6:])
            message = input("File exists, " + str(filesize) +"Bytes, download? (Y/N)? -> ")
            if message == 'Y':
                s.send("OK")
                f = open('new_'+filename, 'wb')
                data = s.recv(1024)
                totalRecv = len(data)
                f.write(data)
                while totalRecv < filesize:
                    data = s.recv(1024)
                    totalRecv += len(data)
                    f.write(data)
                    print ("{0:.2f}".format((totalRecv/float(filesize))*100)+ "% Done")
                print ("Download Complete!")
                f.close()
        else:
            print ("File Does Not Exist!")

    s.close()    

if __name__ == '__main__':
    MainClient()

But when I try to run it I get this error in the server side :

sock.send("ERR ")
TypeError: a bytes-like object is required, not 'str'

Thank you for your help.

Answer 1

You are inventing the protocol so you get to decide when things should be decoded. One option is to say that the base protocol (things like OK and ERR are always ascii and can be managed as bytes) but file names and things like that are utf-8 encoded strings, and need to be decoded. But what about binary files? You either need to define that part of the stream as binary or do some binary-to-ascii encoding on the content.

But its way more complicated than that. TCP is a stream protocol. There is nothing that says a recv gets everything that the other side sends in a single call or that it ends on a convenient utf-8 character boundary. You can't just recv(1024) and expect to get an exact message boundry.

So, time for a radical rethink: define a different protocol. To follow your existing pattern you could have an ascii based header than ends with a new line. The header is some command or status followed by the byte size of each additional parameter, followed by a newline. The parameters would be written directly after that.

You could have commands like

STAT <sizeof utf-8 encoded file name>\n<utf-8-encoded-filename>
EXISTS ascii-encoded-size\n
DOWNLOAD <sizeof utf-8 encoded file name>\n<utf-8-encoded-filename>

And parsers that take hints from the input stream to know what to do next. For instance, headers don't contain newlines but end with newlines, so you can have a function that reads the next header:

def read_header(sock):
    hdr = []
    while True:
        c = sock.recv(1)
        if c == "\n":
            return ''.join(hdr).encode('ascii')

And headers tell you how much data is in the pipe, so you can have a reader for that.

def read_chunk(sock, size, encoding=None):
    buf = io.bytesIO()
    while size:
        tmp = sock.recv(min(size, 4096))
        if not tmp:
            raise OSError("Unexpected end of stream")
        buf.write(tmp)
        size -= len(tmp)
    if encoding:
        return buf.getvalue().encode(encoding)
    else:
        return buf.getvalue()

Reading the stream would be

while True:
    hdr = read_header(sock)
    # try commands
    m = re.match(b"STAT (\d+)", hdr)
    if m:
        stat_size = int(m.group(1))
        file_name = read_chunk(sock, stat_size, 'utf-8')
        # do the stat work...
        continue
    m = re.match(b"DOWNLOAD (\d+)", hdr)
    etc...