CODEWITHSUNDEEP
pythonpython-2.7
Tony Tannous
Tony Tannous

Reputation: 14886

Convert list to string and then back to list

Background
I am converting my data to binary as server side expects binary type.

Question:
How to convert a list of numbers to String and then reconstruct the list ?

File content:

1 1 1
1 2 1
1 3 1
1 4 1
1 5 1
1 6 1
1 7 1
1 8 1
1 9 1
1 10 1
1 11 1
1 12 1
1 13 1
1 14 1
1 15 1


In client: I am reading the whole file, appending each value to a list. Then list is converted to array which is converted to string before data is sent to server.

In server: I am mapping the string back to a list of values. The list is then converted to a list of tuples (x, y, w) using grouper. Then (x, y, z) is fed to Point and the newly constructed object is appended to a list.

Note I can't use bytearray as this is an artificial data sample, I'll have numbers much greated than a byte can represent.

Code:

from itertools import izip_longest
import array

def grouper(iterable, n, fillvalue=None):
    #Collect data into fixed-length chunks or blocks
    args = [iter(iterable)] * n
    return izip_longest(fillvalue=fillvalue, *args)

class Point:
    def __init__(self, x, y, w):
        self.x = x
        self.y = y
        self.w = w


if __name__ == "__main__":
    myList = []
    listOfObjects = []
    with open('data10mb.txt') as f:
        for line in f:
            s = line.split()
            x, y, w = [int(v) for v in s]
            myList.append(x)
            myList.append(y)
            myList.append(w)
    L = array.array('h', myList).tostring()


Data sent to server

Data received

    myList = list(map(ord, list(L)))
    myList = list(grouper(myList, 3))
    s = len(myList)
    for i in range (0, s):
        x, y, w = myList[i]
        obj = Point(x, y, w)
        listOfObjects.append(obj)

Expected Output:

1      <---- first line in file
1 
1
-------- 
1      <--- sixth line in file
7
1

Actual Output:

1
0
1
1
0
4

I am not sure what I've done wrong.. I've asked this question 4 days ago. "How to convert .txt file to a binary object".

server specifies that the data that should be sent is: binary: A byte array. I can't have a simple bytearray here as for python bytearray is limited to hold numbers 0-256 and the numbers represented in my file are much bigger.

What should I use instead ? As for the upper section its clear data is being mixed and I am not parsing correctly on server side, or perhaps I've done something wrong in code and I don't see it...


EDIT! I've tried sending list instead of string but server doesn't accept.

TypeError: write() argument 1 must be string or buffer, not list.

Thanks in advance!

Upvotes: 0

Views: 1238

Answers (1)

Jon Clements
Jon Clements

Reputation: 142156

In your code:

L = array.array('h', myList).tostring()

You are creating a bytestring packed as two byte integers. On the server side, you're then using list(L) which takes each element of L to generate a list, but in this case it doesn't retain the 2-byte packing as it sees each element in the bytestring as a single byte, eg:

>>> import array
>>> a = array.array('h', [1234, 5678])
>>> s = a.tostring()
>>> s
b'\xd2\x04.\x16'
>>> list(s)
[210, 4, 46, 22] # oops - wrong!

So, rebuild the array from the source data to get back what you sent.

>>> array.array('h', s)
array('h', [1234, 5678]) # better!

Also note that in your comment you say range up to 1,000,000 - the 'h' format is a 2 byte signed integer so you'll need to use 'l' for a signed long instead to represent the values sufficiently... (see type codes in the array documentation for available options)

to_send = array.array('l', [1000000, 12345]).tostring()
recieved = array.array('l', to_send)

Upvotes: 1

Related Questions