CODEWITHSUNDEEP
pythonpython-3.xnumpypipepython-3.6
Kilsen
Kilsen

Reputation: 171

How can I send a byte-object over a pipe in python?

I am trying to send a numpy array over stdout from one process to another.

Sending it over the pipe requires me to convert it to a string. On the other side, I receive a byte object. This byte object now has the original byte string encapsulated.

I find it now impossible to restore the original byte object as a byte object. If I decode the byte object, I receive a string which is incompatible with all the restoring functions I tried (np.frombuffer, pickle.loads).

server.py

import subprocess
import numpy as np
p = subprocess.Popen(['python3', 'writer.py'], stdout=subprocess.PIPE)

while 1:
    tmp = p.stdout.readline()    
    # doesn't fail but wrong size
    array = np.frombuffer(tmp, dtype=np.uint8)
    tmp = bytes.decode(tmp)
    # fails because byte object is necessary
    array = np.frombuffer(tmp, dtype=np.uint8)
    array = array.reshape([1, 3, 5, 5, 1])
    print(array.shape)

writer.py

import numpy as np
import sys
while 1:
    array = np.zeros([1, 3, 5, 5, 1], dtype=np.int8)
    string = array.tobytes()
    sys.stdout.write(str(string))
    sys.stdout.flush()

Is there anyway to convert the string into a byteobject, without encoding it? How else could this work? I want to use pipes instead of shared memory as proposed in some other solutions to keep it more simple. Furthermore, I need it to be parallel but blocking, so Pipes seemed ideal to me.

Thanks

Upvotes: 1

Views: 3686

Answers (1)

ymonad
ymonad

Reputation: 12100

you can use pickle to marshal the data, and sys.stdout.buffer instead of sys.stdout to write bytes to stdout.

Reference: sys.stdout

server.py:

import subprocess
import numpy as np
import pickle
p = subprocess.Popen(['python3', 'writer.py'], stdout=subprocess.PIPE)

while 1:
  array = pickle.load(p.stdout)
  array = array.reshape([1, 3, 5, 5, 1])
  print(array.shape)

writer.py:

import numpy as np
import pickle
import sys
while 1:
  array = np.zeros([1, 3, 5, 5, 1], dtype=np.int8)
  pickle.dump(array, sys.stdout.buffer)

Upvotes: 3

Related Questions