CODEWITHSUNDEEP
pythonmultiprocessingipc
Razer
Razer

Reputation: 8211

Share complex object with another process

I do have a complex data-object. What's the best way to share it with another python process?

Background: To avoid the GIL I spawn a second process which does all the calculation. It logs to the main-process using UDP. This calculation process needs input data which is this complex object. So how I share it the most easiest way?

Upvotes: 0

Views: 159

Answers (2)

Razer
Razer

Reputation: 8211

I'm using now following:

process = multiprocessing.Process(target=start_remote_runner, args=(context,))
process.start()

This passes context-object to the remote_runner. This indeed uses pickle internally to pass the object.

Upvotes: 0

Paul
Paul

Reputation: 7355

Have you considered serializing it using pickle?

import pickle
fid = open(filename,'w')
fid.write(pickle.dumps(data))
fid.close()

then load it in the other process:

fid = open(filename,'r')
directData = fid.read()
data = pickle.loads(directData)
fid.close()

Something similar can be done with other formats import json fid = open('jsonOutput','w') fid.write(json.dumps(data)) fid.close()

If you didn't want to transfer via files you could transfer over a network connection

Upvotes: 1

Related Questions