CODEWITHSUNDEEP
pythonsynchronizationqueuemultiprocessingpickle
Manuel Faux
Manuel Faux

Reputation: 2457

Cannot put object on Queue

I want to put an instance of scapy.layers.dhcp.BOOTP on a multiprocessing.Queue. Every time I call put() the following exception occures:

Traceback (most recent call last):
  File "/usr/lib/python2.6/multiprocessing/queues.py", line 242, in _feed
    send(obj)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

Of cause trying to pickle the instance directly using pickle.dumps() also fails. But why is this class not picklable?

For all those who don't have scapy installed:

class BOOTP(Packet):
    name = "BOOTP"
    fields_desc = [ ByteEnumField("op",1, {1:"BOOTREQUEST", 2:"BOOTREPLY"}),
                    ByteField("htype",1),
                    ByteField("hlen",6),
                    ByteField("hops",0),
                    IntField("xid",0),
                    ShortField("secs",0),
                    FlagsField("flags", 0, 16, "???????????????B"),
                    IPField("ciaddr","0.0.0.0"),
                    IPField("yiaddr","0.0.0.0"),
                    IPField("siaddr","0.0.0.0"),
                    IPField("giaddr","0.0.0.0"),
                    Field("chaddr","", "16s"),
                    Field("sname","","64s"),
                    Field("file","","128s"),
                    StrField("options","") ]
    def guess_payload_class(self, payload):
        if self.options[:len(dhcpmagic)] == dhcpmagic:
            return DHCP
        else:
            return Packet.guess_payload_class(self, payload)
    def extract_padding(self,s):
        if self.options[:len(dhcpmagic)] == dhcpmagic:
            # set BOOTP options to DHCP magic cookie and make rest a payload of DHCP options
            payload = self.options[len(dhcpmagic):]
            self.options = self.options[:len(dhcpmagic)]
            return payload, None
        else:
            return "", None
    def hashret(self):
        return struct.pack("L", self.xid)
    def answers(self, other):
        if not isinstance(other, BOOTP):
            return 0
        return self.xid == other.xid

Are there any other ways to "transport" this instance to another subprocess?

Upvotes: 0

Views: 2954

Answers (3)

geekscrap
geekscrap

Reputation: 1045

A solution I use is to str the packet and then put it on the queue...

Upvotes: 0

Marvin
Marvin

Reputation: 3045

The other thing that may help to diagnose problems like these is to use the pickle module instead of cPickle (which must be getting used implicitly by queues.py)

I had a similar situation, getting a completely unhelpful message,

Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

I wandered into the debugger, found the object being pickled, and tried passing it to

pickle.dump(myobj,open('outfile','w'),-1)

and got a much more helpful:

PicklingError: Can't pickle <function findAllRefs at 0x105809f50>: 
   it's not found as buildsys.repoclient.findAllRefs

Which pointed much more directly at the problematic code.

Upvotes: 0

Jochen Ritzel
Jochen Ritzel

Reputation: 107638

Well, the problem is that you can't pickle the function type. It's what you get when you do type(some_user_function). See this:

>>> import types
>>> pickle.dumps(types.FunctionType)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'picke' is not defined
>>> pickle.dumps(types.FunctionType)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python26\lib\pickle.py", line 1366, in dumps
    Pickler(file, protocol).dump(obj)
  File "C:\Python26\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "C:\Python26\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python26\lib\pickle.py", line 748, in save_global
    (obj, module, name))
pickle.PicklingError: Can't pickle <type 'function'>: it's not found as __built
n__.function

So such a function type is stored somewhere on the object you try to send. It's not in the code you pasted, so i guess it's on the superclass.

Maybe you can simply send all the arguments required to create a instance of scapy.layers.dhcp.BOOTP instead of the instance to avoid the problem?

Upvotes: 1

Related Questions