CODEWITHSUNDEEP
pythonpython-3.5
Don M R
Don M R

Reputation: 13

Memory corruption on exit (or stupid user error)

I have a problem where the objects holding imported python modules are being corrupted, but only after sys.exit() is explicitly called. This is also intermittent. I have stripped out a large portion of the code except for the "standard" pyusb module (I have used the very latest version of that). Very little actually happens.

If the program calls sys.exit() then some object IDs get changed to garbage values. If I leave that out then there are no errors!

Is this really a bug in Python? That seems unlikely. What did I do wrong?

===== main program "bug_happens.py"

#!/usr/bin/python3

import sys

import usbtmc

scope = usbtmc.Instrument()
print('in Main')
print('   id of usbtmc.usb: 0x{:x}'.format(id(usbtmc.usb)))
print('   id of usbtmc.usb.util: 0x{:x}'.format(id(usbtmc.usb.util)))
print('   id of usbtmc.usb.util.dispose_resources: 0x{:x}'.format(id(usbtmc.usb.util.dispose_resources)))
print()

print('calling Exit()')
sys.exit()

===== very stripped down usbtmc module

"""
****** Very stripped down partial Version for debugging a memory corruption issue ******

Python USBTMC driver

Copyright (c) 2012-2017 Alex Forencich

...
"""

import usb.util

class Instrument(object):
    def __init__(self, *args, **kwargs):

        self.connected = False

        print('usbtmc.Instrument created 0x{:x}'.format(id(self)))
        print('   id of usb: 0x{:x}'.format(id(usb)))
        print('   id of usb.util: 0x{:x}'.format(id(usb.util)))
        print('   id of usb.util.dispose_resources: 0x{:x}'.format(id(usb.util.dispose_resources)))
        print()

    def __del__(self):
        print('usbtmc.Instrument.__del__() called for 0x{:x}'.format(id(self)))
        print('   id of usb: 0x{:x}'.format(id(usb)))
        print('   id of usb.util: 0x{:x}'.format(id(usb.util)))
        print('   id of usb.util.dispose_resources: 0x{:x}'.format(id(usb.util.dispose_resources)))
        print('   end of usbtmc.Instrument.__del__()')
        print()

======================================================

===== Many times this works fine:

usbtmc.Instrument created 0x7fee5775d9b0
   id of usb: 0x7fee562800e8
   id of usb.util: 0x7fee562247c8
   id of usb.util.dispose_resources: 0x7fee561ad048

in Main
   id of usbtmc.usb: 0x7fee562800e8
   id of usbtmc.usb.util: 0x7fee562247c8
   id of usbtmc.usb.util.dispose_resources: 0x7fee561ad048

calling Exit()
usbtmc.Instrument.__del__() called for 0x7fee5775d9b0
   id of usb: 0x7fee562800e8
   id of usb.util: 0x7fee562247c8
   id of usb.util.dispose_resources: 0x7fee561ad048
   end of usbtmc.Instrument.__del__()

======================================================

===== About 50% of the time is does this:

usbtmc.Instrument created 0x7fec34b829b0
   id of usb: 0x7fec336c3278
   id of usb.util: 0x7fec335dbb88
   id of usb.util.dispose_resources: 0x7fec335e5048

in Main
   id of usbtmc.usb: 0x7fec336c3278
   id of usbtmc.usb.util: 0x7fec335dbb88
   id of usbtmc.usb.util.dispose_resources: 0x7fec335e5048

calling Exit()
usbtmc.Instrument.__del__() called for 0x7fec34b829b0
   id of usb: 0xa40060
Exception ignored in: <bound method Instrument.__del__ of <usbtmc.Instrument object at 0x7fec34b829b0>>
Traceback (most recent call last):
  File "/home/don/Electronics/Projects/HF_Meas/Scope_SA/Bug_usb_close/usbtmc.py", line 27, in __del__
AttributeError: 'NoneType' object has no attribute 'util'

======================================================

===== Occasionally it does this:

usbtmc.Instrument created 0x7ff75968c9b0
   id of usb: 0x7ff7581af0e8
   id of usb.util: 0x7ff7581537c8
   id of usb.util.dispose_resources: 0x7ff7580dc048

in Main
   id of usbtmc.usb: 0x7ff7581af0e8
   id of usbtmc.usb.util: 0x7ff7581537c8
   id of usbtmc.usb.util.dispose_resources: 0x7ff7580dc048

calling Exit()
usbtmc.Instrument.__del__() called for 0x7ff75968c9b0
   id of usb: 0x7ff7581af0e8
   id of usb.util: 0xa40060
Exception ignored in: <bound method Instrument.__del__ of <usbtmc.Instrument object at 0x7ff75968c9b0>>
Traceback (most recent call last):
  File "/home/don/Electronics/Projects/HF_Meas/Scope_SA/Bug_usb_close/usbtmc.py", line 28, in __del__
AttributeError: 'NoneType' object has no attribute 'dispose_resources'

Running on a Mint 18.3 system with python3 3.5.1-3 and either python3-usb 1.0.0~b2-2 or a fresh git pull of pyusb-1.0.2

Upvotes: 1

Views: 210

Answers (1)

wim
wim

Reputation: 363053

When the Python interpreter exits, the runtime is being torn down and the ordering of object deletion is not something you can rely on. In fact, it is not even guaranteed that __del__() methods are called at all when the interpreter exits.

The objects a custom __del__ needs to access (including other modules) may already have been deleted or set to None, which would cause exactly the exceptions you're seeing here ('NoneType' object has no attribute...)

That's not really a "memory corruption", it's just poor code written for the Instrument class. Any necessary setup/teardown context should be managed with __enter__ and __exit__, since __del__ is unsuitable for this purpose. So, this issue is more or less a bug in Instrument. This code could possibly be improved by taking advantage of a datamodel guarantee documented here:

Python guarantees that globals whose name begins with a single underscore are deleted from their module before other globals are deleted; if no other references to such globals exist, this may help in assuring that imported modules are still available at the time when the __del__() method is called.

Upvotes: 1

Related Questions