CODEWITHSUNDEEP
pythonmultithreadingdebugging
sdbbs
sdbbs

Reputation: 5492

Inspecting local variables in threads in Python (pyrasite)?

I have a script that uses threads, that simply freezes after running for an hour or so, which makes it pretty difficult to debug.

Eventually, I found pyrasite, and basically, I could "hook" into the script as soon as I waited it out and it froze. It looks somewhat like this in terminal:

$ pyrasite-shell 3437
Pyrasite Shell 2.0
Connected to 'python3 code/MyTestScript.py'
Python 3.5.3 (default, Sep 27 2018, 17:25:39)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
(DistantInteractiveConsole)

>>> import sys, traceback
>>> for thread_id, frame in sys._current_frames().items():
...     print('Stack for thread {}'.format(thread_id))
...     traceback.print_stack(frame)
...     print('')
...
Stack for thread 1888482416
  File "/usr/lib/python3.5/threading.py", line 882, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "<string>", line 167, in run
  File "/usr/lib/python3.5/code.py", line 228, in interact
    more = self.push(line)
  File "/usr/lib/python3.5/code.py", line 250, in push
    more = self.runsource(source, self.filename)
  File "/usr/lib/python3.5/code.py", line 75, in runsource
    self.runcode(code)
  File "/usr/lib/python3.5/code.py", line 91, in runcode
    exec(code, self.locals)
  File "<console>", line 3, in <module>

Stack for thread 1898968176
  File "/usr/lib/python3.5/threading.py", line 882, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/home/pi/code/my_test_lib.py", line 187, in sendData
    self.evexit.wait(sleep_time_s)
  File "/usr/lib/python3.5/threading.py", line 549, in wait
    signaled = self._cond.wait(timeout)
  File "/usr/lib/python3.5/threading.py", line 297, in wait
    gotit = waiter.acquire(True, timeout)

Stack for thread 1996019264
  File "code/MyTestScript.py", line 1355, in <module>
    main(sys.argv[1:])
  File "code/MyTestScript.py", line 1245, in main
    myObject.waitForJoin() # seems to work fine...
  File "/home/pi/code/my_test_lib.py", line 251, in waitForJoin
    self.myThread.join()
  File "/usr/lib/python3.5/threading.py", line 1054, in join
    self._wait_for_tstate_lock()
  File "/usr/lib/python3.5/threading.py", line 1070, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):

Stack for thread 1908950128
  File "/usr/lib/python3.5/threading.py", line 882, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/home/pi/code/my_testB_lib.py", line 511, in UpdateMyThread
    time.sleep(_update_interval_sec)

This looks great, but what I cannot tell is, whether the script including all its threads is now paused (as if a breakpoint in gdb hass been set), or does the script keep running in the background?

Anyways, I know that in gdb, I could have just issued say thread 1, and then I'd end up in the corresponding stack frame, and then I could print local variables etc.

Here, however, I cannot tell how to change stack frames, nor how to switch threads, so that I could inspect variables.

Is this possible with pyrasite? And if not, is there any other library for Python3, that would allow me the same (that is, ability to hook into an uninstrumented script with threads that freezes), while allowing me to inspect any/all threads and stackframes for local variable values?

Upvotes: 1

Views: 1224

Answers (2)

Anders Elton
Anders Elton

Reputation: 871

This script might be what you are looking for:

import sys, traceback
for thread_id, frame in sys._current_frames().items():
   print('Stack for thread {}'.format(thread_id))
   stack = traceback.StackSummary.extract(traceback.walk_stack(frame), capture_locals=True)
   stack.reverse()
   traceback.print_list(stack)
   print('')

It basically does the same thing as traceback.print_stack(frame), but it also supplies the capture_locals=True, which is false in the print_stack convenience method.

Upvotes: 3

Jeremy Muhlich
Jeremy Muhlich

Reputation: 162

Starting from Pyrasite's included dump_stacks.py payload, I created the snippet below which dumps local variable names and values after each stack trace. However it doesn't seem to work correctly as a Pyrasite payload -- it looks like the globals get displayed instead for some reason? I'm new to Pyrasite myself so I'm not sure if this is a bug or if I'm not using it correctly. If I run pyrasite-shell instead and paste the code in directly, it works as expected (ignoring the extra trace from the shell thread itself).

If you need to see different info, you can either modify the code and re-paste it or just explore interactively within the pyrasite shell.

import sys, traceback

for thread, frame in sys._current_frames().items():
    print('Thread 0x%x' % thread)
    print('=' * 25)
    traceback.print_stack(frame)
    print('-' * 25)
    for name, value in frame.f_locals.items():
        print(f"{name}: {value}")
    print()

Upvotes: 2

Related Questions