"Online" monkey patching of a function

Question

Your program just paused on a pdb.set_trace().

Is there a way to monkey patch the function that is currently running, and "resume" execution?

Is this possible through call frame manipulation?

---

Some context:

Oftentimes, I will have a complex function that processes large quantities of data, without having a priori knowledge of what kind of data I'll find:

def process_a_lot(data_stream):
    #process a lot of stuff
    #...
    data_unit= data_stream.next()
    if not can_process(data_unit)
        import pdb; pdb.set_trace()
    #continue processing

This convenient construction launches a interactive debugger when it encounters unknown data, so I can inspect it at will and change process_a_lot code to handle it properly.

The problem here is that, when data_stream is big, you don't really want to chew through all the data again (let's assume next is slow, so you can't save what you already have and skip on the next run)

Of course, you can replace other functions at will once in the debugger. You can also replace the function itself, but it won't change the current execution context.

Edit: Since some people are getting side-tracked: I know there are a lot of ways of structuring your code such that your processing function is separate from process_a_lot. I'm not really asking about ways to structure the code as much as how to recover (in runtime) from the situation when the code is not prepared to handle the replacement.

Answer 1

No.

You can't moneky-patch a currently running Python function and continue pressing as though nothing else had happened. At least not in any general or practical way.

In theory, it is possible--but only under limited circumstances, with much effort and wizardly skill. It cannot be done with any generality.

To make the attempt, you'd have to:

1. Find the relevant function source and edit it (straightforward)
2. Compile the changed function source to bytecode (straightforward)
3. Insert the new bytecode in place of the old (doable)
4. Alter the function housekeeping data to point at the "logically" "same point" in the program where it exited to pdb. (iffy, under some conditions)
5. "Continue" from the debugger, falling back into the debugged code (iffy)

There are some circumstances where you might achieve 4 and 5, if you knew a lot about the function housekeeping and analogous debugger housekeeping variables. But consider:

1. The bytecode offset at which your pdb breakpoint is called (f_lasti in the frame object) might change. You'd probably have to narrow your goal to "alter only code further down in the function's source code than the breakpoint occurred" to keep things reasonably simple--else, you'd have to be able to compute where the breakpoint is in the newly-compiled bytecode. That might be feasible, but again under restrictions (such as "will only call pdb_trace() once, or similar "leave breadcrumbs for post-breakpoint analysis" stipulations).

2. You're going to have to be sharp at patching up function, frame, and code objects. Pay special attention to func_code in the function (__code__ if you're also supporting Python 3); f_lasti, f_lineno, and f_code in the frame; and co_code, co_lnotab, and co_stacksize in the code.

3. For the love of God, hopefully you do not intend to change the function's parameters, name, or other macro defining characteristics. That would at least treble the amount of housekeeping required.

4. More troubling, adding new local variables (a pretty common thing you'd want to do to alter program behavior) is very, very iffy. It would affect f_locals, co_nlocals, and co_stacksize--and quite possibly, completely rearrange the order and way bytecode accesses values. You might be able to minimize this by adding assignment statements like x = None to all your original locals. But depending on how the bytecodes change, it's possible you'll even have to hot-patch the Python stack, which cannot be done from Python per se. So C/Cython extensions could be required there.

   Here's a very simple example showing that bytecode ordering and arguments can change significantly even for small alterations of very simple functions:

   def a(x):             LOAD_FAST 0 (x)
       y = x + 1         LOAD_CONST 1 (1)
       return y          BINARY_ADD
                         STORE_FAST 1 (y)
                         LOAD_FAST 1 (y)
                         RETURN_VALUE
   ------------------    ------------------
   def a2(x):            LOAD_CONST 1 (2)
       inc = 2           STORE_FAST 1 (inc)
       y = x + inc       LOAD_FAST 0 (x)
       return y          LOAD_FAST 1 (inc)
                         BINARY_ADD
                         STORE_FAST 2 (y)
                         LOAD_FAST 2 (y)
                         RETURN_VALUE
   

5. Be equally sharp at patching some of the pdb values that track where it's debugging, because when you type "continue," those are what dictates where control flow goes next.

6. Limit your patchable functions to those that have rather static state. They must, for example, never have objects that might be garbage-collected before the breakpoint is resumed, but accessed after it (e.g. in your new code). E.g.:

   some = SomeObject()
   # blah blah including last touch of `some`
   # ...
   pdb.set_trace()
   # Look, Ma! I'm monkey-patching!
   if some.some_property:
      # oops, `some` was GC'd - DIE DIE DIE
   

   While "ensuring the execution environment for the patched function is same as it ever was" is potentially problematic for many values, it's guaranteed to crash and burn if any of them exit their normal dynamic scope and are garbage-collected before patching alters their dynamic scope/lifetime.

7. Assert you only ever want to run this on CPython, since PyPy, Jython, and other Python implementations don't even have standard Python bytecodes and do their function, code, and frame housekeeping differently.

I would love to say this super-dynamic patching is possible. And I'm sure you can, with a lot of housekeeping object twiddling, construct simple cases where it does work. But real code has objects that go out of scope. Real patches might want new variables allocated. Etc. Real world conditions vastly multiply the effort required to make the patching work--and in some cases, make that patching strictly impossible.

And at the end of the day, what have you achieved? A very brittle, fragile, unsafe way to extend your processing of a data stream. There is a reason most monkey-patching is done at function boundaries, and even then, reserved for a few very-high-value use cases. Production data streaming is better served adopting a strategy that sets aside unrecognized values for out-of-band examination and accommodation.

Answer 2

First a (prototype) solution, then some important caveats.

# process.py

import sys
import pdb
import handlers

def process_unit(data_unit):
    global handlers
    while True:
        try:
            data_type = type(data_unit)
            handler = handlers.handler[data_type]
            handler(data_unit)
            return
        except KeyError:
            print "UNUSUAL DATA: {0!r}". format(data_unit)
            print "\n--- INVOKING DEBUGGER ---\n"
            pdb.set_trace()
            print
            print "--- RETURNING FROM DEBUGGER ---\n"
            del sys.modules['handlers']
            import handlers
            print "retrying"


process_unit("this")
process_unit(100)
process_unit(1.04)
process_unit(200)
process_unit(1.05)
process_unit(300)
process_unit(4+3j)

sys.exit(0)

And:

# handlers.py

def handle_default(x):
    print "handle_default: {0!r}". format(x)

handler = {
    int: handle_default,
    str: handle_default
}

In Python 2.7, this gives you a dictionary linking expected/known types to functions that handle each type. If no handler is available for a type, the user is dropped own into the debugger, giving them a chance to amend the handlers.py file with appropriate handlers. In the above example, there is no handler for float or complex values. When they come, the user will have to add appropriate handlers. For example, one might add:

def handle_float(x):
    print "FIXED FLOAT {0!r}".format(x)

handler[float] = handle_float

And then:

def handle_complex(x):
    print "FIXED COMPLEX {0!r}".format(x)

handler[complex] = handle_complex

Here's what that run would look like:

$ python process.py
handle_default: 'this'
handle_default: 100
UNUSUAL DATA: 1.04

--- INVOKING DEBUGGER ---

> /Users/jeunice/pytest/testing/sfix/process.py(18)process_unit()
-> print
(Pdb) continue

--- RETURNING FROM DEBUGGER ---

retrying
FIXED FLOAT 1.04
handle_default: 200
FIXED FLOAT 1.05
handle_default: 300
UNUSUAL DATA: (4+3j)

--- INVOKING DEBUGGER ---

> /Users/jeunice/pytest/testing/sfix/process.py(18)process_unit()
-> print
(Pdb) continue

--- RETURNING FROM DEBUGGER ---

retrying
FIXED COMPLEX (4+3j)

Okay, so that basically works. You can improve and tweak that into a more production-ready form, making it compatible across Python 2 and 3, et cetera.

Please think long and hard before you do it that way.

This "modify the code in real-time" approach is an incredibly fragile pattern and error-prone approach. It encourages you to make real-time hot fixes in the nick of time. Those fixes will probably not have good or sufficient testing. Almost by definition, you have just this moment discovered you're dealing with a new type T. You don't yet know much about T, why it occurred, what its edge cases and failure modes might be, etc. And if your "fix" code or hot patches don't work, what then? Sure, you can put in some more exception handling, catch more classes of exceptions, and possibly continue.

Web frameworks like Flask have debug modes that work basically this way. But those are debug modes, and generally not suited for production. Moreover, what if you type the wrong command in the debugger? Accidentally type "quit" rather than "continue" and the whole program ends, and with it, your desire to keep the processing alive. If this is for use in debugging (exploring new kinds of data streams, maybe), have at.

If this is for production use, consider instead a strategy that sets aside unhandled-types for asynchronous, out-of-band examination and correction, rather than one that puts the developer / operator in the middle of a real-time processing flow.

Answer 3

What I wanted to know is if there's a way to monkey patch the function that is currently running (process_a_lot), and "resume" execution.

So you want to somehow, from within pdb, write a new process_a_lot function, and then transfer control to it at the location of the pdb call?

Or, do you want to rewrite the function outside pdb, and then somehow reload that function from the .py file and transfer control into the middle of the function at the location of the pdb call?

The only possibility I can think of is: from within pdb, import your newly written function, then replace the current process_a_lot byte-code with the byte-code from the new function (I think it's func.co_code or something). Make sure you change nothing in the new function (not even the pdb lines) before the pdb lines, and it might work.

But even if it does, I would imagine it is a very brittle solution.

Answer 4

If I understand correctly:

• you don't want to repeat all the work that has already been done

• you need a way to replace the #continue processing as usual with the new code once you have figured out how to handle the new data

@user2357112 was on the right track: expected_types should be a dictionary of

data_type:(detect_function, handler_function)

and detect_type needs to go through that to find a match. If no match is found, pdb pops up, you can then figure out what's going on, write a new detect_function and handler_funcion, add them to expected_types, and continue from pdb.