How to add a global symbol that can be bound by Python compiler?

Question

This is my first question so please be nice :) I am rather new to Python but I am very experienced in other programming languages (e.g. C++).

---

UPDATE 2 - Solution Found

Thanks everyone for helping :) As the solution is "hidden" in comments I will repost it here.

Instead of

file_symbols = {}

the variable local_symbol must initially be added to the file_symbols dictionary:

file_symbols = { "local_symbol" : local_symbol }

For anyone reading this: all variable / class names posted here are not to be understood as actual useful names as these examples are synthetic in nature ;)

---

Well... now I have to figure out the FULL meaning of:

exec compiled_code in file_symbols

So far I thought it would do nothing more than updating the dictionary file_symbols with the symbols found in compiled_code. But it actually does a bit more as it seems! :)

---

UPDATE 1

Ok, my sample project below seems to be too simple to show the actual problem. Anyway, thanks for your already provided support! :)

In fact I want to first compile multiple *.py files which need access to a local symbol (class instance). All symbols coming from these compiled files shall be collected and then be used as an environment for OTHER code objects.

So I really need to do this (note the following code shows the concept, not actual executable code):

class Functions:
    (...)

global_symbols = {}
local_symbol = Functions()

# note that the external files need to access local_symbol to call its functions!
for file in external_files:
    code = file.load()
    compiled_code = compile(code, "<string>", "exec")
    file_symbols = {}
    exec compiled_code in file_symbols
    global_symbols.update(file_symbols)

some_code = another_file.load()
compiled_code = compile(some_code, "<string>", "exec")
exec(compiled_code, global_symbols)

In this example the line

exec compiled_code in file_symbols

produces a NameError() - because there is no way they could access local_symbol as it is not defined anywhere in the external files although it shall be used!

So the question is how to provide access to local_symbol for the external_files so that they can call the instance's functions??

My import hook solution that some of you regard as "hack" was the only working solution so far. I would love to use a simpler one if there is any!

Thanks again :)

---

My initial question was this:

So here we go. What I intend to do is advanced stuff and I did not find a solution to my problem neither here nor anywhere else.

Assume the following code in Python (2.6.x / 2.7.x):

class Functions:
    def __init__(self):
        (...)
    def func_1(...):
        (...)
    def func_2(...):
        (...)
    (...)
    def func_n(...):
        (...)
functions = Functions()
code = loadSomeFile(...)
compiled_code = compile(code, "<string>", "exec")
(...)
global_env = {}
local_env = {"func":functions}
exec(compiled_code, global_env, local_env)

where code in the example above is loaded from a file with a content that might look like this:

import something
(...)
def aFunction(...):
    a = func.func_1(...)
    b = func.func_2(...)
    return a * b
(...)
aFunction()

Please note that (...) in the code above means that there might be more code that I left out for the sake of simplicity.

The problem I encounter in my example is that the compiler raises an error for this line:

compiled_code = compile(code, "<string>", "exec")

I will get this error: NameError("global name 'func' is not defined")

This error is totally understandable as the compiler can't bind to any global symbol with the name "func". But I still want to compile the code this way.

So the obvious question is:

How can I define global symbols that can be used by the compiler for compile() statements so that the compiler will bind any "unknown" symbol to an object of my choice?

In my example I would like to define a global symbol func that is bound to an instance of class Functions so that the compiler will find this symbol when compiling code which makes use of func as seen in the example above.

So how can this be accomplished?

Important: Please note that I am aware that directly executing the code using exec(...) would fix the compilation problem because the dict local_env in the example above would provide the symbol that is required for successful execution. HOWEVER I can't do it this way because the code that shall be compiled is not small at all. It might consist of hundreds of code lines and this code is also not executed only once but many many times. So for performance reasons I really need to compile the code rather than directly executing it.

Thanks for helping me :)

Answer 1

The simple explanation is that you pass in func in the local namespace. aFunction does not have access to the locals you passed in (it has its own locals). All aFunction has access to is its own locals and its module's globals. func is in neither of these and so the function call fails.

Most normal modules work with their globals and locals as the same namespace (you can check this out yourself assert globals() is locals()). This is why you can define things at the module level and have them available to any defined functions (all names in a module are automatically global).

To make this work you need to make locals and globals the same dict, or just not pass locals at all. If you don't want globals to be mutated, then just copy globals and then add func to it.

src = """
def aFunction():
    a = func.func_1()
    b = func.func_2()
    return a + b
value = aFunction()
"""

class Functions:
    def func_1(self):
        return "first"
    def func_2(self):
        return "second"
functions = Functions()

compiled_code = compile(src, "<string>", "exec")

global_env = {}
local_env = {"func":functions}
namespace = dict(global_env)
namespace.update(local_env)
exec(compiled_code, namespace)
print(namespace["value"])

Answer 2

Don't provide separate globals and locals dicts to exec. That causes the executed code to behave as if it's embedded in a class definition. That means that any variable lookups in functions defined in the executed code bypass locals.

>>> exec("""
... def f():
...     print a
... f()""", {}, {"a": 3})
Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
  File "<string>", line 4, in <module>
  File "<string>", line 3, in f
NameError: global name 'a' is not defined
>>> exec("""
... def f():
...     print a
... f()""", {"a": 3})
3

Just pass a globals dict.

Answer 3

Well, obviously it works by simply adding (and potentially removing - if using multiple "func" instances) the instance to sys.modules:

sys.modules["func"] = functions
(...)
compiled_code = compile(code, "<string>", "exec")

The complete and working solution including an importer hook (which intercepts "import func" lines in source code files) would look like this:

import sys

class Functions:
    def __init__(self):
        (...)
    def func_1(...):
        (...)
    def func_2(...):
        (...)
    (...)
    def func_n(...):
        (...)

functions = Functions()
code = loadSomeFile(...)

hook_name = "func"

class ImporterHook:
    def __init__(self, path):
        if path != hook_name:
            raise ImportError()
    def find_module(self, fullname, path=None):
        return self
    def load_module(self, path):
        if sys.modules.has_key(path):
            return sys.modules[path]
        else:
            sys.modules[path] = functions
            return functions

sys.path_hooks.append(ImporterHook)
sys.path.insert(0, hook_name)
compiled_code = compile(code, "<string>", "exec")
(...)
exec(compiled_code)

Not so hard as it seems :) For more information see here:

https://www.python.org/dev/peps/pep-0302/#specification-part-2-registering-hooks

and here:

https://pymotw.com/2/sys/imports.html

THANKS :)

Answer 4

it is an interesting question, thanks for posting it. I've been taking a look at how to change the globals table in compile time. Apparently, you can call the __import__() function directly and:

pass your globals in order to determine how to interpret the name in a package context.

Source: Package documentation