Where to store constants in Python

Question

I am currently working on a project where I have a really large number of global constants to have hardcoded in my code. The project is written in Python. And as we all know, Python performance when it comes to accessing global variables drops down significantly.

I could move those constants, that I use only in one method, to the local scope of the methods I use them from, but that removes readability. Then I will still have some globals which are used in multiple methods and I really cannot hide them in the scope of only one function.

What's the solution to that? I saw a guy making a game (Ludum Dare 31), and f.e in 3:30 you can see that he just has one big file constants.py with a hell lotta global variables there (without global keyword). Is this a good practic?

Answer 1

as we all know, Python performance when it comes to accessing global variables drops down significantly.

Not "significantly" - local lookups are a bit cheaper indeed, but defining a local variable also has a cost, so unless you're looking up a global in a very tight loop the chances you ever notice the difference are really small, and then you can always alias the global locally, ie from:

FOO = 42

def foo():
    for i in range(100000000):
        x = i * FOO

to

FOO = 42

def foo():
    localfoo = FOO
    for i in range(100000000):
        x = i * localfoo

In other words, you should really not worry about the performance issues here until performances are a real issue AND the profiler identified this global lookup as a major bottleneck (which is really really really unlikely), and even then I seriously doubt that you'll ever get any significant boost in the end - if the cost of a global lookup is already too much for your application then Python is not the right tool and it's time to rewrite this part in C.

I could move those constants, that I use only in one method, to the local scope of the methods I use them from, but that removes readability.

And, as mentionned above, will not necessarily improve perfs:

>>> import dis
>>> import timeit
>>> 
>>> FOO = 42
>>> def foo1():
...     return FOO
... 
>>> def foo3():
...     foo = 42
...     return foo
... 
>>> dis.dis(foo1)
  2           0 LOAD_GLOBAL              0 (FOO)
              3 RETURN_VALUE        
>>>
>>> dis.dis(foo3
... )
  2           0 LOAD_CONST               1 (42)
              3 STORE_FAST               0 (foo)

  3           6 LOAD_FAST                0 (foo)
              9 RETURN_VALUE  
>>> timeit.timeit("func()", "from __main__ import foo1 as func")
0.06334185600280762
>>> timeit.timeit("func()", "from __main__ import foo3 as func")
0.06805109977722168

Then I will still have some globals which are used in multiple methods and I really cannot hide them in the scope of only one function. What's the solution to that?

What's the problem actually ?

I saw a guy making a game (...) you can see that he just has one big file constants.py with a hell lotta global variables there (without global keyword).

All names defined at the top-level of a module (by assignation, import, function definition or class definition) are "global" to the module (and this is the only kind of "global" you'll find in Python - there's NO "application-wide globals"). The global keyword is only to be used within functions, and only when you actually want to assign to that global within the function - something we all know we should NOT do, do we ?

Is this a good practic?

Depends on how and where those "constants" are used. If you have constants that are used by more than one single module AND there's no other dependancies between those modules then it makes sense indeed, but most of the time constants are either only used by one single module or the other modules using them also need other names (functions, classes etc) from the same module.

To make a long story short: constants are nothing special, they are just names referencing objects (you may not realize but all your functions and classes ARE "constants" too), so you just want to apply the same guidelines as for anything else: your modules should have a strong cohesion (everything in a module is strongly related) and low coupling (your module depends on as few other modules as possible). From this point of view, defining tens of unrelated constants in a single file that 10+ other unrelated modules depend on is just plain wrong - it breaks cohesion and introduce strong coupling.

Note that you might have some other reason to "centralize" constants (at least some of them) that way: making configuration easier - but this only apply to constants that you want to make configurable (would you make the value "pi" configurable ?), and is a totally different question.

Answer 2

If all you care about is performance of your code in the global namespace lookup, you can probably do

globals()['your_constant_name'] # inside your function/method

which will directly look up things in the global namespace. Note that if for some reason the constant doesn't exist, then a 'KeyError' will be raised instead of an 'AttributeError'.

Also, per Python documentation

This is always the dictionary of the current module (inside a function or method, this is the module where it is defined, not the module from which it is called)

So use it with care.

UPDATE:

This is a bit of an extreme case (unlikely to happen in any realistic scenario), but dictionary lookup does increase performance a little bit if the stack frame is huge, despite the dictionary construction etc. as @brunodesthuilliers has mentioned. Testing code:

import itertools,timeit
globals().update({''.join(n):i for i,n in enumerate(itertools.permutations('ABCDEFGHI'))})


def with_dict():
    def func():
        try:
            func()
        except RecursionError:
            globals()['ABCDEFGHI']


def without_dict():
    def func():
        try:
            func()
        except RecursionError:
            ABCDEFGHI

print(timeit.timeit(with_dict)) # output: 0.33404375400277786
print(timeit.timeit(without_dict)) # output: 0.3390919269877486

Although, according to python wiki, dictionary lookup has an average time complexity of O(1)