CODEWITHSUNDEEP
pythonpython-3.xcompatibilitypython-2.x
ababak
ababak

Reputation: 1793

Python 2.x and 3.x sourceless .pyc deployment

In Python 2.x the .pyc files are stored inside the same directory as the source .py file. In Python 3.2+ the .pyc files are stored inside the __pycache__ directory.

If the code is written properly, it can be run by both interpreters and the .pyc files can coexist fine. But as soon as I want to get rid of the source code I get a problem: the Python 2.x interpreter still imports the correct .pyc file but the Python 3.2+ interpreter refuses to import its compiled .pyc from the __pycache__ directory and tries to import the .pyc file from the previous version and fails as the magic numbers do not match.

Is it possible to make both Python interpreters happy without exposing the source code?

[Updated]

I tried this (it looked like a solution):

  1. Deploy sources as usual
  2. Compile each .py file with Python 2.x compiler (this will produce .pyc file in the same directory)
  3. Compile each .py file with Python 3.2+ compiler (this will produce .pyc file in the __pycache__ directory)
  4. Clear the .py file contents while preserving its modification date

The Python 2 interpreter is happy. The Python 3 interpreter suspects that the source is modified and recompiles the modified source. What's the reason for it to recompile? Does it store the source file size as well? The checksum?

Upvotes: 2

Views: 2340

Answers (1)

Martijn Pieters
Martijn Pieters

Reputation: 1121744

Python 3 will make use of .pyc files in the same location if there is no .py file or __pycache__ directory. Note that Python 3 explicitly refuses to load __pycache__ files if the source file is missing, for source-less distributions, only the legacy locations are supported.

The compileall module and command can be told to produce .pyc files in the same directory for that very reason:

python -m compileall -b

but take into account that byte cache files remain Python-version specific. You'll have to produce and distribute cache files per Python bytecache magic number (which generally changes with each .x release).

So no, you can't have .pyc source-less distributions that work on multiple Python versions. That was never the case before Python 3.2 either, Python 2.6, 2.7, 3.0, and 3.1 .pyc bytecache files each have a unique bytecache version number, the 'magic number' Python will complain about if you give it a .pyc file from another version.

Upvotes: 6

Related Questions