PyInstaller, how to include data files from an external package that was installed by pip?

Question

Problem

I am attempting to use PyInstaller to create an application for internal use within my company. The script works great from a working python environment, but loses something when translated to a package.

I know how to include and reference data files that I myself need within my package, but I am having trouble including or referencing files that should come in when imported.

I am using a pip-installable package called tk-tools, which includes some nice images for panel-like displays (looks like LEDs). The problem is that when I create a pyinstaller script, any time that one of those images is referenced, I get an error:

DEBUG:aspen_comm.display:COM23 19200
INFO:aspen_comm.display:adding pump 1 to the pump list: [1]
DEBUG:aspen_comm.display:updating interrogation list: [1]
Exception in Tkinter callback
Traceback (most recent call last):
  File "tkinter\__init__.py", line 1550, in __call__
  File "aspen_comm\display.py", line 206, in add
  File "aspen_comm\display.py", line 121, in add
  File "aspen_comm\display.py", line 271, in __init__
  File "aspen_comm\display.py", line 311, in __init__
  File "lib\site-packages\tk_tools\visual.py", line 277, in __init__
  File "lib\site-packages\tk_tools\visual.py", line 289, in to_grey
  File "lib\site-packages\tk_tools\visual.py", line 284, in _load_new
  File "tkinter\__init__.py", line 3394, in __init__
  File "tkinter\__init__.py", line 3350, in __init__
_tkinter.TclError: couldn't open "C:\_code\tools\python\aspen_comm\dist\aspen_comm\tk_tools\img/led-grey.png": no such file or directory

I looked within that directory in the last line - which is where my distribution is located - and found that there is no tk_tools directory present.

Question

How to I get pyinstaller to collect the data files of imported packages?

Spec File

Currently, my datas is blank. Spec file, created with pyinstaller -n aspen_comm aspen_comm/__main__.py:

# -*- mode: python -*-

block_cipher = None


a = Analysis(['aspen_comm\\__main__.py'],
             pathex=['C:\\_code\\tools\\python\\aspen_comm'],
             binaries=[],
             datas=[],
             hiddenimports=[],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher)

pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)

exe = EXE(pyz,
          a.scripts,
          exclude_binaries=True,
          name='aspen_comm',
          debug=False,
          strip=False,
          upx=True,
          console=True )

coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               name='aspen_comm')

When I look within /build/aspen_comm/out00-Analysis.toc and /build/aspen_comm/out00-PYZ.toc, I find an entry that looks like it found the tk_tools package. Additionally, there are features of the tk_tools package that work perfectly before getting to the point of finding data files, so I know that it is getting imported somewhere, I just don't know where. When I do searches for tk_tools, I can find no reference to it within the file structure.

I have also tried the --hidden-imports option with the same results.

Partial Solution

If I 'manually' add the path to the spec file using datas = [('C:\\_virtualenv\\aspen\\Lib\\site-packages\\tk_tools\\img\\', 'tk_tools\\img\\')] and datas=datas in the Analysis, then all works as expected. This will work, but I would rather PyInstaller find the package data since it is clearly installed. I will keep looking for a solution, but - for the moment - I will probably use this non-ideal workaround.

If you have control of the package...

Then you can use stringify on the subpackage, but this only works if it is your own package.

Answer 1

Possibly this option was added after this question was asked, but PyInstaller provides a few command line options that helped:

--collect-submodules MODULENAME

Collect all submodules from the specified package or module. This option can be used multiple times.

--collect-data MODULENAME, --collect-datas MODULENAME

Collect all data from the specified package or module. This option can be used multiple times.

--collect-binaries MODULENAME

Collect all binaries from the specified package or module. This option can be used multiple times.

--collect-all MODULENAME

Collect all submodules, data files, and binaries from the specified package or module. This option can be used multiple times.

I simply used the name of the external package as MODULENAME

Answer 2

Here's a one-liner using the same idea as Turn mentioned. In my case I needed a package (zbarcam) that was inside of kivy_garden. But I tried to generalize the process here.

from os.path import join, dirname, abspath, split
from os import sep
import glob
import <package>

pkg_dir = split(<package>.__file__)[0]
pkg_data = []
pkg_data.extend((file, dirname(file).split("site-packages")[1]) for file in glob.iglob(join(pkg_dir,"**{}*".format(sep)), recursive=True))

Answer 3

A bit late to the party, but wrote a help article on how I did this:

https://www.linkedin.com/pulse/loading-external-module-data-during-pyinstaller-bundling-deguzis/?published=t

Snippet:

import os
import pkgutil
import PyInstaller.__main__
import platform
import shutil
import sys


# Get mypkg data not imported automagically
# Pre-create location where data is expected
if not os.path.exists('ext_module'):
    os.mkdir('ext_module')

with open ('ext_module' + os.sep + 'some-env.ini', 'w+') as f:
    data = pkgutil.get_data( 'ext_module', 'some-env.ini' ).decode('utf-8', 'ascii')
    f.write(data)

# Set terminator (PyInstaller does not provide an easy method for this)
# ':' for OS X / Linux
# ';' for Windows
if platform.system() == 'Windows':
    term = ';'
else:
    term = ':'


PyInstaller.__main__.run([
        '--name=%s' % 'mypkg',
        '--onefile',
        '--add-data=%s%smypkg' % (os.path.join('mypkg' + os.sep + 'some-env.ini'),term),
        os.path.join('cli.py'),
    ])


# Cleanup
shutil.rmtree('mypkg')

Answer 4

Mac OS X 10.7.5 Pyinstaller; adding images using 1 line of code, instead of individual lines, for each image in the app.spec file. This is all the code I used to get my images to compile with my script. Add this function to the top of: yourappname.py:

# Path to the resources, (pictures and files) needed within this program def resource_path(relative_path): """ Get absolute path to resource, works for dev and for PyInstaller """ try:

# PyInstaller creates a temp folder and stores path in _MEIPASS base_path = sys._MEIPASS

   except Exception:
        base_path = os.path.abspath(".")
        return os.path.join(base_path, relative_path)`  

Also, in the appname.py script, add this 'resource_path' to get your images from your resource, like this:

yourimage = PhotoImage(file=resource_path("yourimage.png"))

In your appname.spec file replace 'datas=[], with your pathway to the images, you want to use. I used only '*.png' image files and this worked for me:

datas=[("/Users/rodman/PycharmProjects/tkinter/*.png", ".")],

Make sure you replace /Users/rodman/PycharmProjects/tkinter/ with your path, to the folder where your images are located. Forgive the sloppy code formmating, I am not used to these code tags, and thanks Steampunkery, for stearring me in the right direction to figure this Mac os x answer.

Answer 5

I solved this by taking advantage of the fact that the spec file is Python code that gets executed. You can get the root of the package dynamically during the PyInstaller build phase and use that value in the datas list. In my case I have something like this in my .spec file:

import os
import importlib

package_imports = [['package_name', ['file0', 'file1']]

datas = []
for package, files in package_imports:
    proot = os.path.dirname(importlib.import_module(package).__file__)
    datas.extend((os.path.join(proot, f), package) for f in files)

And use the the resulting datas list as a parameters to Analysis.

Answer 6

Edited to Add

To solve this problem more permanently, I created a pip-installable package called stringify which will take a file or directory and convert it into a python string so that packages such as pyinstaller will recognize them as native python files.

Check out the project page, feedback is welcome!

---

Original Answer

The answer is a bit roundabout and deals with the way that tk_tools is packaged rather than pyinstaller.

Someone recently made me aware of a technique in which binary data - such as image data - could be stored as a base64 string:

with open(img_path, 'rb') as f:
    encoded_string = base64.encode(f.read())

The encoded string actually stores the data. If the originating package simply stores the package files as strings instead of as image files and creates a python file with that data accessible as a string variable, then it is possible to simply include the binary data within the package in a form that pyinstaller will find and detect without intervention.

Consider the below functions:

def create_image_string(img_path):
    """
    creates the base64 encoded string from the image path 
    and returns the (filename, data) as a tuple
    """

    with open(img_path, 'rb') as f:
        encoded_string = base64.b64encode(f.read())

    file_name = os.path.basename(img_path).split('.')[0]
    file_name = file_name.replace('-', '_')

    return file_name, encoded_string


def archive_image_files():
    """
    Reads all files in 'images' directory and saves them as
    encoded strings accessible as python variables.  The image
    at images/my_image.png can now be found in tk_tools/images.py
    with a variable name of my_image
    """

    destination_path = "tk_tools"
    py_file = ''

    for root, dirs, files in os.walk("images"):
        for name in files:
            img_path = os.path.join(root, name)
            file_name, file_string = create_image_string(img_path)

            py_file += '{} = {}\n'.format(file_name, file_string)

    py_file += '\n'

    with open(os.path.join(destination_path, 'images.py'), 'w') as f:
        f.write(py_file)

If archive_image_files() is placed within the setup file, then the <package_name>/images.py is automatically created anytime the setup script is run (during wheel creation and installation).

I may improve on this technique in the near future. Thank you all for your assistance,

j

Answer 7

The following code will put all the PNG files in your directory into a folder in the top level of the bundled app called imgs:

datas=[("C:\\_code\\tools\\python\\aspen_comm\\dist\\aspen_comm\\tk_tools\\img\\*.png", "imgs")],

You can then reference them with os.path.join("imgs", "your_image.png") in your code.