py2exe not including the modules from "includes"

Question

I am using the latest version of Anaconda Python (3.4) and latest version of py2exe (which is supposed to support Python 3.3 and 3.4) on Windows 7, 64-bit.

Here is the py2exe script I am using:

from distutils.core import setup
import py2exe, sys, os

sys.argv.append('py2exe')

setup(
    options = {
                    'py2exe': {
                            'bundle_files': 2,
                            'optimize': 2,
                            'compressed': True,
                    }
               },
    windows = [{'script': "ImageSound GUI.py"}],
    data_files = [('images',['c:/Users/EvilDragon/Documents/GitHub/ImageSound/mario.png'])]
)

Here is the output that py2exe returns:

  84 missing Modules                 
  ------------------                 
? Cookie                              imported from requests.compat, tornado.httputil, tornado.web
? IPython.current_nbformat            imported from IPython.nbformat.validator
? Image                               imported from scipy.misc.pilutil
? ImageFilter                         imported from scipy.misc.pilutil
? OpenGL                              imported from IPython.lib.inputhook, IPython.lib.inputhookglut
? PIL._imagingagg                     imported from PIL.ImageDraw
? PyQt5                               imported from IPython.external.qt_loaders, PIL.ImageQt, matplotlib.pyplot
? PySide                              imported from IPython.external.qt_loaders, PIL.ImageQt
? Queue                               imported from IPython.kernel.blocking.channels, IPython.kernel.blocking.client, IPython.kernel.inprocess.blocking, IPython.kernel.inprocess.socket, IPython.kernel.manager, requests.packages.urllib3.connectionpool, zmq.eventloop.zmqstream
? SocketServer                        imported from zmq.ssh.forward
? UserDict                            imported from pytz.lazy
? _abcoll                             imported from requests.packages.urllib3.packages.ordered_dict
? _grabscreen                         imported from PIL.ImageGrab
? _imaging_gif                        imported from PIL.GifImagePlugin
? _subprocess                         imported from IPython.kernel.launcher, IPython.kernel.zmq.parentpoller
? _util                               imported from PIL.ImageCms
? appnope                             imported from IPython.external.appnope
? backports                           imported from requests.packages.urllib3.packages.ssl_match_hostname, setuptools.ssl_support, tornado.netutil
? cloud                               imported from IPython.utils.pickleutil
? cookielib                           imported from requests.compat
? coverage                            imported from IPython.testing.iptestcontroller
? dill                                imported from IPython.utils.pickleutil
? gi                                  imported from IPython.kernel.zmq.gui.gtk3embed, IPython.lib.inputhookgtk3
? glut_support                        imported from IPython.lib.inputhook
? gobject                             imported from IPython.kernel.zmq.gui.gtkembed, IPython.lib.inputhookgtk, matplotlib.pyplot
? gtk                                 imported from IPython.kernel.zmq.gui.gtkembed, IPython.lib.inputhook, IPython.lib.inputhookgtk
? isodate                             imported from jsonschema._format
? matplotlib.backends._macosx         imported from matplotlib.backends.backend_macosx
? mklfft                              imported from numpy.fft
? monotime                            imported from tornado.platform.auto, zmq.eventloop.minitornado.platform.auto
? mpl_toolkits.natgrid                imported from matplotlib.mlab
? multiprocessing.SimpleQueue         imported from concurrent.futures.process
? ndg                                 imported from requests.packages.urllib3.contrib.pyopenssl
? netifaces                           imported from IPython.utils.localinterfaces
? packaging                           imported from pkg_resources
? paramiko                            imported from zmq.ssh.tunnel
? path                                imported from IPython.external.path
? pexpect                             imported from IPython.external.pexpect, zmq.ssh.tunnel
? pyczmq                              imported from zmq.sugar.context
? pydb                                imported from IPython.core.debugger, IPython.core.interactiveshell
? pyglet                              imported from IPython.lib.inputhookpyglet
? pysqlite2                           imported from IPython.core.history
? pywintypes                          imported from -
? repr                                imported from IPython.core.debugger
? resource                            imported from IPython.external.pexpect._pexpect, IPython.utils.timing
? rfc3987                             imported from jsonschema._format
? scikits                             imported from scipy.sparse.linalg.dsolve.linsolve
? scipy.arange                        imported from scipy.sparse.linalg.isolve.minres
? scipy.ones                          imported from scipy.sparse.linalg.isolve.minres
? setuptools_svn                      imported from setuptools.command.egg_info
? simplegeneric                       imported from IPython.external.simplegeneric
? simplejson                          imported from requests.compat
? singledispatch                      imported from tornado.gen
? strict_rfc3339                      imported from jsonschema._format
? unittest._WritelnDecorator          imported from nose.plugins.multiprocess
? urllib.getproxies                   imported from requests.compat
? urllib.pathname2url                 imported from setuptools.compat
? urllib.proxy_bypass                 imported from requests.compat
? urllib.quote                        imported from matplotlib.textpath, requests.compat
? urllib.quote_plus                   imported from requests.compat
? urllib.splittag                     imported from setuptools.compat
? urllib.unquote                      imported from jsonschema.compat, requests.compat
? urllib.unquote_plus                 imported from requests.compat
? urllib.url2pathname                 imported from setuptools.compat
? urllib.urlencode                    imported from IPython.lib.display, requests.compat, requests.packages.urllib3.request, tornado.httputil, tornado.web
? urllib.urlopen                      imported from IPython.core.interactiveshell, IPython.utils.openpy
? urllib.urlretrieve                  imported from IPython.core.extensions, IPython.html.nbextensions
? webcolors                           imported from jsonschema._format
? win32api                            imported from IPython.utils._process_win32, distutils.msvccompiler, platform, win32evtlogutil
? win32com                            imported from setuptools.sandbox
? wincertstore                        imported from setuptools.ssl_support
? zmq.EAGAIN                          imported from zmq.error
? zmq.ETERM                           imported from zmq.devices.basedevice, zmq.error, zmq.eventloop.ioloop
? zmq.PUB                             imported from zmq.devices.monitoredqueuedevice
? zmq.QUEUE                           imported from zmq.devices.basedevice
? zmq.backend.Context                 imported from zmq.sugar.context
? zmq.backend.Frame                   imported from zmq.sugar.frame, zmq.sugar.tracker
? zmq.backend.Socket                  imported from zmq.sugar.socket
? zmq.backend.constants               imported from zmq.sugar.constants
? zmq.backend.strerror                imported from zmq.error
? zmq.backend.zmq_errno               imported from zmq.error
? zmq.backend.zmq_poll                imported from zmq.sugar.poll
? zmq.backend.zmq_version_info        imported from zmq.sugar.version
? zmq.device                          imported from zmq.devices, zmq.devices.basedevice

  5 submodules that appear to be missing, but could also be global names in the parent package
  ---------------------------------------------------------------------------------------------
? numpy.conj                          imported from scipy.io.mmio, scipy.linalg.decomp
? numpy.log2                          imported from scipy.sparse.linalg.matfuncs
? scipy.spatial.cKDTree               imported from scipy.spatial
? scipy.special.ellipeinc             imported from scipy.special
? scipy.special.ellipkinc             imported from scipy.special
Building 'dist\ImageSound GUI.exe'.
Building shared code archive 'dist\library.zip'.
Copy c:\program files\anaconda\python34.dll to dist
Copy DLL C:\Program Files\Anaconda\DLLs\tcl86t.dll to dist
Copy DLL C:\Program Files\Anaconda\DLLs\tk86t.dll to dist
Copy DLL C:\Program Files\Anaconda\lib\site-packages\numpy\core\libiomp5md.dll to dist
Copy DLL C:\Program Files\Anaconda\DLLs\sqlite3.dll to dist
Copy DLL C:\Program Files\Anaconda\lib\site-packages\PyQt4\QtCore4.dll to dist
Copy DLL C:\Program Files\Anaconda\lib\site-packages\portaudio_x64.dll to dist
Copy DLL C:\Program Files\Anaconda\lib\site-packages\PyQt4\QtSvg4.dll to dist
Copy DLL C:\Program Files\Anaconda\lib\site-packages\PyQt4\QtGui4.dll to dist

Then when I try to run my EXE I get this traceback:

Traceback (most recent call last):
  File "ImageSound.py", line 5, in <module>
  File "C:\Program Files\Anaconda\lib\site-packages\zipextimporter.py", line 116, in load_module
    raise zipimport.ZipImportError("can't find module %s" % fullname)
zipimport.ZipImportError: can't find module skimage.draw

This doesn't change even if I add skimage.draw to includes... And also traceback changes if I set bundle_files to 3:

Traceback (most recent call last):
  File "ImageSound.py", line 5, in <module>
  File "C:\Program Files\Anaconda\lib\site-packages\skimage\draw\__init__.py", line 1, in <module>
    from .draw import circle, ellipse, set_color
  File "C:\Program Files\Anaconda\lib\site-packages\skimage\draw\draw.py", line 3, in <module>
    from ._draw import _coords_inside_image
  File "<loader>", line 10, in <module>
  File "<loader>", line 8, in __load
ImportError: (No module named 'skimage._shared') 'C:\\Users\\EvilDragon\\Documents\\GitHub\\ImageSound\\dist\\skimage.draw._draw.pyd'

And this is completely beyond me, because it points to bytecode, however the only function I use from skimage.draw is Line, and that's it! I am completely baffled at this point.

EDIT: I have tried running py2exe with various includes and excludes, so that it tries to recursively find what it needs, but to no avail. It would always report missing skimage.draw or no module named skimage._shared. Makes no sense, they are RIGHT THERE, in my lib/site-packages/ folder!

Also anyone interested in giving it a shot with py2exe can check out the code of the program here: https://github.com/mkruselj/ImageSound. Necessary libraries: numpy, scipy, skimage, PIL, pyaudio.

Here are all the includes from the two .py files I am using (one .py file is imported into another).

ImageSound.py:

from tkinter import *
from tkinter import filedialog, ttk, messagebox
from PIL import Image, ImageTk
from numpy import array
from skimage.draw import line as skline
import time
import DSP  # this is the second .py file

DSP.py:

from numpy import linspace, sin, pi, int16, array, append, multiply
from scipy.io.wavfile import write as writewav
from scipy.interpolate import UnivariateSpline as interpolate
from math import sqrt
import pyaudio

Thanks for any insight!

EDIT #2: cx_freeze and PyInstaller don't want to cooperate, either. I have tried compiling stuff directly using nuitka, but that's even worse as it necessitates both py2 and py3 installed along with a C++ compiler. I failed miserably.

Answer 1

Well, since cgohlke isn't posting his suggestion as an answer, let me do so.

This setup.py has resulted in a working executable for me, using py2exe:

from distutils.core import setup
import py2exe, sys, os, skimage, PIL, numpy, scipy, pyaudio

sys.argv.append('py2exe')

setup(
    options = {
                    'py2exe': {
                            'bundle_files': 3,
                            'optimize': 2,
                            'includes': ['pyaudio', 'scipy', 'numpy', 'skimage', 'PIL', 'difflib', 'locale', 'inspect', 'skimage._shared', 'skimage._shared.geometry', 'scipy.special._ufuncs_cxx', 'scipy.linalg.cython_blas', 'scipy.linalg.cython_lapack', 'scipy.integrate', 'scipy.sparse.csgraph._validation'],
                            'excludes': ['pkg_resources','doctest', 'pdb', 'calendar', 'optparse', 'jsonschema', 'tornado', 'setuptools', 'distutils', 'matplotlib']
                    }
               },
    windows = [{'script': 'ImageSound.py'}],
    data_files = [('images',['images/author.png'])]
)

The trick was in including all the used libraries in setup.py as well, and not use bundled files option (unfortunately for me)!

Answer 2

Instead of using "includes" I would suggest to use "packages" for the packages you have problems with.

Have a look at "http://wiki.wxpython.org/py2exe-python26" a bit outdated, but most if not all still applies for latest py2exe, especially look at "zipfile = r'lib\library.zip'," to get your dist folder better organized.

Answer 3

I had dabbled with pyexe some time ago and had problems as well (though that was with Python2.7 and qt4). I have found pyinstaller to be a better package for executables. As the website says pyinstaller is almost ready for Python 3, it might be worth giving it a try.