How to include dependencies from venv directory when running pyinstaller for project built in Pycharm?

Question

I'm new to Python and PyCharm. I'm running on OSX High Sierra.

I've created a project in PyCharm that uses lxml. I've installed lxml 4.1.1 as a project dependency, and I can see it's files under myProject/venv/lib/site-package/lxml

I can run my script from within PyCharm no problem, but I've been unable to build an executable that includes the lxml library.

I use pyinstaller 3.3.1 from the command line. The project builds, but when I run the executable, I get his for output:

Traceback (most recent call last): File "analyze.py", line 13, in ImportError: No module named lxml [4222] Failed to execute script analyze

I've been looking for some command line switch that tells pyinstaller to include dependencies under the venv directory, with no luck.

Any help would be much appreciated...

Here is my build command and output:

pyinstaller --clean --onefile analyze.py
51 INFO: PyInstaller: 3.3.1
51 INFO: Python: 2.7.10
60 INFO: Platform: Darwin-17.4.0-x86_64-i386-64bit
60 INFO: wrote /Users/gludwig/PycharmProjects/XML_Analyzer/analyze.spec
66 INFO: UPX is not available.
66 INFO: Removing temporary files and cleaning cache in /Users/gludwig/Library/Application Support/pyinstaller
76 INFO: Extending PYTHONPATH with paths
['/Users/gludwig/PycharmProjects/XML_Analyzer',
 '/Users/gludwig/PycharmProjects/XML_Analyzer']
76 INFO: checking Analysis
76 INFO: Building Analysis because out00-Analysis.toc is non existent
76 INFO: Initializing module dependency graph...
79 INFO: Initializing module graph hooks...
130 INFO: running Analysis out00-Analysis.toc
138 INFO: Caching module hooks...
141 INFO: Analyzing /Users/gludwig/PycharmProjects/XML_Analyzer/analyze.py
2363 INFO: Loading module hooks...
2364 INFO: Loading module hook "hook-httplib.py"...
2364 INFO: Loading module hook "hook-encodings.py"...
2874 INFO: Looking for ctypes DLLs
2874 INFO: Analyzing run-time hooks ...
2879 INFO: Looking for dynamic libraries
2995 INFO: Looking for eggs
2995 INFO: Using Python library /System/Library/Frameworks/Python.framework/Versions/2.7/Python
2996 INFO: Warnings written to /Users/gludwig/PycharmProjects/XML_Analyzer/build/analyze/warnanalyze.txt
3009 INFO: Graph cross-reference written to /Users/gludwig/PycharmProjects/XML_Analyzer/build/analyze/xref-analyze.html
3087 INFO: checking PYZ
3087 INFO: Building PYZ because out00-PYZ.toc is non existent
3087 INFO: Building PYZ (ZlibArchive) /Users/gludwig/PycharmProjects/XML_Analyzer/build/analyze/out00-PYZ.pyz
3331 INFO: Building PYZ (ZlibArchive) /Users/gludwig/PycharmProjects/XML_Analyzer/build/analyze/out00-PYZ.pyz completed successfully.
3403 INFO: checking PKG
3403 INFO: Building PKG because out00-PKG.toc is non existent
3403 INFO: Building PKG (CArchive) out00-PKG.pkg
5669 INFO: Building PKG (CArchive) out00-PKG.pkg completed successfully.
5685 INFO: Bootloader /Users/gludwig/Library/Python/2.7/lib/python/site-packages/PyInstaller/bootloader/Darwin-64bit/run
5685 INFO: checking EXE
5685 INFO: Building EXE because out00-EXE.toc is non existent
5685 INFO: Building EXE from out00-EXE.toc
5686 INFO: Appending archive to EXE /Users/gludwig/PycharmProjects/XML_Analyzer/dist/analyze
5703 INFO: Fixing EXE for code signing /Users/gludwig/PycharmProjects/XML_Analyzer/dist/analyze
5716 INFO: Building EXE from out00-EXE.toc completed successfully.

Further, pyinstaller produces a warning file under build, and it contains the following:

missing module named org - imported by copy
missing module named _sha512 - imported by hashlib
missing module named _sha256 - imported by hashlib
missing module named _md5 - imported by hashlib
missing module named _sha - imported by hashlib
missing module named 'org.python' - imported by pickle
missing module named _subprocess - imported by subprocess
missing module named msvcrt - imported by subprocess, getpass
missing module named riscosenviron - imported by os
missing module named riscospath - imported by os
missing module named riscos - imported by os
missing module named ce - imported by os
missing module named _emx_link - imported by os
missing module named os2 - imported by os
missing module named nt - imported by os, ntpath
missing module named _winreg - imported by mimetypes, urllib
missing module named OverrideFrom23 - imported by Carbon.Res
missing module named SOCKS - imported by ftplib
missing module named rourl2path - imported by urllib
missing module named lxml - imported by /Users/gludwig/PycharmProjects/XML_Analyzer/analyze.py

I've also tried supplying paths for the library entries in venv to pyinstaller in this way:

pyi-makespec --paths=~/PycharmProjects/XML_Analyzer \ paths=~/PycharmProjects/XML_Analyzer/venv/lib/python2.7/sitepackages/lxml analyze.py pyinstaller --onefile analyze.py

It didn't help.

Answer 1

I ended up with the package I just started trying to package up, first I ran pyinstaller without using a venv and (due to pandas I think) it grabbed Cuda libs and etc., I ended up with a 5.1GB dist folder! Then I re-ran it in a venv and got the same size!

What I found I had to do on my system to have it use my venv and not just grab python stuff from wherever, I had to install pyinstaller in the virtual environment. If it runs the system-wide copy it feels free to grab from all over the system (whether this is a bug or by design I don't know, once it occured to me I had not installed a local pyinstaller, it seems sensible it should restrict to within the venv, but it also seems sensible that if you wanted to do that you'd install a copy within the venv.) So I did (in the Python venv):

pip install pyinstaller
hash -r

(The hash -r is to make the shell immediately "see" the new copy of pyinstaller, if you closed your terminal, re-opened it and went back into the python venv that'd work fine too.)

Then when I ran pyinstaller, it only grabbed items within the venv, I may still have a few extra items to shave off with --exclude-modules but I have a 120MB dist down from 5.1GB with no exclude-modules needed!

Answer 2

I think I figured it out, hope it still helps. Just install and launch auto-py-to-exe from within the virtual environment and you're done!

Answer 3

The answer provided by George put me in right direction but I opted for another path

I used the --paths command line option (see here) to indicate the location of the site-packages directory and that solved it for me. I am using windows and wanted a single executable so this is the command line

path\to\pyinstaller.exe --onefile --paths path\to\venv\Lib\site-packages file.py

Answer 4

If you use a spec file, you can add it like this:

a = Analysis(['myscript.py'],
             pathex=['.', ".\\path_to_libs", ".\\bla\\path_to_libs2"],
             hiddenimports=['pywinusb','CLogReport','otherLib'],
             ... )

This is useful, if you've got multiple files and/or from different paths.

Answer 5

Finally solved it, posting it here for others in the same boat:

It turns out that when using virtualenv and you want to do a build, you have to add the environment variable PYTHONPATH to point to the site-packages directory of your virtualenv directory. In my case, it looks like this:

PYTHONPATH="/Users/gludwig/PycharmProjects/XML_Analyzer/venv/lib/python2.7/site-packages"
export PYTHONPATH

For convenience sake, I added this to the activate file created by virtualenv so it's automatically set when I activate the virtualenv. I also added an "unset PYTHONPATH" to the deactivate section.

It would be super helpful if this was automatically done when virtualenv sets up the environment!