PyInstaller and Pandas

Question

I have a fairly simple Python module that I am trying to compile into a Windows .exe file. In my script I am using the wxPython and Pandas libraries. The PyInstaller .exe file that is generated only works/opens when the Pandas library is excluded from my module.

I am getting the same issue whether I use --onefile or --onedir in PyInstaller. I found online that the "new" version of PyInstaller (2.1) should have taken care of this bug. Does anyone have any ideas on what to do?

PyInstaller: version 2.1
pandas: version 0.15.2
Python: version 2.7

Answer 1

I had the same error when trying to use pandas. One thing it worked for me is to see the messages that appeared when executing Pyinstaller. It appeared me the following:

2468015 WARNING: Library not found: could not resolve 'msvcp140-ef6047a69b174ada5cb2eff1d2bc9a62.dll', dependency of 'C:\\Users\\ASUS\\anaconda3\\lib\\site-packages\\pandas\\_libs\\window\\aggregations.cp39-win_amd64.pyd'

So apparently the problem was that msvcp140-ef6047a69b174ada5cb2eff1d2bc9a62.dll is required for pandas but pyinstaller doesn't consider it. What I did is to copy the file that is in my computer and paste it in the file that pyinstaller has generated and that contains all the dll files. In my case is called _internal folder. It is IMPORTANT that pyinstaller is executed in --onedir to have the folder where dll are located. If not it wont be possible.

Answer 2

My solution found in 2023 March:

ADD pyinstaller to the .venv package list

Background:

Environment:
    MacOS Ventura (13.2.1)
    Developed in PyCharm @022.3.2 (Professional Edition) 
    Using pipenv version 2023.2.18 (brew installation)
    pandas 1.5.3
    pyinstaller 5.9.0

The solution I found was to make SURE I had installed pyinstaller as a package into Pycharm's Settings/Project/Python Interpreter/package list --- added pyinstaller

And now everything WORKS fine.

Answer 3

This issue has been solved in pyinstaller 4.10

Answer 4

if anyone here is using python 3.10 :

you will probably encounter "IndexError: tuple index out of range" error.

to fix it:

I found a solution to this problem. This is a python 3.10 bug, and after applying the fix everything related to pyinstaller started working properly.

" so basically you have to go to the folder "Python310\Lib" and edit the file 'dis.py'. In the 'dis.py' file you have to find this "def _unpack_opargs" and inside the else statement write a new line with this: "extended_arg = 0".

I did something like that:

    else:
        arg = None
        extended_arg = 0 
    yield (i, op, arg)

and everything is working fine now. " https://github.com/pyinstaller/pyinstaller/issues/6301#issuecomment-974927257

Answer 5

For anyone looking to convert Python scripts with pandas involved to exe in 2021

I struggled a lot with pyinstaller but got perfect results with Nuitka. Perks:

• low file size (500kB)
• quick build and execution time (similar to Python execution time)
• and best of all: pandas support out-of-the-box without virtual envs!

Just install nuitka

pip install nuitka

and build the exe

nuitka hello.py

You can even use upx afterwards to compress further!

upx hello.exe

In my case reduced file size to 1/3, just ~150kB .exe file size for a simple pandas script reading and saving an excel-file.

See my blog post for further details.

Answer 6

I had exactly the same problem, and I found another solution (the only one that worked for me actually) :

I followed pretty much this : https://medium.com/@liron92/pyinstaller-with-pandas-problems-solutions-and-workflow-with-code-examples-c72973e1e23f

Except for the fact that I run my virtual environnement on Anaconda.

Quick step-by-step sollution

Before I start : these are the steps that I followed for my particuliar case, you might want to adapt a little bit depending on your situation.

1) Create and configure your virtual environnement

I used Anaconda to create my env :

conda create --name myenv

Then I installed all the module I needed :

conda install -n myenv pandas
conda install -n myenv -c conda-forge python-docx
etc.

2) Activate your environnement and switch to project path

On Anaconda Prompt :

conda activate myenv
cd path/to/your/project/folder

3) Create and modify your *.spec file

Still on the same Anaconda Prompt window :

pyi-makespec project.py

Then open your project.spec file, it will looks like that:

# -*- mode: python ; coding: utf-8 -*-

block_cipher = None


a = Analysis(['project.py'],
             pathex=['path/to/your/project/folder'],
             binaries=[],
             datas=[],
             hiddenimports=[],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          [],
          exclude_binaries=True,
          name='main',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          console=False )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               upx_exclude=[],
               name='main')

You just modify hiddenimports = [] and add all the implicit import (which include pandas). In my case, I was also using Tkinter, so I specified :

hiddenimports=['pandas', 'tkinter']

4) Finally run pyinstaller

On the same Anaconda Prompt window (environnement activated, on your project directory) :

pyinstaller main.spec

And then you're done !

Disclaimer

I see everywhere people telling you that you should use --onefile when you compile a Python project with Pyinstaller, I honestly think you shouldn't : it makes the *.exe way slower. Maybe I'm missing something with this so please if you use it explain to me.

Specs

Windows 10
Anaconda 4.8.2
Python 3.7.6
Pandas 1.0.5

Answer 7

With python version=3.8 and pyinstaller=3.6, no need to custom the pyinstaller or add pandas hook, the hook-pandas.py already exists in Lib\site-packages\PyInstaller\hooks and everything works fine.

Answer 8

I solved the same problem by using a hook file in the project directory (per pyinstaller document), hook-pandas.py

hiddenimports = [
    'pandas._libs.tslibs.timedeltas',
    'pandas._libs.tslibs.nattype',
    'pandas._libs.tslibs.np_datetime',
    'pandas._libs.skiplist',
]

then adding one line in the spec file:

...
a = Analysis([...
hookspath=['.'],
...],
...

I tried to include hiddenimports=[..., 'pandas', ...] in the spec file, somehow that didn't work out as expected.

Answer 9

Just as another solution, adding --hidden-import=pandas._libs.tslibs.timedelta or whatever the module is missing to the pyinstaller command also works.

This may be helpful if you don't want to touch the source of pyinstaller.

Answer 10

I had a similar issue with pyinstaller version 3.3. The solution was that there was a missing hiddenimport hook as described here

I created a new file under Pyinstaller/hooks/ called hook-pandas.py and put the content as described in this commit here and reinstalled pyinstaller manually via python setup.py install in the Pyinstaller directory.

The issue did not recur when I created exe from my pandas script with pyinstaller using the --onefile option.

Answer 11

I ran into the same issue. I boiled it down to a simple script like this Hello.py:

import pandas
print "hello world, pandas was imported successfully!"

To get pandas to import at run-time correctly I had to modify the Hello.spec to the following:

# -*- mode: python -*-

block_cipher = None

def get_pandas_path():
    import pandas
    pandas_path = pandas.__path__[0]
    return pandas_path

a = Analysis(['Hello.py'],
         pathex=['C:\\ScriptsThatRequirePandas'],
         binaries=None,
         datas=None,
         hiddenimports=[],
         hookspath=None,
         runtime_hooks=None,
         excludes=None,
         win_no_prefer_redirects=None,
         win_private_assemblies=None,
         cipher=block_cipher)

dict_tree = Tree(get_pandas_path(), prefix='pandas', excludes=["*.pyc"])
a.datas += dict_tree
a.binaries = filter(lambda x: 'pandas' not in x[0], a.binaries)

pyz = PYZ(a.pure, a.zipped_data,
         cipher=block_cipher)
exe = EXE(pyz,
      a.scripts,
      exclude_binaries=True,
      name='Hello',
      debug=False,
      strip=None,
      upx=True,
      console=True )
scoll = COLLECT(exe,
           a.binaries,
           a.zipfiles,
           a.datas,
           strip=None,
           upx=True,
           name='Hello')

I then ran:

$pyinstaller Hello.spec --onefile

from the command prompt and got the 'hello world' message I expected. I still don't completely understand why this is necessary. I have a custom build of pandas - which is hooked into the MKL libraries - but it isn't clear to me that this is causing the run failure.

This is similar to the answer here: Pyinstaller not correclty importing pycripto... sometimes