CODEWITHSUNDEEP
pythonpython-3.xscikit-learnpyinstallerlightgbm
javapyscript
javapyscript

Reputation: 737

Pyinstaller exe conversion fail - using lightgbm and sklearn

I am trying to convert my code into an exe using pyinstaller spec. Pyinstaller initially failed with the following reason:

main__.PyInstallerImportError: Failed to load dynlib/dll 
'C:\\Users\\...\\lightgbm\\../compile\\lib_lightgbm.dll'. Most probably this 
dynlib/dll was not found when the application was frozen.

I tried to correct it by adding the following line to my pathex list in the spec:

'myenv\\lib\\site-packages\\lightgbm'

Note: myenv is my virtualenv created for this project.

Which led to ImportError for sklearn. I added sklearn to hidden imports. This is my final spec fie:

# -*- mode: python -*-
import sys
sys.setrecursionlimit(5000)

block_cipher = None


a = Analysis(['myscript.py'],
             pathex=['C:\\project_folder', 'myenv\\lib\\site-packages\\lightgbm'],
             binaries=[],
             datas=[('lib_lightgbm.dll', '.')],
             hiddenimports=['cython', 'sklearn', 'sklearn.feature_extraction','sklearn.pipeline', 'sklearn.ensemble', 'sklearn.neighbors.typedefs', 'sklearn.neighbors.quad_tree', 'sklearn.tree._utils'],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          [],
          exclude_binaries=True,
          name='myscript',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          console=True )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               name='myscript')

This leads to an error I find very weird:

File "myenv\lib\site-packages\lightgbm\sklearn.py", line 9, in <module>
ImportError: attempted relative import with no known parent package
[12692] Failed to execute script myscript

I have no idea why it would be searching for sklearn.py inside lightgbm? There is sklearn.py inside lightgbm, but I am not sure how to fix this error. Also, there is no sklearn folder in the dist/myscript folder. Ideally it should have been present. I tried to manually copy the sklearn folder, although, as suspected, it made no difference. Could someone please let me know where the spec file is wrong?

Upvotes: 2

Views: 1277

Answers (1)

afp
afp

Reputation: 103

Let's go back to the first error you had. That error suggests lib_lightgbm.dll was not seen when your application was frozen.

Assuming you run pyinstaller from Windows cmd.exe, you can fix this by passing a hook for lightgbm to pyinstaller, so that it knows where to get it from, e.g.

pyinstaller --additional-hooks-dir dir_with_lightgbm_hook --hidden-import lightgbm myscript.py

The name of the hook should be hook-lightgbm.py and for its contents, you can look here: https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/hooks/hook-numpy.py , it should be something similar. Just make sure that the library is added to the binaries list, i.e.

binaries.append((os.path.join(os.path.dirname(
get_module_file_attribute('lightgbm')), "lib_lightgbm.dll"), "lightgbm"))

The second "lightgbm" argument is the location where the .dll will be copied, relative to your distribution directory, so in your case it should end up as dist\myscript\lightgbm\lib_lightgbm.dll.

Note: Adding lib_lightgbm.dll through a hook will make pyinstaller add the dependencies for it to the distribution, so it is better to do it like this than just copying it manually.

Second note: When copying your distribution package to another machine, you might need to install the appropriate Visual Studio redistributable used by the library (e.g. VS 2015 redistributable).

EDIT: Forgot to mention that if you do it like this, you can remove the path you added to your pathex.

Upvotes: 2

Related Questions