spacy 2.2.3 FileNotFoundError: [Errno 2] No such file or directory: 'thinc\\neural\\_custom_kernels.cu' in pyinstaller

Question

I was trying to create a executable file using pyinstaller. I got bellow issue while executing the issue.

 File "test_env2_live\main.py", line 2, in <module>
 File "C:\Users\rajesh.das\AppData\Local\Continuum\anaconda3\envs\test_env2\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
exec(bytecode, module.__dict__)
File "test_env2_live\controller\main.py", line 2, in <module>
File "C:\Users\rajesh.das\AppData\Local\Continuum\anaconda3\envs\test_env2\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
exec(bytecode, module.__dict__)
File "test_env2_live\controller\inflow\job_controller.py", line 175, in <module>
File "C:\Users\rajesh.das\AppData\Local\Continuum\anaconda3\envs\test_env2\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
exec(bytecode, module.__dict__)
File "site-packages\spacy\__init__.py", line 10, in <module>
File "C:\Users\rajesh.das\AppData\Local\Continuum\anaconda3\envs\test_env2\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
exec(bytecode, module.__dict__)
File "site-packages\thinc\neural\__init__.py", line 4, in <module>
File "C:\Users\rajesh.das\AppData\Local\Continuum\anaconda3\envs\test_env2\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
exec(bytecode, module.__dict__)
File "site-packages\thinc\neural\_classes\model.py", line 11, in <module>
File "C:\Users\rajesh.das\AppData\Local\Continuum\anaconda3\envs\test_env2\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
exec(bytecode, module.__dict__)
File "site-packages\thinc\neural\train.py", line 7, in <module>
File "optimizers.pyx", line 14, in init thinc.neural.optimizers
File "ops.pyx", line 24, in init thinc.neural.ops
ImportError: cannot import name _custom_kernels

So I added --hiddenimport thinc.neural._custom_kernels at the time of build executable file. Bellow code for build executable file

pyinstaller main.py --hiddenimport preshed.maps --hiddenimport srsly.msgpack.util --hidden-import="sklearn.utils._cython_blas" --hidden-import="sklearn.neighbors.typedefs" --hidden-import="sklearn.neighbors.quad_tree" --hidden-import="sklearn.tree._utils" --hidden-import cymem.cymem --hidden-import thinc.linalg --hidden-import murmurhash.mrmr --hidden-import cytoolz.utils --hidden-import cytoolz._signatures --hidden-import spacy.strings --hidden-import spacy.morphology --hidden-import spacy.lexeme --hidden-import spacy.tokens --hidden-import spacy.gold --hidden-import spacy.tokens.underscore --hidden-import spacy.parts_of_speech --hidden-import dill --hidden-import spacy.tokens.printers --hidden-import spacy.tokens._retokenize --hidden-import spacy.syntax --hidden-import spacy.syntax.stateclass --hidden-import spacy.syntax.transition_system --hidden-import spacy.syntax.nonproj --hidden-import spacy.syntax.nn_parser --hidden-import spacy.syntax.arc_eager --hidden-import thinc.extra.search --hidden-import spacy.syntax._beam_utils --hidden-import spacy.syntax.ner --hidden-import thinc.neural._classes.difference --hidden-import spacy.vocab --hidden-import spacy.lemmatizer --hidden-import spacy._ml --hidden-import spacy.lang.en --hiddenimport thinc.neural._custom_kernels

But now executable file gives below error

File "site-packages\thinc\neural\train.py", line 7, in <module>
File "optimizers.pyx", line 14, in init thinc.neural.optimizers
File "ops.pyx", line 24, in init thinc.neural.ops
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "C:\Users\rajesh.das\AppData\Local\Continuum\anaconda3\envs\test_env2\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
exec(bytecode, module.__dict__)
File "site-packages\thinc\neural\_custom_kernels.py", line 36, in <module>
File "pathlib.py", line 1183, in open
File "pathlib.py", line 1037, in _opener
File "pathlib.py", line 387, in wrapped
FileNotFoundError: [Errno 2] No such file or directory: 'D:\\rajesh\\python\\console\\test_env2_live\\dist\\main\\thinc\\neural\\_custom_kernels.cu'

Not sure why it gives FileNotFoundError error. Can any one help me?

Configuration: Python 3.6.7, conda 4.8.0, spacy v2.2.3, Windows 10

I can successfully run executable file with lower version of spacy(2.0.18). But it gives error in 2.2.3 version.

Thanks in Advance

Answer 1

# -*- mode: python ; coding: utf-8 -*-

block_cipher = None

import PyInstaller

# HOOK FILE FOR SPACY
from PyInstaller.utils.hooks import collect_all
from PyInstaller.utils.hooks import collect_data_files

# ----------------------------- SPACY -----------------------------
data = collect_all('spacy')

datas = data[0]
binaries = data[1]
hiddenimports = data[2]

# ----------------------------- THINC -----------------------------
data = collect_all('thinc')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- CYMEM -----------------------------
data = collect_all('cymem')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- PRESHED -----------------------------
data = collect_all('preshed')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- BLIS -----------------------------

data = collect_all('blis')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- OTHER ----------------------------

hiddenimports += ['srsly.msgpack.util']


# ----------------------------- Sacy Model -------------------------
datas += collect_data_files("en_core_web_sm")


a = Analysis(['ttest_spacy.py'],
             pathex=['PATH/TO/PYFILE'],
             binaries=binaries,
             datas=datas,
             hiddenimports = hiddenimports,
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          [],
          exclude_binaries=True,
          name='ttest_spacy',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          console=True )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               upx_exclude=[],
               name='ttest_spacy')

Answer 2

I agree with @Legorooj. Adding to his answer, I had to add another hidden import for srsly.msgpack.util in the end. Also in my script I was using "en_core_web_sm":

import en_core_web_sm
nlp = en_core_web_sm.load()

So I added a hook file for "en_core_web_sm" as well. The hook-spacy.py file become:

# HOOK FILE FOR SPACY
from PyInstaller.utils.hooks import collect_all

# ----------------------------- SPACY -----------------------------
data = collect_all('spacy')

datas = data[0]
binaries = data[1]
hiddenimports = data[2]

# ----------------------------- THINC -----------------------------
data = collect_all('thinc')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- CYMEM -----------------------------
data = collect_all('cymem')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- PRESHED -----------------------------
data = collect_all('preshed')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- BLIS -----------------------------

data = collect_all('blis')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- OTHER ----------------------------

hiddenimports += ['srsly.msgpack.util']

And the hook-en_core_web_sm file is:

from PyInstaller.utils.hooks import collect_data_files
datas = collect_data_files("en_core_web_sm")

Answer 3

I adjusted .spec file and added main.spec file with below details and it works.

# -*- mode: python ; coding: utf-8 -*-

import PyInstaller

datas = []
datas.extend(PyInstaller.utils.hooks.collect_data_files('spacy.lang', include_py_files = True))
datas.extend(PyInstaller.utils.hooks.collect_data_files('thinc'))

block_cipher = None
a = Analysis(['main.py'],
         pathex=['D:\\rajesh\\python\\console\\live'],
         binaries=[],
         datas=datas,
         hiddenimports = [
            'spacy.kb',
            'spacy.lexeme',
            'spacy.matcher._schemas',
            'spacy.morphology',
            'spacy.parts_of_speech',
            'spacy.syntax._beam_utils',
            'spacy.syntax._parser_model',
            'spacy.syntax.arc_eager',
            'spacy.syntax.ner',
            'spacy.syntax.nn_parser',
            'spacy.syntax.stateclass',
            'spacy.syntax.transition_system',
            'spacy.tokens._retokenize',
            'spacy.tokens.morphanalysis',
            'spacy.tokens.underscore',

            'spacy._align',

            'blis',
            'blis.py',

            'cymem',
            'cymem.cymem',

            'murmurhash',
            'murmurhash.mrmr',

            'preshed.maps',

            'srsly.msgpack.util',

            'thinc.extra.search',
            'thinc.linalg',
            'thinc.neural._aligned_alloc',
            'thinc.neural._custom_kernels',

            'sklearn.utils._cython_blas',
            'sklearn.neighbors.typedefs',
            'sklearn.neighbors.quad_tree',
            'sklearn.tree._utils'
        ],
         hookspath=[],
         runtime_hooks=[],
         excludes=[],
         win_no_prefer_redirects=False,
         win_private_assemblies=False,
         cipher=block_cipher,
         noarchive=False)
   pyz = PYZ(a.pure, a.zipped_data,
         cipher=block_cipher)
    exe = EXE(pyz,
      a.scripts,
      [],
      exclude_binaries=True,
      name='main',
      debug=False,
      bootloader_ignore_signals=False,
      strip=False,
      upx=True,
      console=True )
   coll = COLLECT(exe,
           a.binaries,/
           a.zipfiles,
           a.datas,
           strip=False,
           upx=True,
           upx_exclude=[],
           name='main')

Answer 4

The FileNotFound error is because PyInstaller isn't packaging thinc properly; thinc needed a hook. I've found that as script containing from spacy import * will work with the hook file below. The command I used was:

pyinstaller test-spacy.py --additional-hooks-dir=.

This worked because I added a hook for spacy (and it's submodules) in the same directory as the test-spacy.py script. Simply copy the below text into a file called hook-spacy.py, that's in the same directory as your script.

# HOOK FILE FOR SPACY
from PyInstaller.utils.hooks import collect_all

# ----------------------------- SPACY -----------------------------
data = collect_all('spacy')

datas = data[0]
binaries = data[1]
hiddenimports = data[2]

# ----------------------------- THINC -----------------------------
data = collect_all('thinc')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- CYMEM -----------------------------
data = collect_all('cymem')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- PRESHED -----------------------------
data = collect_all('preshed')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- BLIS -----------------------------

data = collect_all('blis')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]
# This hook file is a bit of a hack - really, all of the libraries should be in seperate hook files. (Eg hook-blis.py with the blis part of the hook)

Ask in the comments if you need any more help.