CODEWITHSUNDEEP
pythonspacypython-packaging
Osvald Laurits
Osvald Laurits

Reputation: 1364

Package spacy model

I want to include the spacy model de_core_news_sm in a python package.

Here is my project: https://github.com/michaelhochleitner/package_de_core_news_sm .

I package and install the project with the following commands.

python setup.py sdist bdist_wheel
pip install dist/example-pkg-mh-0.0.1.tar.gz

I want to import the module example_pkg.import-model.py .

$ python
>>> import example_pkg.import_model
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mh/PycharmProjects/packaging_tutorial/example_pkg/import_model.py", line 2, in <module>
    import de_core_news_sm
ModuleNotFoundError: No module named 'de_core_news_sm'

How can I include the module 'de_core_news_sm' into the package so that it's installed after running the following command?

pip install dist/example-pkg-mh-0.0.1.tar.gz

Upvotes: 2

Views: 1069

Answers (2)

Arne
Arne

Reputation: 20217

If you want to spare your users from running that download, you'll have to package and distribute it with your own source. This process is called vendoring (see this great post for an in-depth explanation of how to best do it in python, or the pip-project's _vendor/__init__.py for a commented example), and it can be quite convenient, but prone to get you into annoying issues if overdone.

Simply put, you create an additional python package called _vendor (or something similar) in the source code directory of your package and copy the downloaded de_core_news_sm package into it:

example_pkg
├── import_model.py
├── __init__.py
└── _vendor
    ├── __init__.py
    └── de_core_news_sm
        ├── de_core_news_sm-2.1.0
        │   ├── accuracy.json
        │   ├── meta.json
        │   ├── ner/
        │   ├── parser/
        │   ├── tagger/
        │   ├── vocab/
        │   └── tokenizer
        ├── __init__.py
        └── meta.json

You can find the package in the site_packages of the python interpreter that you installed it into with python -m spacy download de_core_news_sm, i.e. $(which python)/site_packages/de_core_news_sm.

Finally, you need to change all your imports of the model from import de_core_news_sm to from example_pkg._vendor import de_core_news_sm, and then it should work.

Upvotes: 2

g.lahlou
g.lahlou

Reputation: 478

Did you try on command line:

python -m spacy download de_core_news_sm

Upvotes: 1

Related Questions