CODEWITHSUNDEEP
pythonpipenvkedro
jayBana
jayBana

Reputation: 425

How to use Kedro with Pipenv?

I am currently using kedro, version 0.15.4 with pipenv, version 2018.11.26.

At the moment, I have to do the following if I want to use Pipenv (For this example, I want this project to reside in the kedro-pipenv directory):

  1. mkdir kedro-pipenv && cd kedro-pipenv
    • This is needed so that the virtualenv created is "tied" to the project directory which only really means that it has a name that is based on the directory name from which the pipenv install kedro or pipenv shell commands are executed.
    • Unfortunately, at the moment Pipenv doesn't have the functionality to support custom virtualenv names.
  2. pipenv install kedro
    • In my example, this generates a virtualenv with name kedro-pipenv-AB9IGRnB which resides in the following location ~/.local/share/virtualenvs/kedro-pipenv-AB9IGRnB/
  3. pipenv shell
  4. kedro info
    • all is looking good, kedro was successfully installed in the virualenv handled by Pipenv
  5. cd .. & kedro new
    • I navigate one directory up and provide kedro-pipenv as the directory name for the project. Given that the directory was already created before in step 1, this fails which is expected and I get the following message:
cookiecutter.exceptions.OutputDirExistsException:
Error: "/Users/xyz/projects/kedro-pipenv" directory already exists
Run with --verbose to see the full exception
Error: Failed to generate project.

In order to "work around" this, I do the following whilst still in the same virtualenv as before:

I am familiar with conda as well and it seems that it is a much cleaner way of handling environments for Kedro at the moment.

However, for most of my other projects I have been using pyenv in conjunction with Pipenv for environment and dependency management. This allows me to have environment information tied to specific project spaces by having a Pipfile in each of my projects' root directory.

Has anyone got any suggestions on how to improve the above workflow?

Upvotes: 2

Views: 381

Answers (1)

Lorena Balan
Lorena Balan

Reputation: 81

If you need your Pipenv environment root to point to Kedro project root then your solution is probably the optimal one as things stand. If you can live with Pipenv environment root directory and Kedro project directory having different names, then you can do something like:

mkdir kedro-pipenv && cd kedro-pipenv
pipenv install kedro
pipenv run kedro new  # create Kedro project inside 'kedro-pipenv'
cd <project-dir>
pipenv run kedro run  # still works even in nested directory

As you already pointed out, it's indeed easier with conda, since its environment is not tied to any particular directory, unlike pipenv.

Hoping this helps!

Upvotes: 1

Related Questions