How to install this Python package from sources?

Question

There is a nuScenes-devkit: https://github.com/nutonomy/nuscenes-devkit
I made some modifications in a private repository and want to install it from source but I can't.
If I do pip install . from setup folder, I have the following error:

    ERROR: Command errored out with exit status 1:
     command: /mnt/nvme1n1/venvs/enp2/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-wfs5ujhi/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-wfs5ujhi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-req-build-wfs5ujhi/pip-egg-info
         cwd: /tmp/pip-req-build-wfs5ujhi/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-wfs5ujhi/setup.py", line 5, in <module>
        with open('../README.md', 'r') as fh:
    FileNotFoundError: [Errno 2] No such file or directory: '../README.md'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Although, README.md certainly exists, I can check it by vim ../README.md from the same folder.

UPD: Also I tried python setup.py install from setup folder. It has the following error:

running install
running bdist_egg
running egg_info
writing python-sdk/nuscenes_devkit.egg-info/PKG-INFO
writing dependency_links to python-sdk/nuscenes_devkit.egg-info/dependency_links.txt
writing requirements to python-sdk/nuscenes_devkit.egg-info/requires.txt
writing top-level names to python-sdk/nuscenes_devkit.egg-info/top_level.txt
error: package directory 'python-sdk/nuscenes_devkit/egg-info' does not exist

UPD2: I have Ubuntu 20.04 LTS, Python 3.8.10, pip 22.1.2 (for convenience, I checked Python and pip versions at the end of the instruction of Bastian Venthur). Also, I tried to do it in my own virtual environment, which is also created using python venv.

UPD3: Here is the full error from pip install -e . from the answer of Bastian Venthur:

Obtaining file:///tmp/nuscenes-devkit/setup
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [8 lines of output]
      running egg_info
      creating /tmp/pip-pip-egg-info-02j0_bgt/nuscenes_devkit.egg-info
      writing /tmp/pip-pip-egg-info-02j0_bgt/nuscenes_devkit.egg-info/PKG-INFO
      writing dependency_links to /tmp/pip-pip-egg-info-02j0_bgt/nuscenes_devkit.egg-info/dependency_links.txt
      writing requirements to /tmp/pip-pip-egg-info-02j0_bgt/nuscenes_devkit.egg-info/requires.txt
      writing top-level names to /tmp/pip-pip-egg-info-02j0_bgt/nuscenes_devkit.egg-info/top_level.txt
      writing manifest file '/tmp/pip-pip-egg-info-02j0_bgt/nuscenes_devkit.egg-info/SOURCES.txt'
      error: package directory 'python-sdk/nuscenes_devkit/egg-info' does not exist
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Also, I tried to do it with sudo pip install -e .:

Obtaining file:///tmp/nuscenes-devkit/setup
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/nuscenes-devkit/setup/setup.py'"'"'; __file__='"'"'/tmp/nuscenes-devkit/setup/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info
         cwd: /tmp/nuscenes-devkit/setup/
    Complete output (6 lines):
    running egg_info
    writing python-sdk/nuscenes_devkit.egg-info/PKG-INFO
    writing dependency_links to python-sdk/nuscenes_devkit.egg-info/dependency_links.txt
    writing requirements to python-sdk/nuscenes_devkit.egg-info/requires.txt
    writing top-level names to python-sdk/nuscenes_devkit.egg-info/top_level.txt
    error: package directory 'python-sdk/nuscenes_devkit/egg-info' does not exist
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Answer 1

When I follow guy szweigman's accepted answer using the setup.py file on top level, I still get the error message below.

Insights appreciated if this is another error of nuscenes-devkit or of setuptools in general.

Running setup.py develop for nuscenes-devkit
    error: subprocess-exited-with-error
    
    × python setup.py develop did not run successfully.
    │ exit code: 1
    ╰─> [66 lines of output]
        running develop
        /home/michaelresearch/miniconda3/envs/research/lib/python3.10/site-packages/setuptools/command/develop.py:40: EasyInstallDeprecationWarning: easy_install command is deprecated.
        !!
        
                ********************************************************************************
                Please avoid running ``setup.py`` and ``easy_install``.
                Instead, use pypa/build, pypa/installer or other
                standards-based tools.
        
                See https://github.com/pypa/setuptools/issues/917 for details.
                ********************************************************************************
        
        !!
          easy_install.initialize_options(self)
        /home/michaelresearch/miniconda3/envs/research/lib/python3.10/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
        !!
        
                ********************************************************************************
                Please avoid running ``setup.py`` directly.
                Instead, use pypa/build, pypa/installer or other
                standards-based tools.
        
                See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
                ********************************************************************************
        
        !!
          self.initialize_options()
        running egg_info
        writing python-sdk/nuscenes_devkit.egg-info/PKG-INFO
        writing dependency_links to python-sdk/nuscenes_devkit.egg-info/dependency_links.txt
        writing requirements to python-sdk/nuscenes_devkit.egg-info/requires.txt
        writing top-level names to python-sdk/nuscenes_devkit.egg-info/top_level.txt
        reading manifest file 'python-sdk/nuscenes_devkit.egg-info/SOURCES.txt'
        adding license file 'LICENSE.txt'
        writing manifest file 'python-sdk/nuscenes_devkit.egg-info/SOURCES.txt'
        running build_ext
        Creating /home/michaelresearch/miniconda3/envs/research/lib/python3.10/site-packages/nuscenes-devkit.egg-link (link to python-sdk)
        Adding nuscenes-devkit 1.1.9 to easy-install.pth file
        Traceback (most recent call last):
          File "<string>", line 2, in <module>
          File "<pip-setuptools-caller>", line 34, in <module>
          File "/home/michaelresearch/git/perceptest/third_party/nuscenes-devkit/setup.py", line 27, in <module>
            setuptools.setup(
          File "/home/michaelresearch/miniconda3/envs/research/lib/python3.10/site-packages/setuptools/__init__.py", line 107, in setup
            return distutils.core.setup(**attrs)
          File "/home/michaelresearch/miniconda3/envs/research/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
            return run_commands(dist)
          File "/home/michaelresearch/miniconda3/envs/research/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
            dist.run_commands()
          File "/home/michaelresearch/miniconda3/envs/research/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
            self.run_command(cmd)
          File "/home/michaelresearch/miniconda3/envs/research/lib/python3.10/site-packages/setuptools/dist.py", line 1234, in run_command
            super().run_command(command)
          File "/home/michaelresearch/miniconda3/envs/research/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
            cmd_obj.run()
          File "/home/michaelresearch/miniconda3/envs/research/lib/python3.10/site-packages/setuptools/command/develop.py", line 34, in run
            self.install_for_development()
          File "/home/michaelresearch/miniconda3/envs/research/lib/python3.10/site-packages/setuptools/command/develop.py", line 126, in install_for_development
            self.process_distribution(None, self.dist, not self.no_deps)
          File "/home/michaelresearch/miniconda3/envs/research/lib/python3.10/site-packages/setuptools/command/easy_install.py", line 726, in process_distribution
            self.update_pth(dist)
          File "/home/michaelresearch/miniconda3/envs/research/lib/python3.10/site-packages/setuptools/command/easy_install.py", line 1235, in update_pth
            self.pth_file.save()
          File "/home/michaelresearch/miniconda3/envs/research/lib/python3.10/site-packages/setuptools/command/easy_install.py", line 1642, in save
            self.dirty |= last_dirty or self.paths != self._init_paths
        TypeError: unsupported operand type(s) for |=: 'list' and 'bool'
        [end of output]
    
    note: This error originates from a subprocess, and is likely not a problem with pip.

Answer 2

TL;DR: Create a setup.py file in the root directory of the nuscenes-devkit repository with the following content (then, to install, run pip install . from the root directory):

from pathlib import Path

import setuptools

package_dir = 'python-sdk'
setup_dir = Path(__file__).parent / 'setup'

# Since nuScenes 2.0 the requirements are stored in separate files.
requirements = []
for req_path in (setup_dir / 'requirements.txt').read_text().splitlines():
    if req_path.startswith('#'):
        continue
    req_path = req_path.replace('-r ', '')
    requirements += (setup_dir / req_path).read_text().splitlines()


# Get subfolders recursively

setuptools.setup(
    name='nuscenes-devkit',
    version='1.1.9',
    author='Holger Caesar, Oscar Beijbom, Qiang Xu, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, '
           'Sergi Widjaja, Kiwoo Shin, Caglayan Dicle, Freddy Boulton, Whye Kit Fong, Asha Asvathaman, Lubing Zhou '
           'et al.',
    author_email='nuscenes@motional.com',
    description='The official devkit of the nuScenes dataset (www.nuscenes.org).',
    long_description=Path('README.md').read_text(),
    long_description_content_type='text/markdown',
    url='https://github.com/nutonomy/nuscenes-devkit',
    python_requires='>=3.6',
    install_requires=requirements,
    packages=setuptools.find_packages(package_dir),
    package_dir={'': package_dir},
    package_data={'': ['*.json']},
    include_package_data=True,
    classifiers=[
        'Programming Language :: Python :: 3.6',
        'Operating System :: OS Independent',
        'License :: Free for non-commercial use'
    ],
    license='cc-by-nc-sa-4.0'
)

Explanation: When running in a virtual environment, pip copies the directory that contains the setup.py script to a temporary directory and runs the installation from there. In nuscenes_devkit, the python-sdk directory which is the root package of the project is not copied to this temporary directory since it's a sibling of the setup directory, hence, you get a FileNotFoundError.

At any case, having the setup.py script in the root directory is definitely the best practice (even better approach would be to use setup.cfg file but this is out of scope).

I personally think this issue should be considered as a bug in nuscenes-devkit.

So the solution is to move the setup.py script to the root directory and align it accordingly (I also took the liberty of using setuptools.find_packages to simplify the script).

Last note: You may take a look at the advanced installation doc, the instructions there doesn't mention pip or setup.py, they suggest modifying PYTHONPATH instead. I personally don't like this approach, but it may be an alternative solution.

Answer 3

Here is what worked for me:

cd /tmp/                  # get the package sources from github
git clone https://github.com/nutonomy/nuscenes-devkit.git
cd nuscenes-devkit/
cd setup/

python3 -m venv venv      # provide a clean environment
. venv/bin/activate       # and use it
pip install -e .          # install the package in dev-mode

pip show nuscenes-devkit  # voila
Name: nuscenes-devkit
Version: 1.1.9
Summary: The official devkit of the nuScenes dataset (www.nuscenes.org).
Home-page: https://github.com/nutonomy/nuscenes-devkit
Author: Holger Caesar, Oscar Beijbom, Qiang Xu, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Sergi Widjaja, Kiwoo Shin, Caglayan Dicle, Freddy Boulton, Whye Kit Fong, Asha Asvathaman, Lubing Zhou et al.
Author-email: nuscenes@motional.com
License: cc-by-nc-sa-4.0
Location: /tmp/nuscenes-devkit/python-sdk
Requires: cachetools, descartes, fire, jupyter, matplotlib, numpy, opencv-python, Pillow, pycocotools, pyquaternion, scikit-learn, scipy, Shapely, tqdm
Required-by:

Sometimes it helps to upgrade pip itself before doing the pip install dance:

pip install --upgrade pip

Answer 4

Installing nuscenes-devkit :-

via pip:-

pip install nuscenes-devkit

from source:-

git clone https://github.com/nutonomy/nuscenes-devkit.git   #clone the repo

cd nuscenes-devkit/setup               #change directory to setup

python setup.py install                #install using setup.py

pip show nuscenes-devkit               #confirm ur installation