Is it possible to create docs for multiple packages within the same git repository with sphinx-apidoc?

Question

I have a git repository which includes multiple packages which follow a namespace (i.e PEP 420.

I am trying to create a ReadTheDocs documentation using Sphinx.

The git repository looks like this repo: https://github.com/pypa/sample-namespace-packages

In order to test this on my local machine, I'm using Sphinx's docker image sphinxdoc/sphinx.

I have tried to use different ways to generate the documentation for all of my packages, but each ends up with a different problem.

sphinx-apidoc

docker run -it -v pwd:/repo --rm rtd bash -c 'make clean && rm -rf /repo/docs/_source/* && sphinx-apidoc -F -o /repo/docs/_source /repo && make html'

The problem with this is that it will generate wrong packages as sphinx-apidoc uses the subfolders to generate the packages, which is wrong. This will end up with pkg_resourcespkg_a.example_pkg.a which does not exist and should actually be example_pkg.a

autoapi.extension

# conf.py
def install(package):
    subprocess.check_call([sys.executable, "-m", "pip", "install", package, "--no-deps"])


rootfolder=os.path.abspath('../')
add_module_names = False
autoapi_dirs = []
pathlist = Path(rootfolder).glob('repo-*/repo/*/')
for path in pathlist:
     path_in_str = str(path)
     autoapi_dirs.append(path_in_str)
     print(path_in_str)

...
...

extensions = [
    'sphinx.ext.napoleon',
    'sphinx.ext.autosummary', 
    'sphinx.ext.autodoc',
    'sphinx.ext.viewcode',
    'sphinx.ext.coverage',
    'autoapi.extension',
    'sphinx_rtd_theme',
]
autoapi_type = 'python'

autodoc_mock_imports = [
    'tensorflow',
    'flask',
    'numpy',
    'plotly',
    'tqdm',
    'StringIO',
    'lime',
    'vis',
    'efficientnet_pytorch',
    'pycocotools',
    'repo.trainer.self_trainer_model',
    'theano',
    'sklearn',
    'torch',
    'telegram',
    'msvcrt',
    'bs4',
    'livereload',
    'repo.common.config',
    'plotting_server',
    'experiments',
    'cropper',
    "anytree",
    "skimage"
]

I have also tried this, but unfortunately, this ends up without showing anything about my packages in the HTML while also throwing the following warnings:

/repo/docs/autoapi/repo/data/characteristics/detection/kmeanboxes/index.rst: WARNING: document isn't included in any toctree
/repo/docs/autoapi/repo/data/index.rst: WARNING: document isn't included in any toctree
/repo/docs/autoapi/repo/data_structure/index.rst: WARNING: document isn't included in any toctree
/repo/docs/autoapi/repo/detection/index.rst: WARNING: document isn't included in any toctree
/repo/docs/autoapi/repo/generators/index.rst: WARNING: document isn't included in any toctree
/repo/docs/autoapi/repo/inference/index.rst: WARNING: document isn't included in any toctree
/repo/docs/autoapi/repo/mine/repo_eye_naveyaar/index.rst: WARNING: document isn't included in any toctree
/repo/docs/autoapi/repo/mine/index.rst: WARNING: document isn't included in any toctree
/repo/docs/autoapi/repo/mine/miner_vieweryoungweedscropped/index.rst: WARNING: document isn't included in any toctree
/repo/docs/autoapi/repo/trainer/index.rst: WARNING: document isn't included in any toctree
/repo/docs/autoapi/repo/utils/dataset_specific/repoeyeweedsbackgrounds/index.rst: WARNING: document isn't included in any toctree
/repo/docs/autoapi/repo/utils/dataset_specific/repoeyeweedslabdetection/index.rst: WARNING: document isn't included in any toctree
/repo/docs/autoapi/repo/utils/index.rst: WARNING: document isn't included in any toctree
/repo/docs/autoapi/repocommon/repo/common/index.rst: WARNING: document isn't included in any toctree
/repo/docs/autoapi/repodatasets/repo/data_sets/index.rst: WARNING: document isn't included in any toctree
/repo/docs/autoapi/repometrics/repo/metrics/index.rst: WARNING: document isn't included in any toctree
/repo/docs/autoapi/repomodels/repo/models/index.rst: WARNING: document isn't included in any toctree
/repo/docs/autoapi/repooptimizers/repo/optimizers/index.rst: WARNING: document isn't included in any toctree
/repo/docs/autoapi/index.rst: WARNING: document isn't included in any toctree

So, my question is if it is possible to create docs for multiple packages within the same git repository with sphinx-apidoc?

Answer 1

After quite some digging I figured few things which weren't clear to me while posting my question:

1. sphinx-apidoc is limited only for python and might be tricky to use as it actually imports the python code, which might require installing dependencies or mocking them using autodoc_mock_imports.

2. sphinx-autoapi works with more programming languages as stated on its documentation and does not require installing dependencies.

However, in order to create the documentation for multiple packages within the same git repository with either sphinx-apidoc or sphinx-autoapi, I had to add the following code to docs/conf.py:

from glob import glob
from distutils.dir_util import copy_tree

currentfolder = os.path.abspath('./') # aka docs/
rootfolder = os.path.abspath('../')
tmpfolder = "{}/tmp".format(currentfolder)
globdir = '{}/repo-*/repo'.format(rootfolder)
subprocess.check_call(["mkdir", "-p", tmpfolder]) # make sure temporary folder exists 

all_sub_dir_paths = glob(globdir) # returns list of sub directory paths
for subdir in all_sub_dir_paths:
    print("subdir: {} to tmp folder: {}".format(subdir, tmpfolder))
    copy_tree(subdir, tmpfolder) # copy all folders to docs/tmp/repo/*

autoapi_dirs = [tmpfolder]
sys.path.insert(0, tmpfolder)
autoapi_python_use_implicit_namespaces = True

This solution will copy all of the packages within the repo to a temporary folder, which will allow the docstring tool to scan and then generate the relevant docs.

I have ended up using sphinx-autoapi, but it did work also with sphinx-apidoc which might require you to install these packages:

def install(package):
     subprocess.check_call([sys.executable, "-m", "pip", "install", package, "--no-deps"])

def list_files(dir):
    r = []
    for root, dirs, files in os.walk(dir):
        for name in dirs:
            if name.startswith('repo-'):
                l = "{}".format(os.path.join(root, name))
                # print("Added to path: {}".format(l))
                sys.path.insert(0, l)
                install(l) # might no be required 
        break # stop os.walk recursion
    return r
list_files(currentfolder)

autodoc_mock_imports = [
    'tensorflow',
    'flask',
]

You can debugging this locally using docker:

docker run -it -v `pwd`:/repo --rm rtd bash -c 'python -m sphinx -T -E  -d _build/doctrees-readthedocs -D language=en . _build/html'

If you are using ReadTheDocs Business, all you have to do is to push your code to the relevant branch and the rest will happen automatically for you.