Can sphinx ignore certain tags in python docstrings?

Question

I'm documenting my project with sphinx and using the sphinxcontrib.napoleon extension which lets me use google style docstrings.

Here is a function from my project

def nn_normalized_weight(normweight_fn, qaid2_nns, qreq_):
    """
    Weights nearest neighbors using the chosen function

    Args:
        normweight_fn (func): chosen weight function e.g. lnbnn
        qaid2_nns (dict): query descriptor nearest neighbors and distances. qaid -> (qfx2_nnx, qfx2_dist)
        qreq_ (QueryRequest): hyper-parameters

    Returns:
        tuple(dict, dict) : (qaid2_weight, qaid2_selnorms)

    Example:
        >>> from ibeis.model.hots.nn_weights import *
        >>> from ibeis.model.hots import nn_weights
        >>> ibs, daid_list, qaid_list, qaid2_nns, qreq_ = nn_weights.testdata_nn_weights()
        >>> qaid = qaid_list[0]
        >>> #----
        >>> normweight_fn = lnbnn_fn
        >>> tup1 = nn_weights.nn_normalized_weight(normweight_fn, qaid2_nns, qreq_)
        >>> (qaid2_weight1, qaid2_selnorms1) = tup1
        >>> weights1 = qaid2_weight1[qaid]
        >>> selnorms1 = qaid2_selnorms1[qaid]
        >>> #---
        >>> # test NN_WEIGHT_FUNC_DICT
        >>> #---
        >>> nn_normonly_weight = nn_weights.NN_WEIGHT_FUNC_DICT['lnbnn']
        >>> tup2 = nn_normonly_weight(qaid2_nns, qreq_)
        >>> (qaid2_weight2, qaid2_selnorms2) = tup2
        >>> selnorms2 = qaid2_selnorms2[qaid]
        >>> weights2 = qaid2_weight2[qaid]
        >>> assert np.all(weights1 == weights2)
        >>> assert np.all(selnorms1 == selnorms2)

    Ignore:
        #from ibeis.model.hots import neighbor_index as hsnbrx
        #nnindexer = hsnbrx.new_ibeis_nnindexer(ibs, daid_list)
    """
    #utool.stash_testdata('qaid2_nns')
    #
    K = qreq_.qparams.K

    Knorm = qreq_.qparams.Knorm
    rule  = qreq_.qparams.normalizer_rule
    # Prealloc output
    qaid2_weight = {qaid: None for qaid in six.iterkeys(qaid2_nns)}
    qaid2_selnorms = {qaid: None for qaid in six.iterkeys(qaid2_nns)}
    # Database feature index to chip index
    for qaid in six.iterkeys(qaid2_nns):
        (qfx2_idx, qfx2_dist) = qaid2_nns[qaid]
        # Apply normalized weights
        (qfx2_normweight, qfx2_normmeta) = apply_normweight(
            normweight_fn, qaid, qfx2_idx, qfx2_dist, rule, K, Knorm, qreq_)
        # Output
        qaid2_weight[qaid]   = qfx2_normweight
        qaid2_selnorms[qaid] = qfx2_normmeta
    return (qaid2_weight, qaid2_selnorms)

When I parse this with sphinx-apidoc It parses the args, returns, and example section correctly, but then it tags on the ignore section as well.

The ignore section looks very ugly as it has had all its formating stripped away. I would like to remove it. Is there any way to configure sphinx to ignore certain tags like Ignore:?

I'm aware I could take it out of the docstr, but this is very inconvient as I'd like to have a place without leading # sybmols where I can copy and paste test code to and from ipython.

Answer 1

Ok, I think I've got a solution for you:

sphinx.ext.autodoc offers a listener sphinx.ext.autodoc.between that can be used to determine what parts of the docstrings autodoc collects should be kept or discarded:

sphinx.ext.autodoc.between(marker, what=None, keepempty=False, exclude=False)

Return a listener that either keeps, or if exclude is True excludes, lines between lines that match the marker regular expression. If no line matches, the resulting docstring would be empty, so no change will be made unless keepempty is true.

If what is a sequence of strings, only docstrings of a type in what will be processed.

sphinxcontrib.napoleon works on the docstrings that autodoc collects, so this should work for napoleon as well.

Usage example

Change your docstring like this:

"""
Args:
    ...

Returns:
    ...

IGNORE:
    #from ibeis.model.hots import neighbor_index as hsnbrx
    #nnindexer = hsnbrx.new_ibeis_nnindexer(ibs, daid_list)
IGNORE
"""

So make sure to surround the part you want to exclude with two lines that contain a unique marker (in this example the uppercase word IGNORE).

Add the following to your Sphinx project's conf.py (I'd probably append it all at the bottom as one block, but that's your call):

from sphinx.ext.autodoc import between

def setup(app):
    # Register a sphinx.ext.autodoc.between listener to ignore everything
    # between lines that contain the word IGNORE
    app.connect('autodoc-process-docstring', between('^.*IGNORE.*$', exclude=True))
    return app

(If your conf.py already contains a setup() function, just extend it accordingly).

This will create and register a listener that gets called everytime autodoc processes a docstring. The listener then gets the chance to filter the docstring. In this example, the listener will discard everything between lines that match the regular expression ^.*IGNORE.*$ - so it's up to you to choose this expression so that it's specific enough for your project, but doesn't require a marker that adds too much noise.

(Note: If all you change is your conf.py, Sphinx won't pick up that change because the doctree didn't change. So make sure you run make clean (or rm -rf _build/*) before building your docs again).

Answer 2

I stumbled upon this question I wrote 7 years ago today.

At the time I thought Lukas Graf's answer would be fine, but being forced to have and open and close Ignore tag was too much overhead (espeically in Python), so I stopped doing it and I forgot about it. Fast forward, my API docs are littered with ignores, and I really want to get rid of them. I looked at the answer with older and wiser eyes, and I have an actual solution that isn't hacky. This actually solves the problem.

def setup(app):
    # https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html
    from sphinx.application import Sphinx
    from typing import Any, List

    what = None
    # Custom process to transform docstring lines
    # Remove "Ignore" blocks
    def process(app: Sphinx, what_: str, name: str, obj: Any, options: Any, lines: List[str]
                ) -> None:
        if what and what_ not in what:
            return
        orig_lines = lines[:]

        ignoring = False
        new_lines = []
        for i, line in enumerate(orig_lines):
            if line.startswith('Ignore'):
                # We will start ignoring everything indented after this
                ignoring = True
            else:
                # if the line startswith anything but a space stop
                # ignoring the indented region.
                if ignoring and line and not line.startswith(' '):
                    ignoring = False

            if not ignoring:
                new_lines.append(line)

        lines[:] = new_lines
        # make sure there is a blank line at the end
        if lines and lines[-1]:
            lines.append('')

    app.connect('autodoc-process-docstring', process)
    return app

The idea is very similar, except I looked at the source code of what "between" does, and it is possible to just write your own custom function that processes your docstring in whatever way you want (although it is postprocessed by neopoleon, but that doesn't matter in this case).

Given the lines of each processsed docstring I can check to see if it starts with Ignore, then ignore everything until the indendation scope ends. No regex "between" tags needed.