Validation is not strict enough

[stefanv]

The docs suggest that python -m numpydoc can be used to validate docstrings. However, numpydoc reports only the most egregious breakages of the standard.

If we want to help users write high quality docstrings, we should provide a stronger checker.

In the following example, only the incorrect header gets caught:

def foo(x):
    """The first line should be a one line description, not a multiple
    paragraph sentence.

    References
    ----------
    These are out of place and incorrectly formatted.

    Some Heading
    ------------
    Unknown heading

    Parameters
    ----------
    x :
    I should specify a type for this parameter

    """
    return x

[jnothman]

Is there a way to run a restructured text or sphinx linter over it?

[stefanv]

Easiest is probably to build s small sphinx project and compile it. I haven't had much luck using Sphinx as a library, but maybe in 2.0 it's easier.

[anntzer]

FWIW, I have a small wrapper at https://github.com/anntzer/speedoc that's intended to convert docstrings to man pages on the fly (for use à la man foo), and takes care of setting up a throwaway sphinx project behind the scenes. May be useful for this use case.
(right now only master is compatible with sphinx 2.0.)

[stefanv]

@anntzer Very nice!

[calincru]

Double that.

I stumbled upon a similar issue by following the dependency chain YCM -> ycmd -> jedi -> numpydoc: jedi tries to extract information about parameters and return values from docstrings. But even if the Parameters section exists, numpydoc raises an exception if, for instance, the Examples sections appears twice -- I got here by using networkx in vim with YCM which has this exact problem in its Graph class (1, 2).

IMO the solution would be to separate semantic analysis from parsing. Another quicker but hackier solution would be to throw the exceptions lazily, once an erroneous section is accessed (e.g., when trying to access Returns or Yields when both of them are specified -- which is a semantic error, or when trying to access Examples when it appears twice, etc).

[datapythonista]

In pandas we implemented stricter validation of docstrings. We defined a set of standards that extend numpydoc, and we enforce them in the CI (gradually, since fixing all the docstrings is a huge amount of work).

We mainly have a script that can be used to validate a single docstring and provide detailed information on errors, can be used to validate all the docs (for the CI), and also can generate a JSON report with the errors found.

The list of errors we validate can be found here:
https://github.com/pandas-dev/pandas/blob/master/scripts/validate_docstrings.py#L76

Some examples of things we validate:

• Paragraphs in the summary must start with a capital letter and end with a period.
• Parameters must have a type, str, int must be used instead of string, integer...
• Parameters description must start with a capital letter and end with a period.
• The order of the sections must be correct, and no unknown sections should be found
• Examples must be valid doctests, and respect pep8
• ...

We currently have few things that are pandas specific, but it shouldn't be difficult to make the script 100% generic and move it to numpydoc. From our side we agree that the script could be moved to numpydoc if there is agreement with the rest of the community [1]. The main question is whether there is interest outside of pandas in a stricter standard, and if there can be agreement on what the standard should be.

1. DEPS: make a package for the ci/code_checks.sh pandas-dev/pandas#27342

[larsoner]

The main question is whether there is interest outside of pandas in a stricter standard

Yes from me, I've written and used validators multiple times, mostly for ensuring that all parameters are documented (in proper order).

and if there can be agreement on what the standard should be.

I think as long as the checks can be turned off via options, like you can with PEP8, we can have all the checks listed above.

For example, I have a fairly large codebase that I think will respect all of the above except:

Parameters description must start with a capital letter and end with a period.

Eventually I'd probably/maybe want it to, but not at first.

[larsoner]

(See also #13 where this was originally brought up five years ago!)

[jnothman]

I'd be very happy to see this tooling centralised and it makes sense to include it within numpydoc. @datapythonista do you intend to submit a PR?

[rth]

I'd be very happy to see this tooling centralised and it makes sense to include it within numpydoc.

Really looking forward to having some form of more advanced docstring validation script as part of numpydoc.

If you need some help with that @datapythonista please let us know. Otherwise maybe someone from the pandas team would also be interested/available in contributing it?

[datapythonista]

Yes, for what I know looks like there is agreement on moving our script to numpydoc.

I don't know much how numpydoc works internally, and I don't have time now to check in detail. Does someone have an opinion on whether we should move our code as-is, removing all pandas specific stuff. Or if it makes more sense to integrate it with the rest of numpydoc?

[rth]

I'm not too familiar with numpydoc either but I imagine that backward compatibility on python -m numpydoc would still need to be preserved. So maybe add an option here that would default to current validation but can be changed to the validation code from pandas and tag it as experimental?

Also I think running the doctests from the pandas validation script could be dropped, since pytest does that fine.