Cryptic error message from failing to parse invalid rst docstrings

[nialov]

Problem

A cryptic error message is raised by process_docstring failing to parse a rst docstring within Python code.

sphinx.errors.ExtensionError: Handler <function process_docstring at 0x7efe073368b0> for event 'autodoc-process-docstring' threw an exception (exception: )

Example of invalid docstrings:

  @staticmethod
  def conditional_linemerge(
      first: LineString, second: LineString, tolerance: float, buffer_value: float
  ) -> Union[None, LineString]:
      """
      Conditionally merge two LineStrings (first and second).

      Merge occurs if:
          1. Their endpoints are within buffer_value of each other.
          2. Their total orientations are within tolerance (degrees) of each
             other.

      """

The issue is the indented part after the colon with the numbers.

What I expected

The issue of invalid rst is of course an user error and it is of course great that sphinx-autodoc-typehints finds these invalid portions but the error report should clarify where the error occurs and possibly why. Otherwise it is difficult for the user to understand what is wrong.

Further info

I managed to debug the issue myself by running sphinx-build with the verbose flags (-vvv). The issue is further exacerbated by the fact that the sphinx log can contain additional logging outputs between the exception reporting and the log that displays which part of the code sphinx is currently processing. See the log as a gist here: https://gist.github.com/nialov/e92dcf8aa43c9108b31c37df28725188. At the very start, the emitting event parts tell where the parsing is currently occurring. After it a number of irrelevant warnings are raised, which I have mostly deleted to keep the log cleaner, and then the exception is printed out. The exception itself contains no useful information, only that the function process_docstring failed somewhere in the documentation. The log without the verbosity flags -vvv is much more cryptic as only the end exception is outputted.

[jkittner]

I was able to trace this back to this commit: 7b2f213 / #300 and find a little simpler reproduction. It has to do with :returns: ?!.

this does not work and errors with above error..

def f() -> None:
    """
    abc
        1. abc
    """
    ...

however if you add a :returns:, it works fine

def f() -> None:
    """
    abc
        1. abc

    :returns: abc
    """
    ...

also this does not error, but results in broken docs

def f() -> None:
    """
    abc
    
    1. abc
    """
    ...

For some reason line becomes None here, and the assertion raises:

sphinx-autodoc-typehints/src/sphinx_autodoc_typehints/__init__.py

Line 663 in 7b2f213

line = node.line

I hope this helps.

[gaborbernat]

@hoodmane can you take a look?

[hoodmane]

Sure, will do.

[soxofaan]

our sphinx doc build also started to fail with this error (Open-EO/openeo-python-client#366)

Run python -m sphinx -b html docs build
Running Sphinx v5.3.0
...
building [html]: targets for 23 source files that are out of date
updating environment: [new config] 23 added, 0 changed, 0 removed
reading sources... [  4%] api

Extension error (sphinx_autodoc_typehints):
Handler <function process_docstring at 0x7f908c7e0430> for event 'autodoc-process-docstring' threw an exception (exception: )
Error: Process completed with exit code 2.

This apparently happens with sphinx-autodoc-typehints==1.21.4
When downgrading to sphinx-autodoc-typehints==1.21.3 the build works again

[hoodmane]

The problem does not even seem to require a syntax error in the rst. The root cause is that the docutils rst parser does not add line numbers to definition nodes, only to the definition_list children sigh. I wish docutils had documentation.