Merge pull request #13192 from jakkdl/raisesgroup

add RaisesGroup & RaisesExc. Make raises use RaisesExc. Allow xfail to accept a RaisesGroup/RaisesExc

Author: jakkdl

changelog/11538.feature.rst

@@ -0,0 +1 @@
+Added :class:`pytest.RaisesGroup` as an equivalent to :func:`pytest.raises` for expecting :exc:`ExceptionGroup`. Also adds :class:`pytest.RaisesExc` which is now the logic behind :func:`pytest.raises` and used as parameter to :class:`pytest.RaisesGroup`. ``RaisesGroup`` includes the ability to specify multiple different expected exceptions, the structure of nested exception groups, and flags for emulating :ref:`except* <except_star>`. See :ref:`assert-matching-exception-groups` and docstrings for more information.

changelog/12504.feature.rst

@@ -0,0 +1 @@
+:func:`pytest.mark.xfail` now accepts :class:`pytest.RaisesGroup` for the ``raises`` parameter when you expect an exception group. You can also pass a :class:`pytest.RaisesExc` if you e.g. want to make use of the ``check`` parameter.

doc/en/conf.py

@@ -106,6 +106,8 @@
     ("py:obj", "_pytest.fixtures.FixtureValue"),
     ("py:obj", "_pytest.stash.T"),
     ("py:class", "_ScopeName"),
+    ("py:class", "BaseExcT_1"),
+    ("py:class", "ExcT_1"),
 ]
 
 add_module_names = False

doc/en/getting-started.rst

@@ -97,30 +97,6 @@ Use the :ref:`raises <assertraises>` helper to assert that some code raises an e
         with pytest.raises(SystemExit):
             f()
 
-You can also use the context provided by :ref:`raises <assertraises>` to
-assert that an expected exception is part of a raised :class:`ExceptionGroup`:
-
-.. code-block:: python
-
-    # content of test_exceptiongroup.py
-    import pytest
-
-
-    def f():
-        raise ExceptionGroup(
-            "Group message",
-            [
-                RuntimeError(),
-            ],
-        )
-
-
-    def test_exception_in_group():
-        with pytest.raises(ExceptionGroup) as excinfo:
-            f()
-        assert excinfo.group_contains(RuntimeError)
-        assert not excinfo.group_contains(TypeError)
-
 Execute the test function with “quiet” reporting mode:
 
 .. code-block:: pytest
@@ -133,6 +109,8 @@ Execute the test function with “quiet” reporting mode:
 
     The ``-q/--quiet`` flag keeps the output brief in this and following examples.
 
+See :ref:`assertraises` for specifying more details about the expected exception.
+
 Group multiple tests in a class
 --------------------------------------------------------------
 

doc/en/how-to/assert.rst

@@ -145,8 +145,93 @@ Notes:
 
 .. _`assert-matching-exception-groups`:
 
-Matching exception groups
-~~~~~~~~~~~~~~~~~~~~~~~~~
+Assertions about expected exception groups
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When expecting a :exc:`BaseExceptionGroup` or :exc:`ExceptionGroup` you can use :class:`pytest.RaisesGroup`:
+
+.. code-block:: python
+
+    def test_exception_in_group():
+        with pytest.RaisesGroup(ValueError):
+            raise ExceptionGroup("group msg", [ValueError("value msg")])
+        with pytest.RaisesGroup(ValueError, TypeError):
+            raise ExceptionGroup("msg", [ValueError("foo"), TypeError("bar")])
+
+
+It accepts a ``match`` parameter, that checks against the group message, and a ``check`` parameter that takes an arbitrary callable which it passes the group to, and only succeeds if the callable returns ``True``.
+
+.. code-block:: python
+
+    def test_raisesgroup_match_and_check():
+        with pytest.RaisesGroup(BaseException, match="my group msg"):
+            raise BaseExceptionGroup("my group msg", [KeyboardInterrupt()])
+        with pytest.RaisesGroup(
+            Exception, check=lambda eg: isinstance(eg.__cause__, ValueError)
+        ):
+            raise ExceptionGroup("", [TypeError()]) from ValueError()
+
+It is strict about structure and unwrapped exceptions, unlike :ref:`except* <except_star>`, so you might want to set the ``flatten_subgroups`` and/or ``allow_unwrapped`` parameters.
+
+.. code-block:: python
+
+    def test_structure():
+        with pytest.RaisesGroup(pytest.RaisesGroup(ValueError)):
+            raise ExceptionGroup("", (ExceptionGroup("", (ValueError(),)),))
+        with pytest.RaisesGroup(ValueError, flatten_subgroups=True):
+            raise ExceptionGroup("1st group", [ExceptionGroup("2nd group", [ValueError()])])
+        with pytest.RaisesGroup(ValueError, allow_unwrapped=True):
+            raise ValueError
+
+To specify more details about the contained exception you can use :class:`pytest.RaisesExc`
+
+.. code-block:: python
+
+    def test_raises_exc():
+        with pytest.RaisesGroup(pytest.RaisesExc(ValueError, match="foo")):
+            raise ExceptionGroup("", (ValueError("foo")))
+
+They both supply a method :meth:`pytest.RaisesGroup.matches` :meth:`pytest.RaisesExc.matches` if you want to do matching outside of using it as a contextmanager. This can be helpful when checking ``.__context__`` or ``.__cause__``.
+
+.. code-block:: python
+
+    def test_matches():
+        exc = ValueError()
+        exc_group = ExceptionGroup("", [exc])
+        if RaisesGroup(ValueError).matches(exc_group):
+            ...
+        # helpful error is available in `.fail_reason` if it fails to match
+        r = RaisesExc(ValueError)
+        assert r.matches(e), r.fail_reason
+
+Check the documentation on :class:`pytest.RaisesGroup` and :class:`pytest.RaisesExc` for more details and examples.
+
+``ExceptionInfo.group_contains()``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. warning::
+
+   This helper makes it easy to check for the presence of specific exceptions, but it is very bad for checking that the group does *not* contain *any other exceptions*. So this will pass:
+
+    .. code-block:: python
+
+       class EXTREMELYBADERROR(BaseException):
+           """This is a very bad error to miss"""
+
+
+       def test_for_value_error():
+           with pytest.raises(ExceptionGroup) as excinfo:
+               excs = [ValueError()]
+               if very_unlucky():
+                   excs.append(EXTREMELYBADERROR())
+               raise ExceptionGroup("", excs)
+           # This passes regardless of if there's other exceptions.
+           assert excinfo.group_contains(ValueError)
+           # You can't simply list all exceptions you *don't* want to get here.
+
+
+   There is no good way of using :func:`excinfo.group_contains() <pytest.ExceptionInfo.group_contains>` to ensure you're not getting *any* other exceptions than the one you expected.
+   You should instead use :class:`pytest.RaisesGroup`, see :ref:`assert-matching-exception-groups`.
 
 You can also use the :func:`excinfo.group_contains() <pytest.ExceptionInfo.group_contains>`
 method to test for exceptions returned as part of an :class:`ExceptionGroup`:
@@ -194,12 +279,12 @@ exception at a specific level; exceptions contained directly in the top
         assert not excinfo.group_contains(RuntimeError, depth=2)
         assert not excinfo.group_contains(TypeError, depth=1)
 
-Alternate form (legacy)
-~~~~~~~~~~~~~~~~~~~~~~~
+Alternate `pytest.raises` form (legacy)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-There is an alternate form where you pass
-a function that will be executed, along ``*args`` and ``**kwargs``, and :func:`pytest.raises`
-will execute the function with the arguments and assert that the given exception is raised:
+There is an alternate form of :func:`pytest.raises` where you pass
+a function that will be executed, along with ``*args`` and ``**kwargs``. :func:`pytest.raises`
+will then execute the function with those arguments and assert that the given exception is raised:
 
 .. code-block:: python
 
@@ -244,6 +329,18 @@ This will only "xfail" if the test fails by raising ``IndexError`` or subclasses
 * Using :func:`pytest.raises` is likely to be better for cases where you are
   testing exceptions your own code is deliberately raising, which is the majority of cases.
 
+You can also use :class:`pytest.RaisesGroup`:
+
+.. code-block:: python
+
+    def f():
+        raise ExceptionGroup("", [IndexError()])
+
+
+    @pytest.mark.xfail(raises=RaisesGroup(IndexError))
+    def test_f():
+        f()
+
 
 .. _`assertwarns`:
 

doc/en/reference/reference.rst

@@ -1034,6 +1034,23 @@ PytestPluginManager
     :inherited-members:
     :show-inheritance:
 
+RaisesExc
+~~~~~~~~~
+
+.. autoclass:: pytest.RaisesExc()
+    :members:
+
+    .. autoattribute:: fail_reason
+
+RaisesGroup
+~~~~~~~~~~~
+**Tutorial**: :ref:`assert-matching-exception-groups`
+
+.. autoclass:: pytest.RaisesGroup()
+    :members:
+
+    .. autoattribute:: fail_reason
+
 TerminalReporter
 ~~~~~~~~~~~~~~~~
 

src/_pytest/_code/code.py

@@ -459,6 +459,32 @@ def recursionindex(self) -> int | None:
         return None
 
 
+def stringify_exception(
+    exc: BaseException, include_subexception_msg: bool = True
+) -> str:
+    try:
+        notes = getattr(exc, "__notes__", [])
+    except KeyError:
+        # Workaround for https://github.com/python/cpython/issues/98778 on
+        # Python <= 3.9, and some 3.10 and 3.11 patch versions.
+        HTTPError = getattr(sys.modules.get("urllib.error", None), "HTTPError", ())
+        if sys.version_info < (3, 12) and isinstance(exc, HTTPError):
+            notes = []
+        else:
+            raise
+    if not include_subexception_msg and isinstance(exc, BaseExceptionGroup):
+        message = exc.message
+    else:
+        message = str(exc)
+
+    return "\n".join(
+        [
+            message,
+            *notes,
+        ]
+    )
+
+
 E = TypeVar("E", bound=BaseException, covariant=True)
 
 
@@ -736,33 +762,14 @@ def getrepr(
         )
         return fmt.repr_excinfo(self)
 
-    def _stringify_exception(self, exc: BaseException) -> str:
-        try:
-            notes = getattr(exc, "__notes__", [])
-        except KeyError:
-            # Workaround for https://github.com/python/cpython/issues/98778 on
-            # Python <= 3.9, and some 3.10 and 3.11 patch versions.
-            HTTPError = getattr(sys.modules.get("urllib.error", None), "HTTPError", ())
-            if sys.version_info < (3, 12) and isinstance(exc, HTTPError):
-                notes = []
-            else:
-                raise
-
-        return "\n".join(
-            [
-                str(exc),
-                *notes,
-            ]
-        )
-
     def match(self, regexp: str | re.Pattern[str]) -> Literal[True]:
         """Check whether the regular expression `regexp` matches the string
         representation of the exception using :func:`python:re.search`.
 
         If it matches `True` is returned, otherwise an `AssertionError` is raised.
         """
         __tracebackhide__ = True
-        value = self._stringify_exception(self.value)
+        value = stringify_exception(self.value)
         msg = f"Regex pattern did not match.\n Regex: {regexp!r}\n Input: {value!r}"
         if regexp == value:
             msg += "\n Did you mean to `re.escape()` the regex?"
@@ -794,7 +801,7 @@ def _group_contains(
             if not isinstance(exc, expected_exception):
                 continue
             if match is not None:
-                value = self._stringify_exception(exc)
+                value = stringify_exception(exc)
                 if not re.search(match, value):
                     continue
             return True
@@ -828,6 +835,13 @@ def group_contains(
             the exceptions contained within the topmost exception group).
 
         .. versionadded:: 8.0
+
+        .. warning::
+           This helper makes it easy to check for the presence of specific exceptions,
+           but it is very bad for checking that the group does *not* contain
+           *any other exceptions*.
+           You should instead consider using :class:`pytest.RaisesGroup`
+
         """
         msg = "Captured exception is not an instance of `BaseExceptionGroup`"
         assert isinstance(self.value, BaseExceptionGroup), msg

src/_pytest/mark/structures.py

@@ -28,6 +28,7 @@
 from _pytest.deprecated import check_ispytest
 from _pytest.deprecated import MARKED_FIXTURE
 from _pytest.outcomes import fail
+from _pytest.raises_group import AbstractRaises
 from _pytest.scope import _ScopeName
 from _pytest.warning_types import PytestUnknownMarkWarning
 
@@ -473,7 +474,10 @@ def __call__(
             *conditions: str | bool,
             reason: str = ...,
             run: bool = ...,
-            raises: None | type[BaseException] | tuple[type[BaseException], ...] = ...,
+            raises: None
+            | type[BaseException]
+            | tuple[type[BaseException], ...]
+            | AbstractRaises[BaseException] = ...,
             strict: bool = ...,
         ) -> MarkDecorator: ...
 

src/_pytest/outcomes.py

@@ -77,8 +77,8 @@ def __init__(
         super().__init__(msg)
 
 
-# Elaborate hack to work around https://github.com/python/mypy/issues/2087.
-# Ideally would just be `exit.Exception = Exit` etc.
+# We need a callable protocol to add attributes, for discussion see
+# https://github.com/python/mypy/issues/2087.
 
 _F = TypeVar("_F", bound=Callable[..., object])
 _ET = TypeVar("_ET", bound=type[BaseException])