Integrate pytest-subtests (#13738)

Fixes #1367
Fixes pytest-dev/pytest-subtests#71

---------

Co-authored-by: Ran Benita <[email protected]>

Author: nicoddemus

.coveragerc

@@ -25,6 +25,7 @@ exclude_lines =
     ^\s*raise NotImplementedError\b
     ^\s*return NotImplemented\b
     ^\s*assert False(,|$)
+    ^\s*case unreachable:
     ^\s*assert_never\(
 
     ^\s*if TYPE_CHECKING:

changelog/1367.feature.rst

@@ -0,0 +1,28 @@
+**Support for subtests** has been added.
+
+:ref:`subtests <subtests>` are an alternative to parametrization, useful in situations where the parametrization values are not all known at collection time.
+
+**Example**
+
+.. code-block:: python
+
+    def contains_docstring(p: Path) -> bool:
+        """Return True if the given Python file contains a top-level docstring."""
+        ...
+
+
+    def test_py_files_contain_docstring(subtests: pytest.Subtests) -> None:
+        for path in Path.cwd().glob("*.py"):
+            with subtests.test(path=str(path)):
+                assert contains_docstring(path)
+
+
+Each assert failure or error is caught by the context manager and reported individually, giving a clear picture of all files that are missing a docstring.
+
+In addition, :meth:`unittest.TestCase.subTest` is now also supported.
+
+This feature was originally implemented as a separate plugin in `pytest-subtests <https://github.com/pytest-dev/pytest-subtests>`__, but since then has been merged into the core.
+
+.. note::
+
+    This feature is experimental and will likely evolve in future releases. By that we mean that we might change how subtests are reported on failure, but the functionality and how to use it are stable.

doc/en/how-to/index.rst

@@ -16,6 +16,7 @@ Core pytest functionality
    fixtures
    mark
    parametrize
+   subtests
    tmp_path
    monkeypatch
    doctest

doc/en/how-to/parametrize.rst

@@ -20,6 +20,11 @@ pytest enables test parametrization at several levels:
 * `pytest_generate_tests`_ allows one to define custom parametrization
   schemes or extensions.
 
+
+.. note::
+
+    See :ref:`subtests` for an alternative to parametrization.
+
 .. _parametrizemark:
 .. _`@pytest.mark.parametrize`:
 
@@ -203,6 +208,7 @@ To get all combinations of multiple parametrized arguments you can stack
 This will run the test with the arguments set to ``x=0/y=2``, ``x=1/y=2``,
 ``x=0/y=3``, and ``x=1/y=3`` exhausting parameters in the order of the decorators.
 
+
 .. _`pytest_generate_tests`:
 
 Basic ``pytest_generate_tests`` example

doc/en/how-to/subtests.rst

@@ -0,0 +1,109 @@
+.. _subtests:
+
+How to use subtests
+===================
+
+.. versionadded:: 9.0
+
+.. note::
+
+    This feature is experimental. Its behavior, particularly how failures are reported, may evolve in future releases. However, the core functionality and usage are considered stable.
+
+pytest allows for grouping assertions within a normal test, known as *subtests*.
+
+Subtests are an alternative to parametrization, particularly useful when the exact parametrization values are not known at collection time.
+
+
+.. code-block:: python
+
+    # content of test_subtest.py
+
+
+    def test(subtests):
+        for i in range(5):
+            with subtests.test(msg="custom message", i=i):
+                assert i % 2 == 0
+
+Each assertion failure or error is caught by the context manager and reported individually:
+
+.. code-block:: pytest
+
+    $ pytest -q test_subtest.py
+
+
+In the output above:
+
+* Subtest failures are reported as ``SUBFAILED``.
+* Subtests are reported first and the "top-level" test is reported at the end on its own.
+
+Note that it is possible to use ``subtests`` multiple times in the same test, or even mix and match with normal assertions
+outside the ``subtests.test`` block:
+
+.. code-block:: python
+
+    def test(subtests):
+        for i in range(5):
+            with subtests.test("stage 1", i=i):
+                assert i % 2 == 0
+
+        assert func() == 10
+
+        for i in range(10, 20):
+            with subtests.test("stage 2", i=i):
+                assert i % 2 == 0
+
+.. note::
+
+    See :ref:`parametrize` for an alternative to subtests.
+
+
+Verbosity
+---------
+
+By default, only **subtest failures** are shown. Higher verbosity levels (``-v``) will also show progress output for **passed** subtests.
+
+It is possible to control the verbosity of subtests by setting :confval:`verbosity_subtests`.
+
+
+Typing
+------
+
+:class:`pytest.Subtests` is exported so it can be used in type annotations:
+
+.. code-block:: python
+
+    def test(subtests: pytest.Subtests) -> None: ...
+
+.. _parametrize_vs_subtests:
+
+Parametrization vs Subtests
+---------------------------
+
+While :ref:`traditional pytest parametrization <parametrize>` and ``subtests`` are similar, they have important differences and use cases.
+
+
+Parametrization
+~~~~~~~~~~~~~~~
+
+* Happens at collection time.
+* Generates individual tests.
+* Parametrized tests can be referenced from the command line.
+* Plays well with plugins that handle test execution, such as ``--last-failed``.
+* Ideal for decision table testing.
+
+Subtests
+~~~~~~~~
+
+* Happen during test execution.
+* Are not known at collection time.
+* Can be generated dynamically.
+* Cannot be referenced individually from the command line.
+* Plugins that handle test execution cannot target individual subtests.
+* An assertion failure inside a subtest does not interrupt the test, letting users see all failures in the same report.
+
+
+.. note::
+
+    This feature was originally implemented as a separate plugin in `pytest-subtests <https://github.com/pytest-dev/pytest-subtests>`__, but since ``9.0`` has been merged into the core.
+
+    The core implementation should be compatible to the plugin implementation, except it does not contain custom command-line options to control subtest output.

doc/en/how-to/unittest.rst

@@ -22,17 +22,14 @@ their ``test`` methods in ``test_*.py`` or ``*_test.py`` files.
 
 Almost all ``unittest`` features are supported:
 
-* ``@unittest.skip`` style decorators;
-* ``setUp/tearDown``;
-* ``setUpClass/tearDownClass``;
-* ``setUpModule/tearDownModule``;
+* :func:`unittest.skip`/:func:`unittest.skipIf` style decorators
+* :meth:`unittest.TestCase.setUp`/:meth:`unittest.TestCase.tearDown`
+* :meth:`unittest.TestCase.setUpClass`/:meth:`unittest.TestCase.tearDownClass`
+* :func:`unittest.setUpModule`/:func:`unittest.tearDownModule`
+* :meth:`unittest.TestCase.subTest` (since version ``9.0``)
 
-.. _`pytest-subtests`: https://github.com/pytest-dev/pytest-subtests
 .. _`load_tests protocol`: https://docs.python.org/3/library/unittest.html#load-tests-protocol
 
-Additionally, :ref:`subtests <python:subtests>` are supported by the
-`pytest-subtests`_ plugin.
-
 Up to this point pytest does not have support for the following features:
 
 * `load_tests protocol`_;

doc/en/reference/fixtures.rst

@@ -52,6 +52,9 @@ Built-in fixtures
    :fixture:`pytestconfig`
         Access to configuration values, pluginmanager and plugin hooks.
 
+   :fixture:`subtests`
+        Enable declaring subtests inside test functions.
+
    :fixture:`record_property`
        Add extra properties to the test.
 

doc/en/reference/reference.rst

@@ -572,6 +572,19 @@ The ``request`` fixture is a special fixture providing information of the reques
     :members:
 
 
+.. fixture:: subtests
+
+subtests
+~~~~~~~~
+
+The ``subtests`` fixture enables declaring subtests inside test functions.
+
+**Tutorial**: :ref:`subtests`
+
+.. autoclass:: pytest.Subtests()
+    :members:
+
+
 .. fixture:: testdir
 
 testdir
@@ -2600,8 +2613,35 @@ passed multiple times. The expected format is ``name=value``. For example::
             [pytest]
             verbosity_assertions = 2
 
-    Defaults to application wide verbosity level (via the ``-v`` command-line option). A special value of
-    "auto" can be used to explicitly use the global verbosity level.
+    If not set, defaults to application wide verbosity level (via the ``-v`` command-line option). A special value of
+    ``"auto"`` can be used to explicitly use the global verbosity level.
+
+
+.. confval:: verbosity_subtests
+
+    Set the verbosity level specifically for **passed** subtests.
+
+    .. tab:: toml
+
+        .. code-block:: toml
+
+            [pytest]
+            verbosity_subtests = 1
+
+    .. tab:: ini
+
+        .. code-block:: ini
+
+            [pytest]
+            verbosity_subtests = 1
+
+    A value of ``1`` or higher will show output for **passed** subtests (**failed** subtests are always reported).
+    Passed subtests output can be suppressed with the value ``0``, which overwrites the ``-v`` command-line option.
+
+    If not set, defaults to application wide verbosity level (via the ``-v`` command-line option). A special value of
+    ``"auto"`` can be used to explicitly use the global verbosity level.
+
+    See also: :ref:`subtests`.
 
 
 .. confval:: verbosity_test_cases
@@ -2622,8 +2662,8 @@ passed multiple times. The expected format is ``name=value``. For example::
             [pytest]
             verbosity_test_cases = 2
 
-    Defaults to application wide verbosity level (via the ``-v`` command-line option). A special value of
-    "auto" can be used to explicitly use the global verbosity level.
+    If not set, defaults to application wide verbosity level (via the ``-v`` command-line option). A special value of
+    ``"auto"`` can be used to explicitly use the global verbosity level.
 
 
 .. _`command-line-flags`:

src/_pytest/config/__init__.py

@@ -282,6 +282,7 @@ def directory_arg(path: str, optname: str) -> str:
     "logging",
     "reports",
     "faulthandler",
+    "subtests",
 )
 
 builtin_plugins = {
@@ -1878,6 +1879,9 @@ def getvalueorskip(self, name: str, path=None):
     VERBOSITY_ASSERTIONS: Final = "assertions"
     #: Verbosity type for test case execution (see :confval:`verbosity_test_cases`).
     VERBOSITY_TEST_CASES: Final = "test_cases"
+    #: Verbosity type for failed subtests (see :confval:`verbosity_subtests`).
+    VERBOSITY_SUBTESTS: Final = "subtests"
+
     _VERBOSITY_INI_DEFAULT: Final = "auto"
 
     def get_verbosity(self, verbosity_type: str | None = None) -> int:

src/_pytest/deprecated.py

@@ -25,6 +25,7 @@
     "pytest_catchlog",
     "pytest_capturelog",
     "pytest_faulthandler",
+    "pytest_subtests",
 }