Validate enum-valued configuration options with Literal types

changelog/14791.improvement.rst

@@ -0,0 +1,3 @@
+The :confval:`junit_family`, :confval:`junit_logging`, :confval:`junit_duration_report`, :confval:`console_output_style`, :confval:`empty_parameter_set_mark`, :confval:`assertion_text_diff_style` and :confval:`tmp_path_retention_policy` configuration options are now registered with the set of values they accept: an invalid value now fails with a clean usage error instead of being silently ignored or crashing later, and ``pytest --help`` lists the accepted values.
+
+:func:`config.getini <pytest.Config.getini>` now raises :class:`pytest.UsageError` instead of ``TypeError`` or ``ValueError`` when the value read from the configuration file does not match the registered type, so a bad value is always reported as a short usage error message instead of an internal error traceback. Asking for an option name that was never registered still raises ``ValueError``.

doc/en/reference/reference.rst

@@ -1443,7 +1443,7 @@ passed multiple times. The expected format is ``name=value``. For example::
    .. versionadded:: 8.1
 
 .. confval:: console_output_style
-   :type: ``str``
+   :type: ``"classic" | "progress" | "count" | "times" | "progress-even-when-capture-no"``
    :default: ``"progress"``
 
    Sets the console output style while running tests:
@@ -1568,7 +1568,7 @@ passed multiple times. The expected format is ``name=value``. For example::
 
 
 .. confval:: empty_parameter_set_mark
-    :type: ``str``
+    :type: ``"skip" | "xfail" | "fail_at_collect"``
     :default: ``"skip"``
 
     Allows to pick the action for empty parametersets in parameterization
@@ -1733,7 +1733,7 @@ passed multiple times. The expected format is ``name=value``. For example::
 
 
 .. confval:: junit_duration_report
-    :type: ``str``
+    :type: ``"total" | "call"``
     :default: ``"total"``
 
     .. versionadded:: 4.1
@@ -1759,7 +1759,7 @@ passed multiple times. The expected format is ``name=value``. For example::
 
 
 .. confval:: junit_family
-    :type: ``str``
+    :type: ``"legacy" | "xunit1" | "xunit2"``
     :default: ``"xunit2"``
 
     .. versionadded:: 4.2
@@ -1811,7 +1811,7 @@ passed multiple times. The expected format is ``name=value``. For example::
 
 
 .. confval:: junit_logging
-    :type: ``str``
+    :type: ``"no" | "log" | "system-out" | "system-err" | "out-err" | "all"``
     :default: ``"no"``
 
     .. versionadded:: 3.5
@@ -2638,7 +2638,7 @@ passed multiple times. The expected format is ``name=value``. For example::
 
 
 .. confval:: tmp_path_retention_policy
-   :type: ``str``
+   :type: ``"all" | "failed" | "none"``
    :default: ``"all"``
 
    Controls which directories created by the `tmp_path` fixture are kept around,
@@ -2768,7 +2768,7 @@ passed multiple times. The expected format is ``name=value``. For example::
 
 
 .. confval:: assertion_text_diff_style
-    :type: ``str``
+    :type: ``"ndiff" | "block"``
     :default: ``"ndiff"``
 
     Set how pytest renders diffs for string equality assertions.
@@ -3654,7 +3654,7 @@ All the command-line flags can also be obtained by running ``pytest --help``::
     [pytest] configuration options in the first pytest.toml|pytest.ini|tox.ini|setup.cfg|pyproject.toml file found:
 
       markers (linelist):   Register new markers for test functions
-      empty_parameter_set_mark (string):
+      empty_parameter_set_mark ('skip' | 'xfail' | 'fail_at_collect'):
                             Default marker for empty parametersets
       strict_config (bool): Any warnings encountered while parsing the `pytest`
                             section of the configuration file raise errors
@@ -3696,11 +3696,11 @@ All the command-line flags can also be obtained by running ``pytest --help``::
       strict_parametrization_ids (bool):
                             Emit an error if non-unique parameter set IDs are
                             detected
-      console_output_style (string):
+      console_output_style ('classic' | 'progress' | 'count' | 'times' | 'progress-even-when-capture-no'):
                             Console output: "classic", or with additional
                             progress information ("progress" (percentage) |
-                            "count" | "progress-even-when-capture-no" (forces
-                            progress even when capture=no)
+                            "count" | "times" | "progress-even-when-capture-no"
+                            (forces progress even when capture=no)
       verbosity_test_cases (string):
                             Specify a verbosity level for test case execution,
                             overriding the main level. Higher levels will
@@ -3713,10 +3713,9 @@ All the command-line flags can also be obtained by running ``pytest --help``::
                             How many sessions should we keep the `tmp_path`
                             directories, according to
                             `tmp_path_retention_policy`.
-      tmp_path_retention_policy (string):
+      tmp_path_retention_policy ('all' | 'failed' | 'none'):
                             Controls which directories created by the `tmp_path`
                             fixture are kept around, based on test outcome.
-                            (all/failed/none)
       enable_assertion_pass_hook (bool):
                             Enables the pytest_assertion_pass hook. Make sure to
                             delete any previously generated pyc cache files.
@@ -3726,25 +3725,24 @@ All the command-line flags can also be obtained by running ``pytest --help``::
       truncation_limit_chars (string):
                             Set threshold of CHARS after which truncation will
                             take effect
-      assertion_text_diff_style (string):
+      assertion_text_diff_style ('ndiff' | 'block'):
                             Choose how pytest renders diffs for string equality
-                            assertions: ndiff or block
+                            assertions
       verbosity_assertions (string):
                             Specify a verbosity level for assertions, overriding
                             the main level. Higher levels will provide more
                             detailed explanation when an assertion fails.
       junit_suite_name (string):
                             Test suite name for JUnit report
-      junit_logging (string):
-                            Write captured log messages to JUnit report: one of
-                            no|log|system-out|system-err|out-err|all
+      junit_logging ('no' | 'log' | 'system-out' | 'system-err' | 'out-err' | 'all'):
+                            Write captured log messages to JUnit report
       junit_log_passing_tests (bool):
                             Capture log information for passing tests to JUnit
                             report:
-      junit_duration_report (string):
-                            Duration time to report: one of total|call
-      junit_family (string):
-                            Emit XML for schema: one of legacy|xunit1|xunit2
+      junit_duration_report ('total' | 'call'):
+                            Duration time to report
+      junit_family ('legacy' | 'xunit1' | 'xunit2'):
+                            Emit XML for schema
       doctest_optionflags (args):
                             Option flags for doctests
       doctest_encoding (string):

src/_pytest/assertion/__init__.py

@@ -12,6 +12,7 @@
 from _pytest.assertion import rewrite
 from _pytest.assertion import truncate
 from _pytest.assertion import util
+from _pytest.assertion._typing import _AssertionTextDiffStyle
 from _pytest.assertion._typing import NO_TRUNCATION_BUDGET
 from _pytest.assertion._typing import TruncationBudget
 from _pytest.assertion.rewrite import assertstate_key
@@ -61,12 +62,9 @@ def pytest_addoption(parser: Parser) -> None:
     )
     parser.addini(
         "assertion_text_diff_style",
-        default=util.ASSERTION_TEXT_DIFF_STYLE_NDIFF,
-        help=(
-            "Choose how pytest renders diffs for string equality assertions: "
-            f"{util.ASSERTION_TEXT_DIFF_STYLE_NDIFF} or "
-            f"{util.ASSERTION_TEXT_DIFF_STYLE_BLOCK}"
-        ),
+        type=_AssertionTextDiffStyle,
+        default="ndiff",
+        help="Choose how pytest renders diffs for string equality assertions",
     )
 
     Config._add_verbosity_ini(
@@ -80,7 +78,8 @@ def pytest_addoption(parser: Parser) -> None:
 
 
 def pytest_configure(config: Config) -> None:
-    util.validate_assertion_text_diff_style(config)
+    # Eagerly validate the value; it is only read lazily when an assertion fails.
+    config.getini("assertion_text_diff_style")
 
 
 def register_assert_rewrite(*names: str) -> None:
@@ -249,7 +248,7 @@ def pytest_assertrepr_compare(
         right=right,
         verbose=config.get_verbosity(Config.VERBOSITY_ASSERTIONS),
         highlighter=highlighter,
-        assertion_text_diff_style=util.get_assertion_text_diff_style(config),
+        assertion_text_diff_style=config.getini("assertion_text_diff_style"),
         truncation_budget=truncation_budget,
     )
     return truncate.materialize_with_truncation(lines, config) or None

src/_pytest/assertion/util.py

@@ -6,7 +6,6 @@
 from collections.abc import Callable
 from collections.abc import Iterator
 from collections.abc import Sequence
-from typing import Literal
 from unicodedata import normalize
 
 from _pytest import outcomes
@@ -24,7 +23,6 @@
 from _pytest.assertion.compare_text import _notin_text
 from _pytest.assertion.highlight import dummy_highlighter as dummy_highlighter
 from _pytest.config import Config
-from _pytest.config import UsageError
 
 
 # The _reprcompare attribute on the util module is used by the new assertion
@@ -40,32 +38,6 @@
 # Config object which is assigned during pytest_runtest_protocol.
 _config: Config | None = None
 
-ASSERTION_TEXT_DIFF_STYLE_INI = "assertion_text_diff_style"
-ASSERTION_TEXT_DIFF_STYLE_NDIFF: Literal["ndiff"] = "ndiff"
-ASSERTION_TEXT_DIFF_STYLE_BLOCK: Literal["block"] = "block"
-ASSERTION_TEXT_DIFF_STYLE_CHOICES = (
-    ASSERTION_TEXT_DIFF_STYLE_NDIFF,
-    ASSERTION_TEXT_DIFF_STYLE_BLOCK,
-)
-
-
-def get_assertion_text_diff_style(config: Config) -> _AssertionTextDiffStyle:
-    style = str(config.getini(ASSERTION_TEXT_DIFF_STYLE_INI))
-    match style:
-        case "ndiff" | "block":
-            return style
-        case _:
-            choices = ", ".join(
-                repr(choice) for choice in ASSERTION_TEXT_DIFF_STYLE_CHOICES
-            )
-            raise UsageError(
-                f"{ASSERTION_TEXT_DIFF_STYLE_INI} must be one of {choices}; got {style!r}"
-            )
-
-
-def validate_assertion_text_diff_style(config: Config) -> None:
-    get_assertion_text_diff_style(config)
-
 
 def format_explanation(explanation: str) -> str:
     r"""Format an explanation.

src/_pytest/config/__init__.py

@@ -1739,6 +1739,9 @@ def getini(self, name: str) -> Any:
         If the specified name hasn't been registered through a prior
         :func:`parser.addini <pytest.Parser.addini>` call (usually from a
         plugin), a ValueError is raised.
+
+        If the value read from the configuration file does not match the
+        registered ``type``, a :class:`~pytest.UsageError` is raised.
         """
         canonical_name = self._parser._ini_aliases.get(name, name)
         try:
@@ -1784,21 +1787,28 @@ def _getini(self, name: str):
         value = selected.value
         mode = selected.mode
 
-        if not isinstance(type, tuple):
-            return self._getini_value(mode, name, canonical_name, type, value, default)
-
-        # Union: try each member; the first one that accepts the value wins.
-        for member in type:
-            try:
+        # An invalid value is a user error, raised as UsageError so that it is
+        # reported as a short message rather than an internal error traceback.
+        try:
+            if not isinstance(type, tuple):
                 return self._getini_value(
-                    mode, name, canonical_name, member, value, default
+                    mode, name, canonical_name, type, value, default
                 )
-            except (TypeError, ValueError):
-                pass
-        raise TypeError(
-            f"{self.inipath}: config option '{name}' expects one of "
-            f"{_ini_type_repr(type)}, got {builtins.type(value).__name__}: {value!r}"
-        )
+
+            # Union: try each member; the first one that accepts the value wins.
+            for member in type:
+                try:
+                    return self._getini_value(
+                        mode, name, canonical_name, member, value, default
+                    )
+                except (TypeError, ValueError):
+                    pass
+            raise TypeError(
+                f"{self.inipath}: config option '{name}' expects one of "
+                f"{_ini_type_repr(type)}, got {builtins.type(value).__name__}: {value!r}"
+            )
+        except (TypeError, ValueError) as e:
+            raise UsageError(str(e)) from e
 
     def _getini_value(
         self,

src/_pytest/junitxml.py

@@ -15,6 +15,7 @@
 import os
 import platform
 import re
+from typing import Literal
 import xml.etree.ElementTree as ET
 
 from _pytest import nodes
@@ -33,6 +34,10 @@
 
 xml_key = StashKey["LogXML"]()
 
+_JunitLogging = Literal["no", "log", "system-out", "system-err", "out-err", "all"]
+_JunitDurationReport = Literal["total", "call"]
+_JunitFamily = Literal["legacy", "xunit1", "xunit2"]
+
 
 def bin_xml_escape(arg: object) -> str:
     r"""Visually escape invalid XML characters.
@@ -396,8 +401,8 @@ def pytest_addoption(parser: Parser) -> None:
     )
     parser.addini(
         "junit_logging",
-        "Write captured log messages to JUnit report: "
-        "one of no|log|system-out|system-err|out-err|all",
+        "Write captured log messages to JUnit report",
+        type=_JunitLogging,
         default="no",
     )
     parser.addini(
@@ -408,12 +413,14 @@ def pytest_addoption(parser: Parser) -> None:
     )
     parser.addini(
         "junit_duration_report",
-        "Duration time to report: one of total|call",
+        "Duration time to report",
+        type=_JunitDurationReport,
         default="total",
-    )  # choices=['total', 'call'])
+    )
     parser.addini(
         "junit_family",
-        "Emit XML for schema: one of legacy|xunit1|xunit2",
+        "Emit XML for schema",
+        type=_JunitFamily,
         default="xunit2",
     )
 
@@ -422,14 +429,13 @@ def pytest_configure(config: Config) -> None:
     xmlpath = config.option.xmlpath
     # Prevent opening xmllog on worker nodes (xdist).
     if xmlpath and not hasattr(config, "workerinput"):
-        junit_family = config.getini("junit_family")
         config.stash[xml_key] = LogXML(
             xmlpath,
             config.option.junitprefix,
             config.getini("junit_suite_name"),
             config.getini("junit_logging"),
             config.getini("junit_duration_report"),
-            junit_family,
+            config.getini("junit_family"),
             config.getini("junit_log_passing_tests"),
         )
         config.pluginmanager.register(config.stash[xml_key])
@@ -459,9 +465,9 @@ def __init__(
         logfile,
         prefix: str | None,
         suite_name: str = "pytest",
-        logging: str = "no",
-        report_duration: str = "total",
-        family="xunit1",
+        logging: _JunitLogging = "no",
+        report_duration: _JunitDurationReport = "total",
+        family: _JunitFamily = "xunit1",
         log_passing_tests: bool = True,
     ) -> None:
         logfile = os.path.expanduser(os.path.expandvars(logfile))

src/_pytest/mark/__init__.py

@@ -10,6 +10,7 @@
 from typing import TYPE_CHECKING
 
 from .expression import Expression
+from .structures import _EmptyParameterSetMark
 from .structures import _HiddenParam
 from .structures import EMPTY_PARAMETERSET_OPTION
 from .structures import get_empty_parameterset_mark
@@ -124,7 +125,12 @@ def pytest_addoption(parser: Parser) -> None:
     )
 
     parser.addini("markers", "Register new markers for test functions", "linelist")
-    parser.addini(EMPTY_PARAMETERSET_OPTION, "Default marker for empty parametersets")
+    parser.addini(
+        EMPTY_PARAMETERSET_OPTION,
+        "Default marker for empty parametersets",
+        type=_EmptyParameterSetMark,
+        default="skip",
+    )
 
 
 @hookimpl(tryfirst=True)
@@ -288,13 +294,8 @@ def pytest_configure(config: Config) -> None:
     config.stash[old_mark_config_key] = MARK_GEN._config
     MARK_GEN._config = config
 
-    empty_parameterset = config.getini(EMPTY_PARAMETERSET_OPTION)
-
-    if empty_parameterset not in ("skip", "xfail", "fail_at_collect", None, ""):
-        raise UsageError(
-            f"{EMPTY_PARAMETERSET_OPTION!s} must be one of skip, xfail or fail_at_collect"
-            f" but it is {empty_parameterset!r}"
-        )
+    # Eagerly validate the value; it is only read lazily during collection.
+    config.getini(EMPTY_PARAMETERSET_OPTION)
 
 
 def pytest_unconfigure(config: Config) -> None: