Unify end-to-end tests under pytest - #13545
Draft
bneradt wants to merge 2 commits into
Draft
Conversation
AuTest's stalled file-format support leaves replay tests tied to thin Python registration wrappers. Collect replay YAML directly so every scenario has an independent pytest result and sandbox. Add ATS-specific DNS, Proxy Verifier, Traffic Server, log, and metric lifecycle support, then convert all 96 ATSReplayTest registrations to .test.yaml files. Keep bespoke AuTests in place and run both suites from the existing autest targets.
bneradt
marked this pull request as draft
August 13, 2026 18:37
End-to-end coverage is split between replay pytest items and AuTest process definitions, leaving developers and CI with different runners, parallelism rules, and environment setup. This patch makes pytest own the full inventory, runs replay YAML directly, and adapts bespoke Python definitions through the pinned AuTest backend. It adds shared scheduling, filtering, sharding, serial coordination, and e2etest terminology across the public test interface. The source e2etest.sh entry point standardizes Fedora 44 Docker execution while detecting existing CI containers and preserving simple -f single-test use. Deprecated aliases keep current external Jenkins jobs working during migration.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a draft for the all-in end-to-end test migration discussed on the
original pytest replay PR. It supersedes #13035 without depending on changes to
the AuTest project.
Summary
.test.yaml.test.pydefinitions as one pytest item,using the pinned AuTest package only as a compatibility backend
sharding, isolated sandboxes/ports, and shared/exclusive serial-test locking
e2etest:e2etest.sh, CMake options andtargets, presets, documentation, environment variables, and the replay
e2etest:metadata keyEach replay file remains an independent item. A feature with several scenarios
should therefore keep separate
scenario.test.yamlfiles unless the sessionsintentionally share ATS state.
Developer workflow
Run one test from the source tree:
The same
-fworkflow selects native YAML or compatibility Python items andsupports globs and multiple names.
-k,-m,-j,--list, and other pytestselection options remain available. A configured tree generates
<build>/tests/e2etest.shfor the fastest warmed-up single-test path.Container behavior
The source entry point defaults to
ci.trafficserver.apache.org/ats/fedora:44, performs an incremental dedicatedbuild/install, and runs the selected tests there. It launches Docker with the
CI-relevant
--init, host networking, andSYS_PTRACEcapability, but does notuse
--privilegedor mount the Docker socket.Automatic direct execution requires both a container marker and exact Fedora
44 identification, so the existing Jenkins container does not attempt Docker
in Docker.
--run-in-dockerand--no-run-in-dockeroverride the decision.Container-local short
/tmpsandboxes avoid Unix-socket path and bind-mountproblems while regular diagnostics are copied back to the build tree.
CI compatibility
Temporary deprecated aliases retain
ci-fedora-autest,autest.sh, the oldCMake options/targets, and the Jenkins
parallel_modefeature probe. This letsthe current external pipeline execute the unified pytest inventory immediately;
the pipeline can rename its stage, preset, and command in a follow-up. Existing
SHARD/SHARDCNTvalues now shard all pytest items rather than only.test.pyfiles.
Validation
contains one unrelated untracked replay in addition to this PR's 626 items)
passed concurrently