Skip to content

Unify end-to-end tests under pytest - #13545

Draft
bneradt wants to merge 2 commits into
apache:masterfrom
bneradt:pytest-replay-tests
Draft

Unify end-to-end tests under pytest#13545
bneradt wants to merge 2 commits into
apache:masterfrom
bneradt:pytest-replay-tests

Conversation

@bneradt

@bneradt bneradt commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

This is a draft for the all-in end-to-end test migration discussed on the
original pytest replay PR. It supersedes #13035 without depending on changes to
the AuTest project.

Summary

  • make pytest own the complete ATS end-to-end inventory
  • collect all 96 tracked Proxy Verifier replays directly from .test.yaml
  • collect each of the 505 bespoke .test.py definitions as one pytest item,
    using the pinned AuTest package only as a compatibility backend
  • replace the separate parallel runner with pytest-xdist scheduling, stable CI
    sharding, isolated sandboxes/ports, and shared/exclusive serial-test locking
  • rename the public interface to e2etest: e2etest.sh, CMake options and
    targets, presets, documentation, environment variables, and the replay
    e2etest: metadata key

Each replay file remains an independent item. A feature with several scenarios
should therefore keep separate scenario.test.yaml files unless the sessions
intentionally share ATS state.

Developer workflow

Run one test from the source tree:

./tests/e2etest.sh -f cache-control-basic
./tests/e2etest.sh -f cache-auth

The same -f workflow selects native YAML or compatibility Python items and
supports globs and multiple names. -k, -m, -j, --list, and other pytest
selection options remain available. A configured tree generates
<build>/tests/e2etest.sh for the fastest warmed-up single-test path.

Container behavior

The source entry point defaults to
ci.trafficserver.apache.org/ats/fedora:44, performs an incremental dedicated
build/install, and runs the selected tests there. It launches Docker with the
CI-relevant --init, host networking, and SYS_PTRACE capability, but does not
use --privileged or mount the Docker socket.

Automatic direct execution requires both a container marker and exact Fedora
44 identification, so the existing Jenkins container does not attempt Docker
in Docker. --run-in-docker and --no-run-in-docker override the decision.
Container-local short /tmp sandboxes avoid Unix-socket path and bind-mount
problems while regular diagnostics are copied back to the build tree.

CI compatibility

Temporary deprecated aliases retain ci-fedora-autest, autest.sh, the old
CMake options/targets, and the Jenkins parallel_mode feature probe. This lets
the current external pipeline execute the unified pytest inventory immediately;
the pipeline can rename its stage, preset, and command in a follow-up. Existing
SHARD/SHARDCNT values now shard all pytest items rather than only .test.py
files.

Validation

  • Fedora 44 Docker build, install, and 627-item local collection (the worktree
    contains one unrelated untracked replay in addition to this PR's 626 items)
  • all replay items passed or reached their expected capability skips
  • mixed native/compatibility selections passed under xdist
  • representative DNS, config-reload, next-hop, and plugin compatibility tests
    passed concurrently
  • current Jenkins-style 3/12 compatibility shard collected 52 items
  • 25 framework tests and Python static checks passed
  • new and deprecated Fedora CMake presets configured successfully
  • repository formatting and whitespace hooks passed

AuTest's stalled file-format support leaves replay tests tied to thin
Python registration wrappers. Collect replay YAML directly so every
scenario has an independent pytest result and sandbox.

Add ATS-specific DNS, Proxy Verifier, Traffic Server, log, and metric
lifecycle support, then convert all 96 ATSReplayTest registrations to
.test.yaml files. Keep bespoke AuTests in place and run both suites from
the existing autest targets.
@bneradt bneradt added this to the 11.0.0 milestone Aug 13, 2026
Copilot AI lite review requested due to automatic review settings August 13, 2026 18:34
@bneradt bneradt self-assigned this Aug 13, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@bneradt
bneradt marked this pull request as draft August 13, 2026 18:37
End-to-end coverage is split between replay pytest items and AuTest
process definitions, leaving developers and CI with different runners,
parallelism rules, and environment setup.

This patch makes pytest own the full inventory, runs replay YAML
directly, and adapts bespoke Python definitions through the pinned AuTest
backend. It adds shared scheduling, filtering, sharding, serial
coordination, and e2etest terminology across the public test interface.

The source e2etest.sh entry point standardizes Fedora 44 Docker
execution while detecting existing CI containers and preserving simple
-f single-test use. Deprecated aliases keep current external Jenkins
jobs working during migration.
@bneradt bneradt changed the title Add pytest replay test framework Unify end-to-end tests under pytest Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants