Skip to content

test(workbench): order the suite intentionally and group git tests#500

Open
ian-flores wants to merge 2 commits into
mainfrom
workbench-test-ordering
Open

test(workbench): order the suite intentionally and group git tests#500
ian-flores wants to merge 2 commits into
mainfrom
workbench-test-ordering

Conversation

@ian-flores

@ian-flores ian-flores commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

Part of epic #480. Imposes an intentional collection order on the Workbench
BDD suite so the run tells a coherent story instead of falling out of
alphabetical filenames.

Mechanism

One pytestmark = pytest.mark.order(N) per Workbench test module (ranks with
gaps for future insertion). Composes with — does not replace — the existing
workbench/conftest.py::pytest_collection_modifyitems xdist_group hook:
order governs same-worker sequencing, xdist_group governs worker
assignment. Under --interactive-auth all Workbench tests pin to one worker,
so the order holds exactly where the serial run's readability matters.

Adds pytest-order as a runtime dependency (the shipped product tests carry
the markers) and a selftests/test_workbench_ordering.py asserting the
ordering invariants.

Verification

pytest selftests/test_workbench_ordering.py — 4 passed:
auth-before-launch, launch-before-extensions, launch-before-git, extensions-last.

Resulting Workbench collection order:

test_auth → test_ide_launch → test_sessions → test_session_capacity(_k8s)
→ test_session_idle → test_runtime_versions → [in-session: chronicle,
data_sources, git_ops, jobs, packages, publish_to_connect] → test_ide_extensions (last)

ruff check / ruff format --check: clean.

Workbench BDD tests collected in plain alphabetical file order, so
test_ide_extensions.py and test_git_ops.py sorted ahead of
test_ide_launch.py — extension and git checks read as running before the
session launch they depend on.

Impose an intentional order via module-level pytest-order markers (one
pytestmark = pytest.mark.order(N) per module): auth → launch → session
lifecycle → in-session features → extensions. Group git-ops into the
in-session tier after launch (#482). The markers compose with the
existing xdist_group serialization hook rather than replacing it.

Adds pytest-order as a dependency and a selftest asserting the ordering
invariants (auth before launch, launch before extensions/git, extensions
last).

Closes #481, #482.
Copilot AI review requested due to automatic review settings July 17, 2026 21:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR makes the Workbench BDD suite’s collection order intentional (rather than incidental alphabetical order) by adding module-level pytest-order markers, ensuring the run reads in a coherent sequence (auth → launch → lifecycle → in-session features → extensions). It also adds a selftest to assert key ordering invariants and vendors pytest-order as a runtime dependency so the shipped product tests always have the ordering plugin available.

Changes:

  • Add pytestmark = pytest.mark.order(N) to each Workbench test module to enforce a logical suite order (including grouping test_git_ops into the in-session tier).
  • Add pytest-order as a runtime dependency (and lock it in uv.lock).
  • Add selftests/test_workbench_ordering.py to assert ordering invariants, plus a validation demo markdown write-up.

Reviewed changes

Copilot reviewed 17 out of 18 changed files in this pull request and generated no comments.

Show a summary per file
File Description
validation_docs/demo-480-workbench-ordering.md Adds a demo/validation write-up showing the intended Workbench collection order and example commands/output.
pyproject.toml Adds pytest-order>=1.3 to runtime dependencies so order markers are supported when running shipped tests.
uv.lock Locks pytest-order and updates the resolved environment accordingly.
src/vip_tests/workbench/test_auth.py Sets order rank 10 (auth gate).
src/vip_tests/workbench/test_ide_launch.py Sets order rank 20 (session launch).
src/vip_tests/workbench/test_sessions.py Sets order rank 30 (session lifecycle).
src/vip_tests/workbench/test_session_capacity.py Sets order rank 40 (capacity).
src/vip_tests/workbench/test_session_capacity_k8s.py Sets order rank 40 (capacity on k8s).
src/vip_tests/workbench/test_session_idle.py Sets order rank 45 (idle behavior).
src/vip_tests/workbench/test_runtime_versions.py Sets order rank 50 (runtime version checks).
src/vip_tests/workbench/test_chronicle.py Sets order rank 60 (in-session feature group).
src/vip_tests/workbench/test_data_sources.py Sets order rank 60 (in-session feature group).
src/vip_tests/workbench/test_git_ops.py Sets order rank 60 (in-session feature group; grouped after launch).
src/vip_tests/workbench/test_jobs.py Sets order rank 60 (in-session feature group).
src/vip_tests/workbench/test_packages.py Sets order rank 60 (in-session feature group).
src/vip_tests/workbench/test_publish_to_connect.py Sets order rank 60 (in-session feature group).
src/vip_tests/workbench/test_ide_extensions.py Sets order rank 90 (final validation step).
selftests/test_workbench_ordering.py Adds selftests that run --collect-only and assert key ordering invariants.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@github-actions

Copy link
Copy Markdown

@ian-flores
ian-flores marked this pull request as ready for review July 17, 2026 21:22
PR #500 was branched with a validation_docs/ demo following the showboat
workflow that #493 removed (recipes, .gitignore entry, guidance, and all
prior demos). Drop demo-480-workbench-ordering.md to align; the verification
evidence lives in the PR body instead.

@bdeitte bdeitte left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, makes sense to do this in all tests over time

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(workbench): group git/cloning tests after session-launch tests test(workbench): order tests logically (IDE launch before extensions)

3 participants