test(workbench): order the suite intentionally and group git tests#500
test(workbench): order the suite intentionally and group git tests#500ian-flores wants to merge 2 commits into
Conversation
Workbench BDD tests collected in plain alphabetical file order, so test_ide_extensions.py and test_git_ops.py sorted ahead of test_ide_launch.py — extension and git checks read as running before the session launch they depend on. Impose an intentional order via module-level pytest-order markers (one pytestmark = pytest.mark.order(N) per module): auth → launch → session lifecycle → in-session features → extensions. Group git-ops into the in-session tier after launch (#482). The markers compose with the existing xdist_group serialization hook rather than replacing it. Adds pytest-order as a dependency and a selftest asserting the ordering invariants (auth before launch, launch before extensions/git, extensions last). Closes #481, #482.
There was a problem hiding this comment.
Pull request overview
This PR makes the Workbench BDD suite’s collection order intentional (rather than incidental alphabetical order) by adding module-level pytest-order markers, ensuring the run reads in a coherent sequence (auth → launch → lifecycle → in-session features → extensions). It also adds a selftest to assert key ordering invariants and vendors pytest-order as a runtime dependency so the shipped product tests always have the ordering plugin available.
Changes:
- Add
pytestmark = pytest.mark.order(N)to each Workbench test module to enforce a logical suite order (including groupingtest_git_opsinto the in-session tier). - Add
pytest-orderas a runtime dependency (and lock it inuv.lock). - Add
selftests/test_workbench_ordering.pyto assert ordering invariants, plus a validation demo markdown write-up.
Reviewed changes
Copilot reviewed 17 out of 18 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| validation_docs/demo-480-workbench-ordering.md | Adds a demo/validation write-up showing the intended Workbench collection order and example commands/output. |
| pyproject.toml | Adds pytest-order>=1.3 to runtime dependencies so order markers are supported when running shipped tests. |
| uv.lock | Locks pytest-order and updates the resolved environment accordingly. |
| src/vip_tests/workbench/test_auth.py | Sets order rank 10 (auth gate). |
| src/vip_tests/workbench/test_ide_launch.py | Sets order rank 20 (session launch). |
| src/vip_tests/workbench/test_sessions.py | Sets order rank 30 (session lifecycle). |
| src/vip_tests/workbench/test_session_capacity.py | Sets order rank 40 (capacity). |
| src/vip_tests/workbench/test_session_capacity_k8s.py | Sets order rank 40 (capacity on k8s). |
| src/vip_tests/workbench/test_session_idle.py | Sets order rank 45 (idle behavior). |
| src/vip_tests/workbench/test_runtime_versions.py | Sets order rank 50 (runtime version checks). |
| src/vip_tests/workbench/test_chronicle.py | Sets order rank 60 (in-session feature group). |
| src/vip_tests/workbench/test_data_sources.py | Sets order rank 60 (in-session feature group). |
| src/vip_tests/workbench/test_git_ops.py | Sets order rank 60 (in-session feature group; grouped after launch). |
| src/vip_tests/workbench/test_jobs.py | Sets order rank 60 (in-session feature group). |
| src/vip_tests/workbench/test_packages.py | Sets order rank 60 (in-session feature group). |
| src/vip_tests/workbench/test_publish_to_connect.py | Sets order rank 60 (in-session feature group). |
| src/vip_tests/workbench/test_ide_extensions.py | Sets order rank 90 (final validation step). |
| selftests/test_workbench_ordering.py | Adds selftests that run --collect-only and assert key ordering invariants. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Preview Links
|
bdeitte
left a comment
There was a problem hiding this comment.
Nice, makes sense to do this in all tests over time
Summary
Part of epic #480. Imposes an intentional collection order on the Workbench
BDD suite so the run tells a coherent story instead of falling out of
alphabetical filenames.
in-session features → extensions, via module-level
pytest-ordermarkers.test_git_opsinto the in-session tier so it collectsafter session launch. (Its config-less public-clone path already shipped
in fix(workbench): public-repo cloning wrongly appears to require VIP_GIT_TOKEN #483, so ordering is the only remaining work.)
Mechanism
One
pytestmark = pytest.mark.order(N)per Workbench test module (ranks withgaps for future insertion). Composes with — does not replace — the existing
workbench/conftest.py::pytest_collection_modifyitemsxdist_group hook:ordergoverns same-worker sequencing,xdist_groupgoverns workerassignment. Under
--interactive-authall Workbench tests pin to one worker,so the order holds exactly where the serial run's readability matters.
Adds
pytest-orderas a runtime dependency (the shipped product tests carrythe markers) and a
selftests/test_workbench_ordering.pyasserting theordering invariants.
Verification
pytest selftests/test_workbench_ordering.py— 4 passed:auth-before-launch, launch-before-extensions, launch-before-git, extensions-last.
Resulting Workbench collection order:
ruff check/ruff format --check: clean.