Add end-to-end tests for skillz CLI using VHS#9
Merged
Conversation
- Introduced a new directory `test/e2e` containing end-to-end tests for the skillz CLI. - Added `.gitignore` to exclude generated binaries and output files. - Created README.md detailing the purpose and usage of the end-to-end tests. - Implemented multiple flow recordings including: - `add`: Tests interactive skill addition. - `copy`: Tests non-interactive skill copying. - `global`: Tests global skill installation. - `init`: Tests skill scaffolding. - `list`: Tests listing installed skills. - `remove`: Tests interactive skill removal. - `update`: Tests checking for skill updates. - `error`: Tests error handling for missing local sources. - Each flow generates a GIF and a final-frame text snapshot for verification. - Added scripts for running tests and extracting final frames from VHS recordings. - Included sample skills in `test/fixtures/sample-skills` for testing purposes.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a VHS-based end-to-end testing tier for the skillz CLI, recording real terminal sessions in a pinned container and asserting deterministic “final frame” snapshots against committed goldens. It also wires this into CI (artifacts + PR commenting) and updates local developer tooling (devcontainer) to make the flows reproducible.
Changes:
- Added
test/e2eVHS tapes, goldens, scripts (run.sh,extract-frame.sh), and documentation to record/verify representative CLI flows. - Introduced an opt-in xUnit smoke test (
SKILLZ_E2E=1) that runs the full recording/verification suite on Linux with Docker. - Added a GitHub Actions workflow to run recordings, upload artifacts, and post/update a PR comment showing changed/new recordings; added fixture sample skills for hermetic tests.
Reviewed changes
Copilot reviewed 28 out of 36 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
test/Skillz.SmokeTests/VhsRecordingTests.cs |
Opt-in xUnit E2E test that invokes test/e2e/run.sh and asserts all flows match goldens |
test/fixtures/sample-skills/skills/pr-describer/SKILL.md |
Fixture skill used by E2E recordings |
test/fixtures/sample-skills/skills/commit-helper/SKILL.md |
Fixture skill used by E2E recordings |
test/fixtures/sample-skills/skills/changelog-updater/SKILL.md |
Fixture skill used by E2E recordings |
test/e2e/.gitignore |
Ignores published binary/output directories generated by E2E runs |
test/e2e/README.md |
Documents purpose, determinism strategy, and how to run/update goldens |
test/e2e/run.sh |
Publishes skillz, runs the pinned VHS container per flow, extracts final frames, diffs/updates goldens, and produces a CI report |
test/e2e/extract-frame.sh |
Reduces VHS multi-frame .txt captures to a deterministic final frame |
test/e2e/add-flow.tape |
VHS script for interactive skillz add flow |
test/e2e/add-flow.golden.txt |
Golden final-frame snapshot for add flow |
test/e2e/copy-flow.tape |
VHS script for non-interactive skillz add --copy flow |
test/e2e/copy-flow.golden.txt |
Golden final-frame snapshot for copy flow |
test/e2e/global-flow.tape |
VHS script for non-interactive global install flow |
test/e2e/global-flow.golden.txt |
Golden final-frame snapshot for global flow |
test/e2e/init-flow.tape |
VHS script for skillz init scaffolding flow |
test/e2e/init-flow.golden.txt |
Golden final-frame snapshot for init flow |
test/e2e/list-flow.tape |
VHS script for skillz list flow (after hidden setup install) |
test/e2e/list-flow.golden.txt |
Golden final-frame snapshot for list flow |
test/e2e/remove-flow.tape |
VHS script for interactive skillz remove flow |
test/e2e/remove-flow.golden.txt |
Golden final-frame snapshot for remove flow |
test/e2e/update-flow.tape |
VHS script for interactive skillz update flow |
test/e2e/update-flow.golden.txt |
Golden final-frame snapshot for update flow |
test/e2e/error-flow.tape |
VHS script for error-path skillz add ./missing flow |
test/e2e/error-flow.golden.txt |
Golden final-frame snapshot for error flow |
.github/workflows/e2e-demo.yml |
CI workflow to run/verify recordings, upload artifacts, and comment changed/new flows on PRs |
.devcontainer/Dockerfile |
Adds rationale notes for .NET 10 + Docker support via devcontainer features |
.devcontainer/devcontainer.json |
Adds .NET 10 SDK + docker-in-docker features and verifies Docker in postCreateCommand |
.devcontainer/devcontainer-lock.json |
Locks the added devcontainer feature versions/digests |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
🎬 E2E snapshots changedThese flow recordings differ from 🆕 add — added
🆕 copy — added
🆕 error — added
🆕 global — added
🆕 init — added
🆕 list — added
🆕 remove — added
🆕 update — added
Run 27507175339 · full set in the skillz-e2e-snapshots artifact. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test/e2econtaining end-to-end tests for the skillz CLI..gitignoreto exclude generated binaries and output files.add: Tests interactive skill addition.copy: Tests non-interactive skill copying.global: Tests global skill installation.init: Tests skill scaffolding.list: Tests listing installed skills.remove: Tests interactive skill removal.update: Tests checking for skill updates.error: Tests error handling for missing local sources.test/fixtures/sample-skillsfor testing purposes.