test(e2e): add live end-to-end test suite for the jh CLI#43
Merged
Conversation
Add an `e2e`-tagged Go test suite under e2e/ that builds the jh binary and
runs its read/GET commands against a live JuliaHub instance (the platform
nightly instance by default). One file per command category; commands that
need a specific entity (dataset, registry, project) list first and drive the
detail command off the first entry. Tests assert real behaviour and skip
gracefully on backend gaps (missing perms, absent endpoints, disallowed
queries, timeouts) while still failing on genuine CLI defects.
Authentication is purely file-based: the harness materializes a throwaway
~/.juliahub in an isolated HOME from credentials supplied via env vars, so the
CLI's auth path is exercised unchanged and a real login is never touched. The
suite is build-tag gated so `go test ./...` is unaffected; CI compile-checks it.
Also fix `admin group list`, which failed to parse the /app/config/groups
response: the endpoint returns groups bucketed by category
({"juliahub":{"groups":[...]},"site":{"groups":[...]}}) but the code expected a
top-level array. Parse the categorized object, flatten, and sort by id.
Includes e2e/ci/ glue (composite action + example job) for wiring the suite
into the JuliaHub platform nightly CI.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add e2e/scan_test.go for `jh scan`: a deterministic input-validation test (missing manifest) plus a submit→status→results lifecycle that uploads a sample Manifest.toml with --no-wait, then drives `scan status`/`scan results` off the server-assigned run_uuid. Skips when the static-analysis endpoint is absent on the instance (404). Adds an errorLine() harness helper for clearer skip diagnostics. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Self-review cleanup of the e2e suite: - Split the 459-line harness_test.go: run machinery stays in harness_test.go; assertions, output parsers, and backend-gap/skip helpers move to helpers_test.go. - Unify the duplicated packageBackendGap/scanBackendGap into a single backendGap() helper; use the clearer errorLine() in their skip messages. - Fix a temp-dir leak: the freshly built binary's dir is now removed on exit. - Drop the write-only result.err field and the dead expectedName var. - Remove the duplicated e2e/ci/ CI snippets; the canonical wiring lives in the JuliaHub repo. Fold a concise "Running in CI" section into e2e/README.md. - Add a Makefile (make e2e / e2e-fast / e2e-compile / check) so developers don't have to remember the -tags e2e invocations; e2e-fast skips the slow project-list tests (~14s vs ~70s). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Member
|
Here is the coverage report. We should improve it either in this or future
|
Contributor
Author
|
@vdayanand one if the reasons for this coverage report is much of the post/delete flows are not e2e tested, i refrained from that as it is being tested against nightly and didnt want to have dangling data in the instance from each test run |
vdayanand
approved these changes
Jun 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add an e2e-tagged Go test suite under e2e/ that builds the jh binary and runs its read/GET commands against a live JuliaHub instance (nightly by default). One file per command category; entity-specific commands (dataset, registry, project) list first and drive the detail command off the first entry. Tests assert real behaviour and skip on backend gaps (missing perms, absent endpoints, disallowed queries, timeouts) while still failing on genuine CLI defects.
Auth is file-based: the harness writes a throwaway ~/.juliahub in an isolated HOME from env-supplied tokens, so the CLI's auth path is unchanged. Build-tag gated so go test ./... is unaffected; CI compile-checks it.
Also fix admin group list: /app/config/groups returns groups bucketed by category, not a top-level array — parse the object, flatten, sort by id.
https://github.com/JuliaComputing/JuliaHub/issues/22754