diff --git a/README.md b/README.md index 7cee670..44c848d 100644 --- a/README.md +++ b/README.md @@ -118,9 +118,16 @@ that is not one task (`go tool taskgate`). - **The look** is spin-stack/brand's: its palette, checked there for contrast and perceptual distance on real panels and through colour-vision deficiencies, and the organization's mark. `task reference` copies them from brand at its pin (`BRAND=` to point elsewhere) into - `src/styles/brand-*.css`, `src/assets/` and `public/`, with the app's screenshots, and - `task reference:check` fails on a copy that is not brand's, on screenshots of another release - than installations run, and on one whose test spin no longer has; `src/styles/custom.css` says only how the site's parts use them. + `src/styles/brand-*.css`, `src/assets/` and `public/`, and `task reference:check` fails on a + copy that is not brand's; `src/styles/custom.css` says only how the site's parts use them. +- **The dashboard's screenshots** are the documented spin release's own: spin's `site-shots.yml` + takes them on the KVM fleet (`dashboard/e2e/site.spec.ts`) and puts them on the release as + `site-shots.tar.gz`, which `task sources` unpacks into `.sources/spin-site-shots` with the + sources token. `task reference` copies the one of each test that shoots into + `src/assets/shots` and records the release, the commit and the test in `src/data/shots.json`; + `task reference:check` fails on a picture that is not the release's, and on a record of + another release than the site documents. A release without the asset fails `task sources`, + saying to give it one: `gh workflow run site-shots.yml -R spin-stack/spin -f tag=`. IBM Plex is served from the site, and a hero draws a workspace's disk as its layers, worked step by step (`src/components/Hero.astro`, `Disk.astro`). `task links` fails a page that loads a script, a stylesheet, a font or an image from another site. diff --git a/Taskfile.yml b/Taskfile.yml index 6e52c5f..929df55 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -4,12 +4,14 @@ version: '3' # reference pages from the sources they describe, linkcheck fails on a link the build did not # produce. SPIN, TERRAFORM and BRAND are checkouts of spin-stack/spin, spin-stack/spin-terraform-aws # and spin-stack/brand: by default the ones task sources fetches into .sources/, at the commits -# versions.yaml pins. +# versions.yaml pins. SHOTS is where it unpacks the dashboard's screenshots, which spin's +# release carries as site-shots.tar.gz. vars: SPIN: '{{.SPIN | default ".sources/spin"}}' TERRAFORM: '{{.TERRAFORM | default ".sources/spin-terraform-aws"}}' BRAND: '{{.BRAND | default ".sources/brand"}}' + SHOTS: '{{.SHOTS | default ".sources/spin-site-shots"}}' BASE_PATH: '{{.BASE_PATH | default "/website"}}' tasks: @@ -24,7 +26,7 @@ tasks: - pnpm dev sources: - desc: Fetch spin, spin-terraform-aws and brand into .sources/, each at the commit versions.yaml pins. spin is private - SOURCES_TOKEN, or git's own credentials, reads it. + desc: Fetch spin, spin-terraform-aws and brand into .sources/, each at the commit versions.yaml pins, and spin's release's screenshots (site-shots.tar.gz) beside them. spin is private - SOURCES_TOKEN, or git's own credentials, reads it. cmds: - go run ./tools/sources spin spin-terraform-aws brand @@ -47,13 +49,13 @@ tasks: desc: Write the reference pages (CLI, Terraform, config file, API) and what the site copies from the spin, spin-terraform-aws and brand checkouts. cmds: - task: sources - - go run ./tools/refgen -spin {{.SPIN}} -terraform {{.TERRAFORM}} -brand {{.BRAND}} + - go run ./tools/refgen -spin {{.SPIN}} -terraform {{.TERRAFORM}} -brand {{.BRAND}} -shots {{.SHOTS}} reference:check: desc: Fail if a reference page is not what the sources say. cmds: - task: sources - - go run ./tools/refgen -spin {{.SPIN}} -terraform {{.TERRAFORM}} -brand {{.BRAND}} -check + - go run ./tools/refgen -spin {{.SPIN}} -terraform {{.TERRAFORM}} -brand {{.BRAND}} -shots {{.SHOTS}} -check reference:public: desc: Fail if what the site has of spin-terraform-aws and brand is not what they say at the commits versions.yaml pins. Needs no spin checkout, nor any credential. diff --git a/src/assets/shots/workspace.png b/src/assets/shots/workspace.png index eb74d2b..a3f80da 100644 Binary files a/src/assets/shots/workspace.png and b/src/assets/shots/workspace.png differ diff --git a/src/assets/shots/workspaces.png b/src/assets/shots/workspaces.png index cd77f43..1206930 100644 Binary files a/src/assets/shots/workspaces.png and b/src/assets/shots/workspaces.png differ diff --git a/src/content/docs/use/dashboard.md b/src/content/docs/use/dashboard.md index a5bfdcf..bb25b14 100644 --- a/src/content/docs/use/dashboard.md +++ b/src/content/docs/use/dashboard.md @@ -7,7 +7,7 @@ editUrl: false -The dashboard is at `https://app.`. These screenshots are of spin v20261019.02, taken by spin's dashboard suite on its end-to-end stack - real workspaces on a real runner - each by a test that first checks the screen shows what its title says. Everything here is also in the [CLI](/reference/cli/) and the [API](/reference/api/). +The dashboard is at `https://app.`. These screenshots are of spin v20261024.01, taken by spin's dashboard suite on its end-to-end stack - real workspaces on a real runner - each by a test that first checks the screen shows what its title says. Everything here is also in the [CLI](/reference/cli/) and the [API](/reference/api/). ## A person's workspaces diff --git a/src/data/shots.json b/src/data/shots.json index e36216d..f2be707 100644 --- a/src/data/shots.json +++ b/src/data/shots.json @@ -1,6 +1,6 @@ { - "release": "v20261019.02", - "commit": "617175ffdf2bb568cfcacf97874e2287d0ad8a5b", + "release": "v20261024.01", + "commit": "b3598142dbcd6ef24135066332441a400dd35980", "shots": [ { "name": "workspaces", diff --git a/tools/internal/github/github.go b/tools/internal/github/github.go index a6b32e4..e4dc2c0 100644 --- a/tools/internal/github/github.go +++ b/tools/internal/github/github.go @@ -7,9 +7,11 @@ import ( "context" "encoding/base64" "encoding/json" + "errors" "fmt" "io" "net/http" + "net/url" "strings" "time" ) @@ -52,6 +54,30 @@ func (c *Client) File(ctx context.Context, repo, path, ref string) ([]byte, erro return c.send(ctx, http.MethodGet, "/repos/"+repo+"/contents/"+path+"?ref="+ref, nil, "application/vnd.github.raw") } +// ErrNoAsset is a release that has no asset of the name asked for. +var ErrNoAsset = errors.New("no such asset") + +// ReleaseAsset is the asset called name of repo's release tag, as its bytes, read through the +// API as the token can read a private repository's. The API redirects to storage on another +// host, and Go's client sends the Authorization header no further than GitHub. +func (c *Client) ReleaseAsset(ctx context.Context, repo, tag, name string) ([]byte, error) { + var rel struct { + Assets []struct { + ID int64 `json:"id"` + Name string `json:"name"` + } `json:"assets"` + } + if err := c.Get(ctx, "/repos/"+repo+"/releases/tags/"+url.PathEscape(tag), &rel); err != nil { + return nil, err + } + for _, a := range rel.Assets { + if a.Name == name { + return c.send(ctx, http.MethodGet, fmt.Sprintf("/repos/%s/releases/assets/%d", repo, a.ID), nil, "application/octet-stream") + } + } + return nil, fmt.Errorf("%s %s has no %s: %w", repo, tag, name, ErrNoAsset) +} + func (c *Client) send(ctx context.Context, method, path string, body any, accept string) ([]byte, error) { var in io.Reader if body != nil { diff --git a/tools/refgen/brand.go b/tools/refgen/brand.go index 7728001..b30337e 100644 --- a/tools/refgen/brand.go +++ b/tools/refgen/brand.go @@ -17,7 +17,6 @@ var brandFiles = []struct{ from, to string }{ {"logos/mark-512.png", "public/mark.png"}, {"logos/favicon-64.png", "public/favicon.png"}, {"logos/apple-touch-icon.png", "public/apple-touch-icon.png"}, - {"shots/shots.json", "src/data/shots.json"}, } func brandCopies(brand string, check bool) ([]string, error) { @@ -35,38 +34,5 @@ func brandCopies(brand string, check bool) ([]string, error) { stale = append(stale, f.to) } } - // The app's screenshots, every one brand keeps, and none it does not. - pngs, err := filepath.Glob(filepath.Join(brand, "shots", "*.png")) - if err != nil { - return nil, err - } - keep := map[string]bool{} - for _, p := range pngs { - raw, err := os.ReadFile(p) - if err != nil { - return nil, err - } - to := filepath.Join(shotsDir, filepath.Base(p)) - keep[to] = true - differs, err := syncFile(to, raw, check) - if err != nil { - return nil, err - } - if differs { - stale = append(stale, to) - } - } - have, _ := filepath.Glob(filepath.Join(shotsDir, "*.png")) - for _, h := range have { - if keep[h] { - continue - } - stale = append(stale, h) - if !check { - if err := os.Remove(h); err != nil { - return nil, err - } - } - } return stale, nil } diff --git a/tools/refgen/main.go b/tools/refgen/main.go index 2e825fb..3472f82 100644 --- a/tools/refgen/main.go +++ b/tools/refgen/main.go @@ -29,6 +29,7 @@ func run() error { spin := flag.String("spin", "../spin", "a checkout of spin-stack/spin") tf := flag.String("terraform", "../spin-terraform-aws", "a checkout of spin-stack/spin-terraform-aws") brand := flag.String("brand", "../brand", "a checkout of spin-stack/brand") + shotsFrom := flag.String("shots", ".sources/spin-site-shots", "where task sources unpacked spin's release's site-shots.tar.gz") only := flag.String("only", "spin,terraform,brand", "the sources to generate from: all three, or terraform,brand, which needs no spin checkout and keeps what the site has of spin") out := flag.String("out", "src/content/docs/reference", "where the pages go") docs := flag.String("docs", "src/content/docs", "the hand-written pages, whose commands are checked") @@ -65,6 +66,11 @@ func run() error { } r, stale, bad = s.ref, s.stale, s.bad from["spin"] = checkout{*spin, r.Spin.Commit} + shot, err := releaseShots(*spin, *shotsFrom, r.Spin, *check) + if err != nil { + return err + } + stale = append(stale, shot...) dirs = append(dirs, "status", "cli", "configuration", "api", "security") if err := spinConstants(&c, *spin); err != nil { return err @@ -146,7 +152,7 @@ func run() error { return err } stale = append(stale, branded...) - shotBad, shotStale, err := checkShots(*spin, r.Spin.Installs, *check) + shotBad, shotStale, err := checkShots(r.Spin, *check) if err != nil { return err } diff --git a/tools/refgen/release.go b/tools/refgen/release.go index 036381a..ff147b2 100644 --- a/tools/refgen/release.go +++ b/tools/refgen/release.go @@ -220,6 +220,11 @@ var pinned = regexp.MustCompile("\\?ref=([A-Za-z0-9._/-]*)(<[^>\\s`\"]*>?)?|\\bv func pinReleases(docs, release, module string, check bool) ([]string, error) { var bad []string err := handPages(docs, func(path string, raw []byte) error { + // The dashboard's page names the release its screenshots are of: the one the reference + // documents, which installations may not run yet. + if rel, _ := filepath.Rel(docs, path); rel == shotsPageDoc { + return nil + } lines := bytes.Split(raw, []byte("\n")) for i, line := range lines { lines[i] = pinned.ReplaceAllFunc(line, func(m []byte) []byte { diff --git a/tools/refgen/shots.go b/tools/refgen/shots.go index 4aa1d1e..c06bd28 100644 --- a/tools/refgen/shots.go +++ b/tools/refgen/shots.go @@ -11,15 +11,20 @@ import ( // The dashboard's screenshots are taken by spin's own dashboard suite, of its e2e stack - the // user's real workspaces on a real runner - by dashboard/e2e/site.spec.ts, each test checking -// the screen shows what its title says before it shoots, and kept in spin-stack/brand with the -// release of the tree that took them, beside the palette that draws them. refgen copies them -// from brand (brand.go), writes the dashboard's page from them, and fails when they are of -// another release than the site documents, or a test that took one is gone from spin. +// the screen shows what its title says before it shoots. spin's site-shots.yml takes them of +// every release on the KVM fleet and puts them on it as site-shots.tar.gz, which task sources +// unpacks beside the spin checkout. refgen copies them from there with the release, the commit +// and each test's title (releaseShots), writes the dashboard's page from them, and fails when +// they are of another release than the site documents (checkShots). So a release's pictures +// arrive with the pin that moves to it, and nobody runs a stack to take them. const ( shotsSpec = "dashboard/e2e/site.spec.ts" shotsDir = "src/assets/shots" shotsData = "src/data/shots.json" - shotsPage = "src/content/docs/use/dashboard.md" + // shotsPageDoc is the dashboard's page under the docs, written by refgen: it names the + // release its screenshots are of, which pinReleases leaves as it is. + shotsPageDoc = "use/dashboard.md" + shotsPage = "src/content/docs/" + shotsPageDoc ) type shots struct { @@ -51,10 +56,81 @@ func specShots(spin string) ([]shot, error) { return out, nil } -// checkShots holds the screenshots to the release an installation runs and to the spec, and -// writes the dashboard's page from them. With no spin checkout (spin is ""), the spec is not -// read: the release and the pictures are brand's, and checked all the same. -func checkShots(spin, installs string, check bool) (bad, stale []string, err error) { +// shotsStamp is the file task sources writes in the screenshots' directory once they are all +// there: " \n" of the release they were unpacked from. +const shotsStamp = ".release" + +// releaseShots copies, from the directory task sources unpacked spin's release's +// site-shots.tar.gz into, the screenshot of every test of the spec that shoots one, and no +// other, and records them in shots.json with the release and commit spin is at. It fails when +// the directory is of another release, or spin is at none: a screenshot is the documented +// release's or the site has none. +func releaseShots(spin, from string, at spinRef, check bool) (stale []string, err error) { + if at.Tag == "" { + return nil, fmt.Errorf("spin %s is no release, and the dashboard's screenshots are a release's: pin spin at one", at.Commit) + } + raw, err := os.ReadFile(filepath.Join(from, shotsStamp)) + if err != nil { + return nil, fmt.Errorf("%s has no screenshots of a release (%w): run task sources", from, err) + } + if got, want := string(raw), at.Tag+" "+at.Commit+"\n"; got != want { + return nil, fmt.Errorf("%s holds the screenshots of %q, and spin is at %q: run task sources", from, strings.TrimSpace(got), strings.TrimSpace(want)) + } + inSpec, err := specShots(spin) + if err != nil { + return nil, err + } + keep := map[string]bool{} + for _, t := range inSpec { + png, err := os.ReadFile(filepath.Join(from, t.Name+".png")) + if err != nil { + return nil, fmt.Errorf("spin %s's site-shots.tar.gz has no %s.png, which %q shoots: retake them with `gh workflow run site-shots.yml -R spin-stack/spin -f tag=%s`", at.Tag, t.Name, t.Test, at.Tag) + } + to := filepath.Join(shotsDir, t.Name+".png") + keep[to] = true + differs, err := syncFile(to, png, check) + if err != nil { + return nil, err + } + if differs { + stale = append(stale, to) + } + } + have, err := filepath.Glob(filepath.Join(shotsDir, "*.png")) + if err != nil { + return nil, err + } + for _, h := range have { + if keep[h] { + continue + } + stale = append(stale, h) + if !check { + if err := os.Remove(h); err != nil { + return nil, err + } + } + } + b, err := json.MarshalIndent(shots{Release: at.Tag, Commit: at.Commit, Shots: inSpec}, "", "\t") + if err != nil { + return nil, err + } + differs, err := syncFile(shotsData, append(b, '\n'), check) + if err != nil { + return nil, err + } + if differs { + stale = append(stale, shotsData) + } + return stale, nil +} + +// checkShots holds the screenshots shots.json records to the spin release and commit the +// reference documents, and writes the dashboard's page from them. With a spin checkout, +// releaseShots has just written them from the release; without one, what holds the pictures +// to the release is that run's, in Gate with the token, and this holds the record to the +// reference all the same. +func checkShots(at spinRef, check bool) (bad, stale []string, err error) { raw, err := os.ReadFile(shotsData) if err != nil { return nil, nil, err @@ -63,24 +139,10 @@ func checkShots(spin, installs string, check bool) (bad, stale []string, err err if err := json.Unmarshal(raw, &s); err != nil { return nil, nil, fmt.Errorf("%s: %w", shotsData, err) } - if s.Release != installs { - bad = append(bad, fmt.Sprintf("%s: the dashboard's screenshots are of %s and installations run %s: run `task screenshots`", shotsData, s.Release, installs)) - } - var tests map[string]string - if spin != "" { - inSpec, err := specShots(spin) - if err != nil { - return nil, nil, err - } - tests = map[string]string{} - for _, t := range inSpec { - tests[t.Name] = t.Test - } + if at.Tag == "" || s.Release != at.Tag || s.Commit != at.Commit { + bad = append(bad, fmt.Sprintf("%s: the dashboard's screenshots are of spin %s at %s, and the site documents spin %q at %s: run `task reference`", shotsData, s.Release, s.Commit, at.Tag, at.Commit)) } for _, t := range s.Shots { - if tests != nil && tests[t.Name] != t.Test { - bad = append(bad, fmt.Sprintf("%s: %q is no test of spin's %s that shoots %s: run `task screenshots`", shotsData, t.Test, shotsSpec, t.Name)) - } if _, err := os.Stat(filepath.Join(shotsDir, t.Name+".png")); err != nil { bad = append(bad, fmt.Sprintf("%s: no %s.png", shotsDir, t.Name)) } diff --git a/tools/refgen/shots_test.go b/tools/refgen/shots_test.go new file mode 100644 index 0000000..c172315 --- /dev/null +++ b/tools/refgen/shots_test.go @@ -0,0 +1,98 @@ +package main + +import ( + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" +) + +const spec = `test("a person's workspaces", async ({ page }) => { + await shoot(page, "workspaces"); +}); +test("a running workspace", async ({ page }) => { + await shoot(page, "workspace"); +}); +` + +// site is a working directory with a spin checkout holding the spec, and the screenshots task +// sources unpacked from the release at, with a picture the site has from before. +func site(t *testing.T, at spinRef, stamped string, pngs ...string) (spin, from string) { + t.Helper() + root := t.TempDir() + t.Chdir(root) + spin, from = filepath.Join(root, "spin"), filepath.Join(root, "shots") + files := map[string]string{ + filepath.Join(spin, shotsSpec): spec, + filepath.Join(from, shotsStamp): stamped, + filepath.Join(shotsDir, "dashboard.png"): "a screenshot of an older release", + shotsData: "{}", + } + for _, p := range pngs { + files[filepath.Join(from, p+".png")] = p + " of " + at.Tag + } + for p, body := range files { + if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(p, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + } + return spin, from +} + +// The site's screenshots are the release's, one for each test of the spec that shoots, recorded +// with the release and commit spin is at; and -check then finds them the release's. +func TestTheScreenshotsAreTheDocumentedReleases(t *testing.T) { + at := spinRef{Tag: "v20261024.01", Commit: strings.Repeat("b3", 20)} + spin, from := site(t, at, at.Tag+" "+at.Commit+"\n", "workspaces", "workspace", "extra") + if _, err := releaseShots(spin, from, at, false); err != nil { + t.Fatal(err) + } + have, _ := filepath.Glob(filepath.Join(shotsDir, "*.png")) + if want := []string{filepath.Join(shotsDir, "workspace.png"), filepath.Join(shotsDir, "workspaces.png")}; strings.Join(have, " ") != strings.Join(want, " ") { + t.Errorf("the site has %q; want the spec's two, %q", have, want) + } + if got, _ := os.ReadFile(filepath.Join(shotsDir, "workspace.png")); string(got) != "workspace of v20261024.01" { + t.Errorf("workspace.png is %q; want the release's", got) + } + raw, err := os.ReadFile(shotsData) + if err != nil { + t.Fatal(err) + } + var s shots + if err := json.Unmarshal(raw, &s); err != nil { + t.Fatal(err) + } + if s.Release != at.Tag || s.Commit != at.Commit || len(s.Shots) != 2 || s.Shots[1] != (shot{Name: "workspace", Test: "a running workspace"}) { + t.Errorf("shots.json = %+v; want the release, its commit and the spec's tests", s) + } + if stale, err := releaseShots(spin, from, at, true); err != nil || len(stale) > 0 { + t.Errorf("-check after writing = %q, %v; want nothing stale", stale, err) + } + if bad, _, err := checkShots(at, false); err != nil || len(bad) > 0 { + t.Errorf("checkShots = %q, %v; want nothing", bad, err) + } +} + +// Screenshots of another release than the site documents fail: unpacked from another, recorded +// of another, or taken of a spin that is no release. +func TestScreenshotsOfAnotherReleaseFail(t *testing.T) { + at := spinRef{Tag: "v20261024.01", Commit: strings.Repeat("b3", 20)} + spin, from := site(t, at, "v20261019.02 "+strings.Repeat("61", 20)+"\n", "workspaces", "workspace") + if _, err := releaseShots(spin, from, at, true); err == nil || !strings.Contains(err.Error(), "run task sources") { + t.Errorf("releaseShots of another release's directory = %v; want it refused", err) + } + if _, err := releaseShots(spin, from, spinRef{Commit: at.Commit}, true); err == nil || !strings.Contains(err.Error(), "no release") { + t.Errorf("releaseShots of a spin at no release = %v; want it refused", err) + } + old := `{"release": "v20261019.02", "commit": "` + strings.Repeat("61", 20) + `", "shots": []}` + if err := os.WriteFile(shotsData, []byte(old), 0o644); err != nil { + t.Fatal(err) + } + if bad, _, err := checkShots(at, true); err != nil || len(bad) != 1 || !strings.Contains(bad[0], "are of spin v20261019.02") { + t.Errorf("checkShots of another release's record = %q, %v; want it named", bad, err) + } +} diff --git a/tools/refgen/terraform_test.go b/tools/refgen/terraform_test.go index 6097276..074f62e 100644 --- a/tools/refgen/terraform_test.go +++ b/tools/refgen/terraform_test.go @@ -184,7 +184,9 @@ func TestPagesPinTheModuleAndTheRelease(t *testing.T) { "spin_version = \"v20261019.02\"\n" + "pin one (`?ref=v.`)\n" + "`//modules/terminal?ref=v20261001.01` and v20261001.01\n" - docs := writePage(t, map[string]string{"aws/overview.mdx": page}) + // The dashboard's page names its screenshots' release, the documented one. + shotsLine := "These screenshots are of spin v20261024.01.\n" + docs := writePage(t, map[string]string{"aws/overview.mdx": page, shotsPageDoc: shotsLine}) bad, err := pinReleases(docs, "v20261019.02", "v20261024.01", true) if err != nil { t.Fatal(err) @@ -204,4 +206,7 @@ func TestPagesPinTheModuleAndTheRelease(t *testing.T) { if string(raw) != want { t.Errorf("rewritten:\n%s\nwant:\n%s", raw, want) } + if raw, _ := os.ReadFile(filepath.Join(docs, shotsPageDoc)); string(raw) != shotsLine { + t.Errorf("the dashboard's page was rewritten to %q", raw) + } } diff --git a/tools/sources/assets.go b/tools/sources/assets.go new file mode 100644 index 0000000..8b7af9b --- /dev/null +++ b/tools/sources/assets.go @@ -0,0 +1,135 @@ +package main + +import ( + "archive/tar" + "bytes" + "compress/gzip" + "context" + "errors" + "fmt" + "io" + "os" + "path/filepath" + "regexp" + "slices" + "strings" + + "github.com/spin-stack/go-tools/versions" + + "github.com/spin-stack/website/tools/internal/github" +) + +// asset is a file a source's release carries that the site takes as it is, fetched beside the +// source's checkout. +type asset struct { + // name is the release asset, a .tar.gz of files with no directories. + name string + // dir is where it is unpacked, under the directory sources are fetched into. + dir string + // retake is the command that gives a release the asset, its tag as %s. + retake string +} + +// assets are the releases' files the site takes. spin's site-shots.yml takes the dashboard's +// screenshots of each release on the KVM fleet and puts them on it: refgen copies them from +// here, so the pictures are of the release the site documents, with nobody running a stack. +var assets = map[string]asset{ + "spin": { + name: "site-shots.tar.gz", + dir: "spin-site-shots", + retake: "gh workflow run site-shots.yml -R spin-stack/spin -f tag=%s", + }, +} + +// stamp is the file, in an asset's directory, that says which release it was unpacked from, as +// " \n". It is written last, so a directory without it is one a run did not finish; +// refgen reads it to hold the copy to the release it documents. +const stamp = ".release" + +// shotName is what an unpacked file may be called: a PNG named as site.spec.ts's shoot() names +// one, and nothing a path could climb out of the directory with. +var shotName = regexp.MustCompile(`^[a-z0-9][a-z0-9-]*\.png$`) + +// fetchAsset unpacks a's asset of e's release into dir. A directory already stamped with e's +// release and pin is left as it is, so a run after the one with the token needs none. A release +// without the asset fails, saying how to give it one: the site never keeps another release's +// screenshots under this one's name. +func fetchAsset(ctx context.Context, gh *github.Client, dir string, e versions.Entry, a asset) error { + want := e.Version + " " + e.Pin + "\n" + if have, err := os.ReadFile(filepath.Join(dir, stamp)); err == nil && string(have) == want { + return nil + } + if !release.MatchString(e.Version) { + return fmt.Errorf("%s is pinned at %s, which is no release, and %s is a release's: pin a release", e.Name, e.Version, a.name) + } + repo := strings.TrimPrefix(e.Source, "https://github.com/") + raw, err := gh.ReleaseAsset(ctx, repo, e.Version, a.name) + if errors.Is(err, github.ErrNoAsset) { + return fmt.Errorf("%w: give it one with `%s` and fetch again "+ + "(site-shots.yml puts it on every release once Release has finished, about an hour after it)", + err, fmt.Sprintf(a.retake, e.Version)) + } + if err != nil { + return err + } + files, err := untar(raw) + if err != nil { + return fmt.Errorf("%s %s's %s: %w", repo, e.Version, a.name, err) + } + if err := os.RemoveAll(dir); err != nil { + return err + } + if err := os.MkdirAll(dir, 0o755); err != nil { + return err + } + names := make([]string, 0, len(files)) + for name, body := range files { + if err := os.WriteFile(filepath.Join(dir, name), body, 0o644); err != nil { + return err + } + names = append(names, name) + } + slices.Sort(names) + fmt.Printf("%s: %s of %s: %s\n", e.Name, a.name, e.Version, strings.Join(names, ", ")) + return os.WriteFile(filepath.Join(dir, stamp), []byte(want), 0o644) +} + +// untar is the files of a .tar.gz that holds only PNGs, flat. Anything else in it - a directory, +// a link, another name - is refused rather than skipped: the archive is not this repository's, +// and one that is not what site-shots.yml writes is one to look at. +func untar(raw []byte) (map[string][]byte, error) { + gz, err := gzip.NewReader(bytes.NewReader(raw)) + if err != nil { + return nil, err + } + tr := tar.NewReader(gz) + files := map[string][]byte{} + for { + h, err := tr.Next() + if errors.Is(err, io.EOF) { + break + } + if err != nil { + return nil, err + } + name := strings.TrimPrefix(h.Name, "./") + if h.Typeflag == tar.TypeDir && name == "" { + continue + } + if h.Typeflag != tar.TypeReg || !shotName.MatchString(name) { + return nil, fmt.Errorf("%q is not a PNG at the top of the archive", h.Name) + } + if h.Size > 16<<20 { + return nil, fmt.Errorf("%s is %d bytes, more than a screenshot", name, h.Size) + } + body, err := io.ReadAll(tr) + if err != nil { + return nil, err + } + files[name] = body + } + if len(files) == 0 { + return nil, errors.New("it holds no screenshot") + } + return files, nil +} diff --git a/tools/sources/assets_test.go b/tools/sources/assets_test.go new file mode 100644 index 0000000..d91d2ca --- /dev/null +++ b/tools/sources/assets_test.go @@ -0,0 +1,131 @@ +package main + +import ( + "archive/tar" + "bytes" + "compress/gzip" + "context" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/spin-stack/go-tools/versions" + + "github.com/spin-stack/website/tools/internal/github" +) + +func tarGz(t *testing.T, files map[string]string) []byte { + t.Helper() + var b bytes.Buffer + gz := gzip.NewWriter(&b) + tw := tar.NewWriter(gz) + for name, body := range files { + if err := tw.WriteHeader(&tar.Header{Name: name, Mode: 0o644, Size: int64(len(body)), Typeflag: tar.TypeReg}); err != nil { + t.Fatal(err) + } + if _, err := tw.Write([]byte(body)); err != nil { + t.Fatal(err) + } + } + if err := tw.Close(); err != nil { + t.Fatal(err) + } + if err := gz.Close(); err != nil { + t.Fatal(err) + } + return b.Bytes() +} + +// releases serves spin's releases as GitHub's API does, to the token alone: each tag's asset +// list, and an asset's bytes from its id, when asked for them as octet-stream. +func releases(t *testing.T, assets map[string][]byte) *github.Client { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("Authorization") != "Bearer ghs_x" { + http.NotFound(w, r) + return + } + if tag, ok := strings.CutPrefix(r.URL.Path, "/repos/spin-stack/spin/releases/tags/"); ok { + if body, ok := assets[tag]; ok && body == nil { + _, _ = w.Write([]byte(`{"assets":[{"id":1,"name":"checksums.txt"}]}`)) + return + } + if _, ok := assets[tag]; ok { + _, _ = w.Write([]byte(`{"assets":[{"id":1,"name":"checksums.txt"},{"id":7,"name":"site-shots.tar.gz"}]}`)) + return + } + } + if r.URL.Path == "/repos/spin-stack/spin/releases/assets/7" && r.Header.Get("Accept") == "application/octet-stream" { + for _, body := range assets { + if body != nil { + _, _ = w.Write(body) + return + } + } + } + http.NotFound(w, r) + })) + t.Cleanup(srv.Close) + return &github.Client{Base: srv.URL, Token: "ghs_x", HTTP: srv.Client()} +} + +func spinAt(version string) versions.Entry { + return versions.Entry{Name: "spin", Source: "https://github.com/spin-stack/spin", Version: version, Pin: strings.Repeat("b3", 20)} +} + +// The release's screenshots are unpacked as the archive has them and stamped with the release +// and pin, and a later run - Sync's task reference, with no token - asks nothing again. +func TestTheReleasesScreenshotsAreUnpackedAndStamped(t *testing.T) { + ctx := context.Background() + gh := releases(t, map[string][]byte{"v20261024.01": tarGz(t, map[string]string{"workspace.png": "one", "./workspaces.png": "two"})}) + dir := filepath.Join(t.TempDir(), "spin-site-shots") + e := spinAt("v20261024.01") + if err := fetchAsset(ctx, gh, dir, e, assets["spin"]); err != nil { + t.Fatal(err) + } + for name, want := range map[string]string{"workspace.png": "one", "workspaces.png": "two", stamp: "v20261024.01 " + e.Pin + "\n"} { + if got, err := os.ReadFile(filepath.Join(dir, name)); err != nil || string(got) != want { + t.Errorf("%s = %q, %v; want %q", name, got, err, want) + } + } + if err := fetchAsset(ctx, &github.Client{Base: "http://127.0.0.1:1"}, dir, e, assets["spin"]); err != nil { + t.Errorf("a directory stamped with the pin was fetched again: %v", err) + } +} + +// What is not the documented release's screenshots fails, and a release that lacks them says +// how to give it them: nothing falls back to the pictures already there. +func TestScreenshotsThatAreNotTheReleasesFail(t *testing.T) { + ctx := context.Background() + for _, c := range []struct { + name string + version string + archive []byte + want string + }{ + {"a release without the asset", "v20261019.02", nil, "gh workflow run site-shots.yml -R spin-stack/spin -f tag=v20261019.02"}, + {"a pin that is no release", "main", nil, "which is no release"}, + {"an archive with a directory in it", "v20261024.01", tarGz(t, map[string]string{"shots/workspace.png": "x"}), "not a PNG at the top"}, + {"an archive that climbs out", "v20261024.01", tarGz(t, map[string]string{"../workspace.png": "x"}), "not a PNG at the top"}, + {"an empty archive", "v20261024.01", tarGz(t, nil), "no screenshot"}, + } { + dir := filepath.Join(t.TempDir(), "spin-site-shots") + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, "workspace.png"), []byte("old"), 0o644); err != nil { + t.Fatal(err) + } + gh := releases(t, map[string][]byte{c.version: c.archive}) + err := fetchAsset(ctx, gh, dir, spinAt(c.version), assets["spin"]) + if err == nil || !strings.Contains(err.Error(), c.want) { + t.Errorf("%s: fetchAsset = %v; want an error saying %q", c.name, err, c.want) + } + if _, err := os.Stat(filepath.Join(dir, stamp)); err == nil { + t.Errorf("%s: the directory is stamped as the release's", c.name) + } + } +} diff --git a/tools/sources/main.go b/tools/sources/main.go index 7b020bd..8aee5c1 100644 --- a/tools/sources/main.go +++ b/tools/sources/main.go @@ -6,7 +6,9 @@ // // A private repository is fetched with SOURCES_TOKEN, which reaches git only as GIT_CONFIG_* in // the environment of the git commands this runs: nothing is written to a checkout's config, so -// the steps after this one, which build and run what was fetched, never see it. +// the steps after this one, which build and run what was fetched, never see it. What a release +// carries that the site takes as it is - spin's screenshots - is fetched beside its checkout, +// with the same token, through GitHub's API. package main import ( @@ -77,6 +79,11 @@ func run(ctx context.Context) error { if err := fetch(ctx, filepath.Join(*dir, n), e, github.GitAuth(token)); err != nil { return fmt.Errorf("%s: %w", n, err) } + if a, ok := assets[n]; ok { + if err := fetchAsset(ctx, github.New(token), filepath.Join(*dir, a.dir), e, a); err != nil { + return fmt.Errorf("%s: %w", n, err) + } + } } return nil } diff --git a/tools/sources/repo_test.go b/tools/sources/repo_test.go index 1b61cfe..90bb164 100644 --- a/tools/sources/repo_test.go +++ b/tools/sources/repo_test.go @@ -18,7 +18,7 @@ func TestThisRepositoryPassesTheGate(t *testing.T) { "go.sum", // What refgen writes: reference.json names the commits, and the rest link spin's // tree at its commit (the status page's tests, the security pages, the diagrams) or - // copy what spin and brand recorded (the CLI's goldens' digests, the screenshots'). + // copy what spin recorded (the CLI's goldens' digests, its release's screenshots'). "src/data/reference.json", "src/data/figures.json", "src/data/shots.json", diff --git a/versions.yaml b/versions.yaml index 2e27376..39dcf2d 100644 --- a/versions.yaml +++ b/versions.yaml @@ -32,5 +32,5 @@ pin: c882b6f0f3901787525ef16ef29058c9c1418cbe track: branch note: >- - The palette, the logos and the screenshots. brand cuts no releases, so its main is what it - has to say; the screenshots are compared at the release they were taken of. + The palette and the logos. brand cuts no releases, so its main is what it has to say. The + dashboard's screenshots are not brand's: they are spin's release's, site-shots.tar.gz.