From 7b67cdb156a8085b4440db347ff83489109b7876 Mon Sep 17 00:00:00 2001 From: chris lee Date: Fri, 9 Oct 2026 12:43:37 -0400 Subject: [PATCH 1/7] Add stopped-machine build lifecycle and publication foundation --- docs/macos-builds.md | 95 +++++++++++ lib/builds/machine.go | 314 ++++++++++++++++++++++++++++++++++++ lib/builds/machine_test.go | 276 +++++++++++++++++++++++++++++++ lib/images/macos_machine.go | 13 ++ 4 files changed, 698 insertions(+) create mode 100644 docs/macos-builds.md create mode 100644 lib/builds/machine.go create mode 100644 lib/builds/machine_test.go diff --git a/docs/macos-builds.md b/docs/macos-builds.md new file mode 100644 index 000000000..b2a4913dd --- /dev/null +++ b/docs/macos-builds.md @@ -0,0 +1,95 @@ +# macOS image builds (PR5, foundation only) + +Goal: ordinary build jobs provision a digest-pinned installed macOS base inside +an isolated VM, then publish a complete cold-boot OCI machine image. This is not +Linux rootfs conversion, macOS installation/bootstrap, memory forking or safe +identity rekeying. + +This checkpoint adds an **internal lifecycle/publication contract** and tests. +It does not register an HTTP build mode, choose a recipe language, implement the +VZ/GuestService driver or stream an artifact into a production registry. Existing +Linux build behavior and macOS-only build rejection are unchanged. Do not claim +that normal macOS builds work from this foundation alone. + +## Dependency and recipe boundaries + +The branch begins on PR2's machine-image support. The real driver additionally +requires reviewed PR3's system GuestService/readiness/lifecycle changes; desktop +provisioning needs PR4 where applicable. Integrate those dependencies before API +activation. Do not replay stale guest-agent code or bypass identity exclusivity. + +No Macfile syntax or unrestricted Dockerfile/BuildKit compatibility is introduced. +Packer versus a restricted familiar provisioning recipe still needs a deliberate +choice before the public request/source schema is frozen. The internal driver +consumes the existing build job's staged source archive; it must run recipe +commands in the isolated guest, never in a host shell. Public source/log/secret/ +cancellation/provenance/output contracts should remain shared across OS backends. + +## Lifecycle contract + +`MachineBuildRunner` requires a ready installed `darwin/arm64` base pinned by a +canonical SHA256 reference. CPU/memory must match that base (zero inherits), fit +existing build limits, and preserve exact MiB precision. Negative/unbounded +resource or timeout values are rejected before VM start. Timeout defaults to 600 +seconds and is capped at 24 hours. Networking defaults to isolated; egress means +unrestricted VZ NAT, not domain/TAP/policy parity. Domain allowlists are rejected. + +Source input is staged in a private server-owned workspace, with bounded streaming +(default 64MiB compressed input), cancellation checks, SHA256 provenance and optional +expected-hash verification before start. Guest extraction must independently bound +expanded archive size and reject traversal/symlink escapes; this is not implemented +by the host compressed-size bound. + +The successful order is: + +1. Resolve pinned installed base and validate resource policy. +2. Stage/verify source; start one owned builder with a valid identity lease. +3. Provision and sanitize build credentials/source/secret artifacts. +4. Obtain a graceful-stop receipt **and actual VMM exit**. +5. Export matching disk/aux/platform into a separate build-owned directory. +6. Validate the complete bundle, preserve base identity/resources, reject extra + files and unknown platform fields, and make payload modes private. +7. Destroy the stopped builder and remove staged source before publishing. +8. Stream blobs and commit/verify the manifest last through a server-owned, + build-scoped publisher; return a matching immutable digest receipt. + +The driver must reject unprovisioned system agents; it cannot bootstrap via an +unreviewed host SSH/sudo fallback. Default lifecycle APIs that silently force-stop +are insufficient evidence of graceful export readiness. A forced stop or any +pre-publication failure must not export/publish a successful image. Sanitation is +a mandatory driver operation, not something structural bundle validation proves. +Base credentials and reusable-image SSH/account/browser/TCC policy remain a +separately reviewed requirement. + +Cancellation/failure cleanup gets its own bounded 30-second context. It may destroy +instance storage only after a receipt confirms VMM exit. Otherwise the instance +is quarantined for operator recovery, never deleted or published. Instance storage +must be outside the input/export workspace; that private workspace is removed by +the runner. Raw guest error text is masked in ordinary error/log strings while +`errors.Is/As` remain available internally. The concrete log stream must separately +redact injected secrets before retention/forwarding. + +Publication starts only after validation and fallible builder/source cleanup. +A failed network commit may have ambiguous remote effects; partial blobs are not +success, and no ready result is returned without a verified receipt. Atomic +manifest-last behavior and reconciliation belong to the concrete publisher; +these interfaces alone do not prove remote nonpublication after a lost response. + +## Validation and remaining gates + +Synthetic tests cover input/resource/hash/size admission before VM start, exact +phase order, partial-start cleanup, independent cancellation cleanup, forced/ +unconfirmed-stop nonpublication and no deletion of live storage, failed-stage +nonpublication, identity changes, extra/unknown secret metadata rejection, private +output modes, error-text masking and verified digest/source provenance. + +No actual VM is created and no guest commands or registry uploads are executed by +these tests. Fake tiny disk files establish contract behavior, not bootability. +The shared machine-bundle validator is the same structural validator used for OCI +imports, not a macOS/VZ integrity or credential scanner. + +Remaining: concrete driver and streamed publisher, common build queue/status/log/ +secret/API integration, pinned toolchain/version provenance, recipe choice and +source validation, storage floor/quota enforcement, sanitation audit, large-layer +upload transport validation, repeat builds and output cold boot through normal +APIs. Keep PR5 draft and runtime admission disabled until these gates pass. diff --git a/lib/builds/machine.go b/lib/builds/machine.go new file mode 100644 index 000000000..fc0f6f45d --- /dev/null +++ b/lib/builds/machine.go @@ -0,0 +1,314 @@ +package builds + +import ( + "bytes" + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "io" + "net" + "os" + "path/filepath" + "strings" + "time" + + "github.com/kernel/hypeman/lib/images" +) + +// MachineBuildRequest is an internal backend contract, not a new recipe syntax +// or an HTTP API. The normal build manager owns ID, source staging and publish +// authorization. A concrete driver must require a provisioned system agent. +type MachineBuildRequest struct { + ID string + BaseImage string // Installed darwin/arm64 image, pinned by sha256 digest. + SourceHash string // Optional expected lowercase SHA256 of the source archive. + Policy BuildPolicy // Zero CPU/memory inherit the base machine's exact resources. +} + +type MachineStopReceipt struct{ Graceful, VMMExited bool } +type MachinePublication struct{ Reference, Digest string } +type MachineBuildResult struct { + Publication MachinePublication + Provenance BuildProvenance + BuilderInstanceID string +} + +type MachineBuildDriver interface { + ResolveBase(context.Context, string) (*images.Image, error) + // Start owns an isolated instance and identity lease. A partially created + // session must be returned even with an error so cleanup can stop it. + Start(context.Context, *images.Image, BuildPolicy) (MachineBuildSession, error) +} + +type MachineBuildSession interface { + ID() string + Provision(context.Context, string) error // Build-owned staged source, never a client host path. + Sanitize(context.Context) error // Remove build credentials, source and secret injection artifacts. + // Stop must await actual VMM exit. Forced-stop fallback must not claim graceful. + Stop(context.Context) (MachineStopReceipt, error) + Export(context.Context, string) error // Stopped disk.img, aux.img, config.json only. + // Destroy removes instance storage only after proven VMM exit, not the export. + Destroy(context.Context) error +} + +type MachineBuildPublisher interface { + // Publisher owns registry address, scoped token and builds/{id} destination. + // It must stream blobs and commit/verify the manifest last. An ambiguous + // network error is not a verified successful publication. + Publish(context.Context, string, string) (MachinePublication, error) +} + +type MachineBuildRunner struct { + Driver MachineBuildDriver + Publisher MachineBuildPublisher + WorkDir string // Server-owned private build workspace, not supplied by recipes. + MaxSourceBytes int64 // Zero defaults to 64MiB of compressed input. +} + +// machinePhaseError keeps potentially secret-bearing guest/command errors out of +// ordinary log/error text, while retaining errors.Is/As for internal handling. +type machinePhaseError struct { + phase string + cause error +} + +func (e *machinePhaseError) Error() string { return "machine build " + e.phase + " failed" } +func (e *machinePhaseError) Unwrap() error { return e.cause } +func machineFailure(phase string, err error) error { return &machinePhaseError{phase, err} } + +func (r *MachineBuildRunner) Run(ctx context.Context, req MachineBuildRequest, source io.Reader) (result *MachineBuildResult, err error) { + if r.Driver == nil || r.Publisher == nil || !filepath.IsAbs(r.WorkDir) || source == nil { + return nil, fmt.Errorf("machine driver, publisher, private workspace and source required") + } + if len(req.ID) == 0 || len(req.ID) > 128 || strings.Trim(req.ID, "abcdefghijklmnopqrstuvwxyz0123456789-") != "" { + return nil, fmt.Errorf("invalid machine build ID") + } + ref, e := images.ParseNormalizedRef(req.BaseImage) + if e != nil || !ref.IsDigest() || !machineDigest(ref.Digest()) { + return nil, fmt.Errorf("machine base must be pinned by sha256 digest") + } + policy := req.Policy + if policy.TimeoutSeconds < 0 { + return nil, fmt.Errorf("invalid machine build timeout") + } + if policy.TimeoutSeconds == 0 { + policy.TimeoutSeconds = 600 + } + if policy.TimeoutSeconds > 86400 { + return nil, fmt.Errorf("machine build timeout exceeds 24 hours") + } + if len(policy.AllowedDomains) != 0 { + return nil, fmt.Errorf("macOS domain-restricted egress is unsupported") + } + if policy.NetworkMode == "" { + policy.NetworkMode = "isolated" + } + if policy.NetworkMode != "isolated" && policy.NetworkMode != "egress" { + return nil, fmt.Errorf("unsupported machine network mode") + } + ctx, cancel := context.WithTimeout(ctx, time.Duration(policy.TimeoutSeconds)*time.Second) + defer cancel() + base, e := r.Driver.ResolveBase(ctx, ref.String()) + if e != nil { + return nil, machineFailure("resolve_base", e) + } + if base == nil || base.Status != images.StatusReady || base.Platform != "darwin/arm64" || base.MacOS == nil || base.Digest != ref.Digest() || base.MacOS.CPUs < 2 || base.MacOS.CPUs > MaxBuildCPUs || base.MacOS.Memory < 4<<30 || base.MacOS.Memory > uint64(MaxBuildMemoryMB)<<20 || base.MacOS.Memory%(1<<20) != 0 { + return nil, fmt.Errorf("base is not the pinned ready macOS machine") + } + mac, macErr := net.ParseMAC(base.MacOS.MAC) + if len(base.MacOS.HardwareModel) == 0 || len(base.MacOS.MachineIdentifier) == 0 || macErr != nil || len(mac) != 6 { + return nil, fmt.Errorf("base has invalid machine identity") + } + memory := int(base.MacOS.Memory >> 20) + cpus := int(base.MacOS.CPUs) + if (policy.MemoryMB != 0 && policy.MemoryMB != memory) || (policy.CPUs != 0 && policy.CPUs != cpus) { + return nil, fmt.Errorf("machine resources must match installed base") + } + policy.MemoryMB = memory + policy.CPUs = cpus + if e = policy.Validate(); e != nil { + return nil, e + } + workspace, e := os.MkdirTemp(r.WorkDir, "machine-build-") + if e != nil { + return nil, machineFailure("workspace", e) + } + defer os.RemoveAll(workspace) + sourcePath := filepath.Join(workspace, "source.tar.gz") + limit := r.MaxSourceBytes + if limit == 0 { + limit = 64 << 20 + } + if limit < 1 || limit > 1<<30 { + return nil, fmt.Errorf("invalid machine source limit") + } + hash, e := stageMachineSource(ctx, source, sourcePath, limit) + if e != nil { + return nil, machineFailure("stage_source", e) + } + if req.SourceHash != "" && req.SourceHash != hash { + return nil, fmt.Errorf("machine source hash mismatch") + } + if e = ctx.Err(); e != nil { + return nil, e + } + session, e := r.Driver.Start(ctx, base, policy) + if session == nil { + if e == nil { + e = fmt.Errorf("driver returned no session") + } + return nil, machineFailure("start", e) + } + stopped, destroyed := false, false + defer func() { + cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second) + defer cancel() + if !stopped { + receipt, stopErr := session.Stop(cleanup) + stopped = receipt.VMMExited + if !stopped { + err = errors.Join(err, machineFailure("quarantine_unconfirmed_vmm_exit", stopErr)) + result = nil + return + } + } + if !destroyed { + if destroyErr := session.Destroy(cleanup); destroyErr != nil { + err = errors.Join(err, machineFailure("cleanup", destroyErr)) + result = nil + } + } + }() + if e != nil { + return nil, machineFailure("start", e) + } + instanceID := session.ID() + if e = session.Provision(ctx, sourcePath); e != nil { + return nil, machineFailure("provision", e) + } + if e = session.Sanitize(ctx); e != nil { + return nil, machineFailure("sanitize", e) + } + receipt, e := session.Stop(ctx) + stopped = receipt.VMMExited + if e != nil || !receipt.Graceful || !stopped { + return nil, machineFailure("graceful_stop", e) + } + bundle := filepath.Join(workspace, "bundle") + if e = os.Mkdir(bundle, 0700); e != nil { + return nil, machineFailure("export", e) + } + if e = session.Export(ctx, bundle); e != nil { + return nil, machineFailure("export", e) + } + if e = validateMachineExport(bundle, base.MacOS); e != nil { + return nil, machineFailure("validate_export", e) + } + if e = session.Destroy(ctx); e != nil { + return nil, machineFailure("cleanup", e) + } + destroyed = true + if e = os.Remove(sourcePath); e != nil { + return nil, machineFailure("cleanup_source", e) + } + if e = ctx.Err(); e != nil { + return nil, e + } + publication, e := r.Publisher.Publish(ctx, req.ID, bundle) + if e != nil { + return nil, machineFailure("publish", e) + } + output, e := images.ParseNormalizedRef(publication.Reference) + if e != nil || !output.IsDigest() || !machineDigest(publication.Digest) || output.Digest() != publication.Digest { + return nil, fmt.Errorf("machine publication receipt is unverified") + } + return &MachineBuildResult{Publication: publication, BuilderInstanceID: instanceID, Provenance: BuildProvenance{BaseImageDigest: base.Digest, SourceHash: hash, Timestamp: time.Now().UTC()}}, nil +} + +func machineDigest(s string) bool { + if !strings.HasPrefix(s, "sha256:") || len(s) != 71 { + return false + } + return strings.Trim(s[7:], "0123456789abcdef") == "" +} + +type machineContextReader struct { + ctx context.Context + r io.Reader +} + +func (r machineContextReader) Read(p []byte) (int, error) { + if err := r.ctx.Err(); err != nil { + return 0, err + } + return r.r.Read(p) +} +func stageMachineSource(ctx context.Context, source io.Reader, path string, limit int64) (string, error) { + file, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0600) + if err != nil { + return "", err + } + defer file.Close() + h := sha256.New() + n, err := io.Copy(io.MultiWriter(file, h), io.LimitReader(machineContextReader{ctx, source}, limit+1)) + if err != nil { + return "", err + } + if n > limit { + return "", fmt.Errorf("machine source too large") + } + if err = file.Sync(); err != nil { + return "", err + } + return hex.EncodeToString(h.Sum(nil)), nil +} + +func validateMachineExport(root string, base *images.MacOSImage) error { + directory, err := os.Lstat(root) + if err != nil || !directory.IsDir() { + return fmt.Errorf("export root must be a real directory") + } + entries, err := os.ReadDir(root) + if err != nil { + return err + } + if len(entries) != 3 { + return fmt.Errorf("export must contain only machine payload files") + } + for _, entry := range entries { + if entry.Name() != "disk.img" && entry.Name() != "aux.img" && entry.Name() != "config.json" { + return fmt.Errorf("unexpected export file") + } + info, err := os.Lstat(filepath.Join(root, entry.Name())) + if err != nil || !info.Mode().IsRegular() { + return fmt.Errorf("export files must be regular") + } + } + platform, err := images.ValidateMacOSBundle(root) + if err != nil { + return err + } + if !bytes.Equal(platform.HardwareModel, base.HardwareModel) || !bytes.Equal(platform.MachineIdentifier, base.MachineIdentifier) || !strings.EqualFold(platform.MAC, base.MAC) || platform.CPUs != base.CPUs || platform.Memory != base.Memory { + return fmt.Errorf("machine export changed preserved identity/resources") + } + // Reject unknown metadata rather than publishing an accidental credential field. + data, err := os.ReadFile(filepath.Join(root, "config.json")) + if err != nil { + return err + } + decoder := json.NewDecoder(bytes.NewReader(data)) + decoder.DisallowUnknownFields() + var strict images.MacOSImage + if err = decoder.Decode(&strict); err != nil { + return err + } + for _, entry := range entries { + if err = os.Chmod(filepath.Join(root, entry.Name()), 0600); err != nil { + return err + } + } + return nil +} diff --git a/lib/builds/machine_test.go b/lib/builds/machine_test.go new file mode 100644 index 000000000..f2495b752 --- /dev/null +++ b/lib/builds/machine_test.go @@ -0,0 +1,276 @@ +package builds + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "io" + "os" + "path/filepath" + "reflect" + "strings" + "testing" + + "github.com/kernel/hypeman/lib/images" +) + +const machineTestDigest = "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + +type machineFixture struct { + image *images.Image + calls []string + fail string + stop MachineStopReceipt + cancel context.CancelFunc + publisherCalled bool + cleanupContextLive bool + extraFile, changedIdentity, unknownMetadata bool +} + +func newMachineFixture() *machineFixture { + return &machineFixture{image: &images.Image{Digest: machineTestDigest, Status: images.StatusReady, Platform: "darwin/arm64", MacOS: &images.MacOSImage{HardwareModel: []byte{1}, MachineIdentifier: []byte{2}, MAC: "02:00:00:00:00:01", CPUs: 4, Memory: 8 << 30}}, stop: MachineStopReceipt{Graceful: true, VMMExited: true}} +} +func (f *machineFixture) record(phase string) error { + f.calls = append(f.calls, phase) + if f.fail == phase { + return errors.New("sensitive-command-argument") + } + return nil +} +func (f *machineFixture) ResolveBase(context.Context, string) (*images.Image, error) { + if err := f.record("resolve"); err != nil { + return nil, err + } + return f.image, nil +} +func (f *machineFixture) Start(_ context.Context, _ *images.Image, p BuildPolicy) (MachineBuildSession, error) { + if p.CPUs != 4 || p.MemoryMB != 8192 { + return nil, errors.New("wrong resource defaults") + } + return f, f.record("start") +} +func (f *machineFixture) ID() string { return "isolated-test-instance" } +func (f *machineFixture) Provision(ctx context.Context, source string) error { + if _, err := os.ReadFile(source); err != nil { + return err + } + if f.cancel != nil { + f.cancel() + return ctx.Err() + } + return f.record("provision") +} +func (f *machineFixture) Sanitize(context.Context) error { return f.record("sanitize") } +func (f *machineFixture) Stop(ctx context.Context) (MachineStopReceipt, error) { + f.cleanupContextLive = ctx.Err() == nil + return f.stop, f.record("stop") +} +func (f *machineFixture) Export(_ context.Context, root string) error { + if err := f.record("export"); err != nil { + return err + } + platform := *f.image.MacOS + if f.changedIdentity { + platform.MachineIdentifier = []byte{9} + } + data, _ := json.Marshal(platform) + if f.unknownMetadata { + var raw map[string]any + _ = json.Unmarshal(data, &raw) + raw["password"] = "must-not-publish" + data, _ = json.Marshal(raw) + } + for name, content := range map[string][]byte{"disk.img": []byte("disk"), "aux.img": []byte("aux"), "config.json": data} { + if err := os.WriteFile(filepath.Join(root, name), content, 0666); err != nil { + return err + } + } + if f.extraFile { + return os.WriteFile(filepath.Join(root, "secret.txt"), []byte("must-not-publish"), 0600) + } + return nil +} +func (f *machineFixture) Destroy(ctx context.Context) error { + f.cleanupContextLive = ctx.Err() == nil + return f.record("destroy") +} +func (f *machineFixture) Publish(ctx context.Context, id, root string) (MachinePublication, error) { + f.publisherCalled = true + if err := ctx.Err(); err != nil { + return MachinePublication{}, err + } + if id != "build-test" { + return MachinePublication{}, errors.New("wrong job namespace") + } + if _, err := os.Stat(filepath.Join(filepath.Dir(root), "source.tar.gz")); !os.IsNotExist(err) { + return MachinePublication{}, errors.New("source remains during publish") + } + for _, name := range []string{"disk.img", "aux.img", "config.json"} { + info, err := os.Stat(filepath.Join(root, name)) + if err != nil || info.Mode().Perm() != 0600 { + return MachinePublication{}, errors.New("unsafe export mode") + } + } + return MachinePublication{Reference: "localhost/builds/build-test@" + machineTestDigest, Digest: machineTestDigest}, f.record("publish") +} +func machineTestRunner(t *testing.T, f *machineFixture) *MachineBuildRunner { + t.Helper() + return &MachineBuildRunner{Driver: f, Publisher: f, WorkDir: t.TempDir()} +} +func machineTestRequest() MachineBuildRequest { + return MachineBuildRequest{ID: "build-test", BaseImage: "localhost/macos@" + machineTestDigest} +} + +func TestMachineBuildPublicationOrder(t *testing.T) { + f := newMachineFixture() + r := machineTestRunner(t, f) + result, err := r.Run(context.Background(), machineTestRequest(), strings.NewReader("source")) + if err != nil { + t.Fatal(err) + } + expected := []string{"resolve", "start", "provision", "sanitize", "stop", "export", "destroy", "publish"} + if !reflect.DeepEqual(f.calls, expected) { + t.Fatalf("wrong phase order: %v", f.calls) + } + hash := sha256.Sum256([]byte("source")) + if result.Provenance.SourceHash != hex.EncodeToString(hash[:]) || result.Provenance.BaseImageDigest != machineTestDigest { + t.Fatal("unverified provenance") + } + entries, err := os.ReadDir(r.WorkDir) + if err != nil || len(entries) != 0 { + t.Fatal("workspace not cleaned") + } +} + +func TestMachineBuildFailuresDoNotPublish(t *testing.T) { + for _, phase := range []string{"resolve", "start", "provision", "sanitize", "stop", "export", "destroy"} { + t.Run(phase, func(t *testing.T) { + f := newMachineFixture() + f.fail = phase + r := machineTestRunner(t, f) + result, err := r.Run(context.Background(), machineTestRequest(), strings.NewReader("source")) + if err == nil || result != nil || f.publisherCalled { + t.Fatal("failed build reached publication") + } + if strings.Contains(err.Error(), "sensitive-command-argument") { + t.Fatal("raw guest error leaked") + } + }) + } +} + +func TestMachineBuildStopReceipts(t *testing.T) { + for _, receipt := range []MachineStopReceipt{{}, {VMMExited: true}} { + f := newMachineFixture() + f.stop = receipt + r := machineTestRunner(t, f) + _, err := r.Run(context.Background(), machineTestRequest(), strings.NewReader("source")) + if err == nil || f.publisherCalled { + t.Fatal("unconfirmed/forced stop published") + } + for _, phase := range f.calls { + if phase == "export" { + t.Fatal("unconfirmed/forced stop exported") + } + if phase == "destroy" && !receipt.VMMExited { + t.Fatal("destroyed unconfirmed live storage") + } + } + } +} + +func TestMachineBuildCancellationUsesIndependentCleanup(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + f := newMachineFixture() + f.cancel = cancel + r := machineTestRunner(t, f) + _, err := r.Run(ctx, machineTestRequest(), strings.NewReader("source")) + if !errors.Is(err, context.Canceled) || !f.cleanupContextLive || f.publisherCalled { + t.Fatal("cancellation did not retain safe cleanup") + } + if !reflect.DeepEqual(f.calls, []string{"resolve", "start", "stop", "destroy"}) { + t.Fatalf("cancel cleanup: %v", f.calls) + } +} + +func TestMachineBuildInputAdmission(t *testing.T) { + for _, kind := range []string{"floating base", "wrong digest", "linux", "pending", "shaping", "domains", "source hash", "oversized source", "CPU overflow", "memory precision"} { + t.Run(kind, func(t *testing.T) { + f := newMachineFixture() + r := machineTestRunner(t, f) + req := machineTestRequest() + switch kind { + case "floating base": + req.BaseImage = "localhost/macos:latest" + case "wrong digest": + f.image.Digest = "sha256:" + strings.Repeat("b", 64) + case "linux": + f.image.Platform = "linux/arm64" + case "pending": + f.image.Status = images.StatusPending + case "shaping": + req.Policy.MemoryMB = 4096 + case "domains": + req.Policy.AllowedDomains = []string{"example.com"} + case "source hash": + req.SourceHash = strings.Repeat("0", 64) + case "oversized source": + r.MaxSourceBytes = 3 + case "CPU overflow": + f.image.MacOS.CPUs = ^uint(0) + case "memory precision": + f.image.MacOS.Memory++ + } + _, err := r.Run(context.Background(), req, strings.NewReader("source")) + if err == nil { + t.Fatal("bad input accepted") + } + for _, phase := range f.calls { + if phase == "start" { + t.Fatal("bad input started a VM") + } + } + }) + } +} + +type cancelMachineSource struct{ cancel context.CancelFunc } + +func (s cancelMachineSource) Read(p []byte) (int, error) { + s.cancel() + return copy(p, "source"), io.EOF +} +func TestMachineBuildCanceledSourceDoesNotStart(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + f := newMachineFixture() + _, err := machineTestRunner(t, f).Run(ctx, machineTestRequest(), cancelMachineSource{cancel}) + if !errors.Is(err, context.Canceled) || !reflect.DeepEqual(f.calls, []string{"resolve"}) { + t.Fatalf("canceled source started builder: %v", f.calls) + } +} + +func TestMachineBuildExportAdmission(t *testing.T) { + for _, kind := range []string{"identity", "extra file", "unknown metadata"} { + t.Run(kind, func(t *testing.T) { + f := newMachineFixture() + f.changedIdentity = kind == "identity" + f.extraFile = kind == "extra file" + f.unknownMetadata = kind == "unknown metadata" + _, err := machineTestRunner(t, f).Run(context.Background(), machineTestRequest(), strings.NewReader("source")) + if err == nil || f.publisherCalled { + t.Fatal("unsafe export published") + } + if strings.Contains(err.Error(), "must-not-publish") { + t.Fatal("export secret leaked in failure") + } + }) + } +} + +var _ MachineBuildDriver = (*machineFixture)(nil) +var _ MachineBuildSession = (*machineFixture)(nil) +var _ MachineBuildPublisher = (*machineFixture)(nil) diff --git a/lib/images/macos_machine.go b/lib/images/macos_machine.go index 41da00ab4..46002a4e4 100644 --- a/lib/images/macos_machine.go +++ b/lib/images/macos_machine.go @@ -59,6 +59,19 @@ func stageMachineFile(src, dst string) (int64, error) { return info.Size(), nil } +// ValidateMacOSBundle validates a complete installed machine bundle at the fixed +// disk.img/aux.img/config.json paths. It performs no boot, import or publication. +func ValidateMacOSBundle(root string) (*MacOSImage, error) { + payload, err := parseMacOSMachine(root, &containerMetadata{OS: "darwin", Architecture: "arm64", Labels: map[string]string{ + MacOSMachineVersionLabel: "1", MacOSMachineKindLabel: "macos-image", MacOSMachineFormatLabel: "raw", + MacOSMachineDiskLabel: "disk.img", MacOSMachineAuxLabel: "aux.img", MacOSMachinePlatformLabel: "config.json", + }}) + if err != nil { + return nil, err + } + return payload.Platform, nil +} + func parseMacOSMachine(root string, meta *containerMetadata) (*macOSMachinePayload, error) { // Normalize as resolveManifestPlatform does, so aliases such as aarch64 and // case variants such as Darwin select the machine path rather than rootfs. From 7c50c6cc6caae75a63006cdcf7a5e4d8bb1b3d9b Mon Sep 17 00:00:00 2001 From: chris lee Date: Fri, 9 Oct 2026 13:53:54 -0400 Subject: [PATCH 2/7] Expand machine build timeout, export and publication failure QA --- docs/macos-builds.md | 8 +- lib/builds/machine_failure_test.go | 135 +++++++++++++++++++++++++++++ lib/builds/machine_test.go | 16 +++- 3 files changed, 157 insertions(+), 2 deletions(-) create mode 100644 lib/builds/machine_failure_test.go diff --git a/docs/macos-builds.md b/docs/macos-builds.md index b2a4913dd..f4933bcd6 100644 --- a/docs/macos-builds.md +++ b/docs/macos-builds.md @@ -81,7 +81,13 @@ Synthetic tests cover input/resource/hash/size admission before VM start, exact phase order, partial-start cleanup, independent cancellation cleanup, forced/ unconfirmed-stop nonpublication and no deletion of live storage, failed-stage nonpublication, identity changes, extra/unknown secret metadata rejection, private -output modes, error-text masking and verified digest/source provenance. +output modes, error-text masking and verified digest/source provenance. Expanded +failure tests include partial source-read failure, actual deadline expiry with +independent cleanup, ambiguous publisher errors without retry, invalid/tagged/ +mismatched digest receipts, empty disk/aux, symlink/hardlink payloads, truncated/ +oversized metadata, invalid Ethernet MAC and changed resources. Repeated existing +build queue/cache/storage/secret-provider/registry-token regressions, CGO-disabled +machine tests and `go vet ./lib/builds` also pass locally. No actual VM is created and no guest commands or registry uploads are executed by these tests. Fake tiny disk files establish contract behavior, not bootability. diff --git a/lib/builds/machine_failure_test.go b/lib/builds/machine_failure_test.go new file mode 100644 index 000000000..69f36b8ff --- /dev/null +++ b/lib/builds/machine_failure_test.go @@ -0,0 +1,135 @@ +package builds + +import ( + "context" + "errors" + "os" + "path/filepath" + "reflect" + "strings" + "testing" + "time" +) + +type failingMachineSource struct{} + +func (failingMachineSource) Read(p []byte) (int, error) { + return copy(p, "partial"), errors.New("secret-source-error") +} + +func TestMachineBuildSourceReadFailure(t *testing.T) { + f := newMachineFixture() + r := machineTestRunner(t, f) + result, err := r.Run(context.Background(), machineTestRequest(), failingMachineSource{}) + if err == nil || result != nil || strings.Contains(err.Error(), "secret-source-error") { + t.Fatal("source failure was not masked") + } + if !reflect.DeepEqual(f.calls, []string{"resolve"}) { + t.Fatalf("source failure started guest: %v", f.calls) + } + entries, err := os.ReadDir(r.WorkDir) + if err != nil || len(entries) != 0 { + t.Fatal("partial source retained") + } +} + +func TestMachineBuildTimeoutCleanup(t *testing.T) { + f := newMachineFixture() + f.waitForCancel = true + r := machineTestRunner(t, f) + req := machineTestRequest() + req.Policy.TimeoutSeconds = 1 + parent, cancel := context.WithTimeout(context.Background(), 3*time.Second) + defer cancel() + result, err := r.Run(parent, req, strings.NewReader("source")) + if !errors.Is(err, context.DeadlineExceeded) || result != nil || f.publisherCalled || !f.cleanupContextLive { + t.Fatal("timeout cleanup failed") + } + if !reflect.DeepEqual(f.calls, []string{"resolve", "start", "stop", "destroy"}) { + t.Fatalf("timeout cleanup order: %v", f.calls) + } +} + +func TestMachineBuildPublicationReceipt(t *testing.T) { + for _, kind := range []string{"ambiguous error", "tag receipt", "digest mismatch", "invalid digest"} { + t.Run(kind, func(t *testing.T) { + f := newMachineFixture() + r := machineTestRunner(t, f) + switch kind { + case "ambiguous error": + f.fail = "publish" + case "tag receipt": + f.publicationOverride = &MachinePublication{Reference: "localhost/builds/build-test:latest", Digest: machineTestDigest} + case "digest mismatch": + f.publicationOverride = &MachinePublication{Reference: "localhost/builds/build-test@" + machineTestDigest, Digest: "sha256:" + strings.Repeat("b", 64)} + case "invalid digest": + f.publicationOverride = &MachinePublication{Reference: "localhost/builds/build-test@" + machineTestDigest, Digest: "not-a-digest"} + } + result, err := r.Run(context.Background(), machineTestRequest(), strings.NewReader("source")) + if err == nil || result != nil || !f.publisherCalled { + t.Fatal("unverified receipt became ready") + } + if strings.Contains(err.Error(), "sensitive-command-argument") { + t.Fatal("publisher error leaked") + } + count := 0 + for _, phase := range f.calls { + if phase == "publish" { + count++ + } + } + if count != 1 { + t.Fatal("ambiguous publication was retried") + } + }) + } +} + +func TestMachineBuildMalformedExport(t *testing.T) { + for _, kind := range []string{"empty disk", "empty aux", "symlink disk", "hardlink payloads", "truncated config", "oversized config", "resource change", "bad MAC"} { + t.Run(kind, func(t *testing.T) { + f := newMachineFixture() + r := machineTestRunner(t, f) + f.mutateExport = func(root string) error { + disk, aux, config := filepath.Join(root, "disk.img"), filepath.Join(root, "aux.img"), filepath.Join(root, "config.json") + switch kind { + case "empty disk": + return os.Truncate(disk, 0) + case "empty aux": + return os.Truncate(aux, 0) + case "symlink disk": + if err := os.Remove(disk); err != nil { + return err + } + return os.Symlink(aux, disk) + case "hardlink payloads": + if err := os.Remove(aux); err != nil { + return err + } + return os.Link(disk, aux) + case "truncated config": + return os.WriteFile(config, []byte(`{"hardware_model":`), 0600) + case "oversized config": + return os.WriteFile(config, []byte(strings.Repeat(" ", 65<<10)), 0600) + case "resource change": + data, err := os.ReadFile(config) + if err != nil { + return err + } + return os.WriteFile(config, []byte(strings.Replace(string(data), `"cpus":4`, `"cpus":2`, 1)), 0600) + case "bad MAC": + data, err := os.ReadFile(config) + if err != nil { + return err + } + return os.WriteFile(config, []byte(strings.Replace(string(data), "02:00:00:00:00:01", "02:00:00:00:00:01:02:03", 1)), 0600) + } + return nil + } + result, err := r.Run(context.Background(), machineTestRequest(), strings.NewReader("source")) + if err == nil || result != nil || f.publisherCalled { + t.Fatal("malformed export published") + } + }) + } +} diff --git a/lib/builds/machine_test.go b/lib/builds/machine_test.go index f2495b752..941fefdce 100644 --- a/lib/builds/machine_test.go +++ b/lib/builds/machine_test.go @@ -24,6 +24,9 @@ type machineFixture struct { fail string stop MachineStopReceipt cancel context.CancelFunc + waitForCancel bool + mutateExport func(string) error + publicationOverride *MachinePublication publisherCalled bool cleanupContextLive bool extraFile, changedIdentity, unknownMetadata bool @@ -60,6 +63,10 @@ func (f *machineFixture) Provision(ctx context.Context, source string) error { f.cancel() return ctx.Err() } + if f.waitForCancel { + <-ctx.Done() + return ctx.Err() + } return f.record("provision") } func (f *machineFixture) Sanitize(context.Context) error { return f.record("sanitize") } @@ -90,6 +97,9 @@ func (f *machineFixture) Export(_ context.Context, root string) error { if f.extraFile { return os.WriteFile(filepath.Join(root, "secret.txt"), []byte("must-not-publish"), 0600) } + if f.mutateExport != nil { + return f.mutateExport(root) + } return nil } func (f *machineFixture) Destroy(ctx context.Context) error { @@ -113,7 +123,11 @@ func (f *machineFixture) Publish(ctx context.Context, id, root string) (MachineP return MachinePublication{}, errors.New("unsafe export mode") } } - return MachinePublication{Reference: "localhost/builds/build-test@" + machineTestDigest, Digest: machineTestDigest}, f.record("publish") + publication := MachinePublication{Reference: "localhost/builds/build-test@" + machineTestDigest, Digest: machineTestDigest} + if f.publicationOverride != nil { + publication = *f.publicationOverride + } + return publication, f.record("publish") } func machineTestRunner(t *testing.T, f *machineFixture) *MachineBuildRunner { t.Helper() From 747ba207a97d0edead2823a8fb09e0f517d87749 Mon Sep 17 00:00:00 2001 From: chris lee Date: Fri, 9 Oct 2026 17:51:29 -0400 Subject: [PATCH 3/7] Run installed-machine phases through the normal build job lifecycle --- docs/macos-builds.md | 36 +++++--- lib/builds/builder_disk_test.go | 3 +- lib/builds/machine.go | 96 ++++------------------ lib/builds/machine_job_test.go | 59 +++++++++++++ lib/builds/machine_manager.go | 41 +++++++++ lib/builds/machine_manager_failure_test.go | 34 ++++++++ lib/builds/machine_manager_test.go | 96 ++++++++++++++++++++++ lib/builds/machine_test.go | 4 +- lib/builds/manager.go | 74 ++++++++++++++--- lib/builds/source.go | 75 +++++++++++++++++ lib/builds/source_test.go | 36 ++++++++ lib/builds/types.go | 4 + lib/images/macos_machine.go | 13 --- 13 files changed, 452 insertions(+), 119 deletions(-) create mode 100644 lib/builds/machine_job_test.go create mode 100644 lib/builds/machine_manager.go create mode 100644 lib/builds/machine_manager_failure_test.go create mode 100644 lib/builds/machine_manager_test.go create mode 100644 lib/builds/source.go create mode 100644 lib/builds/source_test.go diff --git a/docs/macos-builds.md b/docs/macos-builds.md index f4933bcd6..6d5d19dc7 100644 --- a/docs/macos-builds.md +++ b/docs/macos-builds.md @@ -5,11 +5,14 @@ an isolated VM, then publish a complete cold-boot OCI machine image. This is not Linux rootfs conversion, macOS installation/bootstrap, memory forking or safe identity rekeying. -This checkpoint adds an **internal lifecycle/publication contract** and tests. +This checkpoint adds an **internal machine backend to the normal build manager** +and synthetic tests. `Config.MachineBuild` opts into `CreateBuildRequest.MachineBaseImage` +at the Go boundary only. The existing queue, persisted request, timeout, status/log +completion, source staging/hash check, provenance and image-readiness gate are shared. It does not register an HTTP build mode, choose a recipe language, implement the -VZ/GuestService driver or stream an artifact into a production registry. Existing -Linux build behavior and macOS-only build rejection are unchanged. Do not claim -that normal macOS builds work from this foundation alone. +VZ/GuestService driver or stream an artifact into a production registry. Public +macOS-only build rejection remains unchanged. Do not claim that normal macOS builds +work from this foundation alone. ## Dependency and recipe boundaries @@ -27,16 +30,20 @@ cancellation/provenance/output contracts should remain shared across OS backends ## Lifecycle contract -`MachineBuildRunner` requires a ready installed `darwin/arm64` base pinned by a +`MachineBuildBackend` executes only machine-specific phases. The shared manager +resolves a ready installed `darwin/arm64` base pinned by a canonical SHA256 reference. CPU/memory must match that base (zero inherits), fit existing build limits, and preserve exact MiB precision. Negative/unbounded resource or timeout values are rejected before VM start. Timeout defaults to 600 seconds and is capped at 24 hours. Networking defaults to isolated; egress means unrestricted VZ NAT, not domain/TAP/policy parity. Domain allowlists are rejected. -Source input is staged in a private server-owned workspace, with bounded streaming -(default 64MiB compressed input), cancellation checks, SHA256 provenance and optional -expected-hash verification before start. Guest extraction must independently bound +Source input is staged once in the existing private build-job source directory, +with bounded streaming (64MiB compressed input), cancellation checks, SHA256 +provenance and optional expected-hash verification. Machine execution re-verifies +that file before start, including on recovery; it does not stage another copy. +Linux source staging uses the same helper. Machine mode rejects Linux builder, +Dockerfile, cache, build-argument and secret options rather than ignoring them. Guest extraction must independently bound expanded archive size and reject traversal/symlink escapes; this is not implemented by the host compressed-size bound. @@ -65,7 +72,7 @@ Cancellation/failure cleanup gets its own bounded 30-second context. It may dest instance storage only after a receipt confirms VMM exit. Otherwise the instance is quarantined for operator recovery, never deleted or published. Instance storage must be outside the input/export workspace; that private workspace is removed by -the runner. Raw guest error text is masked in ordinary error/log strings while +the machine backend. Raw guest error text is masked in ordinary error/log strings while `errors.Is/As` remain available internally. The concrete log stream must separately redact injected secrets before retention/forwarding. @@ -94,8 +101,15 @@ these tests. Fake tiny disk files establish contract behavior, not bootability. The shared machine-bundle validator is the same structural validator used for OCI imports, not a macOS/VZ integrity or credential scanner. -Remaining: concrete driver and streamed publisher, common build queue/status/log/ -secret/API integration, pinned toolchain/version provenance, recipe choice and +Synthetic manager tests additionally cover actual shared queue/status completion, +persisted machine requests, source consumption, inherited resource defaults and +unconfigured/mutable-base/unsupported-secret admission. Source tests cover private +exclusive staging and changed/missing/symlinked input rejection on recovery. +The full local Linux build suite remains blocked by missing `mkfs.ext4`; focused +queue/cache/storage/secret/token tests pass, not a full Linux runtime proof. + +Remaining: concrete driver and streamed publisher, recipe-level logs and common +secret/API activation, pinned toolchain/version provenance, recipe choice and source validation, storage floor/quota enforcement, sanitation audit, large-layer upload transport validation, repeat builds and output cold boot through normal APIs. Keep PR5 draft and runtime admission disabled until these gates pass. diff --git a/lib/builds/builder_disk_test.go b/lib/builds/builder_disk_test.go index e3ac5f7f0..74d13f856 100644 --- a/lib/builds/builder_disk_test.go +++ b/lib/builds/builder_disk_test.go @@ -27,7 +27,8 @@ func prepareBuildOnDisk(t *testing.T, mgr *manager, id string, req CreateBuildRe CreatedAt: time.Now(), } require.NoError(t, writeMetadata(mgr.paths, meta)) - require.NoError(t, mgr.storeSource(id, []byte("fake-tarball-data"))) + _, err := mgr.storeSource(context.Background(), id, []byte("fake-tarball-data")) + require.NoError(t, err) config := &BuildConfig{ JobID: id, diff --git a/lib/builds/machine.go b/lib/builds/machine.go index fc0f6f45d..99e919804 100644 --- a/lib/builds/machine.go +++ b/lib/builds/machine.go @@ -3,13 +3,8 @@ package builds import ( "bytes" "context" - "crypto/sha256" - "encoding/hex" - "encoding/json" "errors" "fmt" - "io" - "net" "os" "path/filepath" "strings" @@ -37,7 +32,6 @@ type MachineBuildResult struct { } type MachineBuildDriver interface { - ResolveBase(context.Context, string) (*images.Image, error) // Start owns an isolated instance and identity lease. A partially created // session must be returned even with an error so cleanup can stop it. Start(context.Context, *images.Image, BuildPolicy) (MachineBuildSession, error) @@ -61,11 +55,9 @@ type MachineBuildPublisher interface { Publish(context.Context, string, string) (MachinePublication, error) } -type MachineBuildRunner struct { - Driver MachineBuildDriver - Publisher MachineBuildPublisher - WorkDir string // Server-owned private build workspace, not supplied by recipes. - MaxSourceBytes int64 // Zero defaults to 64MiB of compressed input. +type MachineBuildBackend struct { + Driver MachineBuildDriver + Publisher MachineBuildPublisher } // machinePhaseError keeps potentially secret-bearing guest/command errors out of @@ -79,9 +71,14 @@ func (e *machinePhaseError) Error() string { return "machine build " + e func (e *machinePhaseError) Unwrap() error { return e.cause } func machineFailure(phase string, err error) error { return &machinePhaseError{phase, err} } -func (r *MachineBuildRunner) Run(ctx context.Context, req MachineBuildRequest, source io.Reader) (result *MachineBuildResult, err error) { - if r.Driver == nil || r.Publisher == nil || !filepath.IsAbs(r.WorkDir) || source == nil { - return nil, fmt.Errorf("machine driver, publisher, private workspace and source required") +// runPrepared executes only machine-specific phases. The shared build manager +// owns resolution, source staging/hash verification, deadline, queue and status. +func (r *MachineBuildBackend) runPrepared(ctx context.Context, req MachineBuildRequest, base *images.Image, sourcePath, workDir string) (result *MachineBuildResult, err error) { + if r.Driver == nil || r.Publisher == nil || !filepath.IsAbs(workDir) || !filepath.IsAbs(sourcePath) || len(req.SourceHash) != 64 || strings.Trim(req.SourceHash, "0123456789abcdef") != "" { + return nil, fmt.Errorf("machine driver, publisher, private workspace and verified source required") + } + if _, bounded := ctx.Deadline(); !bounded { + return nil, fmt.Errorf("prepared machine job requires a deadline") } if len(req.ID) == 0 || len(req.ID) > 128 || strings.Trim(req.ID, "abcdefghijklmnopqrstuvwxyz0123456789-") != "" { return nil, fmt.Errorf("invalid machine build ID") @@ -109,17 +106,10 @@ func (r *MachineBuildRunner) Run(ctx context.Context, req MachineBuildRequest, s if policy.NetworkMode != "isolated" && policy.NetworkMode != "egress" { return nil, fmt.Errorf("unsupported machine network mode") } - ctx, cancel := context.WithTimeout(ctx, time.Duration(policy.TimeoutSeconds)*time.Second) - defer cancel() - base, e := r.Driver.ResolveBase(ctx, ref.String()) - if e != nil { - return nil, machineFailure("resolve_base", e) - } if base == nil || base.Status != images.StatusReady || base.Platform != "darwin/arm64" || base.MacOS == nil || base.Digest != ref.Digest() || base.MacOS.CPUs < 2 || base.MacOS.CPUs > MaxBuildCPUs || base.MacOS.Memory < 4<<30 || base.MacOS.Memory > uint64(MaxBuildMemoryMB)<<20 || base.MacOS.Memory%(1<<20) != 0 { return nil, fmt.Errorf("base is not the pinned ready macOS machine") } - mac, macErr := net.ParseMAC(base.MacOS.MAC) - if len(base.MacOS.HardwareModel) == 0 || len(base.MacOS.MachineIdentifier) == 0 || macErr != nil || len(mac) != 6 { + if base.MacOS.Validate() != nil { return nil, fmt.Errorf("base has invalid machine identity") } memory := int(base.MacOS.Memory >> 20) @@ -132,25 +122,13 @@ func (r *MachineBuildRunner) Run(ctx context.Context, req MachineBuildRequest, s if e = policy.Validate(); e != nil { return nil, e } - workspace, e := os.MkdirTemp(r.WorkDir, "machine-build-") + workspace, e := os.MkdirTemp(workDir, "machine-build-") if e != nil { return nil, machineFailure("workspace", e) } defer os.RemoveAll(workspace) - sourcePath := filepath.Join(workspace, "source.tar.gz") - limit := r.MaxSourceBytes - if limit == 0 { - limit = 64 << 20 - } - if limit < 1 || limit > 1<<30 { - return nil, fmt.Errorf("invalid machine source limit") - } - hash, e := stageMachineSource(ctx, source, sourcePath, limit) - if e != nil { - return nil, machineFailure("stage_source", e) - } - if req.SourceHash != "" && req.SourceHash != hash { - return nil, fmt.Errorf("machine source hash mismatch") + if e = verifyBuildSource(ctx, sourcePath, req.SourceHash); e != nil { + return nil, machineFailure("verify_source", e) } if e = ctx.Err(); e != nil { return nil, e @@ -225,7 +203,7 @@ func (r *MachineBuildRunner) Run(ctx context.Context, req MachineBuildRequest, s if e != nil || !output.IsDigest() || !machineDigest(publication.Digest) || output.Digest() != publication.Digest { return nil, fmt.Errorf("machine publication receipt is unverified") } - return &MachineBuildResult{Publication: publication, BuilderInstanceID: instanceID, Provenance: BuildProvenance{BaseImageDigest: base.Digest, SourceHash: hash, Timestamp: time.Now().UTC()}}, nil + return &MachineBuildResult{Publication: publication, BuilderInstanceID: instanceID, Provenance: BuildProvenance{BaseImageDigest: base.Digest, SourceHash: req.SourceHash, Timestamp: time.Now().UTC()}}, nil } func machineDigest(s string) bool { @@ -235,37 +213,6 @@ func machineDigest(s string) bool { return strings.Trim(s[7:], "0123456789abcdef") == "" } -type machineContextReader struct { - ctx context.Context - r io.Reader -} - -func (r machineContextReader) Read(p []byte) (int, error) { - if err := r.ctx.Err(); err != nil { - return 0, err - } - return r.r.Read(p) -} -func stageMachineSource(ctx context.Context, source io.Reader, path string, limit int64) (string, error) { - file, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0600) - if err != nil { - return "", err - } - defer file.Close() - h := sha256.New() - n, err := io.Copy(io.MultiWriter(file, h), io.LimitReader(machineContextReader{ctx, source}, limit+1)) - if err != nil { - return "", err - } - if n > limit { - return "", fmt.Errorf("machine source too large") - } - if err = file.Sync(); err != nil { - return "", err - } - return hex.EncodeToString(h.Sum(nil)), nil -} - func validateMachineExport(root string, base *images.MacOSImage) error { directory, err := os.Lstat(root) if err != nil || !directory.IsDir() { @@ -294,17 +241,6 @@ func validateMachineExport(root string, base *images.MacOSImage) error { if !bytes.Equal(platform.HardwareModel, base.HardwareModel) || !bytes.Equal(platform.MachineIdentifier, base.MachineIdentifier) || !strings.EqualFold(platform.MAC, base.MAC) || platform.CPUs != base.CPUs || platform.Memory != base.Memory { return fmt.Errorf("machine export changed preserved identity/resources") } - // Reject unknown metadata rather than publishing an accidental credential field. - data, err := os.ReadFile(filepath.Join(root, "config.json")) - if err != nil { - return err - } - decoder := json.NewDecoder(bytes.NewReader(data)) - decoder.DisallowUnknownFields() - var strict images.MacOSImage - if err = decoder.Decode(&strict); err != nil { - return err - } for _, entry := range entries { if err = os.Chmod(filepath.Join(root, entry.Name()), 0600); err != nil { return err diff --git a/lib/builds/machine_job_test.go b/lib/builds/machine_job_test.go new file mode 100644 index 000000000..d06ec3b37 --- /dev/null +++ b/lib/builds/machine_job_test.go @@ -0,0 +1,59 @@ +package builds + +import ( + "context" + "fmt" + "io" + "os" + "path/filepath" + "time" +) + +type machineRunnerFixture struct { + MachineBuildBackend + WorkDir string + MaxSourceBytes int64 +} + +// Run is test-only: production invokes the prepared machine phase from runBuild. +// Existing failure fixtures still exercise resolution/staging before that phase. +func (r *machineRunnerFixture) Run(ctx context.Context, req MachineBuildRequest, source io.Reader) (*MachineBuildResult, error) { + if source == nil || !filepath.IsAbs(r.WorkDir) { + return nil, fmt.Errorf("workspace/source required") + } + timeout := req.Policy.TimeoutSeconds + if timeout == 0 { + timeout = 600 + } + if timeout < 0 || timeout > 86400 { + return nil, fmt.Errorf("invalid timeout") + } + ctx, cancel := context.WithTimeout(ctx, time.Duration(timeout)*time.Second) + defer cancel() + base, err := r.Driver.(*machineFixture).ResolveBase(ctx, req.BaseImage) + if err != nil { + return nil, machineFailure("resolve_base", err) + } + job, err := os.MkdirTemp(r.WorkDir, "prepared-job-") + if err != nil { + return nil, err + } + defer os.RemoveAll(job) + limit := r.MaxSourceBytes + if limit == 0 { + limit = 64 << 20 + } + if limit < 1 || limit > 1<<30 { + return nil, fmt.Errorf("invalid source limit") + } + sourcePath := filepath.Join(job, "source.tar.gz") + hash, err := stageBuildSource(ctx, source, sourcePath, limit) + if err != nil { + return nil, machineFailure("stage_source", err) + } + if req.SourceHash != "" && req.SourceHash != hash { + return nil, ErrSourceHashMismatch + } + req.SourceHash = hash + return r.runPrepared(ctx, req, base, sourcePath, r.WorkDir) +} diff --git a/lib/builds/machine_manager.go b/lib/builds/machine_manager.go new file mode 100644 index 000000000..4c5047e4b --- /dev/null +++ b/lib/builds/machine_manager.go @@ -0,0 +1,41 @@ +package builds + +import ( + "context" + "fmt" + "path/filepath" + + "github.com/kernel/hypeman/lib/images" +) + +// executeMachineBuild is a backend of the same queued build job. runBuild owns +// deadline/status/log completion, and CreateBuild already staged verified source. +func (m *manager) executeMachineBuild(ctx context.Context, id string, req CreateBuildRequest, policy *BuildPolicy) (*BuildResult, error) { + backend := m.config.MachineBuild + if backend == nil { + return nil, fmt.Errorf("machine build backend is not configured") + } + base, err := m.imageManager.GetImage(ctx, req.MachineBaseImage) + if err != nil { + return nil, machineFailure("resolve_base", err) + } + sourcePath, err := filepath.Abs(filepath.Join(m.paths.BuildSourceDir(id), "source.tar.gz")) + if err != nil { + return nil, err + } + result, err := backend.runPrepared(ctx, MachineBuildRequest{ID: id, BaseImage: req.MachineBaseImage, SourceHash: req.SourceHash, Policy: *policy}, base, sourcePath, filepath.Dir(filepath.Dir(sourcePath))) + if err != nil { + return nil, err + } + ref, err := images.ParseNormalizedRef(result.Publication.Reference) + if err != nil || ref.Repository() != stripRegistryScheme(m.config.RegistryURL)+"/builds/"+id { + return nil, fmt.Errorf("machine publication is outside the authorized job repository") + } + if meta, err := readMetadata(m.paths, id); err == nil { + meta.BuilderInstance = &result.BuilderInstanceID + if err = writeMetadata(m.paths, meta); err != nil { + return nil, machineFailure("record_builder", err) + } + } + return &BuildResult{Success: true, ImageDigest: result.Publication.Digest, Provenance: result.Provenance}, nil +} diff --git a/lib/builds/machine_manager_failure_test.go b/lib/builds/machine_manager_failure_test.go new file mode 100644 index 000000000..0746f8336 --- /dev/null +++ b/lib/builds/machine_manager_failure_test.go @@ -0,0 +1,34 @@ +package builds + +import ( + "context" + "testing" + "time" + + "github.com/kernel/hypeman/lib/paths" + "github.com/stretchr/testify/require" +) + +func TestMachineFailureUsesSharedFailedStatusWithoutPublication(t *testing.T) { + f := newMachineFixture() + f.fail = "provision" + config := DefaultConfig() + config.RegistrySecret = "synthetic-test-secret" + config.MachineBuild = &MachineBuildBackend{Driver: f, Publisher: f} + api, err := NewManager(paths.New(t.TempDir()), config, nil, nil, nil, machineJobImages{fixture: f}, nil, nil, nil) + require.NoError(t, err) + m := api.(*manager) + build, err := m.CreateBuild(context.Background(), CreateBuildRequest{MachineBaseImage: "registry/base@" + machineTestDigest}, []byte("source")) + require.NoError(t, err) + require.Eventually(t, func() bool { + b, e := m.GetBuild(context.Background(), build.ID) + return e == nil && b.Status == StatusFailed + }, 5*time.Second, 10*time.Millisecond) + done, err := m.GetBuild(context.Background(), build.ID) + require.NoError(t, err) + require.Equal(t, StatusFailed, done.Status) + require.Nil(t, done.ImageDigest) + require.NotContains(t, f.calls, "publish") + require.Contains(t, f.calls, "destroy") + require.Equal(t, "machine build provision failed", *done.Error, "guest error text must remain masked") +} diff --git a/lib/builds/machine_manager_test.go b/lib/builds/machine_manager_test.go new file mode 100644 index 000000000..b967c7598 --- /dev/null +++ b/lib/builds/machine_manager_test.go @@ -0,0 +1,96 @@ +package builds + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "errors" + "path/filepath" + "testing" + "time" + + "github.com/kernel/hypeman/lib/images" + "github.com/kernel/hypeman/lib/paths" + "github.com/stretchr/testify/require" +) + +type machineJobImages struct { + images.Manager + fixture *machineFixture +} + +func (m machineJobImages) GetImage(ctx context.Context, ref string) (*images.Image, error) { + if ref == "registry/base@"+machineTestDigest { + return m.fixture.ResolveBase(ctx, ref) + } + return &images.Image{Status: images.StatusReady, Digest: machineTestDigest}, nil +} + +func (m machineJobImages) WaitForReady(context.Context, string) error { return nil } + +type machineJobPublisher struct { + fixture *machineFixture + publishedID string +} + +func (p *machineJobPublisher) Publish(ctx context.Context, id, bundle string) (MachinePublication, error) { + p.publishedID = id + // Reuse the fixture's export/source-scrubbing assertions with its fixed ID. + receipt, err := p.fixture.Publish(ctx, "build-test", bundle) + if err == nil { + receipt.Reference = "localhost:4973/builds/" + id + "@" + receipt.Digest + } + return receipt, err +} +func TestMachineBuildUsesSharedQueueStatusAndSourceLifecycle(t *testing.T) { + f := newMachineFixture() + publisher := &machineJobPublisher{fixture: f} + config := DefaultConfig() + config.RegistrySecret = "synthetic-test-secret" + config.MachineBuild = &MachineBuildBackend{Driver: f, Publisher: publisher} + api, err := NewManager(paths.New(t.TempDir()), config, nil, nil, nil, machineJobImages{fixture: f}, nil, nil, nil) + require.NoError(t, err) + m := api.(*manager) + source := []byte("synthetic compressed-source bytes") + sum := sha256.Sum256(source) + req := CreateBuildRequest{MachineBaseImage: "registry/base@" + machineTestDigest, SourceHash: hex.EncodeToString(sum[:])} + build, err := m.CreateBuild(context.Background(), req, source) + require.NoError(t, err) + require.Eventually(t, func() bool { + b, e := m.GetBuild(context.Background(), build.ID) + return e == nil && (b.Status == StatusReady || b.Status == StatusFailed) + }, 5*time.Second, 10*time.Millisecond) + done, err := m.GetBuild(context.Background(), build.ID) + require.NoError(t, err) + require.Equal(t, StatusReady, done.Status, "failure: %v", done.Error) + require.Equal(t, build.ID, publisher.publishedID) + require.Equal(t, req.SourceHash, done.Provenance.SourceHash) + require.Equal(t, machineTestDigest, done.Provenance.BaseImageDigest) + require.Equal(t, f.ID(), *done.BuilderInstanceID) + meta, err := readMetadata(m.paths, build.ID) + require.NoError(t, err) + require.Equal(t, req.MachineBaseImage, meta.Request.MachineBaseImage) + require.Equal(t, "isolated", meta.Request.BuildPolicy.NetworkMode) + require.Equal(t, 0, meta.Request.BuildPolicy.CPUs, "machine defaults must not use Linux CPU defaults") + require.NoFileExists(t, filepath.Join(m.paths.BuildSourceDir(build.ID), "source.tar.gz")) +} +func TestMachineBuildAdmissionAndSharedSourceHash(t *testing.T) { + config := DefaultConfig() + config.RegistrySecret = "synthetic-test-secret" + api, err := NewManager(paths.New(t.TempDir()), config, nil, nil, nil, nil, nil, nil, nil) + require.NoError(t, err) + m := api.(*manager) + _, err = m.CreateBuild(context.Background(), CreateBuildRequest{MachineBaseImage: "registry/base@" + machineTestDigest}, []byte("source")) + require.ErrorContains(t, err, "not configured") + f := newMachineFixture() + m.config.MachineBuild = &MachineBuildBackend{Driver: f, Publisher: f} + _, err = m.CreateBuild(context.Background(), CreateBuildRequest{MachineBaseImage: "registry/base:mutable"}, []byte("source")) + require.ErrorContains(t, err, "pinned") + _, err = m.CreateBuild(context.Background(), CreateBuildRequest{MachineBaseImage: "registry/base@" + machineTestDigest, Secrets: []SecretRef{{ID: "unsupported"}}}, []byte("source")) + require.ErrorContains(t, err, "secret options") + _, err = m.CreateBuild(context.Background(), CreateBuildRequest{SourceHash: "wrong"}, []byte("source")) + require.True(t, errors.Is(err, ErrSourceHashMismatch)) + list, err := m.ListBuilds(context.Background()) + require.NoError(t, err) + require.Empty(t, list) +} diff --git a/lib/builds/machine_test.go b/lib/builds/machine_test.go index 941fefdce..4bebe903e 100644 --- a/lib/builds/machine_test.go +++ b/lib/builds/machine_test.go @@ -129,9 +129,9 @@ func (f *machineFixture) Publish(ctx context.Context, id, root string) (MachineP } return publication, f.record("publish") } -func machineTestRunner(t *testing.T, f *machineFixture) *MachineBuildRunner { +func machineTestRunner(t *testing.T, f *machineFixture) *machineRunnerFixture { t.Helper() - return &MachineBuildRunner{Driver: f, Publisher: f, WorkDir: t.TempDir()} + return &machineRunnerFixture{MachineBuildBackend: MachineBuildBackend{Driver: f, Publisher: f}, WorkDir: t.TempDir()} } func machineTestRequest() MachineBuildRequest { return MachineBuildRequest{ID: "build-test", BaseImage: "localhost/macos@" + machineTestDigest} diff --git a/lib/builds/manager.go b/lib/builds/manager.go index b9157d93c..46ab5f29b 100644 --- a/lib/builds/manager.go +++ b/lib/builds/manager.go @@ -2,6 +2,7 @@ package builds import ( "bufio" + "bytes" "context" _ "embed" "encoding/json" @@ -88,6 +89,9 @@ type Manager interface { // Config holds configuration for the build manager type Config struct { + // MachineBuild is an opt-in internal executor. No HTTP recipe mode is exposed. + MachineBuild *MachineBuildBackend + // MaxConcurrentBuilds is the maximum number of concurrent builds MaxConcurrentBuilds int @@ -453,14 +457,47 @@ func (m *manager) CreateBuild(ctx context.Context, req CreateBuildRequest, sourc } req.Tags = tags.Clone(req.Tags) - // Apply defaults to build policy + // Machine resources inherit the installed base, not Linux builder defaults. policy := req.BuildPolicy - if policy == nil { + if req.MachineBaseImage != "" { + if m.config.MachineBuild == nil || m.config.MachineBuild.Driver == nil || m.config.MachineBuild.Publisher == nil { + return nil, fmt.Errorf("machine build backend is not configured") + } + if req.Dockerfile != "" || req.BuilderID != "" || len(req.BuildArgs) != 0 || len(req.Secrets) != 0 || req.CacheScope != "" || req.GlobalCacheKey != "" || req.IsAdminBuild { + return nil, fmt.Errorf("machine builds do not support Linux builder/cache/secret options") + } + if len(sourceData) > maxBuildSourceBytes { + return nil, fmt.Errorf("machine build source too large") + } + ref, err := images.ParseNormalizedRef(req.MachineBaseImage) + if err != nil || !ref.IsDigest() || !machineDigest(ref.Digest()) { + return nil, fmt.Errorf("machine base must be pinned by sha256 digest") + } + if policy == nil { + policy = &BuildPolicy{} + } else { + copy := *policy + policy = © + } + if policy.TimeoutSeconds == 0 { + policy.TimeoutSeconds = 600 + } + if policy.TimeoutSeconds < 1 || policy.TimeoutSeconds > 86400 { + return nil, fmt.Errorf("invalid machine build timeout") + } + if policy.NetworkMode == "" { + policy.NetworkMode = "isolated" + } + if len(policy.AllowedDomains) != 0 || (policy.NetworkMode != "isolated" && policy.NetworkMode != "egress") { + return nil, fmt.Errorf("unsupported machine network policy") + } + } else if policy == nil { defaultPolicy := DefaultBuildPolicy() policy = &defaultPolicy } else { policy.ApplyDefaults() } + req.BuildPolicy = policy m.createMu.Lock() defer m.createMu.Unlock() @@ -498,11 +535,22 @@ func (m *manager) CreateBuild(ctx context.Context, req CreateBuildRequest, sourc } // Store source data - if err := m.storeSource(id, sourceData); err != nil { + hash, err := m.storeSource(ctx, id, sourceData) + if err != nil { deleteBuild(m.paths, id) return nil, fmt.Errorf("store source: %w", err) } + if req.SourceHash != "" && req.SourceHash != hash { + deleteBuild(m.paths, id) + return nil, ErrSourceHashMismatch + } + req.SourceHash = hash + if err := writeMetadata(m.paths, meta); err != nil { + deleteBuild(m.paths, id) + return nil, fmt.Errorf("write source provenance: %w", err) + } + // Generate scoped registry token for this build // Token grants per-repo access based on build type: // - Regular builds: push to builds/{id}, push to cache/{tenant}, pull from cache/global/{runtime} @@ -673,15 +721,12 @@ func (m *manager) BuilderHasBuilds(builderID string) bool { } // storeSource stores the source tarball for a build -func (m *manager) storeSource(buildID string, data []byte) error { +func (m *manager) storeSource(ctx context.Context, buildID string, data []byte) (string, error) { sourceDir := m.paths.BuildSourceDir(buildID) - if err := ensureDir(sourceDir); err != nil { - return err + if err := os.MkdirAll(sourceDir, 0700); err != nil { + return "", err } - - // Write source tarball - sourcePath := sourceDir + "/source.tar.gz" - return writeFile(sourcePath, data) + return stageBuildSource(ctx, bytes.NewReader(data), filepath.Join(sourceDir, "source.tar.gz"), int64(len(data))) } // runBuild executes a build in a builder VM @@ -699,8 +744,10 @@ func (m *manager) runBuild(ctx context.Context, id string, req CreateBuildReques // Mirror base images to the local registry before launching the VM. // BuildKit is configured with our registry as a mirror for docker.io, // so pre-cached images will be served locally without pulling from Docker Hub. - if err := m.mirrorBaseImagesForBuild(buildCtx, id, req); err != nil { - m.logger.Warn("failed to mirror base images", "id", id, "error", err) + if req.MachineBaseImage == "" { + if err := m.mirrorBaseImagesForBuild(buildCtx, id, req); err != nil { + m.logger.Warn("failed to mirror base images", "id", id, "error", err) + } } // Run the build in a builder VM @@ -809,6 +856,9 @@ func (m *manager) runBuild(ctx context.Context, id string, req CreateBuildReques // executeBuild runs the build in a builder VM func (m *manager) executeBuild(ctx context.Context, id string, req CreateBuildRequest, policy *BuildPolicy) (*BuildResult, error) { + if req.MachineBaseImage != "" { + return m.executeMachineBuild(ctx, id, req, policy) + } if !m.builderReady.Load() { return nil, fmt.Errorf("builder image is being prepared, please retry shortly") } diff --git a/lib/builds/source.go b/lib/builds/source.go new file mode 100644 index 000000000..b2075607f --- /dev/null +++ b/lib/builds/source.go @@ -0,0 +1,75 @@ +package builds + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "fmt" + "io" + "os" +) + +const maxBuildSourceBytes = 64 << 20 + +type buildContextReader struct { + ctx context.Context + r io.Reader +} + +func (r buildContextReader) Read(p []byte) (int, error) { + if err := r.ctx.Err(); err != nil { + return 0, err + } + return r.r.Read(p) +} + +// stageBuildSource streams compressed source once into a private, exclusive job +// file. Both backends consume this file and use its hash for input verification. +func stageBuildSource(ctx context.Context, source io.Reader, path string, limit int64) (hash string, err error) { + file, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0600) + if err != nil { + return "", err + } + defer func() { + if closeErr := file.Close(); err == nil { + err = closeErr + } + }() + h := sha256.New() + n, err := io.Copy(io.MultiWriter(file, h), io.LimitReader(buildContextReader{ctx, source}, limit+1)) + if err != nil { + return "", err + } + if n > limit { + return "", fmt.Errorf("build source too large") + } + if err = file.Sync(); err != nil { + return "", err + } + return hex.EncodeToString(h.Sum(nil)), nil +} + +// verifyBuildSource rejects missing/replaced source on recovery before a machine +// is launched. It never writes a second source copy. +func verifyBuildSource(ctx context.Context, path, expected string) error { + info, err := os.Lstat(path) + if err != nil { + return err + } + if !info.Mode().IsRegular() || info.Size() > maxBuildSourceBytes { + return fmt.Errorf("invalid staged source") + } + file, err := os.Open(path) + if err != nil { + return err + } + defer file.Close() + h := sha256.New() + if _, err = io.Copy(h, io.LimitReader(buildContextReader{ctx, file}, maxBuildSourceBytes+1)); err != nil { + return err + } + if hex.EncodeToString(h.Sum(nil)) != expected { + return ErrSourceHashMismatch + } + return nil +} diff --git a/lib/builds/source_test.go b/lib/builds/source_test.go new file mode 100644 index 000000000..1bb7e8a84 --- /dev/null +++ b/lib/builds/source_test.go @@ -0,0 +1,36 @@ +package builds + +import ( + "context" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/stretchr/testify/require" +) + +func TestSharedBuildSourceStagingAndRecoveryVerification(t *testing.T) { + ctx := context.Background() + path := filepath.Join(t.TempDir(), "source.tar.gz") + hash, err := stageBuildSource(ctx, strings.NewReader("source"), path, maxBuildSourceBytes) + require.NoError(t, err) + info, err := os.Stat(path) + require.NoError(t, err) + require.Equal(t, os.FileMode(0600), info.Mode().Perm()) + require.NoError(t, verifyBuildSource(ctx, path, hash)) + _, err = stageBuildSource(ctx, strings.NewReader("replacement"), path, maxBuildSourceBytes) + require.Error(t, err, "exclusive staging must preserve existing source") + require.NoError(t, verifyBuildSource(ctx, path, hash)) + canceled, cancel := context.WithCancel(ctx) + cancel() + require.ErrorIs(t, verifyBuildSource(canceled, path, hash), context.Canceled) + require.NoError(t, os.WriteFile(path, []byte("changed"), 0600)) + require.ErrorIs(t, verifyBuildSource(ctx, path, hash), ErrSourceHashMismatch) + require.NoError(t, os.Remove(path)) + require.Error(t, verifyBuildSource(ctx, path, hash)) + target := filepath.Join(filepath.Dir(path), "target") + require.NoError(t, os.WriteFile(target, []byte("source"), 0600)) + require.NoError(t, os.Symlink(target, path)) + require.ErrorContains(t, verifyBuildSource(ctx, path, hash), "invalid staged source") +} diff --git a/lib/builds/types.go b/lib/builds/types.go index 4ad6008e9..7e80d9a45 100644 --- a/lib/builds/types.go +++ b/lib/builds/types.go @@ -43,6 +43,10 @@ type CreateBuildRequest struct { // The Dockerfile specifies the runtime (e.g., FROM node:20-alpine). Dockerfile string `json:"dockerfile,omitempty"` + // MachineBaseImage selects the internal installed-machine backend. It is not + // exposed by the HTTP schema until recipes and concrete drivers are supported. + MachineBaseImage string `json:"machine_base_image,omitempty"` + // BaseImageDigest optionally pins the base image by digest for reproducibility BaseImageDigest string `json:"base_image_digest,omitempty"` diff --git a/lib/images/macos_machine.go b/lib/images/macos_machine.go index 46002a4e4..41da00ab4 100644 --- a/lib/images/macos_machine.go +++ b/lib/images/macos_machine.go @@ -59,19 +59,6 @@ func stageMachineFile(src, dst string) (int64, error) { return info.Size(), nil } -// ValidateMacOSBundle validates a complete installed machine bundle at the fixed -// disk.img/aux.img/config.json paths. It performs no boot, import or publication. -func ValidateMacOSBundle(root string) (*MacOSImage, error) { - payload, err := parseMacOSMachine(root, &containerMetadata{OS: "darwin", Architecture: "arm64", Labels: map[string]string{ - MacOSMachineVersionLabel: "1", MacOSMachineKindLabel: "macos-image", MacOSMachineFormatLabel: "raw", - MacOSMachineDiskLabel: "disk.img", MacOSMachineAuxLabel: "aux.img", MacOSMachinePlatformLabel: "config.json", - }}) - if err != nil { - return nil, err - } - return payload.Platform, nil -} - func parseMacOSMachine(root string, meta *containerMetadata) (*macOSMachinePayload, error) { // Normalize as resolveManifestPlatform does, so aliases such as aarch64 and // case variants such as Darwin select the machine path rather than rootfs. From b4dafbb7e3e0d1b137ebd6246e8cad46e9e73b4b Mon Sep 17 00:00:00 2001 From: chris lee Date: Sat, 10 Oct 2026 23:35:59 -0400 Subject: [PATCH 4/7] Implement GuestService machine builds and verified streaming publication --- docs/macos-builds.md | 235 +++++++++-------- lib/builds/machine.go | 14 +- lib/builds/machine_cancel.go | 58 +++++ lib/builds/machine_cancel_test.go | 34 +++ lib/builds/machine_guest.go | 177 +++++++++++++ lib/builds/machine_manager.go | 8 +- lib/builds/machine_publisher.go | 240 ++++++++++++++++++ lib/builds/machine_publisher_test.go | 90 +++++++ lib/builds/machine_recipe.go | 169 ++++++++++++ lib/builds/machine_recipe_test.go | 70 +++++ lib/builds/manager.go | 20 +- lib/instances/macos_build_export.go | 5 + lib/instances/macos_build_export_darwin.go | 126 +++++++++ .../macos_build_export_darwin_test.go | 50 ++++ lib/instances/stop.go | 35 ++- 15 files changed, 1199 insertions(+), 132 deletions(-) create mode 100644 lib/builds/machine_cancel.go create mode 100644 lib/builds/machine_cancel_test.go create mode 100644 lib/builds/machine_guest.go create mode 100644 lib/builds/machine_publisher.go create mode 100644 lib/builds/machine_publisher_test.go create mode 100644 lib/builds/machine_recipe.go create mode 100644 lib/builds/machine_recipe_test.go create mode 100644 lib/instances/macos_build_export.go create mode 100644 lib/instances/macos_build_export_darwin.go create mode 100644 lib/instances/macos_build_export_darwin_test.go diff --git a/docs/macos-builds.md b/docs/macos-builds.md index 6d5d19dc7..cbbb067df 100644 --- a/docs/macos-builds.md +++ b/docs/macos-builds.md @@ -1,115 +1,126 @@ -# macOS image builds (PR5, foundation only) - -Goal: ordinary build jobs provision a digest-pinned installed macOS base inside -an isolated VM, then publish a complete cold-boot OCI machine image. This is not -Linux rootfs conversion, macOS installation/bootstrap, memory forking or safe -identity rekeying. - -This checkpoint adds an **internal machine backend to the normal build manager** -and synthetic tests. `Config.MachineBuild` opts into `CreateBuildRequest.MachineBaseImage` -at the Go boundary only. The existing queue, persisted request, timeout, status/log -completion, source staging/hash check, provenance and image-readiness gate are shared. -It does not register an HTTP build mode, choose a recipe language, implement the -VZ/GuestService driver or stream an artifact into a production registry. Public -macOS-only build rejection remains unchanged. Do not claim that normal macOS builds -work from this foundation alone. - -## Dependency and recipe boundaries - -The branch begins on PR2's machine-image support. The real driver additionally -requires reviewed PR3's system GuestService/readiness/lifecycle changes; desktop -provisioning needs PR4 where applicable. Integrate those dependencies before API -activation. Do not replay stale guest-agent code or bypass identity exclusivity. - -No Macfile syntax or unrestricted Dockerfile/BuildKit compatibility is introduced. -Packer versus a restricted familiar provisioning recipe still needs a deliberate -choice before the public request/source schema is frozen. The internal driver -consumes the existing build job's staged source archive; it must run recipe -commands in the isolated guest, never in a host shell. Public source/log/secret/ -cancellation/provenance/output contracts should remain shared across OS backends. - -## Lifecycle contract - -`MachineBuildBackend` executes only machine-specific phases. The shared manager -resolves a ready installed `darwin/arm64` base pinned by a -canonical SHA256 reference. CPU/memory must match that base (zero inherits), fit -existing build limits, and preserve exact MiB precision. Negative/unbounded -resource or timeout values are rejected before VM start. Timeout defaults to 600 -seconds and is capped at 24 hours. Networking defaults to isolated; egress means -unrestricted VZ NAT, not domain/TAP/policy parity. Domain allowlists are rejected. - -Source input is staged once in the existing private build-job source directory, -with bounded streaming (64MiB compressed input), cancellation checks, SHA256 -provenance and optional expected-hash verification. Machine execution re-verifies -that file before start, including on recovery; it does not stage another copy. -Linux source staging uses the same helper. Machine mode rejects Linux builder, -Dockerfile, cache, build-argument and secret options rather than ignoring them. Guest extraction must independently bound -expanded archive size and reject traversal/symlink escapes; this is not implemented -by the host compressed-size bound. - -The successful order is: - -1. Resolve pinned installed base and validate resource policy. -2. Stage/verify source; start one owned builder with a valid identity lease. -3. Provision and sanitize build credentials/source/secret artifacts. -4. Obtain a graceful-stop receipt **and actual VMM exit**. -5. Export matching disk/aux/platform into a separate build-owned directory. -6. Validate the complete bundle, preserve base identity/resources, reject extra - files and unknown platform fields, and make payload modes private. -7. Destroy the stopped builder and remove staged source before publishing. -8. Stream blobs and commit/verify the manifest last through a server-owned, - build-scoped publisher; return a matching immutable digest receipt. - -The driver must reject unprovisioned system agents; it cannot bootstrap via an -unreviewed host SSH/sudo fallback. Default lifecycle APIs that silently force-stop -are insufficient evidence of graceful export readiness. A forced stop or any -pre-publication failure must not export/publish a successful image. Sanitation is -a mandatory driver operation, not something structural bundle validation proves. -Base credentials and reusable-image SSH/account/browser/TCC policy remain a -separately reviewed requirement. - -Cancellation/failure cleanup gets its own bounded 30-second context. It may destroy -instance storage only after a receipt confirms VMM exit. Otherwise the instance -is quarantined for operator recovery, never deleted or published. Instance storage -must be outside the input/export workspace; that private workspace is removed by -the machine backend. Raw guest error text is masked in ordinary error/log strings while -`errors.Is/As` remain available internally. The concrete log stream must separately -redact injected secrets before retention/forwarding. - -Publication starts only after validation and fallible builder/source cleanup. -A failed network commit may have ambiguous remote effects; partial blobs are not -success, and no ready result is returned without a verified receipt. Atomic -manifest-last behavior and reconciliation belong to the concrete publisher; -these interfaces alone do not prove remote nonpublication after a lost response. +# macOS image builds (experimental, internal activation only) + +Ordinary build jobs can provision a digest-pinned installed macOS base in an +isolated VZ VM and publish a complete cold-boot OCI machine image. This is not +macOS installation/bootstrap, Linux rootfs conversion, memory forking or identity +rekeying. This branch depends on PR3's shared system GuestService. + +`Config.MachineBuild` opts into `CreateBuildRequest.MachineBaseImage` at the Go +boundary. Queue, persisted request, deadline, source staging/hash, status, log +completion, provenance and image-readiness gate remain owned by the normal build +manager. The concrete implementations are `GuestMachineBuildDriver` and +`StreamingMachinePublisher`. **Public HTTP macOS-only build rejection remains +unchanged.** No public request or SDK activation is claimed. + +## Base and policy + +The ready installed base must be pinned by canonical SHA256, declare a provisioned +system agent on2222 and be `darwin/arm64`. Driver provisioning additionally checks +actual guest UID0. The driver never installs an agent or falls back to host +SSH/sudo. CPU/memory match the base exactly; zero request values inherit them. +Build limits, exact MiB precision and bounded deadlines still apply. Timeout +normally defaults600s, capped24h; full machine-image compression/pull can need a +longer explicitly selected deadline. + +Networking defaults isolated. Optional egress means unrestricted VZ NAT, not +Linux TAP/domain-policy parity. Domain allowlists, Linux Dockerfile/builder/cache/ +build-argument/secret/admin options are rejected, not ignored. Template identity +is preserved: concurrent same-identity guests remain forbidden. + +## Internal provisioning recipe + +Source is the normal build-owned tar.gz, privately staged and SHA256 verified, +including recovery. Compressed machine input is capped64MiB; Linux staging gets +no new size cap. Host extraction into an exclusive0700 temporary directory limits +uncompressed data128MiB, entries1024 and individual regular files64MiB. Only regular +files/directories are accepted; duplicate names, links/devices and path escapes +fail before transfer. Files are0600 or owner-executable0700. Guest tar is not used. + +The archive must contain `hypeman-macos-build.json`, limited64KiB: + +```json +{"version":1,"steps":[{"command":["/bin/sh","build.sh"],"env":{"MODE":"release"}}]} +``` + +At most32 steps execute as root in one exclusive `/var/root/hypeman-build-{id}` +directory. Command executable paths must be absolute; arguments/environment are +bounded and reject NUL. Commands run only in the guest, never a host shell. +Unknown JSON fields and extra JSON objects fail. This recipe is deliberately +internal until public source/schema/toolchain contracts are reviewed. + +Sanitation removes and verifies absence of the build-owned guest source directory +and removes extracted host source before stopping. There is no injected registry +token, host credential or secret in the guest. Guest stdout/stderr are not forwarded +or retained yet; recipe-level log streaming is still a gate. This does not sanitize +inherited template accounts, SSH keys, browser state or credentials: use a separately +reviewed clean base, not a private development guest, for distributable images. + +## Stop, export and cleanup safety + +`StopAndExportMacOSInstance` is an internal optional build capability, not a new +HTTP endpoint. It holds the instance lock across shutdown and both storage clones, +preventing Start from racing export. It rejects a pre-existing Stopped state, +unmanaged/non-macOS guests and any forced-stop fallback. A valid receipt requires +shutdown acknowledgement and actual owned VMM exit. Closed/distinct/nonempty +regular disk/aux files are checked before APFS cloning. The exclusive0700 output +contains only private0600 disk.img, aux.img and bounded platform config.json. +An export failure can retain a proven stop receipt for safe cleanup, but never +claims successful publication. + +Darwin shutdown accepts a detached, short-delayed power-off command so gRPC can +send its acceptance reply before vsock disappears. RPC cancellation after +acceptance does not undo shutdown. The host still independently awaits VMM exit; +an acknowledgement alone is insufficient. + +The backend validates complete output and preserved identity/resources, destroys +the stopped builder and consumes staged source **before** publication. Cancellation +interrupts the active normal machine job, not forced instance deletion. It records +the owned builder before provisioning and keeps Cancelled terminal. Cleanup has +its own30s context and may delete storage only after confirmed exit; otherwise it +retains quarantine. Potentially secret-bearing guest errors remain masked in +ordinary error/log strings while errors.Is/As work internally. + +## Streaming publication + +The publisher uses a server-configured registry and a token scoped only to +`builds/{id}`. Recipes cannot select destinations or credentials. HTTPS uses system +trust; plaintext is permitted only on loopback. Redirects are never followed and +environment HTTP proxies are disabled for these credential-bearing requests. + +A private0600 gzip spool computes compressed digest/size and uncompressed DiffID +with bounded memory. Configuration and layer blobs stream through monolithic +POST uploads, not go-containerregistry's large-buffer PATCH path. Each persisted +blob is independently GET/hash/size verified. The `latest` manifest is committed +last (the normal manager's image-readiness convention), then immutable digest +content is verified. Ambiguous errors do not return successful receipts. Private +spools are removed on success/error. Disk quota/floor policy belongs to deployment; +a compressed-size bound is not a storage quota. ## Validation and remaining gates -Synthetic tests cover input/resource/hash/size admission before VM start, exact -phase order, partial-start cleanup, independent cancellation cleanup, forced/ -unconfirmed-stop nonpublication and no deletion of live storage, failed-stage -nonpublication, identity changes, extra/unknown secret metadata rejection, private -output modes, error-text masking and verified digest/source provenance. Expanded -failure tests include partial source-read failure, actual deadline expiry with -independent cleanup, ambiguous publisher errors without retry, invalid/tagged/ -mismatched digest receipts, empty disk/aux, symlink/hardlink payloads, truncated/ -oversized metadata, invalid Ethernet MAC and changed resources. Repeated existing -build queue/cache/storage/secret-provider/registry-token regressions, CGO-disabled -machine tests and `go vet ./lib/builds` also pass locally. - -No actual VM is created and no guest commands or registry uploads are executed by -these tests. Fake tiny disk files establish contract behavior, not bootability. -The shared machine-bundle validator is the same structural validator used for OCI -imports, not a macOS/VZ integrity or credential scanner. - -Synthetic manager tests additionally cover actual shared queue/status completion, -persisted machine requests, source consumption, inherited resource defaults and -unconfigured/mutable-base/unsupported-secret admission. Source tests cover private -exclusive staging and changed/missing/symlinked input rejection on recovery. -The full local Linux build suite remains blocked by missing `mkfs.ext4`; focused -queue/cache/storage/secret/token tests pass, not a full Linux runtime proof. - -Remaining: concrete driver and streamed publisher, recipe-level logs and common -secret/API activation, pinned toolchain/version provenance, recipe choice and -source validation, storage floor/quota enforcement, sanitation audit, large-layer -upload transport validation, repeat builds and output cold boot through normal -APIs. Keep PR5 draft and runtime admission disabled until these gates pass. +Repeated race tests cover source/recipe admission, shared queue/status/provenance, +cancellation without publication, phase failures/quarantine, bundle identity/modes, +closed-storage/hardlink rejection and authenticated production-registry publication +with manifest-last ordering and redirect isolation. Tiny test disks are synthetic, +not bootability evidence. + +Separate isolated Darwin native QA completed a real normal queued build from a +manually provisioned, pinned installed base: root provisioning command, source +sanitation, acknowledged no-force shutdown, owned VMM exit, matching export, +authenticated streamed28.85GB layer publication, image-readiness gate, fresh +HTTP pull/cache destination and cold boot. Root exec found the build marker and +verified absence of the source workspace. The run took483.74s; test-process RSS +peaked36.88MiB, with313.17GiB minimum free disk. The initial attempt correctly +refused publication on lost shutdown acknowledgement, leading to the guest reply +scheduling fix. This is not HTTP build activation, automatic clean-base installation +or repeat/recovery proof. Final cold-boot test cleanup used the ordinary short +stop timeout and forced fallback; no export used that cleanup stop. + +The supported build suite excludes exactly five existing mkfs.ext4-dependent +Linux config-volume tests on this Darwin host. Independent default autoreview +remains authentication-blocked; no clean independent review is claimed. + +Remaining: HTTP/SDK and server configuration activation, common safe recipe logs, +clean-template sanitation policy/audit, toolchain/version provenance, production +storage quotas/floors, repeat-build/cancel/recovery native QA and Linux runtime +regressions. Keep PR5 draft and HTTP admission disabled until those gates pass. diff --git a/lib/builds/machine.go b/lib/builds/machine.go index 99e919804..655106f2b 100644 --- a/lib/builds/machine.go +++ b/lib/builds/machine.go @@ -18,9 +18,10 @@ import ( // authorization. A concrete driver must require a provisioned system agent. type MachineBuildRequest struct { ID string - BaseImage string // Installed darwin/arm64 image, pinned by sha256 digest. - SourceHash string // Optional expected lowercase SHA256 of the source archive. - Policy BuildPolicy // Zero CPU/memory inherit the base machine's exact resources. + BaseImage string // Installed darwin/arm64 image, pinned by sha256 digest. + SourceHash string // Optional expected lowercase SHA256 of the source archive. + Policy BuildPolicy // Zero CPU/memory inherit the base machine's exact resources. + onStarted func(string) error // Normal manager records the owned builder before provisioning. } type MachineStopReceipt struct{ Graceful, VMMExited bool } @@ -160,10 +161,15 @@ func (r *MachineBuildBackend) runPrepared(ctx context.Context, req MachineBuildR } } }() + instanceID := session.ID() + if req.onStarted != nil { + if recordErr := req.onStarted(instanceID); recordErr != nil { + return nil, machineFailure("record_builder", recordErr) + } + } if e != nil { return nil, machineFailure("start", e) } - instanceID := session.ID() if e = session.Provision(ctx, sourcePath); e != nil { return nil, machineFailure("provision", e) } diff --git a/lib/builds/machine_cancel.go b/lib/builds/machine_cancel.go new file mode 100644 index 000000000..2547142fb --- /dev/null +++ b/lib/builds/machine_cancel.go @@ -0,0 +1,58 @@ +package builds + +import ( + "context" + "fmt" + "sync" + "time" +) + +type machineBuildControl struct { + mu sync.Mutex + cancel context.CancelFunc +} + +func (m *manager) recordMachineBuilder(id, instanceID string) error { + if active, ok := m.machineBuilds.Load(id); ok { + control := active.(*machineBuildControl) + control.mu.Lock() + defer control.mu.Unlock() + } + meta, err := readMetadata(m.paths, id) + if err != nil { + return err + } + if meta.Status == StatusCancelled { + return context.Canceled + } + meta.BuilderInstance = &instanceID + return writeMetadata(m.paths, meta) +} + +func (m *manager) cancelMachineBuild(id string) error { + active, ok := m.machineBuilds.Load(id) + if !ok { + return ErrBuildInProgress + } + control := active.(*machineBuildControl) + control.mu.Lock() + defer control.mu.Unlock() + meta, err := readMetadata(m.paths, id) + if err != nil { + return err + } + if meta.Status != StatusBuilding && meta.Status != StatusPushing { + return fmt.Errorf("machine build no longer running") + } + // Persist cancellation before interrupting phases. The backend performs bounded + // graceful cleanup or quarantine; Cancel must not force-delete mutable storage. + meta.Status = StatusCancelled + now := time.Now() + meta.CompletedAt = &now + if err := writeMetadata(m.paths, meta); err != nil { + return err + } + control.cancel() + m.notifyStatusChange(id, StatusCancelled) + return nil +} diff --git a/lib/builds/machine_cancel_test.go b/lib/builds/machine_cancel_test.go new file mode 100644 index 000000000..d5b5df83c --- /dev/null +++ b/lib/builds/machine_cancel_test.go @@ -0,0 +1,34 @@ +package builds + +import ( + "context" + "testing" + "time" + + "github.com/kernel/hypeman/lib/paths" + "github.com/stretchr/testify/require" +) + +func TestMachineCancellationInterruptsPhasesWithoutPublishing(t *testing.T) { + fixture := newMachineFixture() + fixture.waitForCancel = true + config := DefaultConfig() + config.RegistrySecret = "synthetic secret" + config.MachineBuild = &MachineBuildBackend{Driver: fixture, Publisher: fixture} + api, e := NewManager(paths.New(t.TempDir()), config, nil, nil, nil, machineJobImages{fixture: fixture}, nil, nil, nil) + require.NoError(t, e) + build, e := api.CreateBuild(context.Background(), CreateBuildRequest{MachineBaseImage: "registry/base@" + machineTestDigest}, []byte("synthetic source")) + require.NoError(t, e) + require.Eventually(t, func() bool { + meta, e := readMetadata(api.(*manager).paths, build.ID) + return e == nil && meta.BuilderInstance != nil + }, 5*time.Second, 10*time.Millisecond) + require.NoError(t, api.CancelBuild(context.Background(), build.ID)) + require.Eventually(t, func() bool { _, active := api.(*manager).machineBuilds.Load(build.ID); return !active }, 5*time.Second, 10*time.Millisecond) + done, e := api.GetBuild(context.Background(), build.ID) + require.NoError(t, e) + require.Equal(t, StatusCancelled, done.Status) + require.False(t, fixture.publisherCalled) + require.Contains(t, fixture.calls, "stop") + require.Contains(t, fixture.calls, "destroy") +} diff --git a/lib/builds/machine_guest.go b/lib/builds/machine_guest.go new file mode 100644 index 000000000..efecb44bc --- /dev/null +++ b/lib/builds/machine_guest.go @@ -0,0 +1,177 @@ +package builds + +import ( + "context" + "fmt" + "os" + "path/filepath" + "time" + + "github.com/kernel/hypeman/lib/forkvm" + "github.com/kernel/hypeman/lib/guest" + "github.com/kernel/hypeman/lib/hypervisor" + "github.com/kernel/hypeman/lib/images" + "github.com/kernel/hypeman/lib/instances" +) + +type machineExportManager interface { + StopAndExportMacOSInstance(context.Context, string, string) (instances.MacOSBuildExportReceipt, error) +} + +// GuestMachineBuildDriver uses ordinary instance lifecycle and the shared system +// GuestService. It never installs an agent or silently upgrades an unmanaged base. +type GuestMachineBuildDriver struct { + Instances instances.Manager + WorkDir string +} + +func (d *GuestMachineBuildDriver) Start(ctx context.Context, base *images.Image, policy BuildPolicy) (MachineBuildSession, error) { + exporter, ok := d.Instances.(machineExportManager) + if !ok || !filepath.IsAbs(d.WorkDir) { + return nil, fmt.Errorf("strict machine export and private workspace required") + } + if _, ok = ctx.Deadline(); !ok { + return nil, fmt.Errorf("machine start requires a deadline") + } + if base == nil || base.Status != images.StatusReady || base.Platform != "darwin/arm64" || base.MacOS == nil || !base.MacOS.GuestAgent || base.MacOS.Validate() != nil || !machineDigest(base.Digest) { + return nil, fmt.Errorf("provisioned pinned macOS base required") + } + if err := policy.Validate(); err != nil { + return nil, err + } + if policy.CPUs != int(base.MacOS.CPUs) || policy.MemoryMB != int(base.MacOS.Memory>>20) || len(policy.AllowedDomains) != 0 || (policy.NetworkMode != "isolated" && policy.NetworkMode != "egress") { + return nil, fmt.Errorf("machine policy must preserve resources and supported networking") + } + ref, err := images.ParseNormalizedRef(base.Name) + if err != nil { + return nil, err + } + dir, err := os.MkdirTemp(d.WorkDir, "guest-machine-") + if err != nil { + return nil, err + } + inst, err := d.Instances.CreateInstance(ctx, instances.CreateInstanceRequest{Name: filepath.Base(dir), Image: ref.Repository() + "@" + base.Digest, Platform: "darwin/arm64", Hypervisor: hypervisor.TypeVZ, Size: int64(base.MacOS.Memory), Vcpus: int(base.MacOS.CPUs), NetworkEnabled: policy.NetworkMode == "egress"}) + if inst == nil { + _ = os.RemoveAll(dir) + if err == nil { + err = fmt.Errorf("instance creation returned no instance") + } + return nil, err + } + session := &guestMachineSession{manager: d.Instances, exporter: exporter, instance: inst, workspace: dir, guestDir: "/var/root/hypeman-build-" + inst.Id} + return session, err +} + +type guestMachineSession struct { + manager instances.Manager + exporter machineExportManager + instance *instances.Instance + workspace, guestDir string + receipt MachineStopReceipt + sanitized bool +} + +func (s *guestMachineSession) ID() string { return s.instance.Id } +func (s *guestMachineSession) dialer() (hypervisor.VsockDialer, error) { + return hypervisor.NewVsockDialer(s.instance.HypervisorType, s.instance.VsockSocket, s.instance.VsockCID) +} +func (s *guestMachineSession) exec(ctx context.Context, command []string, cwd string, env map[string]string, wait time.Duration) error { + dialer, err := s.dialer() + if err != nil { + return err + } + exit, err := guest.ExecIntoInstance(ctx, dialer, guest.ExecOptions{Command: command, Cwd: cwd, Env: env, WaitForAgent: wait}) + if err != nil { + return err + } + if exit == nil || exit.Code != 0 { + return fmt.Errorf("machine guest command did not succeed") + } + return nil +} + +func (s *guestMachineSession) Provision(ctx context.Context, source string) error { + root := filepath.Join(s.workspace, "source") + recipe, err := extractMachineRecipe(ctx, source, root) + if err != nil { + return err + } + if err = s.exec(ctx, []string{"/bin/sh", "-c", `test "$(/usr/bin/id -u)" -eq 0`}, "", nil, 30*time.Second); err != nil { + return err + } + if err = s.exec(ctx, []string{"/bin/test", "!", "-e", s.guestDir}, "", nil, 30*time.Second); err != nil { + return err + } + if err = s.exec(ctx, []string{"/bin/mkdir", "-m", "700", s.guestDir}, "", nil, 0); err != nil { + return err + } + dialer, err := s.dialer() + if err != nil { + return err + } + if err = guest.CopyToInstance(ctx, dialer, guest.CopyToInstanceOptions{SrcPath: root, DstPath: s.guestDir}); err != nil { + return err + } + for _, step := range recipe.Steps { + if err = s.exec(ctx, step.Command, s.guestDir, step.Env, 0); err != nil { + return err + } + } + return nil +} + +func (s *guestMachineSession) Sanitize(ctx context.Context) error { + if err := s.exec(ctx, []string{"/bin/rm", "-rf", "--", s.guestDir}, "", nil, 0); err != nil { + return err + } + if err := s.exec(ctx, []string{"/bin/test", "!", "-e", s.guestDir}, "", nil, 0); err != nil { + return err + } + if err := os.RemoveAll(filepath.Join(s.workspace, "source")); err != nil { + return err + } + s.sanitized = true + return nil +} + +func (s *guestMachineSession) Stop(ctx context.Context) (MachineStopReceipt, error) { + if s.receipt.VMMExited { + return s.receipt, nil + } + // Invalid source can fail before the guest finishes booting. Cleanup must + // await the declared agent rather than firing two early shutdown attempts. + if err := s.exec(ctx, []string{"/usr/bin/true"}, "", nil, 20*time.Second); err != nil { + return s.receipt, err + } + receipt, err := s.exporter.StopAndExportMacOSInstance(ctx, s.ID(), filepath.Join(s.workspace, "bundle")) + s.receipt = MachineStopReceipt{Graceful: receipt.Graceful, VMMExited: receipt.VMMExited} + return s.receipt, err +} + +func (s *guestMachineSession) Export(ctx context.Context, destination string) error { + if !s.receipt.Graceful || !s.receipt.VMMExited || !s.sanitized { + return fmt.Errorf("sanitized strict stop receipt required") + } + for _, name := range []string{"disk.img", "aux.img", "config.json"} { + if err := ctx.Err(); err != nil { + return err + } + if err := forkvm.CopyRegularFile(filepath.Join(s.workspace, "bundle", name), filepath.Join(destination, name)); err != nil { + return err + } + if err := os.Chmod(filepath.Join(destination, name), 0600); err != nil { + return err + } + } + return nil +} + +func (s *guestMachineSession) Destroy(ctx context.Context) error { + if !s.receipt.VMMExited { + return fmt.Errorf("unconfirmed VMM exit: instance quarantined") + } + if err := s.manager.DeleteInstance(ctx, s.ID()); err != nil { + return err + } + return os.RemoveAll(s.workspace) +} diff --git a/lib/builds/machine_manager.go b/lib/builds/machine_manager.go index 4c5047e4b..baf7ade84 100644 --- a/lib/builds/machine_manager.go +++ b/lib/builds/machine_manager.go @@ -23,7 +23,7 @@ func (m *manager) executeMachineBuild(ctx context.Context, id string, req Create if err != nil { return nil, err } - result, err := backend.runPrepared(ctx, MachineBuildRequest{ID: id, BaseImage: req.MachineBaseImage, SourceHash: req.SourceHash, Policy: *policy}, base, sourcePath, filepath.Dir(filepath.Dir(sourcePath))) + result, err := backend.runPrepared(ctx, MachineBuildRequest{ID: id, BaseImage: req.MachineBaseImage, SourceHash: req.SourceHash, Policy: *policy, onStarted: func(instanceID string) error { return m.recordMachineBuilder(id, instanceID) }}, base, sourcePath, filepath.Dir(filepath.Dir(sourcePath))) if err != nil { return nil, err } @@ -31,11 +31,5 @@ func (m *manager) executeMachineBuild(ctx context.Context, id string, req Create if err != nil || ref.Repository() != stripRegistryScheme(m.config.RegistryURL)+"/builds/"+id { return nil, fmt.Errorf("machine publication is outside the authorized job repository") } - if meta, err := readMetadata(m.paths, id); err == nil { - meta.BuilderInstance = &result.BuilderInstanceID - if err = writeMetadata(m.paths, meta); err != nil { - return nil, machineFailure("record_builder", err) - } - } return &BuildResult{Success: true, ImageDigest: result.Publication.Digest, Provenance: result.Provenance}, nil } diff --git a/lib/builds/machine_publisher.go b/lib/builds/machine_publisher.go new file mode 100644 index 000000000..8239fdeb8 --- /dev/null +++ b/lib/builds/machine_publisher.go @@ -0,0 +1,240 @@ +package builds + +import ( + "archive/tar" + "bytes" + "compress/gzip" + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "fmt" + "io" + "net" + "net/http" + "net/url" + "os" + "path/filepath" + "strings" + "time" + + v1 "github.com/google/go-containerregistry/pkg/v1" + "github.com/google/go-containerregistry/pkg/v1/types" + "github.com/kernel/hypeman/lib/images" +) + +// StreamingMachinePublisher targets only the configured internal registry and a +// build-scoped repository. Blobs are spooled privately, uploaded with bounded +// memory, independently re-read/hash verified, then the manifest is committed last. +// No credentials, redirect destinations or arbitrary output repositories come from recipes. +type StreamingMachinePublisher struct { + RegistryURL string + Tokens *RegistryTokenGenerator +} + +type machineBlob struct { + path string + descriptor v1.Descriptor +} + +func (p *StreamingMachinePublisher) Publish(ctx context.Context, id, bundle string) (MachinePublication, error) { + deadline, ok := ctx.Deadline() + if !ok || ctx.Err() != nil { + return MachinePublication{}, fmt.Errorf("bounded publication required") + } + if id == "" || len(id) > 128 || strings.Trim(id, "abcdefghijklmnopqrstuvwxyz0123456789-") != "" || p.Tokens == nil { + return MachinePublication{}, fmt.Errorf("scoped machine publisher required") + } + address := p.RegistryURL + if !strings.Contains(address, "://") { + address = "https://" + address + } + registry, err := url.Parse(address) + if err != nil || registry.Host == "" || registry.User != nil || registry.RawQuery != "" || registry.Fragment != "" || (registry.Path != "" && registry.Path != "/") || (registry.Scheme != "http" && registry.Scheme != "https") { + return MachinePublication{}, fmt.Errorf("invalid internal registry address") + } + if registry.Scheme == "http" && registry.Hostname() != "localhost" && !net.ParseIP(registry.Hostname()).IsLoopback() { + return MachinePublication{}, fmt.Errorf("plaintext registry must be loopback") + } + platform, err := images.ValidateMacOSBundle(bundle) + if err != nil { + return MachinePublication{}, err + } + if err = validateMachineExport(bundle, platform); err != nil { + return MachinePublication{}, err + } + spool, err := os.MkdirTemp(filepath.Dir(bundle), "machine-publish-") + if err != nil { + return MachinePublication{}, err + } + defer os.RemoveAll(spool) + layer, diffID, err := spoolMachineLayer(ctx, bundle, spool) + if err != nil { + return MachinePublication{}, err + } + config := v1.ConfigFile{OS: "darwin", Architecture: "arm64", RootFS: v1.RootFS{Type: "layers", DiffIDs: []v1.Hash{diffID}}, Config: v1.Config{Labels: map[string]string{ + images.MacOSMachineVersionLabel: "1", images.MacOSMachineKindLabel: "macos-image", images.MacOSMachineFormatLabel: "raw", images.MacOSMachineDiskLabel: "disk.img", images.MacOSMachineAuxLabel: "aux.img", images.MacOSMachinePlatformLabel: "config.json", + }}} + configBytes, err := json.Marshal(config) + if err != nil { + return MachinePublication{}, err + } + configPath := filepath.Join(spool, "config.json") + if err = os.WriteFile(configPath, configBytes, 0600); err != nil { + return MachinePublication{}, err + } + configBlob := machineBlob{path: configPath, descriptor: v1.Descriptor{MediaType: types.OCIConfigJSON, Digest: machineHash(configBytes), Size: int64(len(configBytes))}} + manifest := v1.Manifest{SchemaVersion: 2, MediaType: types.OCIManifestSchema1, Config: configBlob.descriptor, Layers: []v1.Descriptor{layer.descriptor}} + manifestBytes, err := json.Marshal(manifest) + if err != nil { + return MachinePublication{}, err + } + digest := machineHash(manifestBytes).String() + repo := "builds/" + id + token, err := p.Tokens.GeneratePushToken(id, []string{repo}, time.Until(deadline)) + if err != nil { + return MachinePublication{}, err + } + transport := http.DefaultTransport.(*http.Transport).Clone() + transport.Proxy = nil + defer transport.CloseIdleConnections() + client := &http.Client{Transport: transport, CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }} + base := strings.TrimRight(address, "/") + "/v2/" + repo + for _, blob := range []machineBlob{configBlob, layer} { + file, e := os.Open(blob.path) + if e != nil { + return MachinePublication{}, e + } + response, e := machineRegistryRequest(ctx, client, token, http.MethodPost, base+"/blobs/uploads/?digest="+blob.descriptor.Digest.String(), "application/octet-stream", io.LimitReader(file, blob.descriptor.Size), blob.descriptor.Size) + closeErr := file.Close() + if e != nil { + return MachinePublication{}, e + } + if closeErr != nil { + response.Body.Close() + return MachinePublication{}, closeErr + } + status := response.StatusCode + response.Body.Close() + if status != http.StatusCreated { + return MachinePublication{}, fmt.Errorf("machine blob upload was not committed") + } + response, e = machineRegistryRequest(ctx, client, token, http.MethodGet, base+"/blobs/"+blob.descriptor.Digest.String(), "", nil, 0) + if e != nil { + return MachinePublication{}, e + } + sum := sha256.New() + n, e := io.Copy(sum, io.LimitReader(response.Body, blob.descriptor.Size+1)) + response.Body.Close() + if e != nil || response.StatusCode != http.StatusOK || n != blob.descriptor.Size || hex.EncodeToString(sum.Sum(nil)) != blob.descriptor.Digest.Hex { + return MachinePublication{}, fmt.Errorf("machine blob verification failed") + } + } + target := base + "/manifests/latest" + response, err := machineRegistryRequest(ctx, client, token, http.MethodPut, target, string(types.OCIManifestSchema1), bytes.NewReader(manifestBytes), int64(len(manifestBytes))) + if err != nil { + return MachinePublication{}, err + } + status := response.StatusCode + response.Body.Close() + if status != http.StatusCreated { + return MachinePublication{}, fmt.Errorf("machine manifest commit unconfirmed") + } + response, err = machineRegistryRequest(ctx, client, token, http.MethodGet, base+"/manifests/"+digest, "", nil, 0) + if err != nil { + return MachinePublication{}, err + } + published, err := io.ReadAll(io.LimitReader(response.Body, int64(len(manifestBytes))+1)) + response.Body.Close() + if err != nil || response.StatusCode != http.StatusOK || !bytes.Equal(published, manifestBytes) { + return MachinePublication{}, fmt.Errorf("machine manifest verification failed") + } + return MachinePublication{Reference: registry.Host + "/" + repo + "@" + digest, Digest: digest}, nil +} + +func machineRegistryRequest(ctx context.Context, client *http.Client, token, method, target, media string, body io.Reader, size int64) (*http.Response, error) { + request, err := http.NewRequestWithContext(ctx, method, target, body) + if err != nil { + return nil, err + } + request.ContentLength = size + request.Header.Set("Authorization", "Bearer "+token) + if media != "" { + request.Header.Set("Content-Type", media) + } + request.Header.Set("Accept", string(types.OCIManifestSchema1)) + response, err := client.Do(request) + if err != nil { + return nil, fmt.Errorf("machine registry request failed") + } + return response, nil +} + +func machineHash(data []byte) v1.Hash { + sum := sha256.Sum256(data) + return v1.Hash{Algorithm: "sha256", Hex: hex.EncodeToString(sum[:])} +} + +type machineContextReader struct { + ctx context.Context + reader io.Reader +} + +func (r machineContextReader) Read(data []byte) (int, error) { + if err := r.ctx.Err(); err != nil { + return 0, err + } + return r.reader.Read(data) +} + +func spoolMachineLayer(ctx context.Context, bundle, spool string) (blob machineBlob, diffID v1.Hash, err error) { + name := filepath.Join(spool, "layer.tar.gz") + file, err := os.OpenFile(name, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0600) + if err != nil { + return blob, diffID, err + } + defer file.Close() + compressedHash := sha256.New() + gz, err := gzip.NewWriterLevel(io.MultiWriter(file, compressedHash), gzip.BestSpeed) + if err != nil { + return blob, diffID, err + } + plainHash := sha256.New() + archive := tar.NewWriter(io.MultiWriter(gz, plainHash)) + for _, name := range []string{"disk.img", "aux.img", "config.json"} { + source, e := os.Open(filepath.Join(bundle, name)) + if e != nil { + return blob, diffID, e + } + info, e := source.Stat() + if e != nil { + source.Close() + return blob, diffID, e + } + e = archive.WriteHeader(&tar.Header{Name: name, Mode: 0600, Size: info.Size(), Typeflag: tar.TypeReg}) + if e == nil { + _, e = io.CopyN(archive, machineContextReader{ctx, source}, info.Size()) + } + closeErr := source.Close() + if e != nil { + return blob, diffID, e + } + if closeErr != nil { + return blob, diffID, closeErr + } + } + if err = archive.Close(); err != nil { + return blob, diffID, err + } + if err = gz.Close(); err != nil { + return blob, diffID, err + } + if err = file.Close(); err != nil { + return blob, diffID, err + } + info, err := os.Stat(file.Name()) + if err != nil { + return blob, diffID, err + } + return machineBlob{path: file.Name(), descriptor: v1.Descriptor{MediaType: types.OCILayer, Digest: v1.Hash{Algorithm: "sha256", Hex: hex.EncodeToString(compressedHash.Sum(nil))}, Size: info.Size()}}, v1.Hash{Algorithm: "sha256", Hex: hex.EncodeToString(plainHash.Sum(nil))}, nil +} diff --git a/lib/builds/machine_publisher_test.go b/lib/builds/machine_publisher_test.go new file mode 100644 index 000000000..2f745c822 --- /dev/null +++ b/lib/builds/machine_publisher_test.go @@ -0,0 +1,90 @@ +package builds_test + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "sync/atomic" + "testing" + "time" + + "github.com/kernel/hypeman/lib/builds" + "github.com/kernel/hypeman/lib/images" + "github.com/kernel/hypeman/lib/middleware" + "github.com/kernel/hypeman/lib/paths" + "github.com/kernel/hypeman/lib/registry" + "github.com/stretchr/testify/require" +) + +type machineImportStub struct { + images.Manager + imported chan string +} + +func (m *machineImportStub) ImportLocalImage(_ context.Context, repo, reference, digest string) (*images.Image, error) { + m.imported <- repo + return &images.Image{Status: images.StatusReady, Digest: digest}, nil +} + +func machinePublisherBundle(t *testing.T) string { + t.Helper() + root := t.TempDir() + platform := images.MacOSImage{HardwareModel: []byte{1}, MachineIdentifier: []byte{2}, MAC: "02:00:00:00:00:01", CPUs: 4, Memory: 8 << 30, GuestAgent: true} + data, e := json.Marshal(platform) + require.NoError(t, e) + for name, data := range map[string][]byte{"config.json": data, "disk.img": []byte("synthetic disk"), "aux.img": []byte("synthetic aux")} { + require.NoError(t, os.WriteFile(filepath.Join(root, name), data, 0600)) + } + return root +} + +func TestStreamingMachinePublisherAuthenticatedRegistry(t *testing.T) { + // Tiny storage is intentional: this proves auth/streaming/commit ordering, not bootability. + imports := &machineImportStub{imported: make(chan string, 1)} + reg, e := registry.New(paths.New(t.TempDir()), imports) + require.NoError(t, e) + var uploads atomic.Int32 + handler := middleware.JwtAuth("synthetic registry secret")(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method == http.MethodPost { + require.Greater(t, r.ContentLength, int64(0)) + uploads.Add(1) + } + if r.Method == http.MethodPut { + require.Equal(t, int32(2), uploads.Load(), "manifest must be last") + } + reg.Handler().ServeHTTP(w, r) + })) + server := httptest.NewServer(handler) + defer server.Close() + publisher := &builds.StreamingMachinePublisher{RegistryURL: server.URL, Tokens: builds.NewRegistryTokenGenerator("synthetic registry secret")} + ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second) + defer cancel() + result, e := publisher.Publish(ctx, "qa-build", machinePublisherBundle(t)) + require.NoError(t, e) + require.Contains(t, result.Reference, "/builds/qa-build@sha256:") + select { + case repo := <-imports.imported: + require.Equal(t, "builds/qa-build", repo) + case <-ctx.Done(): + t.Fatal("manifest did not reach production conversion boundary") + } +} + +func TestStreamingMachinePublisherRejectsRedirectWithoutLeakingToken(t *testing.T) { + var visited atomic.Bool + other := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { visited.Store(true) })) + defer other.Close() + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + http.Redirect(w, r, other.URL, http.StatusTemporaryRedirect) + })) + defer server.Close() + publisher := &builds.StreamingMachinePublisher{RegistryURL: server.URL, Tokens: builds.NewRegistryTokenGenerator("synthetic secret")} + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + _, e := publisher.Publish(ctx, "qa-build", machinePublisherBundle(t)) + require.Error(t, e) + require.False(t, visited.Load()) +} diff --git a/lib/builds/machine_recipe.go b/lib/builds/machine_recipe.go new file mode 100644 index 000000000..ffb85349e --- /dev/null +++ b/lib/builds/machine_recipe.go @@ -0,0 +1,169 @@ +package builds + +import ( + "archive/tar" + "bytes" + "compress/gzip" + "context" + "encoding/json" + "fmt" + "io" + "os" + "path" + "path/filepath" + "regexp" + "strings" + "unicode/utf8" +) + +const machineRecipeName = "hypeman-macos-build.json" +const maxMachineExtractBytes = 128 << 20 + +type machineRecipe struct { + Version int `json:"version"` + Steps []machineRecipeStep `json:"steps"` +} +type machineRecipeStep struct { + Command []string `json:"command"` + Env map[string]string `json:"env,omitempty"` +} + +var machineEnvName = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*$`) + +// Extract on the host into an exclusive build-owned directory, never with guest +// tar or a caller-supplied destination. Links/devices, duplicates and bombs fail +// before any build source is copied to the VM. +func extractMachineRecipe(ctx context.Context, source, destination string) (*machineRecipe, error) { + file, err := os.Open(source) + if err != nil { + return nil, err + } + defer file.Close() + compressed, err := gzip.NewReader(file) + if err != nil { + return nil, err + } + defer compressed.Close() + compressed.Multistream(false) + if err = os.Mkdir(destination, 0700); err != nil { + return nil, err + } + complete := false + defer func() { + if !complete { + _ = os.RemoveAll(destination) + } + }() + limited := &io.LimitedReader{R: compressed, N: maxMachineExtractBytes + 1} + reader := tar.NewReader(limited) + seen := map[string]bool{} + var total int64 + for count := 0; ; count++ { + if err = ctx.Err(); err != nil { + return nil, err + } + header, e := reader.Next() + if e == io.EOF { + break + } + if e != nil { + return nil, e + } + if count >= 1024 { + return nil, fmt.Errorf("too many machine source entries") + } + name := path.Clean(header.Name) + if name == "." && header.Typeflag == tar.TypeDir { + continue + } + if !utf8.ValidString(name) || strings.ContainsAny(name, "\\\x00") || !filepath.IsLocal(name) || seen[name] { + return nil, fmt.Errorf("invalid or duplicate machine source path") + } + seen[name] = true + if header.Size < 0 || header.Size > 64<<20 || total+header.Size > maxMachineExtractBytes { + return nil, fmt.Errorf("machine source expands beyond limit") + } + total += header.Size + target := filepath.Join(destination, name) + switch header.Typeflag { + case tar.TypeDir: + if header.Size != 0 { + return nil, fmt.Errorf("directory has data") + } + if err = os.MkdirAll(target, 0700); err != nil { + return nil, err + } + case tar.TypeReg, tar.TypeRegA: + if err = os.MkdirAll(filepath.Dir(target), 0700); err != nil { + return nil, err + } + output, e := os.OpenFile(target, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0600|os.FileMode(header.Mode)&0100) + if e != nil { + return nil, e + } + _, copyErr := io.Copy(output, reader) + closeErr := output.Close() + if copyErr != nil { + return nil, copyErr + } + if closeErr != nil { + return nil, closeErr + } + default: + return nil, fmt.Errorf("machine source must contain only regular files and directories") + } + } + // Consume bounded trailing tar padding to verify gzip integrity without allowing + // an unbounded post-tar decompression stream. + _, err = io.Copy(io.Discard, limited) + if err != nil { + return nil, err + } + if limited.N <= 0 { + return nil, fmt.Errorf("machine source expands beyond limit") + } + metadata, err := os.Open(filepath.Join(destination, machineRecipeName)) + if err != nil { + return nil, err + } + data, readErr := io.ReadAll(io.LimitReader(metadata, 64<<10+1)) + closeErr := metadata.Close() + if readErr != nil { + return nil, readErr + } + if closeErr != nil { + return nil, closeErr + } + if len(data) > 64<<10 { + return nil, fmt.Errorf("machine recipe too large") + } + decoder := json.NewDecoder(bytes.NewReader(data)) + decoder.DisallowUnknownFields() + var recipe machineRecipe + if err = decoder.Decode(&recipe); err != nil { + return nil, err + } + if err = decoder.Decode(new(any)); err != io.EOF { + return nil, fmt.Errorf("recipe must contain one JSON object") + } + if recipe.Version != 1 || len(recipe.Steps) == 0 || len(recipe.Steps) > 32 { + return nil, fmt.Errorf("invalid machine recipe version or step count") + } + for _, step := range recipe.Steps { + if len(step.Command) == 0 || len(step.Command) > 128 || !filepath.IsAbs(step.Command[0]) || len(step.Env) > 128 { + return nil, fmt.Errorf("invalid machine command") + } + for _, arg := range step.Command { + if len(arg) > 8192 || !utf8.ValidString(arg) || strings.ContainsRune(arg, 0) { + return nil, fmt.Errorf("invalid machine argument") + } + } + for key, value := range step.Env { + if !machineEnvName.MatchString(key) || len(value) > 8192 || !utf8.ValidString(value) || strings.ContainsRune(value, 0) { + return nil, fmt.Errorf("invalid machine environment") + } + } + } + complete = true + return &recipe, nil +} diff --git a/lib/builds/machine_recipe_test.go b/lib/builds/machine_recipe_test.go new file mode 100644 index 000000000..cce154654 --- /dev/null +++ b/lib/builds/machine_recipe_test.go @@ -0,0 +1,70 @@ +package builds + +import ( + "archive/tar" + "compress/gzip" + "context" + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/require" +) + +func writeMachineSource(t *testing.T, headers []*tar.Header, bodies []string) string { + t.Helper() + name := filepath.Join(t.TempDir(), "source.tar.gz") + f, e := os.Create(name) + require.NoError(t, e) + gz := gzip.NewWriter(f) + writer := tar.NewWriter(gz) + for i, h := range headers { + require.NoError(t, writer.WriteHeader(h)) + if h.Typeflag == tar.TypeReg { + _, e = writer.Write([]byte(bodies[i])) + require.NoError(t, e) + } + } + require.NoError(t, writer.Close()) + require.NoError(t, gz.Close()) + require.NoError(t, f.Close()) + return name +} + +func TestMachineRecipeExtraction(t *testing.T) { + recipe := `{"version":1,"steps":[{"command":["/bin/sh","build.sh"],"env":{"MODE":"qa"}}]}` + source := writeMachineSource(t, []*tar.Header{{Name: machineRecipeName, Size: int64(len(recipe)), Typeflag: tar.TypeReg}, {Name: "build.sh", Size: 4, Typeflag: tar.TypeReg, Mode: 0755}}, []string{recipe, "true"}) + root := filepath.Join(t.TempDir(), "source") + result, e := extractMachineRecipe(context.Background(), source, root) + require.NoError(t, e) + require.Len(t, result.Steps, 1) + info, e := os.Stat(filepath.Join(root, "build.sh")) + require.NoError(t, e) + require.Equal(t, os.FileMode(0700), info.Mode().Perm()) +} + +func TestMachineRecipeRejectsUnsafeArchivesAndCommands(t *testing.T) { + for _, tc := range []struct { + name string + headers []*tar.Header + bodies []string + }{ + {"traversal", []*tar.Header{{Name: "../outside", Typeflag: tar.TypeReg, Size: 1}}, []string{"x"}}, + {"symlink", []*tar.Header{{Name: "link", Typeflag: tar.TypeSymlink, Linkname: "/etc"}}, []string{""}}, + {"hardlink", []*tar.Header{{Name: "link", Typeflag: tar.TypeLink, Linkname: "other"}}, []string{""}}, + {"duplicate", []*tar.Header{{Name: "a", Typeflag: tar.TypeReg, Size: 1}, {Name: "a", Typeflag: tar.TypeReg, Size: 1}}, []string{"x", "y"}}, + } { + t.Run(tc.name, func(t *testing.T) { + root := filepath.Join(t.TempDir(), "source") + _, e := extractMachineRecipe(context.Background(), writeMachineSource(t, tc.headers, tc.bodies), root) + require.Error(t, e) + _, e = os.Stat(root) + require.True(t, os.IsNotExist(e)) + }) + } + for _, body := range []string{`{"version":2,"steps":[]}`, `{"version":1,"steps":[{"command":["sh"]}]}`, `{"version":1,"steps":[{"command":["/bin/true"],"secret":"no"}]}`, `{"version":1,"steps":[{"command":["/bin/true"]}]} {}`} { + source := writeMachineSource(t, []*tar.Header{{Name: machineRecipeName, Typeflag: tar.TypeReg, Size: int64(len(body))}}, []string{body}) + _, e := extractMachineRecipe(context.Background(), source, filepath.Join(t.TempDir(), "source")) + require.Error(t, e) + } +} diff --git a/lib/builds/manager.go b/lib/builds/manager.go index 46ab5f29b..21a724ab9 100644 --- a/lib/builds/manager.go +++ b/lib/builds/manager.go @@ -154,6 +154,7 @@ type manager struct { logger *slog.Logger metrics *Metrics createMu sync.Mutex + machineBuilds sync.Map // build ID -> *machineBuildControl builderReady atomic.Bool // pendingRecovered is set once RecoverPendingBuilds has re-enqueued // persisted pending builds; until then BuilderHasBuilds also scans @@ -734,12 +735,15 @@ func (m *manager) runBuild(ctx context.Context, id string, req CreateBuildReques start := time.Now() m.logger.Info("starting build", "id", id) - // Update status to building - m.updateStatus(id, StatusBuilding, nil) - - // Create timeout context + // Register machine cancellation before exposing the running status. Ordinary + // Linux builder cancellation retains its existing instance-delete behavior. buildCtx, cancel := context.WithTimeout(ctx, time.Duration(policy.TimeoutSeconds)*time.Second) defer cancel() + if req.MachineBaseImage != "" { + m.machineBuilds.Store(id, &machineBuildControl{cancel: cancel}) + defer m.machineBuilds.Delete(id) + } + m.updateStatus(id, StatusBuilding, nil) // Mirror base images to the local registry before launching the VM. // BuildKit is configured with our registry as a mirror for docker.io, @@ -1167,6 +1171,11 @@ func (m *manager) updateStatus(id string, status string, err error) { // updateBuildComplete updates the build with final results func (m *manager) updateBuildComplete(id string, status string, digest *string, errMsg *string, provenance *BuildProvenance, durationMS *int64) { + if active, ok := m.machineBuilds.Load(id); ok { + control := active.(*machineBuildControl) + control.mu.Lock() + defer control.mu.Unlock() + } meta, readErr := readMetadata(m.paths, id) if readErr != nil { m.logger.Error("read metadata for completion", "id", id, "error", readErr) @@ -1313,6 +1322,9 @@ func (m *manager) CancelBuild(ctx context.Context, id string) error { return ErrBuildInProgress // Was already picked up case StatusBuilding, StatusPushing: + if meta.Request != nil && meta.Request.MachineBaseImage != "" { + return m.cancelMachineBuild(id) + } // Can't cancel a running build easily // Would need to terminate the builder instance if meta.BuilderInstance != nil { diff --git a/lib/instances/macos_build_export.go b/lib/instances/macos_build_export.go new file mode 100644 index 000000000..13539cd15 --- /dev/null +++ b/lib/instances/macos_build_export.go @@ -0,0 +1,5 @@ +package instances + +// MacOSBuildExportReceipt describes one stop/export operation, not persisted state. +// A failed export may still confirm that instance cleanup is safe. +type MacOSBuildExportReceipt struct{ Graceful, VMMExited bool } diff --git a/lib/instances/macos_build_export_darwin.go b/lib/instances/macos_build_export_darwin.go new file mode 100644 index 000000000..ceb4addae --- /dev/null +++ b/lib/instances/macos_build_export_darwin.go @@ -0,0 +1,126 @@ +//go:build darwin + +package instances + +import ( + "context" + "encoding/json" + "fmt" + "os" + "os/exec" + "path/filepath" + + "github.com/kernel/hypeman/lib/forkvm" + "github.com/kernel/hypeman/lib/images" +) + +// StopAndExportMacOSInstance is an internal build capability, not an HTTP endpoint. +// The instance lock spans shutdown and both storage clones so Start cannot race export. +// Neither a pre-existing Stopped state nor forced hypervisor shutdown qualifies. +func (m *manager) StopAndExportMacOSInstance(ctx context.Context, id, destination string) (receipt MacOSBuildExportReceipt, err error) { + if err = ctx.Err(); err != nil { + return + } + if _, bounded := ctx.Deadline(); !bounded { + return receipt, fmt.Errorf("machine export requires a deadline") + } + if !filepath.IsAbs(destination) { + return receipt, fmt.Errorf("export destination must be absolute") + } + lock := m.getInstanceLock(id) + lock.Lock() + defer lock.Unlock() + current, err := m.currentInstanceWithoutHydration(ctx, id) + if err != nil { + return receipt, err + } + if current.MacOS == nil || !current.GuestAgentEnabled() || (current.State != StateRunning && current.State != StateInitializing) { + return receipt, fmt.Errorf("export requires an active macOS guest with a system agent") + } + if rel, e := filepath.Rel(current.DataDir, destination); e != nil || filepath.IsLocal(rel) { + return receipt, fmt.Errorf("export must be outside instance storage") + } + if err = os.Mkdir(destination, 0700); err != nil { + return + } + complete := false + defer func() { + if !complete { + _ = os.RemoveAll(destination) + } + }() + if err = m.markRestartManualStopLocked(ctx, id); err != nil { + return + } + stopped, err := m.stopInstanceWithFallback(ctx, id, false) + if err != nil { + return receipt, err + } + receipt = MacOSBuildExportReceipt{Graceful: true, VMMExited: true} + m.notifyLifecycleEvent(ctx, LifecycleEventStop, stopped) + if err = writeStoppedMacOSBuildBundle(ctx, stopped, m.paths.InstanceOverlay(id), destination); err != nil { + return receipt, err + } + complete = true + return receipt, nil +} + +func writeStoppedMacOSBuildBundle(ctx context.Context, inst *Instance, disk, destination string) error { + if inst.State != StateStopped || inst.MacOS == nil { + return fmt.Errorf("machine must be stopped") + } + aux := filepath.Join(inst.DataDir, "mac-aux.img") + var diskInfo os.FileInfo + for _, source := range []string{disk, aux} { + info, err := os.Lstat(source) + if err != nil { + return err + } + if !info.Mode().IsRegular() || info.Size() == 0 { + return fmt.Errorf("machine storage must be nonempty regular files") + } + if diskInfo != nil && os.SameFile(diskInfo, info) { + return fmt.Errorf("machine storage files must be distinct") + } + diskInfo = info + err = exec.CommandContext(ctx, "/usr/sbin/lsof", "-t", source).Run() + exit, ok := err.(*exec.ExitError) + if !ok || exit.ExitCode() != 1 { + return fmt.Errorf("cannot confirm closed machine storage") + } + } + for _, file := range []struct{ source, name string }{{disk, "disk.img"}, {aux, "aux.img"}} { + if err := ctx.Err(); err != nil { + return err + } + path := filepath.Join(destination, file.name) + if err := forkvm.CopyRegularFile(file.source, path); err != nil { + return err + } + if err := os.Chmod(path, 0600); err != nil { + return err + } + } + data, err := json.Marshal(inst.MacOS) + if err != nil { + return err + } + path := filepath.Join(destination, "config.json") + file, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0600) + if err != nil { + return err + } + _, writeErr := file.Write(data) + closeErr := file.Close() + if writeErr != nil { + return writeErr + } + if closeErr != nil { + return closeErr + } + if err = ctx.Err(); err != nil { + return err + } + _, err = images.ValidateMacOSBundle(destination) + return err +} diff --git a/lib/instances/macos_build_export_darwin_test.go b/lib/instances/macos_build_export_darwin_test.go new file mode 100644 index 000000000..19b69fc90 --- /dev/null +++ b/lib/instances/macos_build_export_darwin_test.go @@ -0,0 +1,50 @@ +//go:build darwin + +package instances + +import ( + "context" + "os" + "path/filepath" + "testing" + + "github.com/kernel/hypeman/lib/images" + "github.com/stretchr/testify/require" +) + +func TestMacOSBuildBundleRequiresClosedDistinctStorage(t *testing.T) { + root := t.TempDir() + disk := filepath.Join(root, "overlay.raw") + aux := filepath.Join(root, "mac-aux.img") + require.NoError(t, os.WriteFile(disk, []byte("synthetic disk"), 0600)) + require.NoError(t, os.WriteFile(aux, []byte("synthetic aux"), 0600)) + inst := &Instance{State: StateStopped, StoredMetadata: StoredMetadata{DataDir: root, MacOS: &images.MacOSImage{HardwareModel: []byte{1}, MachineIdentifier: []byte{2}, MAC: "02:00:00:00:00:01", CPUs: 4, Memory: 8 << 30, GuestAgent: true}}} + file, e := os.Open(disk) + require.NoError(t, e) + require.Error(t, writeStoppedMacOSBuildBundle(context.Background(), inst, disk, t.TempDir())) + require.NoError(t, file.Close()) + out := t.TempDir() + require.NoError(t, writeStoppedMacOSBuildBundle(context.Background(), inst, disk, out)) + _, e = images.ValidateMacOSBundle(out) + require.NoError(t, e) + for _, name := range []string{"disk.img", "aux.img", "config.json"} { + info, e := os.Stat(filepath.Join(out, name)) + require.NoError(t, e) + require.Equal(t, os.FileMode(0600), info.Mode().Perm()) + } + require.NoError(t, os.Remove(aux)) + require.NoError(t, os.Link(disk, aux)) + require.Error(t, writeStoppedMacOSBuildBundle(context.Background(), inst, disk, t.TempDir())) + inst.State = StateRunning + require.Error(t, writeStoppedMacOSBuildBundle(context.Background(), inst, disk, t.TempDir())) +} + +func TestMacOSBuildExportRequiresBoundedAbsoluteDestination(t *testing.T) { + m := &manager{} + _, e := m.StopAndExportMacOSInstance(context.Background(), "unused", t.TempDir()) + require.Error(t, e) + ctx, cancel := context.WithCancel(context.Background()) + cancel() + _, e = m.StopAndExportMacOSInstance(ctx, "unused", t.TempDir()) + require.ErrorIs(t, e, context.Canceled) +} diff --git a/lib/instances/stop.go b/lib/instances/stop.go index 3a2973fdf..298e35ace 100644 --- a/lib/instances/stop.go +++ b/lib/instances/stop.go @@ -42,6 +42,10 @@ func hasVFIODevices(stored *StoredMetadata) bool { // tryGracefulGuestShutdown asks GuestService to shut down and waits for the // hypervisor process to exit. Returns true if the process exited in time. func (m *manager) tryGracefulGuestShutdown(ctx context.Context, inst *Instance, stopTimeout int) bool { + return m.tryGuestShutdown(ctx, inst, stopTimeout, false) +} + +func (m *manager) tryGuestShutdown(ctx context.Context, inst *Instance, stopTimeout int, requireAcknowledgement bool) bool { log := logger.FromContext(ctx) if !inst.StoredMetadata.GuestAgentEnabled() { @@ -65,7 +69,7 @@ func (m *manager) tryGracefulGuestShutdown(ctx context.Context, inst *Instance, return false } if pid == 0 { - return true + return !requireAcknowledgement } inst.HypervisorProcessIdentity.Set(pid) @@ -93,10 +97,18 @@ func (m *manager) tryGracefulGuestShutdown(ctx context.Context, inst *Instance, } waitTimeout := guestShutdownWaitTimeout(inst, stopTimeout, shutdownSent) + if requireAcknowledgement { + if ctx.Err() != nil { + return false + } + if deadline, ok := ctx.Deadline(); ok && time.Until(deadline) < waitTimeout { + waitTimeout = max(0, time.Until(deadline)) + } + } if WaitForProcessExit(pid, waitTimeout) { - log.DebugContext(ctx, "VM shut down gracefully", "instance_id", inst.Id) - return true + log.DebugContext(ctx, "VM exited after guest shutdown", "instance_id", inst.Id, "shutdown_rpc_acknowledged", shutdownSent) + return shutdownSent || !requireAcknowledgement } log.WarnContext(ctx, "graceful shutdown timed out, falling back to hypervisor shutdown", "instance_id", inst.Id) @@ -120,7 +132,11 @@ func guestShutdownWaitTimeout(inst *Instance, stopTimeout int, shutdownSent bool func (m *manager) stopInstance( ctx context.Context, id string, -) (_ *Instance, retErr error) { +) (*Instance, error) { + return m.stopInstanceWithFallback(ctx, id, true) +} + +func (m *manager) stopInstanceWithFallback(ctx context.Context, id string, allowFallback bool) (_ *Instance, retErr error) { start := time.Now() log := logger.FromContext(ctx) log.InfoContext(ctx, "stopping instance", "instance_id", id) @@ -163,19 +179,28 @@ func (m *manager) stopInstance( // 4. Graceful shutdown: send signal to guest init via Shutdown RPC, // then wait for VM to power off cleanly. Fall back to hypervisor shutdown on timeout. stopTimeout := resolveStopTimeout(stored) + if !allowFallback { + // macOS power-off can outlast the ordinary short stop default. + stopTimeout = max(stopTimeout, 30) + } gracefulCtx, gracefulSpanEnd := m.startLifecycleStep(ctx, "graceful_guest_shutdown", attribute.String("instance_id", id), attribute.String("hypervisor", string(stored.HypervisorType)), attribute.String("operation", "graceful_guest_shutdown"), attribute.Int("stop_timeout_seconds", stopTimeout), ) - gracefulShutdown := m.tryGracefulGuestShutdown(gracefulCtx, &inst, stopTimeout) + gracefulShutdown := m.tryGuestShutdown(gracefulCtx, &inst, stopTimeout, !allowFallback) if gracefulShutdown { gracefulSpanEnd(nil) } else { gracefulSpanEnd(errGracefulShutdownFailed) } + // Export callers must never turn a forced stop into a graceful receipt. + if !gracefulShutdown && !allowFallback { + return nil, errGracefulShutdownFailed + } + // 5. Fallback hypervisor shutdown if guest graceful shutdown didn't work if !gracefulShutdown { log.DebugContext(ctx, "shutting down hypervisor (fallback)", "instance_id", id) From eb1d1ed674b7fe979d6249a1e61a8ff71638fa2a Mon Sep 17 00:00:00 2001 From: chris lee Date: Sun, 11 Oct 2026 00:01:36 -0400 Subject: [PATCH 5/7] Enable experimental macOS builds through the normal HTTP API --- cmd/api/api/builds.go | 17 +- cmd/api/api/machine_builds.go | 100 ++++ cmd/api/api/machine_builds_test.go | 121 +++++ cmd/api/config/config.go | 9 +- cmd/api/config/macos_builds_test.go | 19 + cmd/api/main.go | 5 + docs/macos-builds.md | 46 +- lib/oapi/oapi.go | 790 ++++++++++++++-------------- lib/providers/providers.go | 25 +- openapi.yaml | 10 + 10 files changed, 739 insertions(+), 403 deletions(-) create mode 100644 cmd/api/api/machine_builds.go create mode 100644 cmd/api/api/machine_builds_test.go create mode 100644 cmd/api/config/macos_builds_test.go diff --git a/cmd/api/api/builds.go b/cmd/api/api/builds.go index cff961ca2..016c42fbb 100644 --- a/cmd/api/api/builds.go +++ b/cmd/api/api/builds.go @@ -44,9 +44,15 @@ func (s *ApiService) ListBuilds(ctx context.Context, request oapi.ListBuildsRequ // CreateBuild creates a new build job func (s *ApiService) CreateBuild(ctx context.Context, request oapi.CreateBuildRequestObject) (oapi.CreateBuildResponseObject, error) { if s.Config != nil && s.Config.MacOSOnly { - return oapi.CreateBuild400JSONResponse{Code: "unsupported", Message: "Linux builder VMs are unavailable in macOS-only mode"}, nil + if !s.Config.MacOSBuilds { + return oapi.CreateBuild400JSONResponse{Code: "unsupported", Message: "Linux builder VMs are unavailable in macOS-only mode"}, nil + } + req, source, err := parseMachineBuildMultipart(request.Body) + if err != nil { + return oapi.CreateBuild400JSONResponse{Code: "invalid_request", Message: err.Error()}, nil + } + return s.submitBuild(ctx, req, source) } - log := logger.FromContext(ctx) // Parse multipart form fields var sourceData []byte @@ -69,6 +75,8 @@ func (s *ApiService) CreateBuild(ctx context.Context, request oapi.CreateBuildRe } switch part.FormName() { + case "machine_base_image": + return oapi.CreateBuild400JSONResponse{Code: "unsupported", Message: "machine builds require macos_only and macos_builds"}, nil case "source": sourceData, err = io.ReadAll(part) if err != nil { @@ -256,6 +264,11 @@ func (s *ApiService) CreateBuild(ctx context.Context, request oapi.CreateBuildRe } } + return s.submitBuild(ctx, domainReq, sourceData) +} + +func (s *ApiService) submitBuild(ctx context.Context, domainReq builds.CreateBuildRequest, sourceData []byte) (oapi.CreateBuildResponseObject, error) { + log := logger.FromContext(ctx) build, err := s.BuildManager.CreateBuild(ctx, domainReq, sourceData) if err != nil { switch { diff --git a/cmd/api/api/machine_builds.go b/cmd/api/api/machine_builds.go new file mode 100644 index 000000000..4ae43c480 --- /dev/null +++ b/cmd/api/api/machine_builds.go @@ -0,0 +1,100 @@ +package api + +import ( + "fmt" + "io" + "mime/multipart" + "net/http" + "strconv" + + "github.com/kernel/hypeman/lib/builds" + "github.com/kernel/hypeman/lib/images" +) + +// LimitMachineBuildBody must wrap the validator, which also consumes request bodies. +func LimitMachineBuildBody(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method == http.MethodPost && r.URL.Path == "/builds" { + r.Body = http.MaxBytesReader(w, r.Body, 65<<20) + } + next.ServeHTTP(w, r) + }) +} + +// Machine-only HTTP admission is bounded regardless of multipart field order. +// Linux retains its existing parser and source contract. +func parseMachineBuildMultipart(body *multipart.Reader) (builds.CreateBuildRequest, []byte, error) { + req := builds.CreateBuildRequest{BuildPolicy: &builds.BuildPolicy{}} + var source []byte + seen := map[string]bool{} + if body == nil { + return req, nil, fmt.Errorf("multipart source is required") + } + for count := 0; ; count++ { + part, err := body.NextPart() + if err == io.EOF { + break + } + if err != nil { + return req, nil, fmt.Errorf("invalid multipart form") + } + name := part.FormName() + if count >= 16 || seen[name] { + return req, nil, fmt.Errorf("duplicate or excessive multipart fields") + } + seen[name] = true + limit := int64(64 << 10) + switch name { + case "source": + limit = 64 << 20 + case "machine_base_image", "timeout_seconds", "memory_mb", "cpus", "tags": + default: + return req, nil, fmt.Errorf("unsupported machine build field: %s", name) + } + data, err := io.ReadAll(io.LimitReader(part, limit+1)) + if err != nil || int64(len(data)) > limit { + return req, nil, fmt.Errorf("invalid or oversized %s field", name) + } + if err = part.Close(); err != nil { + return req, nil, fmt.Errorf("invalid multipart field") + } + switch name { + case "source": + source = data + case "machine_base_image": + req.MachineBaseImage = string(data) + case "tags": + req.Tags, err = parseTagsJSON(string(data)) + if err != nil { + return req, nil, fmt.Errorf("tags must be a JSON object of strings") + } + default: + value, err := strconv.Atoi(string(data)) + if err != nil || value < 0 { + return req, nil, fmt.Errorf("%s must be a nonnegative integer", name) + } + switch name { + case "timeout_seconds": + req.BuildPolicy.TimeoutSeconds = value + case "memory_mb": + req.BuildPolicy.MemoryMB = value + case "cpus": + req.BuildPolicy.CPUs = value + } + } + } + if len(source) == 0 { + return req, nil, fmt.Errorf("source is required") + } + ref, err := images.ParseNormalizedRef(req.MachineBaseImage) + if err != nil || !ref.IsDigest() || len(ref.Digest()) != 71 || ref.Digest()[:7] != "sha256:" { + return req, nil, fmt.Errorf("machine_base_image must be pinned by sha256 digest") + } + if req.BuildPolicy.TimeoutSeconds > 86400 { + return req, nil, fmt.Errorf("timeout_seconds exceeds 86400") + } + if err = req.BuildPolicy.Validate(); err != nil { + return req, nil, err + } + return req, source, nil +} diff --git a/cmd/api/api/machine_builds_test.go b/cmd/api/api/machine_builds_test.go new file mode 100644 index 000000000..dfcacfcce --- /dev/null +++ b/cmd/api/api/machine_builds_test.go @@ -0,0 +1,121 @@ +package api + +import ( + "bytes" + "context" + "io" + "mime/multipart" + "net/http" + "net/http/httptest" + "strings" + "testing" + + "github.com/kernel/hypeman/cmd/api/config" + "github.com/kernel/hypeman/lib/builds" + "github.com/kernel/hypeman/lib/oapi" + "github.com/stretchr/testify/require" +) + +func machineMultipart(t *testing.T, fields ...[2]string) *multipart.Reader { + t.Helper() + var data bytes.Buffer + writer := multipart.NewWriter(&data) + for _, field := range fields { + require.NoError(t, writer.WriteField(field[0], field[1])) + } + require.NoError(t, writer.Close()) + return multipart.NewReader(&data, writer.Boundary()) +} + +type machineZeroReader struct{} + +func (machineZeroReader) Read(p []byte) (int, error) { clear(p); return len(p), nil } + +func TestMachineBuildBodyBoundBeforeValidation(t *testing.T) { + for _, path := range []string{"/builds", "/images"} { + r := httptest.NewRequest(http.MethodPost, path, io.LimitReader(machineZeroReader{}, (65<<20)+1)) + LimitMachineBuildBody(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + n, err := io.Copy(io.Discard, r.Body) + if path == "/builds" { + require.Error(t, err) + require.EqualValues(t, 65<<20, n) + } else { + require.NoError(t, err) + require.EqualValues(t, (65<<20)+1, n) + } + })).ServeHTTP(httptest.NewRecorder(), r) + } +} + +func TestMachineBuildMultipartAdmission(t *testing.T) { + base := "example.test/base@sha256:" + strings.Repeat("a", 64) + valid := [][2]string{{"source", "synthetic"}, {"machine_base_image", base}} + for _, tc := range []struct { + name string + extra [][2]string + ok bool + }{ + {name: "inherit", ok: true}, + {name: "policy", extra: [][2]string{{"cpus", "4"}, {"memory_mb", "8192"}, {"timeout_seconds", "1200"}}, ok: true}, + {name: "duplicate", extra: [][2]string{{"source", "second"}}}, + {name: "linux", extra: [][2]string{{"dockerfile", "FROM scratch"}}}, + {name: "secret", extra: [][2]string{{"secrets", "[]"}}}, + {name: "invalid number", extra: [][2]string{{"cpus", "nan"}}}, + {name: "negative", extra: [][2]string{{"memory_mb", "-1"}}}, + {name: "timeout", extra: [][2]string{{"timeout_seconds", "86401"}}}, + {name: "metadata bound", extra: [][2]string{{"tags", strings.Repeat("x", (64<<10)+1)}}}, + } { + t.Run(tc.name, func(t *testing.T) { + req, source, err := parseMachineBuildMultipart(machineMultipart(t, append(append([][2]string{}, valid...), tc.extra...)...)) + if !tc.ok { + require.Error(t, err) + return + } + require.NoError(t, err) + require.Equal(t, base, req.MachineBaseImage) + require.Equal(t, []byte("synthetic"), source) + }) + } + _, _, err := parseMachineBuildMultipart(machineMultipart(t, [2]string{"source", "synthetic"}, [2]string{"machine_base_image", "example.test/base:latest"})) + require.Error(t, err) +} + +type machineHTTPManager struct { + builds.Manager + req builds.CreateBuildRequest + calls int +} + +func (m *machineHTTPManager) CreateBuild(_ context.Context, req builds.CreateBuildRequest, source []byte) (*builds.Build, error) { + m.calls++ + m.req = req + return &builds.Build{ID: "synthetic-build", Status: builds.StatusQueued}, nil +} + +func TestMachineBuildHTTPGateAndDispatch(t *testing.T) { + base := "example.test/base@sha256:" + strings.Repeat("a", 64) + for _, tc := range []struct { + name string + cfg *config.Config + accepted bool + }{ + {"disabled", &config.Config{MacOSOnly: true}, false}, + {"linux", &config.Config{}, false}, + {"enabled", &config.Config{MacOSOnly: true, MacOSBuilds: true}, true}, + } { + t.Run(tc.name, func(t *testing.T) { + manager := &machineHTTPManager{} + service := &ApiService{Config: tc.cfg, BuildManager: manager} + response, err := service.CreateBuild(context.Background(), oapi.CreateBuildRequestObject{Body: machineMultipart(t, [2]string{"source", "synthetic"}, [2]string{"machine_base_image", base})}) + require.NoError(t, err) + if tc.accepted { + require.IsType(t, oapi.CreateBuild202JSONResponse{}, response) + require.Equal(t, 1, manager.calls) + require.Equal(t, base, manager.req.MachineBaseImage) + } else { + require.IsType(t, oapi.CreateBuild400JSONResponse{}, response) + require.Zero(t, manager.calls) + } + }) + } +} diff --git a/cmd/api/config/config.go b/cmd/api/config/config.go index fe1fc74ee..d401cbd94 100644 --- a/cmd/api/config/config.go +++ b/cmd/api/config/config.go @@ -281,7 +281,8 @@ type Config struct { JwtSecret string `koanf:"jwt_secret"` Env string `koanf:"env"` Version string `koanf:"version"` - MacOSOnly bool `koanf:"macos_only"` // Experimental: omit Linux boot downloads and reject Linux creates. + MacOSBuilds bool `koanf:"macos_builds"` // Experimental: enable machine builds on the macOS-only server. + MacOSOnly bool `koanf:"macos_only"` // Experimental: omit Linux boot downloads and reject Linux creates. Network NetworkConfig `koanf:"network"` Caddy CaddyConfig `koanf:"caddy"` @@ -590,6 +591,12 @@ func expandHomePath(path string) string { // Validate checks configuration values for correctness. // Returns an error if any configuration value is invalid. func (c *Config) Validate() error { + if c.MacOSBuilds && !c.MacOSOnly { + return fmt.Errorf("macos_builds requires macos_only") + } + if c.MacOSBuilds && (c.Registry.CACertFile != "" || (c.Registry.Insecure && strings.HasPrefix(c.Registry.URL, "https://"))) { + return fmt.Errorf("macos_builds requires system-trusted registry TLS or loopback HTTP") + } if c.MacOSOnly && (runtime.GOOS != "darwin" || runtime.GOARCH != "arm64" || c.Hypervisor.Default != "vz") { return fmt.Errorf("macos_only requires vz on Apple silicon") } diff --git a/cmd/api/config/macos_builds_test.go b/cmd/api/config/macos_builds_test.go new file mode 100644 index 000000000..0fe52e3ee --- /dev/null +++ b/cmd/api/config/macos_builds_test.go @@ -0,0 +1,19 @@ +package config + +import ( + "github.com/stretchr/testify/require" + "testing" +) + +func TestMacOSBuildsFailClosedConfiguration(t *testing.T) { + require.False(t, (&Config{}).MacOSBuilds) + c := Config{MacOSBuilds: true} + require.ErrorContains(t, c.Validate(), "requires macos_only") + c.MacOSOnly = true + c.Registry.CACertFile = "custom.pem" + require.ErrorContains(t, c.Validate(), "system-trusted registry TLS") + c.Registry.CACertFile = "" + c.Registry.URL = "https://registry.example" + c.Registry.Insecure = true + require.ErrorContains(t, c.Validate(), "system-trusted registry TLS") +} diff --git a/cmd/api/main.go b/cmd/api/main.go index 72bf7f87c..4f064a7d1 100644 --- a/cmd/api/main.go +++ b/cmd/api/main.go @@ -533,6 +533,11 @@ func run() error { // even though the build is still running. r.Use(timeoutNonStreamingRequests(60 * time.Second)) + // Bound machine multipart input before OpenAPI validation reads the body. + if app.Config.MacOSBuilds { + r.Use(api.LimitMachineBuildBody) + } + // OpenAPI request validation with authentication validatorOptions := &nethttpmiddleware.Options{ Options: openapi3filter.Options{ diff --git a/docs/macos-builds.md b/docs/macos-builds.md index cbbb067df..d0134223e 100644 --- a/docs/macos-builds.md +++ b/docs/macos-builds.md @@ -1,4 +1,4 @@ -# macOS image builds (experimental, internal activation only) +# macOS image builds (experimental, opt-in HTTP) Ordinary build jobs can provision a digest-pinned installed macOS base in an isolated VZ VM and publish a complete cold-boot OCI machine image. This is not @@ -9,8 +9,29 @@ rekeying. This branch depends on PR3's shared system GuestService. boundary. Queue, persisted request, deadline, source staging/hash, status, log completion, provenance and image-readiness gate remain owned by the normal build manager. The concrete implementations are `GuestMachineBuildDriver` and -`StreamingMachinePublisher`. **Public HTTP macOS-only build rejection remains -unchanged.** No public request or SDK activation is claimed. +`StreamingMachinePublisher`. HTTP activation is disabled by default. Set both +`macos_only: true` and `macos_builds: true` on an Apple-silicon VZ server, with +an explicit `registry.url` reachable from the host (not a guest gateway). HTTPS +uses system trust; custom CA/insecure HTTPS machine configurations are rejected. +Loopback HTTP remains supported. Linux servers reject the machine selector. + +Use the existing authenticated `POST /builds` multipart endpoint: + +```sh +curl -H "Authorization: Bearer $TOKEN" \ + -F "machine_base_image=registry.example/base@sha256:$BASE_DIGEST" \ + -F "source=@source.tar.gz;type=application/gzip" \ + -F "timeout_seconds=1200" "$API/builds" +``` + +The installed base must already be Ready. A202 response queues an ordinary job; +GET /builds/{id}, cancellation and status/provenance use the shared APIs. OpenAPI +and generated server request types include machine_base_image; external language +SDK releases are separate, not claimed here. Only source, machine_base_image, +timeout_seconds, memory_mb, cpus and tags are accepted for machine HTTP builds. +Duplicates/unknown fields, malformed numbers and mutable bases return400. Metadata +is limited64KiB per field, source64MiB and the entire HTTP body65MiB before OpenAPI +validation. These limits do not change Linux source parsing. ## Base and policy @@ -27,7 +48,7 @@ Linux TAP/domain-policy parity. Domain allowlists, Linux Dockerfile/builder/cach build-argument/secret/admin options are rejected, not ignored. Template identity is preserved: concurrent same-identity guests remain forbidden. -## Internal provisioning recipe +## Experimental versioned provisioning recipe Source is the normal build-owned tar.gz, privately staged and SHA256 verified, including recovery. Compressed machine input is capped64MiB; Linux staging gets @@ -45,8 +66,8 @@ The archive must contain `hypeman-macos-build.json`, limited64KiB: At most32 steps execute as root in one exclusive `/var/root/hypeman-build-{id}` directory. Command executable paths must be absolute; arguments/environment are bounded and reject NUL. Commands run only in the guest, never a host shell. -Unknown JSON fields and extra JSON objects fail. This recipe is deliberately -internal until public source/schema/toolchain contracts are reviewed. +Unknown JSON fields and extra JSON objects fail. Version1 remains experimental; +public source/schema/toolchain contracts require review before merge. Sanitation removes and verifies absence of the build-owned guest source directory and removes extracted host source before stopping. There is no injected registry @@ -116,11 +137,20 @@ scheduling fix. This is not HTTP build activation, automatic clean-base installa or repeat/recovery proof. Final cold-boot test cleanup used the ordinary short stop timeout and forced fallback; no export used that cleanup stop. +A subsequent live test used the actual opt-in API server/provider wiring: +unauthenticated/invalid tokens returned401, a read-only token returned403, and +POST /builds returned202. GET reached Ready after the real provision/export/upload +and readiness gate. POST /instances cold-booted the returned image_ref, and the +normal authenticated exec WebSocket read the new root marker and verified source +absence. No fresh cache claim for this second run (the preceding run separately +proved fresh pull). Both runs used the private development base, not production +credential sanitation. External SDK releases remain separate. + The supported build suite excludes exactly five existing mkfs.ext4-dependent Linux config-volume tests on this Darwin host. Independent default autoreview remains authentication-blocked; no clean independent review is claimed. -Remaining: HTTP/SDK and server configuration activation, common safe recipe logs, +Remaining: external SDK releases, common safe recipe logs, clean-template sanitation policy/audit, toolchain/version provenance, production storage quotas/floors, repeat-build/cancel/recovery native QA and Linux runtime -regressions. Keep PR5 draft and HTTP admission disabled until those gates pass. +regressions. Keep PR5 draft and experimental HTTP activation disabled by default. diff --git a/lib/oapi/oapi.go b/lib/oapi/oapi.go index 908752181..7cfc4659e 100644 --- a/lib/oapi/oapi.go +++ b/lib/oapi/oapi.go @@ -2002,6 +2002,15 @@ type CreateBuildMultipartBody struct { // Admin builds can populate the shared global cache that all tenant builds read from. IsAdminBuild *string `json:"is_admin_build,omitempty"` + // MachineBaseImage Experimental installed darwin/arm64 base pinned by sha256 digest. + // Requires server macos_only and macos_builds. Source must contain + // hypeman-macos-build.json (version 1). Machine source is limited to + // 64 MiB compressed; only timeout_seconds, memory_mb, cpus and tags + // are accepted alongside source and this selector. CPU/memory zero + // inherit the base. Linux/cache/secrets/admin options are rejected. + // Output retains the base identity and cannot run concurrently with it. + MachineBaseImage *string `json:"machine_base_image,omitempty"` + // MemoryMb Memory limit for builder VM in MB (default 2048) MemoryMb *int `json:"memory_mb,omitempty"` @@ -19275,395 +19284,398 @@ func (sh *strictHandler) GetVolume(w http.ResponseWriter, r *http.Request, id st // Base64 encoded, gzipped, json marshaled Swagger object var swaggerSpec = []string{ - "H4sIAAAAAAAC/+y9+3IbN7Y3+ioonv1VpBmSoi6+KZX6jmM7jnasWNuyPd/eoQ8FdoMkRt1AB0BLplOu", - "On/NA0ydJ5wnObUWgL4RTbZkS7InrqmayGzcsbCwsC6/9UcvkmkmBRNG9w7/6OlowVKKfz42hkaLtzLJ", - "U/aK/Z4zbeDnTMmMKcMZFkplLswko2YB/4qZjhTPDJeid9g7oWZBLhdMMXKBrRC9kHkSkykjWI/FvX6P", - "vadplrDeYW8nFWYnpob2+j2zzOAnbRQX897Hfk8xGkuRLG03M5onpnc4o4lm/Ua3x9A0oZpAlQHWKdqb", - "SpkwKnofscXfc65Y3Dv8rTqNd0VhOf07iwx0/jg38tRQEU+XJzLh0XJ1si+4yN9jb4TmRqbU8IhoW4dk", - "WIlMqWYxkYLQyPALRriYylzE5PWTExJJIVgEjemxkFPN1AWLyUzJlJgFIwupDZYxikbnxNBpwoZj0es3", - "9oMJ+BJvXqW/LZhZMBUYLNfEtUJmUhGz4JpwAV8jNqxumFE5W13Zfo/HCZsYnjKZm9WF+llekkSKOU7L", - "t0vSXBuyoBeMfGBKkt9zmvDZkot5+yJN2UwqRn5eZiylgmQJjZgm3BAujPSzsWtU0ti9NERcfC6kYpOY", - "acMFhfYnmVRGr47+Jf5BE1Ipi0PD8sQsqPFULqQh54xl9YnSS3peX8bf9vb6j0aj0bt+jxuW2mNF3/M0", - "T3uH9+/d27/X76Vc2H/vFqPnwrA5U72PxS9UKbqsTEfLXEVsEvFYrZtJlHAmDHly9PTVNSfQ2x0N8X87", - "D3v93u6jveHu/Yf47937veq0Vha+PvKP64/eqaEm16s8yJ6miSOUSYVIVmf9a55OmSJyRqJcKSZMsiR4", - "pFjcgehq0x6FtiKSYsbnuWLxat/+yNWWc0E1ocIyjUGDX5SNdTp3ETCxWF6KiWIp5QLWeGUQr/wnAieU", - "uEMEQ4qkMEomCTAFY1iaGe1PUR/YuCA0yxIeIeupHaqDdKR7/Z7Ik4ROV0ZY7jZL+JxjgU5Lw3Vlk3xd", - "YiRhwjBVnPAuS1Nji20dl8sd3I2SL3bqciGTeJILw5PVXt/Az25Nqz1xTRYsiYmczfqEzwgl0Ar8bGm8", - "vux7o737g9HBYHT/9e6Dw9Gjw9G9/+n1ezOpUmp6h72YGjaAXe6yN8i0NRdReHdEk0RSuJAUi+zuFBdW", - "bQOnLJIpI9D02pEffPrIW6+bJ+WmQkHiCpY3a2Crh40roxt1J1SbggnhfnGznNDAmF7zlGlD0wz4EIyh", - "sphtXMg32NwHv/JrF3j3kxZYsPdm4lYoOJ8QfbD3GYvgRpSemxQCBrTnyLtgWbdC44pRLYUVk+Ay/a2X", - "C51ncHWzeJIl1EC7vX7PksEk5VpD1eKHmGvLR/o9T+QTIc1E5ULYgoKZS6nOqyVdKxOe9fq9BdWTi3mW", - "9/rrrq06UWMXLKGZxvbcjqsJU0qqnhWNl5OZVH6Teu+qS7imqZUV0sUVG1ihXr9XW4CCnfu5+HEXuxoc", - "XL/n/poo+6rA5uxkVgdebWt1uMXQ1jN2e4tYIdpvM3GVdZ0DxJzOhdSGR7oTm0fhAbY3lXGAdT4tmiM8", - "ZsLwGWfKydWMKLgfUkZ8IwQaIVyQXDfOQSH6T9gFE0ZPLg4mJspWF6XxsKluXkU2KW/Eyq1cbH9xUjYQ", - "aX3uwYfTBeV4Jp+yCx6xVdnNbc0kVvyCqQD7LgQAywptObIFZx1YiJCCbddWSlzwmNMu7CDGMU14gHpO", - "nhwR+5kcPSVbC/a+3sneg+nDXnuTgqYBWvg5T6kYwIGAYfn2sWy17RcHoZa5TNN8Mlcyz1ZbPnp5fPyG", - "4EciUMKttvhwLySpZhGf0DhWTOvw/P3H6thGo9HokO4djkbDUWiUF0zEUrUuqf0cXtLdUczWNNlpSV37", - "K0v669ujp0ePyROpMqnwzbbx4FSXpzqvKtnUdyVE/z/mPIlXqX4KP7OWdWJKc22YMORHW4woZp9y5HIh", - "NSMRjRaMTO2jBZ/n2F4Xki869rdXaARPuc4k8nny9riU5vBpyN6zKDe1fr8nMdeGi8gJVZXJdRgRqJ8S", - "BvdvSLLAFSCuDJeCGC86XVsIiBSjG7qDEp06Wz3/uSWvSarbWvdFCBck5UnCNYukiHW1Dy7M/YP2yVRO", - "sb02V7p6Bj+TlGlN54xsoVoKnxuWwxOuyYzyhMXb3STstsn8XU4r91rtzCEZDOg02t3bDzK0lM7ZJOZz", - "p1ds0iD8DvIxtGMIT1snghJGt3lgl4rNVvv7Ce8T7ESxGVNMRJ/cXabkBRPUPan+A/vt/V87pcJ1x2lb", - "d3AxT8riH/u933OWs0kmNbcjXGET7guQES41wRrhMeOneLsTRWlD1frzgSU+w0kshc2Na+NUPyBv0fnG", - "Kq+hTJOhIzsqBJwKF2jl289A0gqILFIYJgKr80LOScKFfQYxYdxeoLC3zNgPiZxv9z7bOhTLv3r4YdzX", - "YF72h5bW4Fu/eBUkcl5dzQWjykxZbTFb7lXXUDm61uU/qR2fxgVKNZus5yAnXAgWo87dHWxbEmTr4NsH", - "T9E5N5MLuIFDZw6H9Qs3xJVobSqR0fmMJ2yyoHphR0zjmFuF60ltJgERsmbMoKgk8A2iaIOP6tOfH+/d", - "u09cB4E1dNpfKLA6k0ptaN6WJYaqKU2SIG20k9vV7+hVCglTQKnwbbt7Cgr0hGk5Xc/tpnu853ph/0Le", - "DaPCu6/X70VAXgn8vUK4/d77AbQ6uKAKVx2arwzrv3xPld9+LDut/HpS9F/58ZUbSuWnn/yoKr89KQfo", - "V4Wp1fPgnmk46bBUKbkwAy5wC0AAd5qnqd284pXlNBlEioqEx1QXdr5OsHqyIlKRrVc/Pdnf33+03V24", - "4vp8ovkHNplP18rNbtBOUIZqBKoRLsicz+l0aZgekidUCGnAGBktqJizmNCZgVpurLUH+L3RJiNMq5DE", - "VJuIZD4c0EcP37+n5tF9fqkffUinav73fdpr0y7mumV9UQNH3W5CUaKoCGzi9S7qlL6Hl7+lkpDx89ia", - "q0hRyO2Bbg5hSMoH/Yy/hzU3ZLe20sHFDb/+ChOWkGKQC/57jrudJXS5+gY0jKYDpIjQ8lq+YQ+Q3nx8", - "NJwfW8fP9JJyAyensJm6KfeJTGK4d2ZcadPrbg27kmDA1OcSkWqnLCwxrVJEcwFbL/VinK1nJeEzFi2j", - "hK0yd8++M5U7dWvM4M7BPwsFYlc2zlSAE/sfT4ouaj8/Lfur/f7Mdv6x33tCMzrlCfecuc6nnU1+4hR/", - "m3ap2thTW/WVq/mx35sxanLFAmt5iv4BxOqJBwm7YAlxpYF4D8njkyOiczVDkzlSqy0LanupmR4LxeZU", - "xQnTSOqXCx4tCnUlqB40oaJ0CyCPk0u61CRTTDNhDsdiXGjI9bjXJ2P7+HJ/Wyqxf1vXEF9IzBXT7h+g", - "axj3+tBWIud63BuSn6U2A3guO1nqkIydPkaPe2TL/kkyqrVZKJnPF2ORUkHnLGXC9Ak6aaAaGDhTstwm", - "VMRk3FNSM2PogKV5grwfGnucZQkbi1Oe8EgKktLo5amre8nNgryylSqXJ84YLus+yZzqWdOUjQW1alDY", - "ySXJlJwyktBcRAumCRMzqSK2PSQnTA38Evu9JVQx2AyrxCW5gCPCaLkZ+rd3hAmjnKdDd+YCM7kK/cHS", - "F49ofZWaR7bGx8I6cpXKv7oqwKrcjINvFLUstekVerY3YqHq99dRzRLQHwtcUdxVDqUuBalt2SyhcySV", - "yiGyxKjHIpJpig891wvR7IIpmhQ7BHecRH0C3A3uRhqLKJF5PFgsM6YuuJaqDzcEi9AZQfXJ7yzN+9gp", - "/DVIeaTkRUqkGIsEyHiHpvH9g+0heUwSroE4/Py5JW5HYsgJUB/AYbTjHvU6+XHPToxrAhf/kDwTRnFL", - "c0Rbgpvam7RBXF03r8KsVu423J+rNHZqa6xYOOzPjqD7Kyy2QjclARZkXOGioTtrDQNelcH9uq43Rrnx", - "FbsVUeF2qkadh4QbQrOMUaUJF2NRUpOIkUpdpUwxXA7Njdu6lBmylVKD7AW0tmOBHOI7TWq7vz0kKDda", - "/7CSI0EVoliytANiY+HHfMmTBBWHJEcPBuRw9qChZqziQYGHiYqC9yWltcsZv5xdtDaB2hGxff/Xs+M3", - "RC+1YSmZckHVcptwXfJaS5qrhrmwyOh2D1/uKKrVZx1L9HqCukSKugDZPK8bTQhOBC3pYhOF/SxDLpZU", - "RYuQG5025MnJGwKfuWEREHFtvFSl94OmJKlbmoNeKQqxdr1rzSHT2ThnCUcKR7xpskfFPRIWkQpz/Kq1", - "C2pW7LgsYRHwKutPY99vjDjjMpEpsnTBVmez07pEvnG9qXdNtqTegQlvV24WONMqF0PyUhAUI0hNihgL", - "LMpFlOQx06TCz0mpvvXyRc0NqiD7sfjX//v/WVkiJhecFtIGecuVAR+SD/YVO1M0ZcD06ldatGDR+Vg0", - "xRACrWpJqL9UimXmunqhKD5fGCLk5XAsjuyYigFvaTkzl1SxPINnJhkM3KgHTs7atk5c2BA3DRfN3xp7", - "U1mcK/oS1sx5xYb2VylsE6n+WootdVqdU8MuaeBB/BxIb3DBNbqsIU24suToZEjwM5oSooX9ClwUpFiy", - "ZamCi/lY4BcnEm8TJ9FaMnOmySF5mXJTEL+QxfXibrqxAAfDKWMCrYjJhXVALkl1yZobAN6cu2DgHe6G", - "jsYcxj4xcjLPg3pXf9fFXLHIkLfHAyMHb4+JUXQ24xHQUcaUG7W9YJqDDvJz8DFIWlbaV/QuFMkQHhmK", - "x3MQcmC5nOhvf7OXU8YUjOv14xO44HD1vyfjnqDG1ik5/QAsOTxmMfn18Wu4WqOXp9tu1dzT1Dbcg2sn", - "pATv93Q+Fcy0jd9+RR/c8FagK+0m5muXaGWLNpH3p0k0cPHbBr5bI5OUHAykExBiCxIckhc1+dVWKBxX", - "QARdleiLFvwh/p4Uo/0BBRqSMio0oU7Wr4yIcD0WXvqoCRwgaRO3olBV5cJyKXsfDircF8USK4+gRFY4", - "6V/A/w8utIzOnYuHfWUWnFbQFE93delWJCtuOgk5G1UAlQdLYEl97zXBE94IWtBML6TBV7X/x45i2kjF", - "tuFx7vyl4HtGc83IX0nKUqmWY+HL9+05S6ixPgxF3Rkwph7ZihIpWFWPYB0I6FhoI7OMxc4w8T2ZucNN", - "CydCqQpdtS00Fq4LvWMlAE2KWeAGlBaYZFlucmUm26hqwNkU04I556kd9kKaLMnnAzfPHtlK+AVz04YJ", - "8g/MtgFWFCnFwPlDQtFSd4KfiKFqzoxTPWtsH2lm3BuLLTy3BP8NhpU0FzzC2xzLzbN8UFVx9MjW85M3", - "O6XP0lhUPmMVUOoNuBwkPOXA3rbgB3K085IouKHxZy7mdvRY2E6nmCSWd1Mckrc0yd05RWqH9R0LL/3j", - "SAdG5WYxy5NDcvHBCWH1/Sh2spCKdvf7TtOUoIapVZJ5e0w0vWCeILEWPKm1gWNE4wumDNdOG3tOthw5", - "DZydC+lOE8FYTIQci2pbcJCRTGGITZGfWEWMJo1FGotSbGt4kvurzZsMC1WEF3y6a2zWv2habAyf/mq5", - "whP5tHjUNy6SjLdbVUEXiQeFRuUyVd6vjLzMmIBSsYzylAlDuJhJX7TuRTYa7rV5prV0f1q9Voru9UIq", - "Q+bcEMVAjJPCcbNxbxBzZeD463w24+/xFZmjFsjKNNiS7gOHGvdycS7kpRj36sN8cJ/uzeLp/sbd8MPu", - "19YwuA/I9JxWujVeb70N7clagxmImsRIeDtXX/6Owj+DuawMSYJLTw3grCScxRXa7ltv3TkTDDhXbH1A", - "DfESWovJZxAUZW/cqnQlS0zLjlrP2dYN3TCHeSKneOH5iZROp0NyVF+7PqH4AQXm+ioX0RIVx1CyxYbz", - "IdwuWcQH4Bk6oHuD0WgwalJ7cjCwLu8ZNYYpGOD/8xsdfHg8+J/R4NG78s/JcPDur/8RfJB39Fb1dm03", - "zy1/lvrED7bqwtoc6Cb31mtb1qrDbz+8qFpo3elIMTwGNNk4BvA4eFIp3k7rT45Wvd7sSsUS9NBDLncS", - "PlVULXfEnIv3hyDOaVNft/Vl1ylYAsExVjDyBQjVxGlYftu5oIpTYd7ZMZJxVTkA8hvBWwPu/ac4IjIY", - "+HYc+/Ke9zUDwJD8Kg2hziINT7pDLLTa71zRNKWo7bygCY/xcDgzm+Yxs8pZgd/IhZWSFDO5EuRgNPJf", - "JpVBgdwgtZ/KuOcUpypaEKcQUgRVITZCJmIZdGlkTBthSpWlWD1pcMz+ulP8teX/HL776/b//o/PSupI", - "aGto3KozrsjPGn7eUIhsJfKSqYhqRhIGs9V9EvM5N9oKbzG6RqFC7XsSWWcP67ooFWHCq6ixXJ2c0+WA", - "ZnzgNC/W3P2CiblZ9A7v7wcXd8v9MXj3F/9Ty8KqPAm9117JHDWv+LkaeO3H0Mn44lc3T9DoknJxZKvt", - "roqTn7bDfiLrdto+6lq3Gu6YIipow0BWI/DB4ygNCREXTCk4ipavPTl+Ck+Yc+YYGrwayTgfjfYjLIB/", - "MvcLCHHwlMHftusSj20uJOz81mPRQqIbZJLIqygp+02m3uYluNZGFlrtkvuv+hSiEd2axdHZ3RUkUyXP", - "mUKTDCwxZ5qcs6VVF8xrCk0mLsgFVd40+BrDErCI/4TWIIbPY3jEYnA+xLIy5diie7PZMChGk5JbphTM", - "Xfiu1JHMGDBUVwynRs6YuDgjKc3ss14xqzRPqWGKw3OxVCymLObAo8eC4cEgGYUzH0VSgYoV9hZ1sOUq", - "fKfJmX0mnmHzZ1wAWZ/Zg9lQlf7Re/nm9Y8v3/z6dPLy5Nmvj48mvzz7b/jZVuod/vZHz4JpFI5XPzKq", - "mCL/8QfO96N1nkWfPgh1X0jlXrveRI/+KjTjQ5kxQfkwkmmv3/tL9Z/vPr7z7p42cu+idxgc2MegmOl0", - "nwEffB8ApYlUpecpRVwQZFFe7VBRNzRU+Vbyu9KRwCcKl5NppoOxKU29RSmH7o5Gxz/uOO+Re/4f20Pi", - "TLc4fOBBUjnxWC+AfApcDrSnJYmMXITGrC0E33cVYvBoa80kF2YjcyqLrvKowaD8egVWtDPlYkfDNgyi", - "q6070s21HZWfiQuupMDnOYpL04Tp+ln59eXTZ5Nnv77tHcJFEOcYR9jr905evnrdO+ztj0ajXohA2fuM", - "K6aDjo+Pp1omuWEEC5U+pkNynGt07nTMY5ajHhTlM03SHBRLyZKw91GSa+BTKA0Yk4Tijx8O9nZf7z48", - "HI0OR6P/6eywCqS/gXk/P3ljI9TxvDOamMUETXObKv6MZZ9g0Y99r5vEp31glYo99dpKOL2uDtla1KUr", - "OzmCBDnu7T//0R6o3efjXoUQXShw0YptuLF6z38MrUtFG7Uq6RXfbISbiD3Vr4LRWH19rp2aqGoJHwiK", - "EDtQMqYqJlNJVfx93ZcGa4Ii/ztNUFtBE2cnsqI31rGypGQa38rex6dUtup+YwlQ85NaVSIVhKGl9IID", - "yxmkKZe+Xp0tOWVKTYc6rBiYAnq8irtQr9+Dmbn/+An2+r2LD/WY6s3qQOca0+mtuOERSJOMC7bmFfiF", - "CPugU04kjQe7n1nWr/i7NWBo7If6dhePU0/gK+hOUyriSx6bxQQi4mHIgRvSfSFF4eKafA8zocm//vHP", - "t8elQmT3+TRzd+bu3r1PvDMbtyQ0HYy1KSaSZ+FpvMnCk3h7/K9//NPP5G4nYUXKa4nnbv+f2RaaADUO", - "t8t647fYXQs5rIzPlS229A0YBqErV14wldBly3XyN8WtddEyS1e4ojdef6PsjvyVcm/0HNU2fszwpaKo", - "QbOQNwk5o401r9iYYTKV0mC3zUs7fO98mtJp6KYbSWEoF0zZF09hV3WXhn2TpPcPhuTZ+4wpDiIRTUhM", - "1SUX1q3F13SvakIFQP4kXLABT52NFidv19RyXWgYTGnepUhbl6JVvTyqo2PibkBbWaqGyusmdUeKIeud", - "ZAVW3roD8sqWLp/1YaL7kerC1tqFyAoa2907dn/uPV9hCrvPf4SKnpId//V7YFgKC8aqUhPuSycxR5/z", - "zLpgTOi8iBhdB8p3es4z99TGGt4BwF4PcQ4tI9EPx8KC7sCJxjGDs7w3qxrw79fWpwAuQrxqVsUneLiX", - "l40tbi2oKhd9Ms0NvMalYcTpRVz0P4wFC08ZyUv/zsbb2E1w1WcBl+WcKcGSiX376o4rYysRV6l1cXCq", - "M6odSJgyeVZfr6e/HJ+SradLQVMekV9sq8cyzsFRzx7m7frq9cciU4i8Ap0AEXPXr5wRmZuBnA2MYswP", - "McXGChOJw5+4eH7yxiGYaPAfesVgYZmInS+ql0MKZ1TxnbFW6lqz1f4bi96GUuMt7x2P5KkrXp7J7srC", - "fs+YJIQ7l1h+BPE9NsQBbEoF9fmouz7hgjyXJUyCPdZD8kbDsR6BR4pU6FBAptIsqg+/GWdJrKuPkxLd", - "siwGT8Pw6698aDbO98EiaFKOsnwtxOCF9V8A6aIh7e91YToglCD8UQvfCcK6uHCa1WFZINeqBscJDMUW", - "UFOHvuiqdLYtI+zqRj9Mpzu2T4zOuuOKNnNFi+xVbX90DE9Y3/6R8ANZqzsvtW2f0NepbaS5RK7tvp/Z", - "NVbpqFiThsb98yzPY13RTnZCSLWyh5UrNNk6A4Wmo2NQYZ71ydlfaj8Ae/RyDQjml8SuBrJcAT9V29cr", - "PsQbNKbX9iOm+vr78Vi34luRi11iFBXaerQvaMaG5Ge85zw/gIuHV534hLz8nkj7GvBVxwKGputejV5v", - "rvlcgJMXuqdPGZjMrXLd6ca8p4sOAQt7jdeK5Oy5FV5ZCOPr/NrJmVdyn9XZ6aoKfFVD5HTiK6oB/N2q", - "BFBpYnbS3ED3lkU666/MjYULc1OvQ7k1FO2bXHDcWAql3zX2/7RgF0205ovA/QGTk7PSMlIx0bRZQpws", - "F7bSnLMlbrm3yNAVm0zVGBM2mTjncSeZVKqils5eJVxXLDnWJuPMMOjmWz+iQQPFpq2A9eq8/PU3djDi", - "ZeAnW1KMezZ7JFDPhWBytj9U72mGi49v9kMbQHXWt0oGhpo8IoBYEu8F32weHYDRXfDM/TJ0rZ0V4Xif", - "A927jLOobi2p7KyXjLEZL1kZFvfrsgHgQOvNk7KOoIsWA6Nil4p7RuahJDtKsC4+JXXvqE/TuDyrNLbG", - "9aprE6vxqHZ9K2N29GQhlH24ht0PtDTVsIWbEPsxq+14z2K01Ls8o0lyRrZcoW2i2N8Rf9XtFfiyFdsP", - "eLKOBArHrrfH/bGwXOBsYUw2gf/TEzjFZ83GXF1/wkv484cjfIIeHOzXoyPsgBvN1hXTQdyZ9q0BD6cb", - "8pRqUX97nHn8XNxAgC1Dtgydw9FxAENwT2x/qmqcC82iXLHNZ/0xikcO5AbHlFAuBj+/fn1CFJtzjUG9", - "wfAB64Tegobm3v6lY5ifrpF1/eje/sG9+w8ePhrG52rIIjXM9YBRbQa7Q5rSD1LQSw1i1066pFl2eLG7", - "kc/bLSjG187y/UO1nRRkCizZu/52efQ+KauUprlzLuKuDfwCZTe7l/o3+U0bODLFBnk2VzRmn9m8cW0P", - "IlzN9l3dkHNkvetwrg3oFUqw3a31HsP1xbqQySCmhl7FYdgOd9U3OF3apuyTPNReqyP2adP7uoaQE8w8", - "8KnuXH4soW1pQ++1igQWT4wMBMp62eHoKay5L9sJphTBho2cXMy43BROD0OrB8lida/egCYGWcSd4s2b", - "E3ihfbe6t7fHVS+W4VgMUAo7JE+LDopmiyYpPjFobDXDW1LVInVRIT8FkBHy9nhIXhej/U575bwdEyli", - "NHM03aFQNLAiWXUAOYa0cNOs7rSMFvl4G511pPs2LPKjoD6z0Ieh+mvKG/OxEdS4Ue4uqYK8dNTvrsMB", - "e2Xvo3VYYDXfi3uD0e5g997r3dFVfS9uANw51NbjOm9xXvZV7vPkzdHTPafF3L42/Ndnh38Os7KnZXgA", - "2co1q4S+ShUMCqj43rc4/V/bl/+GXPNLoMt1Ze1KvIaSN4FrHULewiL9ayBPNxnmRnjTyuRWFTrLDNUP", - "5SmpCPJ2R7OIB2OcwTnvR8XoOeL1r97biPgwsfdZ2LMPYTmmS4ShQiWXktLMNCnQWkrl9+7Bg4OH+/cP", - "Ho5GAeTkVYKXEZ+gZbTTAMDZJaHLAkRwCx0GYjJN5LRO6Pf27z98MHq0u9d1HNZa3m0dive3r0W23Ir8", - "1edU819qg9rbe3B/f39/dP/+3kGnUdnGug3Kla2LJA/2HxzsPtw76LQKoXfdM49k3RTgQ2kVHttMRPCv", - "gc5YxGc8IogAR6ACxugvwJzurcP1MzmlsU8+Eb47DOWJXutHZztzJa2+Nc0Tw7OE2W+4IZ1MEjjzp9hS", - "MJuXEEWyj6u15PC/N3pY+bkURVCcidk0n88duF6xdMcubr4UnjhL4kNSPNnW8jnczXJg79rowM2hIzW8", - "kJcFzF2FCOzVZTEhFCMFndhNq83KB+ZwkeVBkmhdyp9yZeEusFFCpzI31iDvsrOUnVjkGIEG6lzE3dQd", - "P0l1vjGSAm7iIglNR4WBD6m3IfeNYPqq0FcYzq17gfuuyStbwyoKy5+zvJ6Br489OYWi8DgAcaE3ds10", - "lS7DcgvqzL37nO2v5J235Ds4mFl3q8/7wlZzhsmXzEaJBSjF+jGdYvHOgVlQcaMipcO6C3Z5G4uOoYUD", - "INsBqFBuZsXXuSgXuoayEJHOsR+QMGHmcQXIqXHSTj22hVvy4Vi4yKbiJ20NaVDRroNZMK6IVHzO6x3X", - "9ay36CR8Fcr0xHVt6qxWbPPd04UZMMQDLLTzzPrcWu/BKrKs25Nev3daZKlyjKm+NP7HwIqUrvyrID/o", - "yXElV99MyRkPIe+gE5H76h5q3gn2xcHodLD7X9ZNH8gPJTYurOMR2BGGjUxDWL7bRfT85M1J25iKNE+k", - "OrqVOa3BE1p1mpnlwiaaquhUqEsA6SDeLACQ7WxIfgTbJ5vNpDKHtoQ1SJVEkNIlUSzOgR6qWGhSELCY", - "4hktegWBH36BCSHyyDSfzZiqwxw8CsnSldKTNKDc+wm+E1ugwFk4/rEuT+8dhJoOc+GTGjXgs31GIzD1", - "d97ugEKwMY1NQIbPT968cnmK2iCeYSmLXEYW3XlIfi0yecFS6xLGaRjQFDboyXpsQ4GJTmTIG+UnxRjB", - "b2UGUTh7aMG2oMCF1wCNERRKCp/HEFzmHUTf7zlVVBguWEze/qS/JyNnMn/7E7FhO44BcV31iay/FB+E", - "trQ1/O5ksdSgqLPLYnOJcFHVUuKR7vzMOCkrOn1u4LERzmLn2QfZgtyFuICnrwZHL9/upDG76NfGBB8v", - "FzJhMO7tCo+98BgQRdk6a71oUxf5rSqSKk7KRI7Ng4Dw006P2aQO2JsRmbKIek/p5uZFRd7jKfpeoNYL", - "bNaGCaKoUzpTUWvEovRpMsuTZOgYoB8yQTXzDPlwBTLVXlVAPvg9o/iSKsSo9wY7ZUTnUcRY3HBfaY66", - "Qm+HJBcpVXpBk+bkDgtcgmhBFY0MU+S799/1womLYHC6K6OvUGdx03Qmy8q9EqDHyrlrO+Jvf6ofT3vV", - "22hkpyDnghtEV0NI7a1ikBYEfEiO6dLpUTK8Mx29lDtGZzOLMVrcJYoljGpMOK7J2582O2YaaWjSNofX", - "8NHxqS2YEJwiGGafZDU2AL9Xwb2q3d4PXhnlfAJmJuywabMYkiMRZnvW+wfh+SzSltZ8LuBtp50DUERF", - "sZIre1e/O4NqoSacYa++cLXp9AMXQIhiQreVDZ9cfeasJkCV53U2Jc83J920jbT3+8RHeDbsWu4t6c8t", - "BoJ6TyRiqzqOYbWjmmVUUcOSpX1cFNRZT6LAVp1KwJP/CiGmz6D4R5s4J1dsYhaKaXAAq2kb9vuryZc1", - "RgxcMJdvzs6pYnszkqRUneMp829pkgu7AvVw0v1NmFILY7IrTAo8JKyCzTB1QZPaXHq7I73iZPOUQZzT", - "lJlLxoSfCtWEVj3Wm4HiuiXnFwTIMMVlfQ17+4F+T20UA5krGjFia/nU7QVbg6PXdSl7+y3DggtH65b9", - "3V23v67qLE+67XFoWLubdthEV9ng109OfO6qIje4X+a91VWG/Av2yLkaG7Z2T6/Poua7EhZ8ekVmnzJM", - "q+YCuKrB397FENPGQfVawHWFOehqiLDrB0+BXaq+PefvOj3wmsc95EsDUUChHOv4wYW+zRGjDwMNVI6v", - "Lx5bScneyZX72UU5gaTFBcAZ1524o1wlvX5vMHOzOtzZAX6fIEzpwf7uw531ntxrXfidx+Ik5utUPK6U", - "83zzMWAuCBEmXSeJHZplHZTgWHPT/YDsaWUTbF51uNsqjyxHaWiCangivqeR8ZkgUSte83qg1WOLYOJ1", - "QQYaTG1yJ3jtPqqez1HQDQV8Suvkv7NC+zAz61YMNGI1jo11tET+ISidShVK9yeVcVG+U+b9jov70Hv1", - "etjsmn189LA6y/v37u3f28SHkNk0jrk7d4Gp2tJ1HANWWW53fl0D8B9dlzncl/VkheuygaZePwmQ1PoF", - "lRkTV1rPewf7e1dbz64TOfKemQ2+FIKBAtwnlInKcN2UGRpTQ+tMBtXJwGUwnxRlKQb1zb5fz1paXJiq", - "uE7XTYb3uRxgWvJivvIw8CkVfIaPJFuy2rNe0L179w9t9t6YzQ7u3R8Oh1dFu3lWfOu2FTvWT7YCfDPU", - "i0/bhxsAtekylz96J49f/wyMLNfKXlo7esrFYeXfxT/LD/iH/eeUizAYTpeEz3y2kui5tr1Z7tDGWXxI", - "ypz+Xu7p4qLXYg/CAAFE4ApCS9YcpW8OQ7KgcV5PFXIFLMk18fEgroB6vXVtrp27WRS6UFPJ2VzVK3TI", - "3wx+nOvcOLzGGcu4Pq2mo0htverccq3k5Hpt/taV9H4ZE0XG1iSxf0VSIJh3KH1r7Yr03zpk/8NrxKX5", - "K7qs/lj0XvnxSXUgld999sDKTy6P67sreqWtFUj/tiqHbuZCXhzdcJjD6v/iVuiaM9thTwbDMe74LryO", - "52a995fz//z9/+iTB3/f/f3F27f/ffH8P5/+yv/7bXLy8pNwjdbDjt4pduhngwtFd8UaZmhXUjoGy8wq", - "PcHzr2WF3RercTDRArP8kik7BH/qF9wwZXNGNuKPIc0Bw5cS1iJSYHI/hzSxDZVPrFMBVP7Di8Efm23E", - "DlJCuQ0p8IV0Po1lSrnYHouxcG0RPxGN7wL4KyYRzWziRy5AMQte2gqzdDorb9l5n/xBs+zjNmQSoYaw", - "9xZBP6OY6HBW2CaE99Fwo7Ke6K44K2CKbVDwWBS3dYG7aU39wzJJB2dJM2yvZVHWv9/cy+nhKIRQiiFl", - "sJGYiAqfIAVlAxkVsW7k4Wh79T234Y1R0NAa8nO+MDbmGGKGA6TYEqf8lMU8Qr7iQ3VtyUqUtKU0p8TL", - "lHy/xL15ZeNHY0Jzs2DCOC/HPomkPOesj1vaR4s0Ol9hTetSs5DZYLocLGRWQKRQZR3OqHVKqT+y/8/A", - "TXTwlik+cz0FkS6ARAJCJx4ZNzMsU2oXVib22ubwEZqjutAWt8mgtM3/Y6MpTK6ET1HDMFEtIvoU1MdC", - "Mvn3JEo4ap30Am14CwTZNNBMCCezNyqi1ug0itms+e+6V9Hevfv4gvX/3t/rHDRul24dleVJ4E2betbX", - "gWNbNokDsOLBxCvCN/gBwg3oXClQT2Ek/veU+IbKE1cwEmvps2Gs2mU8SXQlgHU7GGnojoED/JnQ3HSd", - "3eoprAUudmjBOghhtaQD1NAznAd5/eKUGKZSD2GxFcHu4CmxuDIDrnXuku89fnL8bHvYC4Kr1UxauFVr", - "Axvrgw7Andhz0OoHVepoaMr65OgpBqy7a6V8i2GE0U9SkcTeiuVldIhQO3V1D7VpO4+eOgE0WZZeR1Zs", - "Gfe2fYtZ83o7JK98t4QWQylCj0va8k2Wlwk263xQbfjTSuuNSHW0j7nnn7uPMdgJc5paXowAsa33V3ed", - "o1tx+NjUkF35QqpUbNd/Vfb+c0Opf34Zff9qMrqzQk+yBdUh6l5UrZpYaGXfq4bsOrtX7W57tX4nmgfP", - "1t98ZqfKNWQY3HO16iHs4HuD3d3XuwdXV99dFQW7DodXgcYsgLC7I1jfBBJ0ANeZm0lrfAeBzy6aw+tF", - "3h5DqCSgosHHhnZkd/9BF6UE9to1MqLyEURqHFLBpTwqXuHRb1EGz3mSWAFG87mgCXlEtk6Pnv9y9OLF", - "NhmQly+Pm1uxrkZwf64BiI23AK6jdY4KAKPVUDlIEb77+vUL+BIlDCOgrBx+fn2Y7I2qxQ6w2X5wz0/e", - "QI0F1RPvO90eLkzLkHv2nmujVzERO4UgfApMt61aKsa6TNK24ZwqN2J9/1wDow5CZW7fAEi3jx9ZWc47", - "wK++yzjdLw87ey3a9adCVjs9ww0hVrdeaSG05wYoyb222+362NM3MpwaplOIbVUlHA+icG2w536PBwLI", - "Hzs/PnJ0UmZ3K40RvvnGnB7tDXfvP8QExbujLow9pdGavo8fP+ne+WjP3jKHdHoYxYf4YL+uzcoRtn2C", - "0OQSwmzH/pE47tkLs/K6rRxbW6ZbAEVHTO2NaNpNMa4F6xrFWe+4pCfp2vxCHSKEm7iFae4AXVOeJFyz", - "SIpY12XkBdVEZxbH2ObZKST4scAB9kmR7hylFEKjSOWl6tFJ107ezzNH95jrN5NCM5fs4xe21CTlaAQt", - "ukfXR02KQLR4LLaUD1osohMxzW8MP2AIUN8Fl8R99BrGfEJQYSz0IjfAxLaHBPze8pQpp5UlU44Wo22i", - "c/ukxfHiaiyBYWoeMzUWUCyAlPxH8Tw5vA8wB/1e8ZKDnCijUYiabtT4OXRY5tiCRsBN4bLQobe7ygXJ", - "RcxUkfOfWXJoushd0XD6icDhvno38cpVL+Wq6+GOdxKmPhl/GYfa8j5H79BrPM7vXV9E7xT25+VXF/Dn", - "ak2u4sHgQjLgxTeF284q5Fjs1JCauViPIvbmjc23W5+6cz02EozqakneHh/X3B5chEa3iSOXaNkHmV1p", - "G/Y26Eg2juYq5uUroVV/Osx0U1KpSIifHVS6amH0OASWQmuKrdrDeZ1+DV6VwRgoLuw+AdGsmWAj5bJF", - "bSkdI6ut6qWeOYhVJ0r7mAqHFl8gsJRGfb3UO9Nc72QR37Hf9Q7C4zxEeJyDIH5BzC4meR56GsEn1w95", - "8wYwbPAXhHe2ATK17im9v/tw9PDR4OF09/7gIB7tDuju/v3B3j06mu1HD/Z39/bXBBV1CGi9foxq8MUc", - "8GMuvNYn3ns+5NTcFrvQkE2cP/YlF7G8rF1/QQfZau/O+XZT96uu9Z2HEAwJSqg2Vn3RwsmO8ZJnEbSN", - "5XzQdJFeLazovP96tPup2h8cXMsd8VrlwppVsUBpQkgrA65uVn2c12P5OCAf+LJptaqdd1+00eG9R4f3", - "PnXRfPDGpjE2yekWN7fNI8zDoDeiQ3yQcEWP5BWUPScTWa2+Lb7s9Xu5qP6NwkDDl7r43CmIq+3A9sNs", - "ZN210oJfcFR7r6CnioXBjA9BUvHvEczHUqBkgAj0BHRhpKKLs6iAaIc7qrxdoBk0izkVnQ1ctsEYGBnJ", - "tYvn4wIYMdofoREHdnBInmNZ/ERT+6xzg7D5rqumNxovrb8MnC/ftX1krR/yqXtfYR14bBH4F04blsGp", - "bNc3YaWzQ/KrxDrFa0/Ipu7XFsdn1mrxRlmy5RDzPXgNduZEzUPyUyFeFgKqE0i3NHN/ThzDKjGjtmvI", - "HW7HwZGwsnMVFIp+z65or9/zCwV/BnAr3pRUv3L+qqQYciRjNMGzXIbp54YnDikfZ8K14ZF2wSOwuW1i", - "j4vNZPHEPp7afFJt6KstQ4pKXqp6e0y2EAz1r8QptuFf24X/au2u23t08Oj+g71H9ztBnpUD3CwaP0Fk", - "gtXBbZSToyyfON1I29QhzQuueWS1CoXvC1i7SjLPlATWAzP3DVY7fzR8VEV6i2UOfLgYkoOFtMDSuGFB", - "MMOCF7X4Qf7Okws+m4nfP0Tne39XPN19f1/vTXdbEKxtR2G121HVuWBFR82mA5sRMwzGhQSldCte3Sum", - "cQbklBmC9DMgNMLnTRFP7Ugu17UVDxLWwf7+/sMH9/Y60ZUbXeXgTFAJF7iU3QjKkgRLkq1Xp6dkp0Jw", - "tk0P65IFYA4a54y4vOajukA63B3th6ik5eIuqca1fZG2Lvlb93x0k3KLjsHbxdNy5ZQHV3t/f/Tg4N7D", - "e92OsVMPT9T79RzGlbP9+VwY1Z3fQmny9eMTAq2rGY3quh3vIXalUZkrjQrzuNj8C1cY2MMH9yHmarcb", - "8GLI6cRBitYObJ13BQ5dgCgCuxFYilXW22+7LULilCWwVyxKKE8fRz7EonH72DwLE2WLlZvQ5WJwL/CV", - "i6tD3U7KrUJlZQN0sA6RiuSiyO4z3GyS/SyW1Xauba+HzVw9FJYjYPUcQphNgHiNpcwUu+Ay15+hIWls", - "zOwskVJdqW7bg+UV02B3Q50N1+Tt8XeaOFoj2rCs/oZy1LgGR+2ak7vSea6RSJjI2xar02502fp1E+63", - "nNr+OkCNGjdoRS+MgXPlYrPz5xOaRDnms6LFfsKsEIYPIQGyLFla3/4kkVKQaEEFGklUBXSMAHbCMOgJ", - "C18ms6BXhbwkibS465Bvx6V6soOAaiDCgD5oq5of0ZJSI63pvdQyGZfMp06N99Jw0tYwplIRCg/rSY2s", - "QILbKjVNaCLnGh+FBqMWhs1MFBlVNhiBCpu67CK1b8mAw3VgiA1mHrpR8SMcTPvAdSIHBprblaSRkroC", - "B/b2WA+7xrwVUcybHTrrg+1AutagGbjKLCBd5wyHofsxEM/zKTck0jDGDK5xlfTKyZSKHJM/VQjZKeKH", - "nd0hQRs9KWCurjhYbSaY0QUUeIXlqoi6L/RBvkzwXvSs7TrL5fyOr1V7harCTbUNsJ2nBlc0vFr9ggZD", - "ZLyKK7cWyq4E+GuCWV0FL7JMAcI1tsoryIFkC2NeKmypggK53cVJJvxkhX5WXqsuty/YKboiK64HUjyh", - "ZnEkZnKVE1zFcuo10c5bNWPKgxvGTHAW+7dkYUJ1qi6uXSq4OGdu5bDbGvwdujiAzUp4HRmE+NR5fbPD", - "LuphO4b1CV+wX1eww05yHQ6ofa1yXCskICCeMrS2kxMo15OwOWu1YcXmeUIVAv51HLJepgkX511a18t0", - "KhMeEajQtIvPJAR+TeCT/gHnst1pdlBh0pYd7NQOzsUF2g1p9FtO4QeY5XYjKhk1MTu2/g7U7+Q9FvQU", - "/4knzGFTvhH8fYXQ6xkRDvZGbdHyLY3W4uRXwVmvyrkdyQZPfK4DsYVrpRyf14zFLk0Flsb8bHjUOtxK", - "HgfZmwCvZ9GpB458GjTJE8uvG8AkZMow7sdPbaW5Lmyxy1SC2V0g4d3f5bSuEO3q9hvIGbglSogMxWZB", - "/37c0bUKaVtiZU0qu3sVDApkqzBRrHRFaIdN2RVL/6o2fvJqJdHggrkl436ONulghyQ63v+jgC9wvXbH", - "MWimhAz4KyNKjTZLTK6MGa2WlTynkClMKQSBBwlHCj8bhF2Ryq61x72CaOwlUSylXIwFF4WSFMHUGBHs", - "gqlKlKxU8MiaQ4jG3ypPPPidpZlZunwMqDz/ThMwofkxjkV1kNA4KL+H5LGwmkWVZ6aWsZXF7tUHhIJR", - "y2gEMwpTmHKzIDPF9KI691DaWpDxLqWKW/OBLYkvAtOgaGMlRp4zUWVlRTPBp6FtaGJrrXrx2ZzT+NW9", - "P0ktDTRppnle31+umQoLicWUiiKdXFcqR6XyOMEDCQNw6T3dX5bFFygoHTBPyub/yzdZ/nRSNF7/rVGs", - "gmvikb4fW7VtUAUbZfnq6jQsaRtdVTAMbh2azaotgWx5F2qfI0lvt+Qq6nRPdovEawYL+NHsaBbVez94", - "eO/B/Y7Joj7JWGfRuz63ae4iXWOSa9mp4y52n4f3Hj56tH9w79HelSwsPq6kZX/aYkuq+wPgHwYOa/Kv", - "f/zz7XHD6nMPfbBHVxpUnrUP6U3WYUBvj//1j3/6UV17QCFGswrS32K3b/XSSao76R0F6ia8bkayNe/7", - "xzUlAS3YDNliCMUNTjd23QblYBowIN2kYJrRiJtlgJHTS+vtXhRpYG13aL0x2JDIa9t2qKjAuXQ+LUqA", - "54kr8BdrG27QwsPOOed0Pm2zQ79s9orlSGm1qPo4dHAxsBQRVnAX87mkuubQHTNrd+gXEWar0Ta2xHrU", - "3WYoBAwFgXRrbUWhXAMNedJVqm5/YzsrdsuaWqe54u/WnMP2I3glrW/gRg4ofaMs79pQKbfDBXi9WpNp", - "NRvk2nSbtdSR5a179X47RA+vpiopbrCr91cJmLxKxQatWXp0Y3BLXrbdr5FECzVVYmECCjSZsEEleQGW", - "Jjq3FkE48w5mPhDBGZ3L2ayOdXuvHRsdYX8w2Mv3Qo2Bl0kftP5OZ9EE1rYYP+PePT3uEakgkDId9xpm", - "q2D4ZErfT1wHdWyX0Tqw8jL8vTFI7WcwTWR0bhMdYv78IRmRlFGhSS7w8Desaruj9dahfi+r7I1fR3zc", - "rSyl20wyZQt6wTGfirOpzGuOmOw9NxodRrGdQxJLi/ZUy/LsZgjFbHDjYTlpvHSoWLqGoUEoJ4X3aC3L", - "ooJvhrmlxQemZN+BFQDHfvnyuG8dGND10A6s5t/oJ2pH0Ov3yi4a6RXK38P+w2CswHE34frT1XWsxqSj", - "ZVUxzYx2+N0lOTSIAIKOhGni+KfdHnL1sLLVKykX6Ozn3D9gQX3vlkBIzCI8kXr1LNYJ/RrE3YgbcCsd", - "ChzYD5EwHoo1+XdeOYNwcwBW2VBJ0G7bqfp1WyvhRBuZ9eqnesLeR4zFTcDPcJGuvvKuZtBX/gV1GEFF", - "7nRXGv2dV2c3vLkALxxr22pXffqFFAM4dcWW2sNp8eY8Vk2d0KqrX4W0mITgVUMFOoxfsPfr1/pX9t4g", - "PnqcJyxeXdviSFpW5S6jTSt+7cjGtgMtFduYIfMGMkdaf/Nr5Y60VdldpI90pW4kZeTK7pwy48ueOjJq", - "3aF6opeaScs1QXyRuo+NJaU+cRc82U23GyR4sAhrRRwob8cYTdC3gPF9xt+vIR5bwD6M67Am5UHy1EAs", - "vuhWSt+TgweY+ks3xi74fGGSZd0B5yCApfRJeVUVM0x4RWGXnS9301dc9XZz21ltPSQcn1aggVZSmjiR", - "dLION/tJUcxr5zO6RC1Oq5Hwwf7BaLS/N7oWcLYf1hWW60lZxWUhrbfTFlJXqUdY1WJmOVC1hSLIejW1", - "tU1XV64m0UYxmh5i4E1GI0YSNkOQvCKgdbNlsdn1+sE7gcpF0Rb07zfK7Zu3wddT5hRdiXnlX1b1gXOt", - "YxBVv28wiLawmWgFUi8Qc7c/GN1/vbt/eO/+4e7uTYBdF4vUFu3x4MPu5YNkj84OkofLB7/vLh7M99L9", - "4DvsnIu4K63+AmVbvWzKS7KOZVRjaWTLzSFjqpmzvJnrX7OECzbQRYTU5jDFNbzA2t83nv+r6fntDNbK", - "Dqf1SVZFCGqIDlLW3eBvucmstV00Z3P0dP0srhWB1BxImN6aQ4FSHQeDGSp2e5+IzJCLjtfQm0rBzhfR", - "2qi4TVdRyMKOJz24yy0rHiLvBjBDZdbrLvDVSy6gO51Lxc0iXX9bFMUKGHH0m/6gTVzHexqSo7mQitV/", - "Ltzkqo8oqNzr95IPB/Uz437vjvzlEIgLAnRbXZUKOriRJeyCJetXAYuUDw9lPdmpYrgQP+wOdh+hH0Ly", - "4eCH0eBR3eOgb1eruny7vnTt11GXNaymAPSpo3YfXcnj2q/nOgr6hYcS2JX3ssMmdjRepof3V4cPuK1t", - "cPl5ZY8bSD6tAuinSnrucptUhaYYNLIhbPqKolY3Xo9VIiNTNudCd9Hb7o8Kxe09UNySxw4gHN+yRhb9", - "VJsHWqnSCU9TFnNqcFTKrIlg2Ouoi2u+Ja6Wm8TXCkhrw7C49mgzRMKmgKtN1+TwE+JxP+n12+3Fuw69", - "A/Vq/omKGGJYsE/4DHTQ9fZ9smr76MDASPRXO/RAbSXJOh6gSznQ60n6ZC4NKUPoO+rbctGuFyzGD5k1", - "87WYGZYg9j4LIEoBIMbXsa+jpyRTMs6jMn40wUGXiB8qF8OuQv7God2ofgMDs0HJsVm/0abQ6KafbNvv", - "hm4SCLZ9q3dHm7f6RpQi/V6exZt5mC3UjYNdCbl9QwhiQEVTX/aGJFiZzLsOHP1VdQVX37xWlxyBSJRn", - "3sACNLVKSQFzC5oYQn69T1nCDAs1QmQSl1ESXJdcdDNL3b3/cNFm4kSL1OpAfmEsI9RY/CPsLwVrXmhg", - "Pu1ocZdsjTzat7YGr4FNV+RWqz64Bxslsdatqqpw21IoWC7f0HlbvJTzivrbz6Ymmq0ioHiGXxPSXrWn", - "AHA1vdtbq/74JtRydymkvXSmhwZsqwcVLtDRff+lLzCIdXXiPQiZ50Nk8ZrOMVfeBjtKK6mE8kqikYMS", - "Q11yK0ywCf8kmEVUe8AWLyBgdR/R65INEm4IndNm5oi2tJTWEr3Zh6g90ctrOl+LydsMh62q3x8P/seq", - "28lkeLjzw1//78G7vwTV7g0FgmZqELMZelyds+XAZmECZcWwjkiLKSB62tC5OzOMpqhMQ7R3x5Wq4703", - "Krjn8learkwBXdUqKZR2N07or//R7uhVWcY3eGFsPLufnCHlJjLJGunv5a2Uqbl3qvcRddvDscAgvXO2", - "1KSSmG26rJ3Y73RRpeKKT84suQ+ZuDgjU46ZLvVYwPOeRhHL4Fnlcv1wHIuSyIYVo0m1HZcgzp8XZ5m1", - "jhUQPr0CZ/zyzesfX7759enk5cmzXx8fTX559t+9w55glwPbQzwA2ju4d98laa+u5G4wUcjV810MybGL", - "V3A+D7McX/YIWAY2UghBS5aEvY+SXIPbBC69Mcn1M1usRi1fP1PEJ0IhG5OE3DMcZDfAcMOY7b3qvIu4", - "9sTINWa3d1oeLsiq6GIJZ9zDK8VIL0kE84rAVoRXu9zY+qLfX3S7SyxIbOCwY+gvHLaAFvUF1wjc4Z1g", - "KoXJFobQ+BS89ot1Q70CaO3josGgC+ZnzrQ0evQ5sqG+WZv+9EImA3jntaSMCKrV7VoEQwiwKRua0Wuz", - "vsyngceM03HP+ZwGDC4hw8pnyVrqB7QxdGxl/1vTt4UDOp4282nYY2mXqpH/oaEt0WbQHu+RgnjfgkCM", - "EMs2iJdXnAxrze+kwuy47MIh4JBYIqr7urDt8pR5mMgBVtocjbz2gVmZWWUk7XtzLPPQtqxbIIjHBx6p", - "WGUjsAKLr7lkLkCpAxwN0hvJmCqdd11lFM/R7q7JVqEC80tQhF2v2gXW56E4pu+LHqAEvCaahlicR5kH", - "a/f5j5gz4JXbJWCfrgkcRuOVG0bIr1PRujXxVLW6GVWqWp23LR88eI5XreF+bWerQZxlHzXSDNHj3yg3", - "P0mF7+J28JcbB9rHyz9mCsHwmjD6nTDoecriiczN+vMPxUB6tFd+kR+2zC/sdQAUiTiqxTW38QIPT1KO", - "YXWlYTlYlCtulqA0ci+TKcaD+qS+uJDYEf5cdoyJVD9+RO35LBA585wJpniEaWrhPKZU4IsJYPvKbIU2", - "ceUKcC2KQC+fHDm9i8c+xnc0N0h63gH18ckRGOOZsrqH3mi4PxzhYc6YoBnvHfb2h7vDUQ9fVQuc4s4U", - "0uy7QHL3oi5e8Eexk4R+9IWgpqIpM1jjtwAkAjpguuL4BKHzyiMyo1y5V2SWyLggGA61Me+Cv1AP7a3c", - "t8veWXmM8dYYBsSyl25z36GgjGcHp7k3GjmEd+OuXwxispETO393brRlv52kOrdEgTQEK888L1sWS/+x", - "3zsY7V5pTOuGgmc31PEbQV00M8PX+b3R6OY7PRI2PtFFmzu/sOqJQ0KqnrXf3sGe6TxNqVr6BauuViZ1", - "m2DMQK/lytp3nNEkAlaByZIA0IbZ76ASpdaFW+UCc0z7ikCh9VNg2/abXKA1/Sjj5WdbwlofXkfxsc7O", - "4Lh8XKHnz0c7BRmvbqT75KHGLdXeAgH9SIsE6Xd2Ug5Gj26+U9D9JjyCaFVPwM4xm2v0fUoQON2DMElF", - "fs+loaSIa/iKjrSTWacFufXLq2jnDx5/tMc7YSF7wAlTKRU2SsSW2XDoV46ztc2Ux3ntreYJ/+hpz91U", - "Ho3IXlQoyNWPaPXaagqDq9fRQe+wrU87vfgOCf/gFk64m2yRI/cujxxmJgW78td0nJytcVoKIUFZ7jkz", - "XwrNj27zynLZFP6Ep+hrIeDnrJDwyt1auRR2MpUL+wAOSoCvyshNV++7uvD3uvxScRdCuwY0jXk9jDUw", - "03g5JH5N7aMflL7aAuurC+tuVj9fJzC8L+WE7d3GCcMZF5aib9fUt2tq3Sm31OKngAezcso76CCupIH4", - "8+kfrqx9+KZ76K576KR5EOzSlgY8yCFxrrmRjBnRC8hSCsZmC/zknXAMVcP5B0JVtOAXDNH9MFtdnhie", - "UYUuNikBK4G1obcqJtaqJYrmdqC5gXfILBe4Ceih2QRdYCZtQJylKyYXApS1VHuvmRJXcSWvuj37QQV7", - "0WB5NZLLhdSsADYUpnKbY5y3tq9jbHY4Fq8d4i0sIHqVe16jWYK4vWv0P1IQOhauwveehXiPOE3TknNR", - "heCJ3EJ02m1ZmSqOdKIjGQIdes0EFWagMxbxGXgf4rTO2dI5tgYb7JSACgbsxwl5Ybyj1t52GLgOXZfC", - "KMVPi2/EUVLdfiOkcam5SyOXx1KiCnIfBDOUzBM5pcnErs85C9gEn2MJtyilwaW0JgkZM5tLP1uahRT2", - "73yaC5Pbv6dKXmqmwOY0FhiR4tYavBUKAZFcYka7NJPKWD8H7HPHDnHnj3O2/Dgci8dxyoWtoW0VmmgJ", - "ZgqoZ6QHD7Hcq4Ue7GkK28Gf5NrItAoB6+nODlPmJsuNC63RzPRD8KdjYST5w4Ncftz5o+zxIxqLGY2B", - "TipF7JRQtm4btZ5QmP0EiwbM7QwXYNyDi9S6ecwVFQbHVKJ0knl1S7eKNBGYOra5whEVJJOZTbGBRLWg", - "QHK1NhC0giYJMXiUfF0Q3HEnW+bjMAjTaSsAoUWMaxwjLsjxj5XDNDp4GD5PmkWKhTxK/vP05a8Eb2XY", - "A1usdNfCNeICBAbwosHcIY6nPQOYJGuowqyK4x6Px73CnBtv41hz7dxlBgO0Kf4AQ/vBdtPn8Q/DITRl", - "zZWH5Lc/bCuHcJay1AKijnsf+6TyYc7NIp8W396FF7QNx+20xgjIlr3mtpGTUI6QO5Ub316RoLqT7hYA", - "AEFScqCq48qUC6qW6yIqA0vvVlDOrCdjZTH+GKPn4rh3OPa+i+Nef9xj4gJ/cw6O497H8Ao4q2V7Cj+8", - "zwrjZkFE90ej7c2Q4G59AzbLDoaBz/wGbH0VFflHYQcdHu2fyz7wb/3+/LF4sQnovMRosg9/r3z/Cg0Q", - "FYm9+hINmCAaYjcVEUu82L1Z0XP7xgPYrIglyW0T6F2RZ2EeK1IWfFXkiJtVHqO16vs7prjRbV0qNbX9", - "3dDvV6c/D2jPne6cXXhX53DCFgTjcU9pYguDguIUxzQ4ZcKQZ/jr0P3Xv/0QXPIskfOzQ/t0h9wDJOHC", - "xQNUHJVBPHBriZUsHk9Rz/6zyJa3ZSWJf/3jnzgoLub/+sc/Hcj9v/7xTzzuOxZnDpNtny0YVWbKqDk7", - "JBD1NqAJuJG7yWA6XHbB1JLsj5zOHz+RSs5/J6XpsRiLV8zkSlTiJmziOu0adKYCmA8XOdMOzwgK8pnL", - "qmN9GwN6G3+W7VLe6onuB3AhcQaVCcCt6GkAQfW4zTjuXqK9sMrUzrmmNG26aa44623mL4a9N5Z6B3aA", - "V2QwuMShc4cf3KTJ1unps+0hwdeWpQrMnIRvh7IZ94wYfuNJm3mS5Sh1hoKrbHlTRDM65Qn3KseWtC/2", - "CKY0WgA+U+FfXICu+yYO/UiBx4CrpXOE7GPRsXh5uoMqVsMikyvWd5xAOajVMi+cdHEu2APyL27QO2zg", - "yo7FjFFogxw9tUyggkZeBEYWDQtENEEfV25qKej6Y2EhdR2EMxy8VMYswUrY/5wadkmXfVIk/fVpYhJq", - "4EGs+1B4LGyooVuDAWK2kMowh8jP7JAG3pPXxWwpNgPlKE7E5T/HvrdAr+A8nPtlXKnvzkab2mHBoqU0", - "enkK85vjS1BafSC29PLU78Z2n2hJooQjNURUjMUcHYE8irEUtV0tAsoWVMWDSMIlUEW1Ag/khMXzNh77", - "pNJW7wYlmVo/geP0c5NcvzbhYrE6ATjEFqlvveXuqSvTzXTnWvwz2e7sAl3FeGc1uMzyG7u63wx5HQx5", - "4XXzRr2QZe2ph6K8OY9f28UdOfx62ltdc/ulsmR3odEjWx7jB60iUpGTJ0eExrFiWm//e+v7YKaWSkv5", - "D+5HYMV34XrixiKV3Quvb6kTyNfCDl65URPq59VMNFy933ZqWYhab7oiIVF55d387dHo9CrXSCn0lrT2", - "7SbZ6GzLdQShi1VqGaBolLBCfCnOaZWKNmmVrRtvceWsFZdsKfCRcAfy9vTLrutcNO+GW2CKTxsM8Q4Z", - "YT3Uupo+/Gui5jfFLrp5rVM/f1mkObo9Kei2VdEhMv+anotxY9mAC1qgk9YL9DkzFt7kJt/profAxEHP", - "7U61HejSzrqYlq1KLE4LTgg1Mevfvke2SLenr23vz/TyxeW5isTilvybiNLhsVuu1boH7pHLhX1z79sa", - "ktstu604AgssMnxwummERYL0klQvRbT9zXPls1M0LnblEav8vFlcaLLxe/nOui257rGoJl7nunjXckFm", - "CaDEWiNAzGfoq2eqicxxlHu3MMoiYbiihjkXxa8x7vckT6x1I5LigqkKUmP1St35A51WNz+VPPNae7u+", - "efViwEQk48J40i6Tui+f+cFk6b8Wy3v7p+4rjGflXjxoExg/Yf+tMzkpoDv/195PDrzzf+39RJOMC/a/", - "9h8n1DBttm+MWEa3ddPd9gPmKyY+eL/w+qKtsKYdQ+fVONQ6ZXr42qtQZg121iu7i0Caz0Z9n1/Ia2L1", - "dpLwbo3u4SV124glRz4tgd1SqTx6VQOS+N9b6DutEfSt6+tq3XONI0DB7iv0UpbKo65SB1vNPXE73iTm", - "imm9SRlRlOqoj/Dl/1QqCTvpKyklinX9ppfoopeoLtda1YQteLPKCdvHHVnfC2ILrTZ++ga3dQsGHUeR", - "FbitmoW7BNxaSG3w09eHveCuEl5QXPXa6GiZLA/k2uvDk+7R0z4uJKb9P3pahjbfkp3Sj+PWH96u39sX", - "eh6nUz7PZa6rUdMpNdGCaYcokLA6A/7aVALl9dyqFPiCqXR0m1fHrb/5v9H9DWkjmhtqmbd1Ntgk8/tS", - "XWV+Vx5kfou27FAXXG6kvs+bt90SBOLxlruScQ2W+mO/y7hCbxHINaFJ+Vwg+II4HIv/De+P3wyj6bsf", - "fHh3Phrt3cffmbh494OP8BbHnlSYMIq7NKePf32KHh5zdOLHTKglmERzHCTNtbGk51Oe/Ns9kEonl+4v", - "JE+F315InV5IleVa/0KyBW/4iVRPm3TrbyRPb6EFt9/+pK+kP7nptvaC0/lsxiPOBCafQuWfXvFVti+5", - "b1bba4IlCOcrUXF0rEkinZ+RBdfaIKGXif8/pydhvzUHhSTUGJZmBkCMIga5cG2Enl7kJpaXwqekwAn6", - "7Ga8nE/oevdNTXwj4QDZcIr2ri/dIi3jbT91XcdfaYSqzFyGcfe4LEWb9tfl3RLvzb4pO1y1t/+q/JpJ", - "zD7fVpcugxdCIMWaNfOkuQ3nLWqW6IxD8vr1Cx+6C88T5RP2Gemz9PlMzmNRzdI3JM/K9Ie2gG8Bng8s", - "dqH+UN/nvYsZjRMuGMY6MB2Ksq3n1rzTY/H5JeBw4tDbNnF3OZZ2Q+9QAr4zVnArsmZh0zfOIFsczUpO", - "0eK0eHkTT81Xxa8cAwownpCst0NzIwcODGBnIS1CZBgk9yShEWLkQjEL3+jwVyxea7UpBFVRMkmYsrCc", - "WW68uAUIuW5wXBhZJJ52ktkZND/JheHJWd+6GkLZRBNIem4RZcai1pmT+RAjAfE/cISKZXbEjSy6MGgO", - "qkYohXAG1S4JTS7pUo+FQ1Ww1TH1umKRRbBNkiH5WSKgjc0mXWG8NpXrd3osznicsInDozkjXBO9kMow", - "wWKSygum6/0yqhLOFE7iCYWV0ySlSwSGtBi5dn1kxiz4Yg31RsK/qYg5JgWFnospH44FJXujEUkZFdph", - "WGg6wwvHtUFwELUBfU8oORg9crUa+4bg5X75t+A0KcUuZESnkP8VqNgm297GDUxdkl6b9R22b8aVtvtV", - "6Ddd9sXaxnLtc87GfZKLEqUDdf25KEA1YLtMrgTO01kBGVfFNeiAiaYsorCeQtb7QUhYGUW5Cl2QsNWV", - "bNH/joJjZXqnuFRhDIwEVQYRuNpLxDFc4JmWeJS2v2+hqpKo/hwXTfCQSEUoqdB1qdFgUY6scQshVM/K", - "1KfCpzI/2/7enx04vo4R+OMv5l/T/YREJGez2gHcfDXZA7wu9myVhP+s5/SJY3o1FhdzOhdSGx55ZugR", - "qquP5m8Pwk4PwvUrG6TmmVTn7Q7HP0l13vUF5h0jv66HWHWGX6AhAoaHIPh3b49Abbh9rADR3PojrUlf", - "xSlFoYsbXbgEk0SKOVNVrfytmw2qr7otC2gJl6myxu4CfgweIRP3o02bXXHrRhND5Fq9a14Evd+CMepX", - "aQhPs4SlDNNqDyyxwWaXUHXTpX16lIBtV+OVcKqquAr2Lait/0Hfi0NIV37DtlB6X92uIFMFPM+NgKhF", - "5x79M4CIOhZvtE1VcGZNT2ek4MEg0Nr0I+RywaMFtIO/YfsWPJVm2VkBDL99SJ7jQa5sm+18yyYdAVrT", - "MmEW9PQiTc8OVxNHvz0+xkpYZmFTRJ8dEp8surg/NJSqop3CLBKqDfnVYbhuFY9x3NEz0GFW5rftcFBL", - "4P6xCGGiAqSobZDPyFkFHvWsBbvP89sXcq6/GFNRmW7FzsVI4p6OSJtMxL02Jw+ehA0/u6NRKAtAR5RW", - "O4wbBmldGcwLOS9SvdRImWZZV/J1w0QqvkjTNTRMtkoOQrSJZW7+qk3MlMLKjrrbiJts0cj+w9BzIFQH", - "8OkP9vZYtCyVnWF4qYCp9vo9JvK0d/ib+9dFmvb6PTceqHdpsrSSu+EKQv4G1Ntmgx/7oR2qQNt+E8+v", - "AlpbZ/oV1NrGDeKe1e2S+Stb4E9vNfS6u1uTiN0joSGR2EcrgNTiR1ymuzsYKLk01MtfknBcGW9dFSVk", - "gQeMI3t0GyND0GASJVKzmunp64EcdCq4hjTbrsLyazyA4cW5z9HWxbfm1FU99TW/AK3AJi8WP2bip3vr", - "7iyrI/ia4RP0ymyA/TVw6jb5uXzxhPT5tmRlql0o5BttXl3/2Ykw4QWzyiJchdjm7KS5kSk1PMJ8cdFC", - "Sl0h+wJU3mZ2dGrtgjJR6WPf3y624QxI9cwpyM/cA+fQKfMIrX5yfQyxuouICNfwn8oaP1X0FQXH7/tH", - "CeZU0ZBTS3E2IxnNNQM5Mwcl0zJKmE8QyMBUHdHM5Iph7lNGUi54mqfVbAGwYxcUkY3OdtOzPpnmhiSg", - "uFLFR+8GFMk0ZSK2ctJYLBi94EzBoiTUMBEtB5phzvQLRiBhQSJpjMqPLKZog8Kcq4oBBWLqhZQZGlND", - "UdA5gxM/seFVZ0UadatwEOx9SQ1gjs3F9zYPDDR75gd6RhgmOuB6UaTbjWjMRBRMAHD6ZbOxz68lP2Wm", - "OdE78lm6Fi+9SyemqjbYD+fL8G/6asEqOrD5NUKvbn9U1+NSPBn9ex5pO1c/xzsyfRVLvO4Ufxk2r4Lo", - "vhi7190btqQicW67q5xKJPM/q7WqYChVNzCMebXbeF2TVZFXtFjmK/G8nT/8n0fX0C5+IZyw3/qwb8tg", - "V076S2C5blWvxXPvSK3qdElVndzdsWA3qLsTn6SqcLmvRdlaw4wr+HaVOxlF8fUlxTe23WTbzhXjumzb", - "62ZXnA0qjJyLAXqvhjm4U+O2smqnOvg3jZNpzK7CMu+cRZaWi1sHgPSsMaNLUJX8GdyX19iPIqmUBeag", - "hn1doNUVrWE1cAF1c2VqzL6PI317fLzdxiWUWcsjlPmKOUQlWAiqpfGqAvflBVOKxx6Y88nx09Imq3Ix", - "JC9TboiR5JyxrIy1wXhHwJssIEoaw25ikfR7TBi1zCQXZuMoyqI3M5jyB+mQWr5AUdJlQfhmDu9sDkfN", - "/tfHzpDLEFpMYP3L1FCzMTszFzOpUiuX0anMoXXgQbBMsJ/W2WLGE6aX2rDU+ksCugIcN8yo47Kmu3p2", - "l/voLQwnxwbyZUylXGsuhR4LF8WSMQV9Q3Vov+L6FTQIGFrw1xPLJL8Mt0IYjPWko6Zt1RBMCrMx9w57", - "OzTLdmJqaIvrmhveJwzpJ/QTJHqZTiEmERwNzzXZSvi5fZ6QC00S+GN7raPhBOt97pzw1z9ZsNJHYiaD", - "GTctzRbE/KeKN3NszRsmvwi2dkfvzKCX2FWN3OXB9bxwJltZ7OYM+IrRZIAZ5T28EckNT/gHy3ahEa4N", - "j2xgFi328e1xweCHY3HMjOKRtfdG4NKNuA/40t3JlIx2ABpvP8o4YuTtMxwcMt/2zyn2+OTkDZZLWSrV", - "sj8WVLiGXz8+sZbmGY1Yc6Au9T052nm5wQ38FJfp39h/0k5wLcZDcMO/mSevjsTSeoZ0yxGV2bpnm8z+", - "9A6+Tpr8puP4OnUcQMPlbLYK+DOPWRbWZ1zIJE/hH/aPo03ob4ZGi7dY9IuRvO1wNnbjJ/hVHEo3p5jZ", - "7MR3YoCxC/a1+s/CwvkpoBBT80wM3gKPzZ+Ruj+/KaG6jl+g6dWtqM/8/cWcrdu++dwYPA5JdT2+lmNu", - "Kc3PxMj1mrBLyts1YT8mMjrXDrCmqsIkilFEoYcfS9RwZ65EMQHjZ4kDerJwYkz3x6KhDLW4SJpQYphK", - "uaDJDs7ZNoL4516jRi8kxzD2CGNmBhpU3ZlMEgQ5R5BAmA2UmPkGKtZl7XIjVstUDaNGAhKSTJnHhN8O", - "Pd3+Rrn5Sao6wPuXwhdfV9YfgRMpR93/Bkz79h4/CeP+mL5Ht+04d8ZtP6Kt57L80aql+gT3ZtzbH+lx", - "r0/Gvb103IMdeEJRnUsNuUdSLnIDYJFPra4NA5Xvj4hmkRSx9tD0Xpu4P9ItM3dk2RIDex/r3abY46gK", - "l/KV6yTEHqAcgfoYQES2qgfOncm4j4cuJjI31vTgzpUrFTOD6pHtW7cbV87It7d9F07+N3d8azwKdxnY", - "ZWXrLWfPcr1g7Sq3FzbdU26msA5Femi9IH+XU92HqAOrmQcQuhW+B7VPbAe3kY4BurpKKgY39295GDrk", - "YSjXKgxpaZ094Ur21GFxLdn7TCoDvzsoAEdD+JJAfA1M4vjyydFYRMCKLACjYqlE7uRQ4+0t/Phvp+TZ", - "k1d98hRTFZOf8+n2kLyESBtsVTt70VhYScwyr4gCVrmdQRy6nu3YkXpu0nEdOrij3P/2ZASsPH6vvMN6", - "v7dgNEaJ5I/eC2k7C2Azv3oBBwj2z9Ustr23VvgAHb5aDh7PDFOrzR67mC1RIIu4S9oD9TnBDTvGDrXH", - "pyv7tLKBBRDZ3+sF8EQ+fkuNcfMprm/HYof9uqSEEM33tWa3xYNYMMcQC6xe10VyibaIZcfL1j4wsMuj", - "p5/lUXGjVup1vKuGwP/verpwpl+toSmr7RMQcZGUZqOl1wcqLyxotK1GIprRiJtln9AkcXeUuwkK75hB", - "If5OFaPnoLcfArCX69kHF4Ottu8NtSTm+ty24GyxQwLucTqfFoMjeNCs1RjXHEJ0jSQRTaI8oYYRNpux", - "COOCMcuNbrHlFkPp3eDZKTsJwsJUPOzzry4TYJgmcPdKsmhS3I7d6h3FooTytB2i3Qlq6PyIrgZTaFQK", - "8F1ILMXRSEmtiWtqwBI+55CpAj0lIDkFqLloysYiS6gQTJFcWw99GPogU0zrXDHfANxdjqL6pIRfzJQ0", - "zjUhkVLpsfAU/vaYaMOyNWT2yrZ8jHO+IdnWNu56uiMldWMM7aoQV4TAhlhKsQsOdASqoTtwi7cDumsp", - "8Ws5+K8Vn4Pva6YktUzW+iXZY+2X0x76WvR0a1bQ06JUt6ygRauVCMlK9OBa+LxJiUge967mgRjo/Jy3", - "Iiy6T1eLaP4FKnXsux45Gx6E+/SJswwlOP13zCV6WglY7KrAKin8a1MnVUZeO6q1oN/NEF+do3xvMuq2", - "M5bXnUF4fc3IXbQWytv24P3yCGF0u4gTt52K7uumrRryVu1t2gI/sDnnwBdBgTeTbOCOEVeukWzgi8IA", - "ANq5QyyW4EG9q5j+mu1Zqm/5Am40lN8mDUBouLZQfsv1nPPq2ofSW1em2zPJtfhnkuDtAl1FfvfL/u3V", - "3+HJUFmsTSZoIHiWZmbpKnlbZel0pvkHNmwxBBd+qzdnCr6GS+fnIw9Pp60OnXeIXnaHhpw78Rl1CRa5", - "Bje/1dT8XxneYfXM1S6WHbh1BlRFC37B2pXu9RPslihTbJDJDI0rsV0wtx7+LjNUDecfiGve4b+6f2GG", - "TkxkwGISc8UiA94iwkjkCLaP7zRRUhr3Xaplu5eIPSI/KZk+drPZcB+6M+WUYaWfYbocxNTQwYXnNmtU", - "aJ/g3en9KYHhES7I8x/JFntvlM0DQmbw8oFoc7+k7H3EWKyRJrerA94dhYcIDU/m0y6jXJPR5SX+QRMS", - "5drI1O/90VOyhRni5kzAXoCoP0NJNlPygscsro0RnPftqu62LOhV9a4gVBTp/fzjwg7uTmSYLhfS/APP", - "6myhcImZckFxcBtzptTPlAUUgP4oF94Bx+2RH8W3K8y9/Lb8YwcoEbOVukU0Ulq46e1v19zXfM1Vg6H8", - "nVa77bx7znrldbf4qI5hSzeRhKKInbtdtfXbLyekh+uvMprHqc4vigdpm9r8yyLB0e3dD7etLn/7FYeA", - "grL8orFstgF1ESaYF+jTHbMLlsgsxazxWLbX7+Uq6R32FsZkhzs76Pu9kNocHjx6sN/7+O7j/z8A6dJ9", - "0GD7AQA=", + "H4sIAAAAAAAC/+y9+3IbN7Y3+ioonv1VpBmSoi6WbaVS33Fsx9GOFWtbtufbO8yhwG6QxKgb6ABoyUzK", + "VeeveYCp84TzJKfWAtA3osmWbEn2xDVVE5mNOxYWFtblt/7oRTLNpGDC6N7RHz0dLVhK8c8nxtBo8U4m", + "ecpes99ypg38nCmZMWU4w0KpzIWZZNQs4F8x05HimeFS9I56p9QsyNWCKUYusRWiFzJPYjJlBOuxuNfv", + "sfc0zRLWO+rtpMLsxNTQXr9nlhn8pI3iYt770O8pRmMpkqXtZkbzxPSOZjTRrN/o9gSaJlQTqDLAOkV7", + "UykTRkXvA7b4W84Vi3tHv1Sn8WtRWE7/ziIDnT/JjTwzVMTT5alMeLRcnexLLvL32BuhuZEpNTwi2tYh", + "GVYiU6pZTKQgNDL8khEupjIXMXnz9JREUggWQWN6LORUM3XJYjJTMiVmwchCaoNljKLRBTF0mrDhWPT6", + "jf1gAr7Em1fpbwtmFkwFBss1ca2QmVTELLgmXMDXiA2rG2ZUzlZXtt/jccImhqdM5mZ1oX6UVySRYo7T", + "8u2SNNeGLOglI78zJclvOU34bMnFvH2RpmwmFSM/LjOWUkGyhEZME24IF0b62dg1KmnsQRoiLj4XUrFJ", + "zLThgkL7k0wqo1dH/wr/oAmplMWhYXliFtR4KhfSkAvGsvpE6RW9qC/jL3t7/cej0ejXfo8bltpjRd/z", + "NE97R4cPHuw/6PdSLuy/d4vRc2HYnKneh+IXqhRdVqajZa4iNol4rNbNJEo4E4Y8PX72+oYT6O2Ohvi/", + "nUe9fm/38d5w9/AR/nv3sFed1srC10f+Yf3ROzPU5HqVB9nTNHGEMqkQyeqsf87TKVNEzkiUK8WESZYE", + "jxSLOxBdbdqj0FZEUsz4PFcsXu3bH7naci6oJlRYpjFo8IuysU7nLgImFssrMVEspVzAGq8M4rX/ROCE", + "EneIYEiRFEbJJAGmYAxLM6P9KeoDGxeEZlnCI2Q9tUN1kI50r98TeZLQ6coIy91mCZ9zLNBpabiubJKv", + "S4wkTBimihPeZWlqbLGt43K5g7tR8sVOXS5kEk9yYXiy2utb+NmtabUnrsmCJTGRs1mf8BmhBFqBny2N", + "15d9b7R3OBgdDEaHb3YfHo0eH40e/E+v35tJlVLTO+rF1LAB7HKXvUGmrbmIwrsjmiSSwoWkWGR3p7iw", + "ahs4ZZFMGYGm14784ONH3nrdPC03FQoSV7C8WQNbPWxcGd2oO6HaFEwI94ub5YQGxvSGp0wbmmbAh2AM", + "lcVs40K+weY++JVfu8C7H7XAgr03E7dCwfmE6IO9z1gEN6L03KQQMKA9R94Fy7oTGleMaimsmASX6S+9", + "XOg8g6ubxZMsoQba7fV7lgwmKdcaqhY/xFxbPtLveSKfCGkmKhfCFhTMXEl1US3pWpnwrNfvLaieXM6z", + "vNdfd23ViRq7YAnNNLbndlxNmFJS9axovJzMpPKb1Pu1uoRrmlpZIV1csYEV6vV7tQUo2Lmfix93savB", + "wfV77q+Jsq8KbM5OZnXg1bZWh1sMbT1jt7eIFaL9NhNXWdc5QMzpXEhteKQ7sXkUHmB7UxkHWOezojnC", + "YyYMn3GmnFzNiIL7IWXEN0KgEcIFyXXjHBSi/4RdMmH05PJgYqJsdVEaD5vq5lVkk/JGrNzKxfYXJ2UD", + "kdbnHnw4XVKOZ/IZu+QRW5Xd3NZMYsUvmQqw70IAsKzQliNbcNaBhQgp2HZtpcQljzntwg5iHNOEB6jn", + "9OkxsZ/J8TOytWDv653sPZw+6rU3KWgaoIUf85SKARwIGJZvH8tW2355EGqZyzTNJ3Ml82y15eNXJydv", + "CX4kAiXcaouP9kKSahbxCY1jxbQOz99/rI5tNBqNjuje0Wg0HIVGeclELFXrktrP4SXdHcVsTZOdltS1", + "v7KkP787fnb8hDyVKpMK32wbD051earzqpJNfVdC9P99zpN4leqn8DNrWSemNNeGCUO+t8WIYvYpR64W", + "UjMS0WjByNQ+WvB5ju11IfmiY397hUbwjOtMIp8n705KaQ6fhuw9i3JT6/dbEnNtuIicUFWZXIcRgfop", + "YXD/hiQLXAHiynApiPGi042FgEgxuqE7KNGps9Xzn1vymqS6rXVfhHBBUp4kXLNIilhX++DCHB60T6Zy", + "iu21udLVc/iZpExrOmdkC9VS+NywHJ5wTWaUJyze7iZht03m73JauddqZw7JYECn0e7efpChpXTOJjGf", + "O71ikwbhd5CPoR1DeNo6EZQwus0Du1RsttrfD3ifYCeKzZhiIvro7jIlL5mg7kn1H9hv7//aKRWuO07b", + "uoOLeVoW/9Dv/ZaznE0yqbkd4QqbcF+AjHCpCdYIjxk/xdudKEobqtafDyzxCU5iKWxuXBun+gF5i843", + "VnkDZZoMHdlRIeBUuEAr334OklZAZJHCMBFYnZdyThIu7DOICeP2AoW9Zca+S+R8u/fJ1qFY/tXDD+O+", + "AfOyP7S0Bt/6xasgkfPqai4YVWbKaovZcq+6hsrRtS7/ae34NC5QqtlkPQc55UKwGHXu7mDbkiBbB98+", + "eIouuJlcwg0cOnM4rJ+4Ia5Ea1OJjC5mPGGTBdULO2Iax9wqXE9rMwmIkDVjBkUlgW8QRRt8VJ/9+GTv", + "wSFxHQTW0Gl/ocDqTCq1oXlblhiqpjRJgrTRTm7Xv6NXKSRMAaXCt+3uKSjQE6bldD23m+7xnuuF/Qt5", + "N4wK775evxcBeSXw9wrh9nvvB9Dq4JIqXHVovjKs//I9VX77vuy08utp0X/lx9duKJWffvCjqvz2tByg", + "XxWmVs+De6bhpMNSpeTCDLjALQAB3GmepnbzileW02QQKSoSHlNd2Pk6werpikhFtl7/8HR/f//xdnfh", + "iuuLiea/s8l8ulZudoN2gjJUI1CNcEHmfE6nS8P0kDylQkgDxshoQcWcxYTODNRyY609wB+MNhlhWoUk", + "ptpEJPP7AX386P17ah4f8iv9+Pd0quZ/36e9Nu1irlvWFzVw1O0mFCWKisAm3uyiTul7ePlbKgkZP0+s", + "uYoUhdwe6OYQhqR80M/4e1hzQ3ZrKx1c3PDrrzBhCSkGueC/5bjbWUKXq29Aw2g6QIoILa/lG/YA6c3H", + "R8P5sXX8TK8oN3ByCpupm3KfyCSGe2fGlTa97tawawkGTH0qEal2ysIS0ypFNBew9VIvxtl6VhI+Y9Ey", + "Stgqc/fsO1O5U7fGDO4c/LNQIHZl40wFOLH/8bToovbzs7K/2u/Pbecf+r2nNKNTnnDPmet82tnkJ07x", + "t2mXqo09s1Vfu5of+r0ZoyZXLLCWZ+gfQKyeeJCwS5YQVxqI94g8OT0mOlczNJkjtdqyoLaXmumxUGxO", + "VZwwjaR+teDRolBXgupBEypKtwDyJLmiS00yxTQT5mgsxoWGXI97fTK2jy/3t6US+7d1DfGFxFwx7f4B", + "uoZxrw9tJXKux70h+VFqM4DnspOljsjY6WP0uEe27J8ko1qbhZL5fDEWKRV0zlImTJ+gkwaqgYEzJctt", + "QkVMxj0lNTOGDliaJ8j7obEnWZawsTjjCY+kICmNXp25ulfcLMhrW6lyeeKM4bLuk8ypnjVN2VhQqwaF", + "nVySTMkpIwnNRbRgmjAxkypi20NyytTAL7HfW0IVg82wSlySCzgijJaboX/5lTBhlPN06M5cYCbXoT9Y", + "+uIRra9T89jW+FBYR65T+WdXBViVm3HwjaKWpTa9Qs/2RixU/f46qlkC+mOBK4q7yqHUlSC1LZsldI6k", + "UjlElhj1WEQyTfGh53ohml0yRZNih+COk6hPgLvB3UhjESUyjweLZcbUJddS9eGGYBE6I6g++Y2leR87", + "hb8GKY+UvEyJFGORABnv0DQ+PNgekick4RqIw8+fW+J2JIacAPUBHEY77lGvkx/37MS4JnDxD8lzYRS3", + "NEe0JbipvUkbxNV18yrMauVuw/25TmNntsaKhcP+7Ai6v8JiK3RTEmBBxhUuGrqz1jDgVRncr+t6Y5Qb", + "X7FbERVup2rUeUS4ITTLGFWacDEWJTWJGKnUVcoUw+XQ3LitS5khWyk1yF5AazsWyCG+0aS2+9tDgnKj", + "9Q8rORJUIYolSzsgNhZ+zFc8SVBxSHL0YEAOZw8aasYqHhR4mKgoeF9SWruc8cvZRWsTqB0R2/d/PT95", + "S/RSG5aSKRdULbcJ1yWvtaS5apgLi4xu9/DljqJafdaxRK8nqEukqAuQzfO60YTgRNCSLjZR2I8y5GJJ", + "VbQIudFpQ56eviXwmRsWARHXxktVehg0JUnd0hz0SlGItetdaw6ZzsY5SzhSOOJNkz0u7pGwiFSY41et", + "XVCzYsdlCYuAV1l/Gvt+Y8QZl4lMkaULtjqbndYl8o3rTb1rsiX1Dkx4u3KzwJlWuRiSV4KgGEFqUsRY", + "YFEuoiSPmSYVfk5K9a2XL2puUAXZj8W//t//z8oSMbnktJA2yDuuDPiQ/G5fsTNFUwZMr36lRQsWXYxF", + "Uwwh0KqWhPpLpVhmrqsXiuLzhSFCXg3H4tiOqRjwlpYzc0UVyzN4ZpLBwI164OSsbevEhQ1x03DR/KWx", + "N5XFuaYvYc2cV2xof5XCNpHqz6XYUqfVOTXsigYexC+A9AaXXKPLGtKEK0uOT4cEP6MpIVrYr8BFQYol", + "W5YquJiPBX5xIvE2cRKtJTNnmhySVyk3BfELWVwv7qYbC3AwnDIm0IqYXFoH5JJUl6y5AeDNuQsG3uFu", + "6GjMYewTIyfzPKh39XddzBWLDHl3MjBy8O6EGEVnMx4BHWVMuVHbC6Y56CA/Bx+DpGWlfUXvQpEM4ZGh", + "eDwHIQeWy4n+9jd7OWVMwbjePDmFCw5X/1sy7glqbJ2S0w/AksNjFpOfn7yBqzV6dbbtVs09TW3DPbh2", + "Qkrwfk/nU8FM2/jtV/TBDW8FutJuYr52iVa2aBN5f5xEAxe/beCbNTJJycFAOgEhtiDBIXlZk19thcJx", + "BUTQVYm+aMEf4m9JMdrvUKAhKaNCE+pk/cqICNdj4aWPmsABkjZxKwpVVS4sl7L34aDCfVEssfIISmSF", + "k/4l/P/gUsvowrl42FdmwWkFTfF0V5duRbLippOQs1EFUHmwBJbU914TPOGNoAXN9EIafFX7f+wopo1U", + "bBse585fCr5nNNeM/JWkLJVqORa+fN+es4Qa68NQ1J0BY+qRrSiRglX1CNaBgI6FNjLLWOwME9+SmTvc", + "tHAilKrQVdtCY+G60DtWAtCkmAVuQGmBSZblJldmso2qBpxNMS2Yc57aYS+kyZJ8PnDz7JGthF8yN22Y", + "IP+d2TbAiiKlGDh/SCha6k7wEzFUzZlxqmeN7SPNjHtjsYXnluC/wbCS5oJHeJtjuXmWD6oqjh7ZenH6", + "dqf0WRqLymesAkq9AZeDhKcc2NsW/ECOd14RBTc0/szF3I4eC9vpFJPE8m6KQ/KOJrk7p0jtsL5j4aV/", + "HOnAqNwsZnlyRC5/d0JYfT+KnSykot39vtM0JahhapVk3p0QTS+ZJ0isBU9qbeAY0fiSKcO108ZekC1H", + "TgNn50K600QwFhMhx6LaFhxkJFMYYlPkJ1YRo0ljkcaiFNsanuT+avMmw0IV4QWf7hqb9S+aFhvDx79a", + "rvFEPise9Y2LJOPtVlXQReJBoVG5TJX3KyOvMiagVCyjPGXCEC5m0hete5GNhnttnmkt3Z9Vr5Wie72Q", + "ypA5N0QxEOOkcNxs3BvEXBk4/jqfzfh7fEXmqAWyMg22pPvAoca9XFwIeSXGvfowHx7SvVk83d+4G37Y", + "/doaBvcBmZ7TSrfG6623oT1dazADUZMYCW/n6svfUfgnMJeVIUlw6akBnJWEs7hC233rrTtnggHniq0P", + "qCFeQmsx+QyCouytW5WuZYlp2VHrOdu6oRvmME/kFC88P5HS6XRIjutr1ycUP6DAXF/lIlqi4hhKtthw", + "PoTbJYv4ADxDB3RvMBoNRk1qTw4G1uU9o8YwBQP8f36hg9+fDP5nNHj8a/nnZDj49a//EXyQd/RW9XZt", + "N88tf5b6xA+26sLaHOgm99YbW9aqw28/vKhaaN3pSDE8BjTZOAbwOHhaKd5O60+PV73e7ErFEvTQQy53", + "Ej5VVC13xJyL90cgzmlTX7f1ZdcpWALBMVYw8gUI1cRpWH7ZuaSKU2F+tWMk46pyAOQ3grcG3PvPcERk", + "MPDtOPblPe9rBoAh+VkaQp1FGp50R1hotd+5omlKUdt5SRMe4+FwZjbNY2aVswK/kUsrJSlmciXIwWjk", + "v0wqgwK5QWo/lXHPKU5VtCBOIaQIqkJshEzEMujSyJg2wpQqS7F60uCY/XWn+GvL/zn89a/b//s/Pimp", + "I6GtoXGrzrgmP2v4eUMhspXIK6YiqhlJGMxW90nM59xoK7zF6BqFCrVvSWSdPazrolSECa+ixnJ1ck6X", + "A5rxgdO8WHP3SybmZtE7OtwPLu6W+2Pw61/8Ty0Lq/Ik9F57LXPUvOLnauC1H0Mn44tf3TxBo0vKxbGt", + "trsqTn7cDvuJrNtp+6hr3Wq4Y4qooA0DWY3AB4+jNCREXDKl4Chavvb05Bk8YS6YY2jwaiTjfDTaj7AA", + "/sncLyDEwVMGf9uuSzy2uZCw80uPRQuJbpBJIq+jpOw3mXqbl+BaG1lotUvuv+pTiEZ0axZHZ3dXkEyV", + "vGAKTTKwxJxpcsGWVl0wryk0mbgkl1R50+AbDEvAIv4TWoMYPo/hEYvB+RDLypRji+7NZsOgGE1KbplS", + "MHfhu1JHMmPAUF0xnBo5Z+LynKQ0s896xazSPKWGKQ7PxVKxmLKYA48eC4YHg2QUznwUSQUqVthb1MGW", + "q/CNJuf2mXiOzZ9zAWR9bg9mQ1X6R+/V2zffv3r787PJq9PnPz85nvz0/L/hZ1upd/TLHz0LplE4Xn3P", + "qGKK/McfON8P1nkWffog1H0hlXvtehM9+qvQjA9lxgTlw0imvX7vL9V//vrhV+/uaSP3LntHwYF9CIqZ", + "TvcZ8MH3AVCaSFV6nlLEBUEW5dUOFXVDQ5VvJb9rHQl8onA5mWY6GJvS1FuUcujuaHTy/Y7zHnng/7E9", + "JM50i8MHHiSVE4/1AsinwOVAe1qSyMhFaMzaQvB9VyEGj7bWTHJhNjKnsugqjxoMyq/XYEU7Uy52NGzD", + "ILreuiPd3NhR+bm45EoKfJ6juDRNmK6flZ9fPXs+ef7zu94RXARxjnGEvX7v9NXrN72j3v5oNOqFCJS9", + "z7hiOuj4+GSqZZIbRrBQ6WM6JCe5RudOxzxmOepBUT7TJM1BsZQsCXsfJbkGPoXSgDFJKP740WBv983u", + "o6PR6Gg0+p/ODqtA+huY94vTtzZCHc87o4lZTNA0t6nij1j2KRb90Pe6SXzaB1ap2FOvrYTT6+qQrUVd", + "urKTI0iQ497+i+/tgdp9Me5VCNGFAhet2IYbq/fi+9C6VLRRq5Je8c1GuInYU/0qGI3V1+faqYmqlvCB", + "oAixAyVjqmIylVTF39Z9abAmKPK/0QS1FTRxdiIremMdK0tKpvGt7H18SmWr7jeWADU/qVUlUkEYWkov", + "ObCcQZpy6evV2ZJTptR0qMOKgSmgx6u4C/X6PZiZ+4+fYK/fu/y9HlO9WR3oXGM6vRU3PAJpknHB1rwC", + "PxNhH3TKiaTxYPcTy/oVf7cGDI39UN/u4nHqCXwF3WlKRXzFY7OYQEQ8DDlwQ7ovpChcXJPvYSY0+dc/", + "/vnupFSI7L6YZu7O3N178JF3ZuOWhKaDsTbFRPIsPI23WXgS707+9Y9/+pnc7ySsSHkj8dzt/3PbQhOg", + "xuF2WW/8FrtrIYeV8bmyxZa+AcMgdOXKS6YSumy5Tv6muLUuWmbpClf0xutvlN2Rv1IejF6g2saPGb5U", + "FDVoFvImIWe0seYVGzNMplIa7LZ5aYfvnY9TOg3ddCMpDOWCKfviKeyq7tKwb5L08GBInr/PmOIgEtGE", + "xFRdcWHdWnxN96omVADkT8IFG/DU2Whx8nZNLdeFhsGU5l2KtHUpWtXLozo6Ju4GtJWlaqi8blN3pBiy", + "3klWYOWtOyCvbenyWR8muu+pLmytXYisoLHdvRP3596LFaaw++J7qOgp2fFfvweGpbBgrCo14b50EnP0", + "Bc+sC8aEzouI0XWgfGcXPHNPbazhHQDs9RDn0DIS/XAsLOgOnGgcMzjLe7OqAf9+bX0K4CLEq2ZVfIKH", + "e3nZ2OLWgqpy0SfT3MBrXBpGnF7ERf/DWLDwlJG89O9svI3dBFd9FnBZLpgSLJnYt6/uuDK2EnGVWhcH", + "pzqj2oGEKZNn9fV69tPJGdl6thQ05RH5ybZ6IuMcHPXsYd6ur15/LDKFyCvQCRAxd/3KGZG5GcjZwCjG", + "/BBTbKwwkTj8icsXp28dgokG/6HXDBaWidj5ono5pHBGFd8Ya6WuNVvtv7HobSg13vLe8UieueLlmeyu", + "LOz3jElCuHOJ5UcQ32NDHMCmVFCfj7rrEy7IC1nCJNhjPSRvNRzrEXikSIUOBWQqzaL68JtxlsS6+jgp", + "0S3LYvA0DL/+yodm43wfLIIm5SjL10IMXlr/BZAuGtL+XhemA0IJwh+18J0grIsLp1kdlgVyrWpwnMBQ", + "bAE1deiLrkpn2zLCrm70w3S6Y/vE6Kw7rmgzV7TIXtX2R8fwhPXtHws/kLW681Lb9hF9ndlGmkvk2u77", + "md1glY6LNWlo3D/N8jzRFe1kJ4RUK3tYuUKTrXNQaDo6BhXmeZ+c/6X2A7BHL9eAYH5F7GogyxXwU7V9", + "veJDvEFjemM/Yqpvvh9PdCu+FbncJUZRoa1H+4JmbEh+xHvO8wO4eHjViU/Iq2+JtK8BX3UsYGi67tXo", + "9eaazwU4eaF7+pSBydwq151uzHu66BCwsNd4rUjOnlvhlYUwvs6vnZx7Jfd5nZ2uqsBXNUROJ76iGsDf", + "rUoAlSZmJ80NdG9ZpLP+ytxYuDA39TqUW0PRvskFx42lUPrdYP/PCnbRRGu+DNwfMDk5Ky0jFRNNmyXE", + "yXJhK80FW+KWe4sMXbHJVI0xYZOJcx53kkmlKmrp7FXCdcWSY20yzgyDbr71Ixo0UGzaClivzstff2MH", + "I14GfrIlxbhns0cC9VwIJmf7Q/WeZrj4+GY/sgFU532rZGCoySMCiCXxXvDN5tEBGN0Fz90vQ9faeRGO", + "9ynQvcs4i+rWksrOeskYm/GSlWFxvy4bAA603jwp6wi6aDEwKnaluGdkHkqyowTr4lNS9476OI3L80pj", + "a1yvujaxGo9q17cyZkdPFkLZh2vY/UBLUw1buAmxH7PajvcsRku9y3OaJOdkyxXaJor9HfFX3V6BL1ux", + "/YAn60igcOx6d9IfC8sFzhfGZBP4Pz2BU3zebMzV9Se8hD9/NMIn6MHBfj06wg640WxdMR3EnWnfGvBw", + "uiVPqRb1t8eZx8/FDQTYMmTL0DkcHQcwBPfE9seqxrnQLMoV23zWn6B45EBucEwJ5WLw45s3p0SxOdcY", + "1BsMH7BO6C1oaO7tXzqG+ekaWdeP7u0fPDh8+OjxML5QQxapYa4HjGoz2B3SlP4uBb3SIHbtpEuaZUeX", + "uxv5vN2CYnztLN8/VNtJQabAkr3rb5dH79OySmmau+Ai7trAT1B2s3upf5PftoEjU2yQZ3NFY/aJzRs3", + "9iDC1Wzf1Q05R9a7DufagF6hBNvdWu8xXF+sS5kMYmrodRyG7XBXfYPTpW3KPslD7bU6Yp81va9rCDnB", + "zAMf687lxxLaljb0XqtIYPHEyECgrJcdjp/BmvuynWBKEWzYyMnljMtN4fQwtHqQLFb36g1oYpBF3Cne", + "vDmBF9p3q3t7d1L1YhmOxQClsCPyrOigaLZokuITg8ZWM7wlVS1SFxXyUwAZIe9OhuRNMdpvtFfO2zGR", + "IkYzR9MdCkUDK5JVB5BjSAs3zepOy2iRj7fRWUe6b8MiPwrqMwt9GKq/prwxHxtBjRvl7pIqyEtH/e46", + "HLDX9j5ahwVW8714MBjtDnYfvNkdXdf34hbAnUNtPanzFudlX+U+T98eP9tzWsztG8N/fXL45zAre1aG", + "B5CtXLNK6KtUwaCAiu99i9P/jX35b8k1vwS6XFfWrsQbKHkbuNYh5C0s0r8B8nSTYW6EN61MblWhs8xQ", + "/VCekoogb3c0i3gwxhmc875XjF4gXv/qvY2IDxN7n4U9+xCWY7pEGCpUcikpzUyTAq2lVH7vHjw8eLR/", + "ePBoNAogJ68SvIz4BC2jnQYAzi4JXRYgglvoMBCTaSKndUJ/sH/46OHo8e5e13FYa3m3dSje374W2XIr", + "8lefU81/qQ1qb+/h4f7+/ujwcO+g06hsY90G5crWRZKH+w8Pdh/tHXRahdC77rlHsm4K8KG0Ck9sJiL4", + "10BnLOIzHhFEgCNQAWP0F2BO99bh+pmc0tgnnwjfHYbyRK/1o7OduZJW35rmieFZwuw33JBOJgmc+TNs", + "KZjNS4gi2cf1WnL43xs9rPxciiIozsRsms/nDlyvWLoTFzdfCk+cJfERKZ5sa/kc7mY5sF/b6MDNoSM1", + "vJRXBcxdhQjs1WUxIRQjBZ3YTavNygfmcJHlQZJoXcofcmXhLrBRQqcyN9Yg77KzlJ1Y5BiBBupcxN3U", + "HT9IdbExkgJu4iIJTUeFgQ+ptyH3jWD6qtBXGM6te4H7rslrW8MqCsufs7yega+PPTmFovA4AHGhN3bN", + "dJUuw3IL6sy9+5ztr+Sdd+Q7OJhZd6tP+8JWc4bJl8xGiQUoxfoxnWHxzoFZUHGjIqXDugt2dReLjqGF", + "AyDbAahQbmfF17koF7qGshCRzrEfkDBh5nEFyKlx0s48toVb8uFYuMim4idtDWlQ0a6DWTCuiFR8zusd", + "1/Wsd+gkfB3K9MR1Y+qsVmzz3dOFGTDEAyy088z63FrvwSqyrNuTXr93VmSpcoypvjT+x8CKlK78qyA/", + "6MlxLVffTMkZDyHvoBOR++oeat4J9uXB6Gyw+1/WTR/IDyU2LqzjEdgRho1MQ1i+20X04vTtaduYijRP", + "pDq6lTmtwRNadZqZ5cImmqroVKhLAOkg3iwAkO1sSL4H2yebzaQyR7aENUiVRJDSJVEszoEeqlhoUhCw", + "mOIZLXoFgR9+gQkh8sg0n82YqsMcPA7J0pXSkzSg3PsBvhNboMBZOPm+Lk/vHYSaDnPh0xo14LN9RiMw", + "9Xfe7oBCsDGNTUCGL07fvnZ5itognmEpi1xGFt15SH4uMnnBUusSxmkY0BQ26Ml6bEOBiU5kyBvlB8UY", + "wW9lBlE4e2jBtqDAhdcAjREUSgqfxxBc5h1E3285VVQYLlhM3v2gvyUjZzJ/9wOxYTuOAXFd9YmsvxQf", + "hra0NfzudLHUoKizy2JziXBR1VLike78zDgtKzp9buCxEc5i59kH2YLchbiAZ68Hx6/e7aQxu+zXxgQf", + "rxYyYTDu7QqPvfQYEEXZOmu9bFMX+a0qkipOykSOzYOA8NNOj9mkDtibEZmyiHpP6ebmRUXe4yn6XqDW", + "C2zWhgmiqFM6U1FrxKL0aTLLk2ToGKAfMkE18wz5cAUy1V5VQD74PaP4kirEqPcGO2VE51HEWNxwX2mO", + "ukJvRyQXKVV6QZPm5I4KXIJoQRWNDFPkm/ff9MKJi2Bwuiujr1BncdN0JsvKvRKgx8q5azvi736oH097", + "1dtoZKcg54IbRFdDSO2tYpAWBHxITujS6VEyvDMdvZQ7RmczizFa3CWKJYxqTDiuybsfNjtmGmlo0jaH", + "N/DR8aktmBCcIhhmn2Q1NgC/V8G9qt0eBq+Mcj4BMxN22LRZDMmxCLM96/2D8HwWaUtrPhfwttPOASii", + "oljJlb2r351BtVATzrBXX7jadPqBCyBEMaHbyoZPrj5zVhOgyos6m5IXm5Nu2kba+33qIzwbdi33lvTn", + "FgNBvScSsVUdx7DaUc0yqqhhydI+LgrqrCdRYKtOJeDJf40Q0+dQ/INNnJMrNjELxTQ4gNW0Dfv91eTL", + "GiMGLpnLN2fnVLG9GUlSqi7wlPm3NMmFXYF6OOn+JkyphTHZNSYFHhJWwWaYuqRJbS693ZFecbJ5xiDO", + "acrMFWPCT4VqQqse681Acd2S8wsCZJjisr6Gvf1Av2c2ioHMFY0YsbV86vaCrcHR67qUvf2WYcGFo3XL", + "/u6u219XdZYn3fY4NKzdTTtsouts8Junpz53VZEb3C/z3uoqQ/4Fe+RcjQ1bu6fXZ1HzXQkLPr0is08Z", + "plVzAVzV4G/vYohp46B6LeC6whx0NUTY9YOnwC5V357zXzs98JrHPeRLA1FAoRzr+MGFvs0Row8DDVSO", + "ry8eW0nJ3smV+9lFOYGkxQXAGdeduKNcJb1+bzBzszra2QF+nyBM6cH+7qOd9Z7ca134ncfiJObrVDyu", + "lPN88zFgLggRJl0niR2aZR2U4Fhz0/2A7GllE2xedbjbKo8sR2logmp4Ir6nkfGZIFErXvN6oNVji2Di", + "dUEGGkxtcid47T6uns9R0A0FfErr5L+zQvswM+tWDDRiNY6NdbRE/ntQOpUqlO5PKuOifKfM+x0X96H3", + "6vWw2TX7+OhRdZaHDx7sP9jEh5DZNI65O3eBqdrSdRwDVllud35dA/AfXZc53Jf1ZIXrsoGm3jwNkNT6", + "BZUZE9dazwcH+3vXW8+uEzn2npkNvhSCgQLcJ5SJynDdlBkaU0PrTAbVycBlMJ8UZSkG9c2+Xc9aWlyY", + "qrhON02G96kcYFryYr72MPApFXyGjyRbstqzXtC9B4dHNntvzGYHDw6Hw+F10W6eF9+6bcWO9ZOtAN8M", + "9eLj9uEWQG26zOWP3umTNz8CI8u1spfWjp5ycVT5d/HP8gP+Yf855SIMhtMl4TOfrSR6rm1vlju0cRYf", + "kTKnv5d7urjotdiDMEAAEbiC0JI1R+nbw5AsaJzXU4VcA0tyTXw8iCugXm9dmxvnbhaFLtRUcjZX9Qod", + "8jeDH+c6Nw6vccYyrk+r6ShSW686t9woOblem791Jb1fxkSRsTVJ7F+RFAjmHUrfWrsi/bcO2f/wGnFp", + "/oouqz8WvVd+fFodSOV3nz2w8pPL4/rrNb3S1gqkf1uVQzdzIS+ObjjMYfV/cSt0zZntsCeD4Rj3fBfe", + "xHOz3vur+X/+9n/06cO/7/728t27/7588Z/Pfub//S45ffVRuEbrYUfvFTv0k8GFortiDTO0KymdgGVm", + "lZ7g+deywu6L1TiYaIFZfsmUHYE/9UtumLI5Ixvxx5DmgOFLCWsRKTC5n0Oa2IbKp9apACr/4cXgD802", + "YgcpodyGFPhCOp/GMqVcbI/FWLi2iJ+IxncB/BWTiGY28SMXoJgFL22FWTqdlbfsvE/+oFn2YRsyiVBD", + "2HuLoJ9RTHQ4K2wTwvtouFFZT3RXnBUwxTYoeCyK27rA3bSm/mGZpIOzpBm217Io699v7uX0aBRCKMWQ", + "MthITESFT5CCsoGMilg38mi0vfqe2/DGKGhoDfk5XxgbcwwxwwFSbIlTfsZiHiFf8aG6tmQlStpSmlPi", + "ZUq+X+LevLbxozGhuVkwYZyXY59EUl5w1sct7aNFGp2vsKZ1qVnIbDBdDhYyKyBSqLIOZ9Q6pdQf2f9n", + "4CY6eMcUn7megkgXQCIBoROPjJsZlim1CysTe2Nz+AjNUV1oi9tkUNrm/7HRFCZXwqeoYZioFhF9Cupj", + "IZn8WxIlHLVOeoE2vAWCbBpoJoST2RsVUWt0GsVs1vx33ato78EhvmD9v/f3OgeN26VbR2V5EnjTpp71", + "deDYlk3iAKx4MPGK8A1+gHADOlcK1FMYif89I76h8sQVjMRa+mwYq3YZTxJdCWDdDkYaumPgAH8mNDdd", + "Z7d6CmuBix1asA5CWC3pADX0HOdB3rw8I4ap1ENYbEWwO3hKLK7MgGudu+R7T56ePN8e9oLgajWTFm7V", + "2sDG+qADcCf2HLT6QZU6GpqyPjl+hgHr7lop32IYYfSDVCSxt2J5GR0h1E5d3UNt2s7jZ04ATZal15EV", + "W8a9bd9i1rzejshr3y2hxVCK0OOStnyT5WWCzTofVBv+tNJ6I1Id7WPu+efuYwx2wpymlhcjQGzr/dVd", + "5+hWHD42NWTXvpAqFdv1X5W9/9RQ6p9eRt+/nozurNCTbEF1iLoXVasmFlrZ96ohu87uVbvbXq3fiebB", + "s/U3n9mpcg0ZBvdcrXoIO/jBYHf3ze7B9dV310XBrsPhVaAxCyDs7gjWt4EEHcB15mbSGt9B4LOL5vB6", + "kXcnECoJqGjwsaEd2d1/2EUpgb12jYyofASRGodUcCmPild49FuUwQueJFaA0XwuaEIek62z4xc/Hb98", + "uU0G5NWrk+ZWrKsR3J8bAGLjLYDraJ2jAsBoNVQOUoTvvnnzEr5ECcMIKCuHX9wcJnujarEDbLYf3IvT", + "t1BjQfXE+063hwvTMuSevefa6FVMxE4hCB8D022rloqxLpO0bTinyo1Y3z/WwKiDUJnbtwDS7eNHVpbz", + "HvCr7zNO9/PDzl6Ldv2xkNVOz3BLiNWtV1oI7bkBSvKg7Xa7Ofb0rQynhukUYltVCceDKNwY7Lnf44EA", + "8ifOj48cn5bZ3UpjhG++MafHe8Pdw0eYoHh31IWxpzRa0/fJk6fdOx/t2VvmiE6PovgIH+w3tVk5wrZP", + "EJpcQZjt2D8Sxz17YVZet5Vja8t0C6DoiKm9EU27Kca1YF2jOOsdl/QkXZtfqEOEcBO3MM0doGvKk4Rr", + "FkkR67qMvKCa6MziGNs8O4UEPxY4wD4p0p2jlEJoFKm8VD066drJ+3nm6B5z/WZSaOaSffzElpqkHI2g", + "Rffo+qhJEYgWj8WW8kGLRXQipvmN4QcMAeq74JK4j17DmE8IKoyFXuQGmNj2kIDfW54y5bSyZMrRYrRN", + "dG6ftDheXI0lMEzNY6bGAooFkJL/KJ4nR4cAc9DvFS85yIkyGoWo6VaNn0OHZY4taATcFC4LHXq7q1yQ", + "XMRMFTn/mSWHpovcNQ2nHwkc7qt3E69c9VKuuhnueCdh6qPxl3GoLe9z9A69weP8wc1F9E5hf15+dQF/", + "rtbkOh4MLiQDXnxTuO2sQo7FTg2pmYv1KGJv3tp8u/WpO9djI8Gorpbk3clJze3BRWh0mzhyiZZ9kNm1", + "tmFvg45k42iuY16+Flr1x8NMNyWVioT4yUGlqxZGj0NgKbSm2Ko9nNfp1+BVGYyB4sLuExDNmgk2Ui5b", + "1JbSMbLaql7qmYNYdaK0j6lwaPEFAktp1NdLvTPN9U4W8R37Xe8gPM4jhMc5COIXxOxykuehpxF8cv2Q", + "t28BwwZ/QXhnGyBT657Sw91Ho0ePB4+mu4eDg3i0O6C7+4eDvQd0NNuPHu7v7u2vCSrqENB68xjV4Is5", + "4MdceK1PvPd8yKm5LXahIZs4f+wrLmJ5Vbv+gg6y1d6d8+2m7ldd6zsPIRgSlFBtrPqihZOd4CXPImgb", + "y/mg6SK9WljRefhmtPux2h8cXMsd8UblwppVsUBpQkgrA65uVn2cN2P5OCAf+LJptaqdd1+00dGDx0cP", + "PnbRfPDGpjE2yekON7fNI8zDoDeiQ3yQcEWP5BWUPScTWa2+Lb7s9Xu5qP6NwkDDl7r43CmIq+3A9sNs", + "ZN210oJfcFx7r6CnioXBjI9AUvHvEczHUqBkgAj0FHRhpKKLs6iAaIc7rrxdoBk0izkVnQ1ctsEYGBnJ", + "tYvn4wIYMdofoREHdnBEXmBZ/ERT+6xzg7D5rqumNxovrb8MnC/ftX1krR/ymXtfYR14bBH4F04blsGp", + "bNc3YaWzI/KzxDrFa0/Ipu7XFsdn1mrxRlmy5RDzPXgNduZEzSPyQyFeFgKqE0i3NHN/ThzDKjGjtmvI", + "HW7HwZGwsnMVFIp+z65or9/zCwV/BnAr3pZUv3L+qqQYciRjNMGzXIbp54YnDikfZ8K14ZF2wSOwuW1i", + "j4vNZPHEPp7afFJt6KstQ4pKXqp6d0K2EAz1r8QptuFf24X/au2u23t88Pjw4d7jw06QZ+UAN4vGTxGZ", + "YHVwG+XkKMsnTjfSNnVI84JrHlmtQuH7AtaukswzJYH1wMx9g9XOHw8fV5HeYpkDHy6G5GAhLbA0blgQ", + "zLDgRS1+kL/x5JLPZuK336OLvb8rnu6+P9R7090WBGvbUVjtdlx1LljRUbPpwGbEDINxIUEp3YpX95pp", + "nAE5Y4Yg/QwIjfB5U8RTO5LLdW3Fg4R1sL+//+jhg71OdOVGVzk4E1TCBS5lN4KyJMGSZOv12RnZqRCc", + "bdPDumQBmIPGOSMur/moLpAOd0f7ISppubhLqnFtX6atS/7OPR/dpNyiY/B28bRcOeXB1d7fHz08ePDo", + "Qbdj7NTDE/V+PYdx5Wx/PhdGdee3UJp88+SUQOtqRqO6bsd7iF1rVOZao8I8Ljb/wjUG9ujhIcRc7XYD", + "Xgw5nThI0dqBrfOuwKELEEVgNwJLscp6+223RUicsgT2mkUJ5emTyIdYNG4fm2dhomyxchO6XAzuBb5y", + "cXWo20m5VaisbIAO1iFSkVwU2X2Gm02yn8Sy2s617fWwmauHwnIErJ5DCLMJEG+wlJlil1zm+hM0JI2N", + "mZ0lUqpr1W17sLxmGuxuqLPhmrw7+UYTR2tEG5bV31COGtfgqN1wctc6zzUSCRN522J12o0uW79uwv2W", + "U9tfB6hR4wat6IUxcK5cbHb+fEqTKMd8VrTYT5gVwvAhJECWJUvr258kUgoSLahAI4mqgI4RwE4YBj1h", + "4ctkFvSqkFckkRZ3HfLtuFRPdhBQDUQY0AdtVfMjWlJqpDV9kFom45L51KnxQRpO2hrGVCpC4WE9qZEV", + "SHBbpaYJTeRc46PQYNTCsJmJIqPKBiNQYVOXXab2LRlwuA4MscHMQzcqfoSDaR+4TuTAQHO7kjRSUlfg", + "wN6d6GHXmLciinmzQ2d9sB1I1xo0A1eZBaTrnOEwdD8G4nk+5oZEGsaYwTWukl45mVKRY/KnCiE7Rfyw", + "szskaKMnBczVNQerzQQzuoACr7BcFVH3hT7Ilwnei5613WS5nN/xjWqvUFW4qbYBtvPU4IqGV6tf0GCI", + "jFdx5dZC2ZUAf00wq+vgRZYpQLjGVnkFOZBsYcxLhS1VUCC3uzjJhJ+s0M/Ka9Xl9gU7RVdkxfVAiqfU", + "LI7FTK5ygutYTr0m2nmrZkx5cMOYCc5i/5YsTKhO1cW1SwUX58ytHHZbg79DFwewWQmvI4MQnzqvb3bY", + "RT1sx7A+4Qv26wp22EmuwwG1b1SOa4UEBMRThtZ2cgLlehI2Z602rNg8T6hCwL+OQ9bLNOHiokvreplO", + "ZcIjAhWadvGZhMCvCXzS3+FctjvNDipM2rKDndnBubhAuyGNfsspfAez3G5EJaMmZsfW34H6nbzHgp7i", + "P/CEOWzKt4K/rxB6PSPCwd6oLVq+pdFanPwqOOt1Obcj2eCJz3UgtnCtlOPzmrHYpanA0pifDY9ah1vJ", + "4yB7E+DNLDr1wJGPgyZ5avl1A5iETBnG/fiprTTXhS12mUowuwskvPu7nNYVol3dfgM5A7dECZGh2Czo", + "3487ulYhbUusrElld6+DQYFsFSaKla4J7bApu2LpX9XGT16vJBpcMLdk3M/RJh3skETH+38U8AWu1+44", + "Bs2UkAF/ZUSp0WaJyZUxo9WykucUMoUphSDwIOFI4WeDsCtS2bX2uFcQjb0kiqWUi7HgolCSIpgaI4Jd", + "MlWJkpUKHllzCNH4W+WJB7+zNDNLl48BleffaAImND/GsagOEhoH5feQPBFWs6jyzNQytrLYvfqAUDBq", + "GY1gRmEKU24WZKaYXlTnHkpbCzLelVRxaz6wJfFFYBoUbazEyAsmqqysaCb4NLQNTWytVS8+m3Mav7r3", + "J6mlgSbNNM/r+8s1U2EhsZhSUaST60rlqFQeJ3ggYQAuvaf7y7L4AgWlA+ZJ2fx/+SbLn06Lxuu/NYpV", + "cE080vcTq7YNqmCjLF9dnYYlbaOrCobBrUOzWbUlkC3vQu1zJOntllxFne7JbpF4zWABP5odzaJ67weP", + "Hjw87Jgs6qOMdRa961Ob5i7TNSa5lp066WL3efTg0ePH+wcPHu9dy8Li40pa9qcttqS6PwD+YeCwJv/6", + "xz/fnTSsPg/QB3t0rUHlWfuQ3mYdBvTu5F//+Kcf1Y0HFGI0qyD9LXb7Vi+dpLqT3lGgbsLrZiRb875/", + "UlMS0ILNkC2GUNzgdGPXbVAOpgED0k0KphmNuFkGGDm9st7uRZEG1naH1huDDYm8tm2HigqcS+fTogR4", + "nrgCf7G24QYtPOqcc07n0zY79Ktmr1iOlFaLqo9DBxcDSxFhBXcxnyuqaw7dMbN2h34RYbYabWNLrEfd", + "bYZCwFAQSLfWVhTKNdCQJ12l6vY3trNit6ypdZor/uuac9h+BK+l9Q3cyAGlb5TlXRsq5Xa4AG9WazKt", + "ZoNcm26zljqyvHWv32+H6OHVVCXFDXb9/ioBk9ep2KA1S49uDG7Jy7b7NZJooaZKLExAgSYTNqgkL8DS", + "ROfWIghn3sHMByI4ows5m9Wxbh+0Y6Mj7A8Ge/leqDHwMumD1t/pLJrA2hbjZ9x7oMc9IhUEUqbjXsNs", + "FQyfTOn7ieugju0yWgdWXoa/Nwap/QymiYwubKJDzJ8/JCOSMio0yQUe/oZVbXe03jrU72WVvfHriI+7", + "laV0m0mmbEEvOeZTcTaVec0Rk73nRqPDKLZzRGJp0Z5qWZ7dDKGYDW48KieNlw4VS9cwNAjlpPAerWVZ", + "VPDNMLe0+J0p2XdgBcCxX7066VsHBnQ9tAOr+Tf6idoR9Pq9sotGeoXy97D/MBgrcNxNuP50dR2rMelo", + "WVVMM6MdfndJDg0igKAjYZo4/mm3h1w9rGz1SsoFOvs59w9YUN+7JRASswhPpF49i3VCvwFxN+IG3EqH", + "Agf2QySMh2JN/p3XziDcHIBVNlQStNt2qn7d1ko40UZmvfqpnrD3EWNxE/AzXKSrr7yrGfSVf0kdRlCR", + "O92VRn/n1dkNby/AC8fattpVn34hxQBOXbGl9nBavDmPVVMntOrqVyEtJiF41VCBDuMX7P36tf6ZvTeI", + "jx7nCYtX17Y4kpZVucto04rfOLKx7UBLxTZmyLyFzJHW3/xGuSNtVXYf6SNdqVtJGbmyO2fM+LJnjoxa", + "d6ie6KVm0nJNEF+k7mNjSalP3AVPdtPtBgkeLMJaEQfK2zFGE/QtYHyf8fdriMcWsA/jOqxJeZA8NRCL", + "L7qV0vfk4CGm/tKNsQs+X5hkWXfAOQhgKX1UXlXFDBNeUdhl58vd9BVXvd3cdlZbDwnHZxVooJWUJk4k", + "nazDzX5aFPPa+YwuUYvTaiR8uH8wGu3vjW4EnO2HdY3lelpWcVlI6+20hdRV6hFWtZhZDlRtoQiyXk1t", + "bdPVlatJtFGMpkcYeJPRiJGEzRAkrwho3WxZbHa9fvBOoHJRtAX9+41y++Zt8PWUOUVXYl75l1V94Fzr", + "GETV7xsMoi1sJlqB1AvE3O0PRodvdvePHhwe7e7eBth1sUht0R4Pf9+9epjs0dlB8mj58LfdxcP5Xrof", + "fIddcBF3pdWfoGyrl015SdaxjGosjWy5OWRMNXOWN3P9a5ZwwQa6iJDaHKa4hhdY+/vG8389Pb+dwVrZ", + "4aw+yaoIQQ3RQcq6H/wtN5m1tovmbI6frZ/FjSKQmgMJ01tzKFCq42AwQ8Vu7yORGXLR8Rp6WynY+SJa", + "GxW36SoKWdjxpAd3uWXFQ+TdAGaozHrdBb56yQV0p3OpuFmk62+LolgBI45+079rE9fxnobkeC6kYvWf", + "Cze56iMKKvf6veT3g/qZcb93R/5yCMQFAbqtrkoFHdzIEnbJkvWrgEXKh4eynuxUMVyI73YHu4/RDyH5", + "/eC70eBx3eOgb1eruny7vnTt11GXNaymAPSpo3YfX8vj2q/nOgr6iYcS2JX3ssMmdjRepof3V4cPuK1t", + "cPl5ZY8bSD6tAujHSnrucptUhaYYNLIhbPqKolY3Xo9VIiNTNudCd9Hb7o8Kxe0DUNySJw4gHN+yRhb9", + "VJsHWqnSCU9TFnNqcFTKrIlg2Ouoi2u+Ja6Xm8TXCkhrw7C49ngzRMKmgKtN1+TwI+JxP+r12+3Fuw69", + "A/Vq/omKGGJYsE/4DHTQ9fZ9smr76MDASPRXO/JAbSXJOh6gSznQ60n6ZC4NKUPoO+rbctGuFyzGD5k1", + "87WYGZYg9j4JIEoBIMbXsa/jZyRTMs6jMn40wUGXiB8qF8OuQv7God2qfgMDs0HJsVm/0abQ6KafbNvv", + "hm4SCLZ9q3dHm7f6VpQi/V6exZt5mC3UjYNdC7l9QwhiQEVTX/aGJFiZzK8dOPrr6gquvnmtLjkCkSjP", + "vIEFaGqVkgLmFjQxhPx6n7GEGRZqhMgkLqMkuC656GaWunv4aNFm4kSL1OpAfmIsI9RY/CPsLwVrXmhg", + "Pu1ocZdsjTzat7YGr4FNV+RWqz64hxslsdatqqpw21IoWC7f0HlbvJSLivrbz6Ymmq0ioHiGXxPSXren", + "AHA1vdtbq/74NtRy9ymkvXKmhwZsqwcVLtDRff+lLzCIdXXiPQiZ50Nk8YbOMVfeBjtKK6mE8kqikYMS", + "Q11yK0ywCf8kmEVUe8AWLyBgdR/R65INEm4IndNm5oi2tJTWEr3Zh6g90csbOl+LydsMh62q358M/seq", + "28lkeLTz3V//78Gvfwmq3RsKBM3UIGYz9Li6YMuBzcIEyophHZEWU0D0tKFzd2YYTVGZhmjvjitVx/tg", + "VHDP5c80XZkCuqpVUijtbpzQX/+j3dGrsoxv8cLYeHY/OkPKbWSSNdLfy1spU3PvVO8j6raHY4FBehds", + "qUklMdt0WTux3+iiSsUVn5xbch8ycXlOphwzXeqxgOc9jSIG9uyhy/XDcSxKIhtWjCbVdlyCOH9enGXW", + "OlZA+PQKnPGrt2++f/X252eTV6fPf35yPPnp+X/3jnqCXQ1sD/EAaO/gwaFL0l5dyd1gopDr57sYkhMX", + "r+B8HmY5vuwRsAxspBCCliwJex8luQa3CVx6Y5KbZ7ZYjVq+eaaIj4RCNiYJuWc4yG6A4YYx23vVeRdx", + "7YmRa8xu77Q8XJBV0cUSzriHV4qRXpII5hWBrQivdrmx9UU/XHS7SyxIbOCwY+gvHLaAFvUl1wjc4Z1g", + "KoXJFobQ+BS8PoJ4+3qgtU+KBoMumJ8409Lo8afIhvp2bfrTS5kM4J3XkjIiqFa3axEMIcCmbGhGr836", + "Mp8GHjNOxz3ncxowuIQMK58ka6kf0MbQsZX9b03fFg7oeNbMp2GPpV2qRv6HhrZEm0F7vEcK4n0LAjFC", + "LNsgXl5xMqw1v5MKs+OyC4eAQ2KJqO7rwrbLU+ZhIgdYaXM08toHZmVmlZG0782JzEPbsm6BIB4feKRi", + "lY3ACiy+4ZK5AKUOcDRIbyRjqnTedZVRPEe7uyZbhQrML0ERdr1qF1ifh+KEvi96gBLwmmgaYnEeZR6s", + "3RffY86A126XgH26JnAYjVduGCG/TkXr1sRT1epmVKlqdd62fPDgOV61hvu1na0GcZZ91EgzRI9/o9z8", + "IBW+i9vBX24daB8v/5gpBMNrwuh3wqDnKYsnMjfrzz8UA+nRXvlFftgyv7DXAVAk4qgW19zGCzw8STmG", + "1ZWG5WBRrrhZgtLIvUymGA/qk/riQmJH+HPZMSZS/fABteezQOTMCyaY4hGmqYXzmFKBLyaA7SuzFdrE", + "lSvAtSgCvXp67PQuHvsY39HcIOl5B9Qnp8dgjGfK6h56o+H+cISHOWOCZrx31Nsf7g5HPXxVLXCKO1NI", + "s+8Cyd2LunjBH8dOEvreF4KaiqbMYI1fApAI6IDpiuMThM4rj8iMcuVekVki44JgONTGvAv+Qj2yt3Lf", + "Lntn5THGW2MYEMteuc39FQVlPDs4zb3RyCG8G3f9YhCTjZzY+btzoy377STVuSUKpCFYeeZ52bJY+g/9", + "3sFo91pjWjcUPLuhjt8K6qKZGb7OH4xGt9/psbDxiS7a3PmFVU8cElL1rP3yK+yZztOUqqVfsOpqZVK3", + "CcYM9FqurH3HGU0iYBWYLAkAbZj9DipRal24VS4wx7SvCBRaPwW2bb/JBVrT9zJefrIlrPXhdRQf6uwM", + "jsuHFXr+dLRTkPHqRrpPHmrcUu0dEND3tEiQfm8n5WD0+PY7Bd1vwiOIVvUE7ByzuUbfpwSB0z0Ik1Tk", + "t1waSoq4hi/oSDuZdVqQW7+8inb+4PEHe7wTFrIHnDKVUmGjRGyZDYd+5Thb20x5nNfeap7wj5/13E3l", + "0YjsRYWCXP2IVq+tpjC4eh0d9I7a+rTTi++R8A/u4IS7yRY5cu/zyGFmUrArf0nHydkap6UQEpTlXjDz", + "udD86C6vLJdN4U94ir4UAn7BCgmv3K2VS2EnU7mwD+CgBPi6jNx09b6pC39vyi8VdyG0a0DTmNfDWAMz", + "jZdD4tfUPvpB6astsL66tO5m9fN1CsP7XE7Y3l2cMJxxYSn6ek19vabWnXJLLX4KeDArp7yDDuJaGog/", + "n/7h2tqHr7qH7rqHTpoHwa5sacCDHBLnmhvJmBG9gCylYGy2wE/eCcdQNZz/TqiKFvySIbofZqvLE8Mz", + "qtDFJiVgJbA29FbFxFq1RNHcDjQ38A6Z5QI3AT00m6ALzKQNiLN0xeRCgLKWau81U+IqruRVt2c/qGAv", + "GiyvRnK1kJoVwIbCVG5zjPPW9nWMzQ7H4o1DvIUFRK9yz2s0SxC3d43+RwpCx8JV+NazEO8Rp2laci6q", + "EDyRW4hOuy0rU8WRTnQkQ6BDb5igwgx0xiI+45Gb1gVbOsfWYIOdElDBgP04IS+Md9Ta2w4D16HrUhil", + "+FnxjThKqttvhDQuNXdp5PJYSlRB7oNghpJ5Iqc0mdj1uWABm+ALLOEWpTS4lNYkIWNmc+lnS7OQwv6d", + "T3Nhcvv3VMkrzRTYnMYCI1LcWoO3QiEgkivMaJdmUhnr54B97tgh7vxxwZYfhmPxJE65sDW0rUITLcFM", + "AfWM9OAhlnu10IM9TWE7+NNcG5lWIWA93dlhytxkuXGhNZqZfgj+dCyMJH94kMsPO3+UPX5AYzGjMdBJ", + "pYidEsrWbaPWEwqzn2DRgLmd4QKMe3CRWjePuaLC4JhKlE4yr27pVpEmAlPHNlc4ooJkMrMpNpCoFhRI", + "rtYGglbQJCEGj5KvC4I77mTLfFIaLbhgk5LBBYxf7zOmeMoEBHGiSTUBN9mYqisudmyKc+R2nvMtYYR7", + "Dw4d4xuOxesi3MDePCmNpJ4gSwL9lP2nHXJxSyAkAxw0BHldWKvOAIsOLIODa5FsOQsP2d0ekhM7HX/s", + "gAot7hAxciwOD8gJ/74Scv2ti7WxdjafuKnvgscm6bRPgMvgGA1tOoIRCvg4msdFf1gOaL/kr09P3+7Y", + "5gggAMF9tmCKG0vNVLMheYkJ3+1J0yxSzOgdJDIiM/seg14V+zsiuYF/G5I/UQyWRhctlc8yGEdEBTAk", + "lQtYxBJHBnWo3LTRg595KyAlLmiTrXJBTr6vMNfRwaMwf3XzW239P89e/UxQSoMzaYuV7nt4ZriAFQCv", + "Kswl4+645wCbZQ2XmGVz3OPxuFeY9+NtHGuunfvUYIA25u9gaN/Zbvo8/m44hKas+fqI/PKHbeUIeGuW", + "WoDcce9Dn1Q+zLlZ5NPi26/hBW3D9TurXQxky4o9257gYYYVCdCKTLCt0kkFcHJIeSNVHZmmXFC1XBdh", + "G1h6t4JyZj1bK4vxxxg9Wce9o7H3ZR33+uMeE5f4m3N4Hfc+hFegcbpalLuFsbsgosPRaHszRLxb34AN", + "u4Oh6BPrBFpfyUU+WthBh0/857IX/VvrI74vXvDIcktea28Kb4z5Ag1SlRdcVTMRMEk1nmFURCzxz7DN", + "ir+7NybBZkUMBJm7JdD7Is/CXFqksPiiyBE3qzxGa80590xxo7u6VGpmnPuh3y/OnhKwpjhbCrv0ru/h", + "BD4IzuRUK8QWBoXVGY5pcMaEIc/x16H7r9cFINjoeSLn50dWlQO5KEjChYsPqTiuo9xu1xIrWXymop79", + "Z5E9cctKEv/6xz9xUFzM//WPf7qkB//6xz/xuO9Y3EFMvn6+YFSZKaPm/IhAFOSAJhBW4CaD6ZHZJVNL", + "sj9yNiD8BEMSLCqc7GVu9FjAo87kSlTiaGwiQ+0adKYjmA8XOdMO3woK8pnLsmR9XQN6PH+W7VLe6Ynu", + "B3BCcQaVCcCt6GkAQRa5zUDvNBO9sArdzrmmRG+67a44b27mL4a9N5Z6B3aA12QwuMShc4cf3KTJ1tnZ", + "8+0hwdeWpQrMpIVvh7IZ94wYfuVJm3mS5Sh1hoKrbHlTRDM65Qn3KuiWNED2CDo1TulvXoDw+yaO/EiB", + "x4DrrVOb9LHoWLw620GVu2GRyRXrO06gHPRumSdQurgn7MEpE9BbcODKjsWMUWiDHD+zTKCCTl8EyhYN", + "C0S4QZ9nbmopCftjYSGWHaQ3HLxUxizBStj/nBp2RZd9UiSB9mmDEmrgQaz7UHgsbOipW4MBYviQyjCH", + "yM/skAbes9vF8Ck2A2UOTsTlw8e+t0Cv4Dze+2Wcse/ORh/bYcGipTR6dQbzm+NLUFr9MLb06szvxnaf", + "aEmihCM1RFSMxRwdwzyqtRS1XS0CDBdUxYNIwiVQRTkDj/SExfM2Hvu00lbvFiWZWj+B4/Rjk1y/NOFi", + "sToBOMQWuXG9JfeZK9PNlOta/DPZcu0CXceYazX6zPIbu7pfDbsdDLvhdfNG3pCl9ZmHJr09D3DbxT05", + "gHvaW11z+6WyZPeh0SNbHvMJrWRSkdOnx4TGsWJab/976/tgppZKS/kP7kdgxffhiuTGIpXdC69vqRPI", + "l8IOXrtRE+rn1Uw8Xb3fdmpZqVpvuiJBVXnl3f7t0ej0OtdIKfSWtPb1JtnofM11BKGsVWoZoGiUsEJ8", + "Kc5plYo2aZWtW3dx5awVl2wp8JlxB/Lu9Muu61w074Y7YIrPGgzxHhlhPfS+mk7+S6Lmt8UuunmtUz9/", + "XqQ5ujsp6K5V0SEy/5Kei3Fj2YALWuCb1gv0BTMW7uY23+muh8DEQc/tTrUd6NLOupiWrUosbg9OCDUx", + "69++x7ZIt6evbe/P9PLF5bmOxOKW/KuI0uGxW67VugfuscuNfnvv2xqy3x27rTgCCywyfHC6aYTJgnSj", + "VC9FtP3Vc+WTUzQuduURq/y8WVxosvF7+c66K7kO8tyXifi5Lt61XJBZAqjB1ggQ8xn66plqYnsc5d4d", + "jLJIIK+oYc5F8UuMAz/NE2vdiKS4ZKqC3Fm9Unf+QCfmzU8lz7zW3q5vX78cMBHJuDCetMuk7ssnfjBZ", + "+q/Fdt/9qfsC45u5Fw/aBMaP2H8bXEAKKNf/tfeDA3P9X3s/0CTjgv2v/ScJNUyb7VsjltFd3XR3/YD5", + "gokP3i+8vmgrrGnH0Hk1LrlOmR7O+DqUWYMh9sruIrDqk1HfpxfymtjNnSS8O6N7eEndNYLNsU9TYbdU", + "Ko9m1oCo/vcW+s5qBH3n+rpa91zjCFCw+wK9lKXyKLzUwZhzT9yON4k5BvxsUEYUpTrqI3z5P5VKwk76", + "WkqJYl2/6iW66CWqy7VWNWEL3q5ywvZxT9b3gthCq42fvsKv3YFBx1FkBX6tZuEuAdgWUhv89OVhcbir", + "hBcUV702OlomywO59vrwpHv8rI8L2YelO35WhrrfkZ3Sj+POH96u37sXep6kUz7PZa6rUfQpNdGCaYcw", + "kbA6A/7SVALl9dyqFPiMqXR0l1fHnb/5v9L9LWkjmhtqmbd1Ntgk8/tSXWV+Vx5RDhB926FwuFxZfZ9H", + "cbslCMTjb3cl4xpM+Yd+l3GF3iKQe0ST8rlA8AVxNBb/G94fvxhG01+/8+Hd+Wi0d4i/M3H563c+wluc", + "eFJhwiju0t4++fkZenjM0YkfM+OW4CLNcVhUByQ9nwLn3+6BVDq5dH8heSr8+kLq9EKqLNf6F5IteMtP", + "pHoarTt/I3l6Cy24/fYnfSX9yU23tReczmczHnEmMBkZKv/0iq+yfcl9tdreECxBOF+JiqNjTRLp/Iws", + "uNYGCb3IcP1JPQn7rTlJJKHGsDQzAGoVMciNbCP09CI3sbwSPkUJTtBnu+PlfELXu29q4hsJB8iGU/Z3", + "fekWaTrv+qnrOv5CI1Rl5jLOu8dlKdq0vy7vl3hv903Z4aq9+1fll0xi9vm2unQZvBACKfesmSfNbThv", + "UbNE6xySN29e+tBdi5vmEjga6bM2+szeY1HN2jgkz8t0mLZAHXnNhfpDfZ8HMWY0TrhgGOvAdCjKtp5r", + "9V6PxaeXgMOJZO/axN3lWNoNvUcJ+N5YwZ3ImoVN3ziDbHE0Kzlmi9Pi5U08NV8Uv3IMKMB4QrLeDs2N", + "HDgwgJ2FtIihYdDk04RGiJkMxSycp8NfsQiP1aYQVEXJJGHKwrRmufHi1lgUg+PCyCIRuZPMzqH5SS4M", + "T877HigyYokmkATfIsqMRa0zJ/MhRgLif+AIFcvsiBtZlWHQHFSNUArhDKpdEppc0aUeC4eqYKtjKn7F", + "IotonCRD8qNEQBubXbzCeG1q32/0WJzzOGETh0dzTrgmeiGVYYLFJJWXTNf7ZVQlnCmcxFMAMFWapHSJ", + "wJAWM9muj8yYBV+sod5I+DcVMccksdBzMeWjsaBkbzQiKaNCOwwLTWd44bg2CA6iNqBvCSUHo8euVmPf", + "EMzeL/8WnCal2KWM6BTyAQMV2+Tr27iBqUvarJg2UsH2zbjSdr8K/abLxlnbWK59DuK4T3JRonSgrj8X", + "BagGbJfJlcB5Oisg46q4Bh0w0ZRFFNZTyHo/CBEsoyhXoQsStrqSPfzfUXCsTO8MlyqMgZGgyiACV3uJ", + "OIYLPNMSj9L2ty1UVRLVn+OiCR4SqQglFbouNRosypE1biGE6nmZClf41Pbn29/6swPH1zECf/zF/Eu6", + "n5CI5GxWO4CbryZ7gNfFnq2S8J/1nD51TK/G4mJO50JqwyPPDD1iefXR/PVB2OlBuH5lg9Q8k+qi3eH4", + "B6kuur7AvGPkl/UQq87wMzREwPAwKcL92yNQG24fK0A0d/5Ia9JXcUpR6OJGFy7BBODlmapq5e/cbFB9", + "1W1ZQEu4TJU1dhfwY/AImbgfbRr1ils3mhgi1+p98yLo/Q6MUT9LQ3iaJSxlmGZ9YIkNNruEqpsu7dOj", + "BGy7Hq+EU1XFVbBvQW39D/peHEK68hu2hdL76nYFmSrgeW4ERC069+ifAUTUsXirbeqKc2t6OicFDwaB", + "1qZLIFcLHi2gHfwN27fgqTTLzgtg+O0j8gIPcmXbbOdbNgkN0JqWCbOgp5dpen60mkj83ckJVsIyLrnE", + "+RHxycOL+0NDqSraKcwiodqQnx2G61bxGMcdPQcdZmV+2w4HtQTuH4sQJipAitoG+YycV+BRz1uw+zy/", + "fSnn+rMxFZXpd+xcjCTu6Yi0yUTca3Py4EnY8LM7GoWyAHREabXDuGWQ1pXBvJTzIvVPjZRplnUlXzdM", + "pOLLNF1Dw2Sr5CBEm1jm5q/axEwprOyou424yRaN7D8MvQBCdQCf/mBvj0XLUtkZhpcKmGqv32MiT3tH", + "v7h/XaZpr99z44F6VyZLK7kbriHkb0C9bTb4oR/aoQq07Vfx/DqgtXWmX0Gtbdwg7lndLpm/tgX+9FZD", + "r7u7M4nYPRIaEol9tAJILX7EZbq/g4GSS0O9/DkJx5Xx1lVRQhZ4wDiyx3cxMgQNJlEiNauZnr4cyEGn", + "gmtIs+0qLL/GAxhenPucfV18a85c1TNf8zPQCmzyYvFjJn66d+7OsjqCLxk+Qa/MBthfA6duk5/LZ09I", + "n25LVqbahUK+0ub19Z+dCBNeMKsswlWIbQ5XmhuZUsMjzBcXLaTUFbIvQOVtpk+n1i4oE5U+9v3tYhvO", + "gVTPnYL83D1wjpwyj9DqJ9fHEKu7iIhwDf+prPFDRV9RcPy+f5RgThUNObUUZzOS0VwzkDNzUDIto4T5", + "BIEMTNURzUyuGObCZSTlgqd5Ws0WADt2SRHZ6Hw3Pe+TaW5IAoorVXz0bkCRTFMmYisnjcWC0UvOFCxK", + "Qg0T0XKgGebQv2QEEhYkksao/MhiijYozDapGFAgpl5ImaExNRQFnXM48RMbXnVepNW3CgfB3pfUAObY", + "XHxr88BAs+d+oOeEYaIDrhdF+uWIxkxEwQQAZ583G/v0WvIzZpoTvSefpRvx0vt0Yqpqg/1wPg//pi8W", + "rKIDm18j9Or2R3U9LsWT0b/nkbZz9XO8J9NXscTrTvHnYfMqiO6zsXvdv2FLKhLntrvKqUQy/7NaqwqG", + "UnUDw5hXu403NVkVeUWLZb4Wz9v5w/95fAPt4mfCCfutD/u2DHblpD8HlutW9UY8957Uqk6XVNXJ3R8L", + "doO6P/FJqgqX+1KUrTXMuIJvV7mTURRfX1J8ZdtNtu1cMW7Ktr1udsXZoMLIuRig92qYgzs1biurdqqD", + "f9M4mcbsKizz3llkabm4cwBIzxozugRVyZ/BfXmN/SiSSllgDmrYlwVaXdEaVgMXUDdXpsbs+zjSdycn", + "221cQpm1PEKZL5hDVIKFoFoarypwX10ypXjsgTmfnjwrbbIqF0PyKuWGGEkuGMvKWBuMdwS8yQKipDHs", + "JhZJv8eEUctMcmE2jqIsejuDKX+QDqnlMxQlXRaEr+bwzuZw1Ox/eewMuQyhxQTWv0wNNRuzM3Mxkyq1", + "chmdyhxaBx4EywT7aZ0tZjxheqkNS62/JKArwHHDjDoua7qrZ3e5j97CcHJsIF/GVMq15lLosXBRLBlT", + "0DdUh/Yrrl9Bg4ChBX89tUzy83ArhMFYTzpq2lYNwaQwG3PvqLdDs2wnpoa2uK654X3EkH5AP0Gil+kU", + "YhLB0fBCk62EX9jnCbnUJIE/ttc6Gk6w3qfOCX/zkwUrfSxmMphx09JsQcx/qngzx9a8YfKzYGv39M4M", + "eold18hdHlzPC2eylcVuzoCvGE0GmFHewxuR3PCE/27ZLjTCteGRDcyixT6+OykY/HAsTphRPLL23ghc", + "uhH3AV+6O5mS0Q5A4+1HGUeMvH2Gg0Pm2/45xR6fnr7FcilLpVr2x4IK1/CbJ6fW0jyjEWsO1KW+J8c7", + "rza4gZ/hMv0b+0/aCa7FeAhu+Ffz5PWRWFrPkG45ojJb92yT2Z/ewddJk191HF+mjgNouJzNVgF/5jHL", + "wvqMS5nkKfzD/nG8Cf3N0GjxDot+NpK3Hc7GbvwEv4hD6eYUM5ud+F4MMHbBvlT/WVg4PwUUYmqeicFb", + "4In5M1L3pzclVNfxMzS9uhX1mb8/m7N11zefG4PHIamux5dyzC2l+ZkYuV4TdkV5uybs+0RGF9oB1lRV", + "mEQxiij08GOJGu7MlSgmYPwscUBPFk6M6f5YNJShFhdJE0oMUykXNNnBOdtGEP/ca9TopeQYxh5hzMxA", + "g6o7k0mCIOcIEgizgRIz30DFuqxdbsRqmaph1EhAQpIp85jw26Gn298oNz9IVQd4/1z44pvK+iNwIuWo", + "+9+Aad/e40dh3J/Q9+i2HefOuO1HtPVClj9atVSf4N6Me/sjPe71ybi3l457sANPKapzqSEPSMpFbgAs", + "8pnVtWGg8uGIaBZJEWsPTe+1ifsj3TJzR5YtMbCHWO8uxR5HVbiUr10nIfYA5QjUxwAislU9cO5Mxn08", + "dDGRubGmB3euXKmYGVSPbN+53bhyRr6+7btw8r+541vjUbjLwC4rW285e5brBWtXub206Z5yM4V1KNJD", + "6wX5u5zqPkQdWM08gNCt8D2ofWo7uIt0DNDVdVIxuLl/zcPQIQ9DuVZhSEvr7AlXsqcOi2vJ3mdSGfjd", + "QQE4GsKXBOJrYBLHV0+PxyICVmQBGBVLJXInhxpvb+Enfzsjz5++7pNnmKqY/JhPt4fkFUTaYKva2YvG", + "wkpilnlFFLDK7Qzi0PVsx47Uc5uO69DBPeX+tycjYOXxe+Ud1vu9BaMxSiR/9F5K21kAm/n1SzhAsH+u", + "ZrHtvbXCB+jw1XLwZGaYWm32xMVsiQJZxF3SHqjPCW7YMXaoPT5d2aeVDSyAyP5eL4An8uFraozbT3F9", + "NxY77NclJYRovi81uy0exII5hlhg9boukku0RSw7Xrb2gYFdHj/7JI+KW7VSr+NdNQT+f9fThTP9Yg1N", + "WW2fgIiLpDQbLb0+UHlhQaNtNRLRjEbcLPuEJom7o9xNUHjHDArxd6oYvQC9/RCAvVzPPrgYbLV9b6gl", + "MdcXtgVnix0ScI/T+bQYHMGDZq3GuOYQomskiWgS5Qk1jLDZjEUYF4xZbnSLLbcYSu8Wz07ZSRAWpuJh", + "n39xmQDDNIG7V5JFk+J27FbvKBYllKftEO1OUEPnR3Q1mEKjUoDvQmIpjkZKak1cUwOW8DmHTBXoKQHJ", + "KUDNRVM2FllChWCK5Np66MPQB5liWueK+Qbg7nIU1Scl/GKmpHGuCYmUSo+Fp/B3J0Qblq0hs9e25ROc", + "8y3JtrZx19M9KakbY2hXhbgiBDbEUopdcKAjUA3dg1u8HdB9S4lfysF/o/gcfF8zJallstYvyR5rv5z2", + "0Neip1uzgp4VpbplBS1arURIVqIH18LnTUpE8rh3PQ/EQOcXvBVh0X26XkTzT1CpY9/1yNnwINynj5xl", + "KMHpv2Mu0bNKwGJXBVZJ4V+aOqky8tpRrQX9bob46hzle5tRt52xvO4NwutLRu6itVDetgfv50cIo7tF", + "nLjrVHRfNm3VkLdqb9MW+IHNOQc+Cwq8nWQD94y4coNkA58VBgDQzj1isQQP6n3F9Ndsz1J9zRdwq6H8", + "NmkAQsO1hfJbruecV9c+lN65Mt2eSa7FP5MEbxfoOvK7X/avr/4OT4bKYm0yQQPBszQzS1fJ2ypLpzPN", + "f2fDFkNw4bd6e6bgG7h0fjry8HTa6tB5j+hl92jIuRefUZdgkWtw81tNzf+F4R1Wz1ztYtmBW2dAVbTg", + "l6xd6V4/wW6JMsUGmczQuBLbBXPr4e8yQ9Vw/jtxzTv8V/cvzNCJiQxYTGKuWGTAW0QYiRzB9vGNJkpK", + "475LtWz3ErFH5Acl0yduNhvuQ3emnDKs9DNMl4OYGjq49NxmjQrtI7w7vT8lMDzCBXnxPdli742yeUDI", + "DF4+EG3ul5S9jxiLNdLkdnXAu6PwEKHhyXzaZZRrMrq8wj9oQqJcG5n6vT9+RrYwQ9ycCdgLEPVnKMlm", + "Sl7ymMW1MYLzvl3V3ZYFva7eFYSKIr2ff1zYwd2LDNPlQpr/zrM6WyhcYqZcUBzcxpwp9TNlAQWgP8qF", + "d8Bxe+RH8fUKcy+/Lf/YAUrEbKVuEY2UFm56++s19yVfc9VgKH+n1W47756zXnndLT6qY9jSbSShKGLn", + "7lZt/e7zCenh+ouM5nGq88viQdqmNv+8SHB0d/fDXavL333BIaCgLL9sLJttQF2GCeYl+nTH7JIlMksx", + "azyW7fV7uUp6R72FMdnRzg76fi+kNkcHjx/u9z78+uH/HwCvG9+xcP0BAA==", } // GetSwagger returns the content of the embedded swagger specification file diff --git a/lib/providers/providers.go b/lib/providers/providers.go index d8aa41d4e..3ba171c61 100644 --- a/lib/providers/providers.go +++ b/lib/providers/providers.go @@ -5,6 +5,7 @@ import ( "fmt" "log/slog" "os" + "path/filepath" "strconv" "strings" "time" @@ -417,9 +418,13 @@ func ProvideBuildManager(p *paths.Paths, cfg *config.Config, instanceManager ins log.Info("registry CA certificate loaded", "file", cfg.Registry.CACertFile) } - registryURL, err := builderRegistryURL(cfg.Registry.URL, networkManager) - if err != nil { - return nil, err + registryURL := cfg.Registry.URL + if !cfg.MacOSBuilds { + var err error + registryURL, err = builderRegistryURL(registryURL, networkManager) + if err != nil { + return nil, err + } } if registryURL != cfg.Registry.URL { log.Info("resolved registry URL for builder VMs", "original", cfg.Registry.URL, "resolved", registryURL) @@ -436,6 +441,20 @@ func ProvideBuildManager(p *paths.Paths, cfg *config.Config, instanceManager ins DockerSocket: cfg.Build.DockerSocket, } + if cfg.MacOSBuilds { + if !cfg.MacOSOnly || registryURL == "" { + return nil, fmt.Errorf("machine builds require macOS-only mode and an explicit host registry URL") + } + workDir := filepath.Join(p.BuildsDir(), "machine-work") + if err := os.MkdirAll(workDir, 0700); err != nil { + return nil, err + } + buildConfig.MachineBuild = &builds.MachineBuildBackend{ + Driver: &builds.GuestMachineBuildDriver{Instances: instanceManager, WorkDir: workDir}, + Publisher: &builds.StreamingMachinePublisher{RegistryURL: registryURL, Tokens: builds.NewRegistryTokenGenerator(cfg.JwtSecret)}, + } + } + // Configure secret provider (use NoOpSecretProvider as fallback to avoid nil panics) var secretProvider builds.SecretProvider if cfg.Build.SecretsDir != "" { diff --git a/openapi.yaml b/openapi.yaml index b23af2c15..33bb9eab2 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -4668,6 +4668,16 @@ paths: dockerfile: type: string description: Dockerfile content. Required if not included in the source tarball. + machine_base_image: + type: string + description: | + Experimental installed darwin/arm64 base pinned by sha256 digest. + Requires server macos_only and macos_builds. Source must contain + hypeman-macos-build.json (version 1). Machine source is limited to + 64 MiB compressed; only timeout_seconds, memory_mb, cpus and tags + are accepted alongside source and this selector. CPU/memory zero + inherit the base. Linux/cache/secrets/admin options are rejected. + Output retains the base identity and cannot run concurrently with it. base_image_digest: type: string description: Optional pinned base image digest From d5871ad91061f57328be0d7bd10dc8c48954cbbd Mon Sep 17 00:00:00 2001 From: chris lee Date: Sun, 11 Oct 2026 00:39:52 -0400 Subject: [PATCH 6/7] Abort failed machine builds without exporting; split strict stop from export --- cmd/api/config/config.go | 36 +++++++-- cmd/api/config/macos_builds_test.go | 18 ++++- docs/macos-builds.md | 28 ++++--- lib/builds/builder_disk_test.go | 2 +- lib/builds/machine.go | 10 ++- lib/builds/machine_abort_test.go | 74 +++++++++++++++++++ lib/builds/machine_cancel_test.go | 2 +- lib/builds/machine_failure_test.go | 2 +- lib/builds/machine_guest.go | 54 +++++++++----- lib/builds/machine_manager.go | 3 +- lib/builds/machine_test.go | 6 +- lib/builds/manager.go | 31 ++++++-- lib/instances/macos_build_export_darwin.go | 60 ++++++++------- .../macos_build_export_darwin_test.go | 8 +- 14 files changed, 252 insertions(+), 82 deletions(-) create mode 100644 lib/builds/machine_abort_test.go diff --git a/cmd/api/config/config.go b/cmd/api/config/config.go index d401cbd94..9b273bf17 100644 --- a/cmd/api/config/config.go +++ b/cmd/api/config/config.go @@ -2,6 +2,7 @@ package config import ( "fmt" + "net" "os" "path/filepath" "runtime" @@ -591,15 +592,12 @@ func expandHomePath(path string) string { // Validate checks configuration values for correctness. // Returns an error if any configuration value is invalid. func (c *Config) Validate() error { - if c.MacOSBuilds && !c.MacOSOnly { - return fmt.Errorf("macos_builds requires macos_only") - } - if c.MacOSBuilds && (c.Registry.CACertFile != "" || (c.Registry.Insecure && strings.HasPrefix(c.Registry.URL, "https://"))) { - return fmt.Errorf("macos_builds requires system-trusted registry TLS or loopback HTTP") - } if c.MacOSOnly && (runtime.GOOS != "darwin" || runtime.GOARCH != "arm64" || c.Hypervisor.Default != "vz") { return fmt.Errorf("macos_only requires vz on Apple silicon") } + if err := c.validateMacOSBuilds(); err != nil { + return err + } if strings.TrimSpace(c.Metrics.ListenAddress) == "" { return fmt.Errorf("metrics.listen_address must not be empty") } @@ -846,3 +844,29 @@ func (c *Config) validateFirecrackerUFFDGraduation() error { func intPtr(v int) *int { return &v } + +// validateMacOSBuilds enforces the experimental machine-build registry boundary. +func (c *Config) validateMacOSBuilds() error { + if c.MacOSBuilds && !c.MacOSOnly { + return fmt.Errorf("macos_builds requires macos_only") + } + if c.MacOSBuilds && c.Registry.CACertFile != "" { + return fmt.Errorf("macos_builds requires system-trusted registry TLS or loopback HTTP") + } + if c.MacOSBuilds && c.Registry.Insecure && strings.HasPrefix(c.Registry.URL, "https://") { + return fmt.Errorf("macos_builds requires system-trusted registry TLS or loopback HTTP") + } + if c.MacOSBuilds && strings.HasPrefix(c.Registry.URL, "http://") { + host := strings.TrimPrefix(c.Registry.URL, "http://") + if i := strings.IndexAny(host, "/?#"); i >= 0 { + host = host[:i] + } + if h, _, err := net.SplitHostPort(host); err == nil { + host = h + } + if host != "localhost" && !net.ParseIP(host).IsLoopback() { + return fmt.Errorf("macos_builds plaintext registry must be loopback") + } + } + return nil +} diff --git a/cmd/api/config/macos_builds_test.go b/cmd/api/config/macos_builds_test.go index 0fe52e3ee..559ac6b5d 100644 --- a/cmd/api/config/macos_builds_test.go +++ b/cmd/api/config/macos_builds_test.go @@ -2,18 +2,30 @@ package config import ( "github.com/stretchr/testify/require" + "runtime" "testing" ) func TestMacOSBuildsFailClosedConfiguration(t *testing.T) { + if runtime.GOOS != "darwin" || runtime.GOARCH != "arm64" { + t.Skip("macos_only requires Apple silicon") + } require.False(t, (&Config{}).MacOSBuilds) c := Config{MacOSBuilds: true} - require.ErrorContains(t, c.Validate(), "requires macos_only") + require.ErrorContains(t, c.validateMacOSBuilds(), "requires macos_only") c.MacOSOnly = true + c.Hypervisor.Default = "vz" c.Registry.CACertFile = "custom.pem" - require.ErrorContains(t, c.Validate(), "system-trusted registry TLS") + require.ErrorContains(t, c.validateMacOSBuilds(), "system-trusted registry TLS") c.Registry.CACertFile = "" c.Registry.URL = "https://registry.example" c.Registry.Insecure = true - require.ErrorContains(t, c.Validate(), "system-trusted registry TLS") + require.ErrorContains(t, c.validateMacOSBuilds(), "system-trusted registry TLS") + c.Registry.Insecure = false + c.Registry.URL = "http://registry.example:4973" + require.ErrorContains(t, c.validateMacOSBuilds(), "must be loopback") + c.Registry.URL = "http://127.0.0.1:4973" + require.NoError(t, c.validateMacOSBuilds()) + c.Registry.URL = "http://localhost:4973" + require.NoError(t, c.validateMacOSBuilds()) } diff --git a/docs/macos-builds.md b/docs/macos-builds.md index d0134223e..f9f17ea82 100644 --- a/docs/macos-builds.md +++ b/docs/macos-builds.md @@ -78,15 +78,21 @@ reviewed clean base, not a private development guest, for distributable images. ## Stop, export and cleanup safety -`StopAndExportMacOSInstance` is an internal optional build capability, not a new -HTTP endpoint. It holds the instance lock across shutdown and both storage clones, -preventing Start from racing export. It rejects a pre-existing Stopped state, -unmanaged/non-macOS guests and any forced-stop fallback. A valid receipt requires -shutdown acknowledgement and actual owned VMM exit. Closed/distinct/nonempty -regular disk/aux files are checked before APFS cloning. The exclusive0700 output -contains only private0600 disk.img, aux.img and bounded platform config.json. -An export failure can retain a proven stop receipt for safe cleanup, but never -claims successful publication. +The publication path is two instance operations, not one copy-inside-stop: +`StopMacOSBuildInstance` performs only the strict graceful stop (no storage copy) +and rejects forced fallback, a pre-existing Stopped state, unmanaged/non-macOS +guests and any stop without shutdown acknowledgement plus owned VMM exit. +`ExportStoppedMacOSBuildInstance` later copies the closed storage of a +still-Stopped instance under the instance lock, so a concurrent Start makes export +fail rather than race it. Each storage file is copied once, into a private +exclusive 0700 output that holds only 0600 disk.img, aux.img and bounded config.json. + +Failure cleanup uses `Abort`, not `Stop`. Nothing from a failed build is exported, +so cleanup first attempts the strict stop, then uses the ordinary forced +hypervisor stop and deletes storage only after confirmed VMM exit. A VM that still +has no confirmed exit remains quarantined for operator recovery; there is no +automatic reaper yet. Cleanup has a 2-minute budget (strict stop window plus forced +fallback and deletion), separate from the build deadline. Darwin shutdown accepts a detached, short-delayed power-off command so gRPC can send its acceptance reply before vsock disappears. RPC cancellation after @@ -96,9 +102,7 @@ an acknowledgement alone is insufficient. The backend validates complete output and preserved identity/resources, destroys the stopped builder and consumes staged source **before** publication. Cancellation interrupts the active normal machine job, not forced instance deletion. It records -the owned builder before provisioning and keeps Cancelled terminal. Cleanup has -its own30s context and may delete storage only after confirmed exit; otherwise it -retains quarantine. Potentially secret-bearing guest errors remain masked in +the owned builder before provisioning and keeps Cancelled terminal. Potentially secret-bearing guest errors remain masked in ordinary error/log strings while errors.Is/As work internally. ## Streaming publication diff --git a/lib/builds/builder_disk_test.go b/lib/builds/builder_disk_test.go index 74d13f856..f06f7a1ce 100644 --- a/lib/builds/builder_disk_test.go +++ b/lib/builds/builder_disk_test.go @@ -27,7 +27,7 @@ func prepareBuildOnDisk(t *testing.T, mgr *manager, id string, req CreateBuildRe CreatedAt: time.Now(), } require.NoError(t, writeMetadata(mgr.paths, meta)) - _, err := mgr.storeSource(context.Background(), id, []byte("fake-tarball-data")) + _, err := mgr.storeSource(context.Background(), id, []byte("fake-tarball-data"), false) require.NoError(t, err) config := &BuildConfig{ diff --git a/lib/builds/machine.go b/lib/builds/machine.go index 655106f2b..d91cdbc75 100644 --- a/lib/builds/machine.go +++ b/lib/builds/machine.go @@ -44,6 +44,8 @@ type MachineBuildSession interface { Sanitize(context.Context) error // Remove build credentials, source and secret injection artifacts. // Stop must await actual VMM exit. Forced-stop fallback must not claim graceful. Stop(context.Context) (MachineStopReceipt, error) + // Abort is failure-only cleanup. It may force-stop because nothing is exported. + Abort(context.Context) (MachineStopReceipt, error) Export(context.Context, string) error // Stopped disk.img, aux.img, config.json only. // Destroy removes instance storage only after proven VMM exit, not the export. Destroy(context.Context) error @@ -143,10 +145,10 @@ func (r *MachineBuildBackend) runPrepared(ctx context.Context, req MachineBuildR } stopped, destroyed := false, false defer func() { - cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second) + cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), machineCleanupTimeout) defer cancel() if !stopped { - receipt, stopErr := session.Stop(cleanup) + receipt, stopErr := session.Abort(cleanup) stopped = receipt.VMMExited if !stopped { err = errors.Join(err, machineFailure("quarantine_unconfirmed_vmm_exit", stopErr)) @@ -212,6 +214,10 @@ func (r *MachineBuildBackend) runPrepared(ctx context.Context, req MachineBuildR return &MachineBuildResult{Publication: publication, BuilderInstanceID: instanceID, Provenance: BuildProvenance{BaseImageDigest: base.Digest, SourceHash: req.SourceHash, Timestamp: time.Now().UTC()}}, nil } +// machineCleanupTimeout covers a strict graceful stop attempt (30s power-off +// budget) plus forced fallback and storage deletion. Export is outside cleanup. +const machineCleanupTimeout = 2 * time.Minute + func machineDigest(s string) bool { if !strings.HasPrefix(s, "sha256:") || len(s) != 71 { return false diff --git a/lib/builds/machine_abort_test.go b/lib/builds/machine_abort_test.go new file mode 100644 index 000000000..c0b0020d1 --- /dev/null +++ b/lib/builds/machine_abort_test.go @@ -0,0 +1,74 @@ +package builds + +import ( + "context" + "errors" + "testing" + + "github.com/kernel/hypeman/lib/instances" + "github.com/stretchr/testify/require" +) + +type abortInstances struct { + instances.Manager + state instances.State + forced bool + forcedErr error +} + +func (a *abortInstances) GetInstance(context.Context, string) (*instances.Instance, error) { + return &instances.Instance{State: a.state}, nil +} + +func (a *abortInstances) StopInstance(context.Context, string) (*instances.Instance, error) { + a.forced = true + if a.forcedErr != nil { + return nil, a.forcedErr + } + return &instances.Instance{State: a.state}, nil +} + +type abortExporter struct { + receipt instances.MacOSBuildExportReceipt + err error +} + +func (e abortExporter) StopMacOSBuildInstance(context.Context, string) (instances.MacOSBuildExportReceipt, error) { + return e.receipt, e.err +} + +func (abortExporter) ExportStoppedMacOSBuildInstance(context.Context, string, string) error { + return errors.New("abort must not export") +} + +func newAbortSession(m *abortInstances, e abortExporter) *guestMachineSession { + return &guestMachineSession{manager: m, exporter: e, instance: &instances.Instance{StoredMetadata: instances.StoredMetadata{Id: "builder"}}} +} + +func TestMachineAbortUsesStrictStopBeforeForcedFallback(t *testing.T) { + m := &abortInstances{state: instances.StateRunning} + s := newAbortSession(m, abortExporter{receipt: instances.MacOSBuildExportReceipt{Graceful: true, VMMExited: true}}) + receipt, err := s.Abort(context.Background()) + require.NoError(t, err) + require.True(t, receipt.VMMExited) + require.False(t, m.forced, "a confirmed graceful exit must not force-stop") +} + +func TestMachineAbortForcedFallbackConfirmsVMMExit(t *testing.T) { + m := &abortInstances{state: instances.StateStopped} + s := newAbortSession(m, abortExporter{err: errors.New("agent never became ready")}) + s.instance = &instances.Instance{StoredMetadata: instances.StoredMetadata{Id: "builder"}} + receipt, err := s.Abort(context.Background()) + require.NoError(t, err) + require.True(t, receipt.VMMExited) + require.False(t, receipt.Graceful) +} + +func TestMachineAbortWithoutConfirmedExitStaysQuarantined(t *testing.T) { + m := &abortInstances{state: instances.StateRunning, forcedErr: errors.New("hypervisor still alive")} + s := newAbortSession(m, abortExporter{err: errors.New("agent never became ready")}) + receipt, err := s.Abort(context.Background()) + require.Error(t, err) + require.False(t, receipt.VMMExited) + require.Error(t, s.Destroy(context.Background()), "Destroy must refuse without confirmed VMM exit") +} diff --git a/lib/builds/machine_cancel_test.go b/lib/builds/machine_cancel_test.go index d5b5df83c..7c08d30da 100644 --- a/lib/builds/machine_cancel_test.go +++ b/lib/builds/machine_cancel_test.go @@ -29,6 +29,6 @@ func TestMachineCancellationInterruptsPhasesWithoutPublishing(t *testing.T) { require.NoError(t, e) require.Equal(t, StatusCancelled, done.Status) require.False(t, fixture.publisherCalled) - require.Contains(t, fixture.calls, "stop") + require.Contains(t, fixture.calls, "abort") require.Contains(t, fixture.calls, "destroy") } diff --git a/lib/builds/machine_failure_test.go b/lib/builds/machine_failure_test.go index 69f36b8ff..e02c36cc4 100644 --- a/lib/builds/machine_failure_test.go +++ b/lib/builds/machine_failure_test.go @@ -45,7 +45,7 @@ func TestMachineBuildTimeoutCleanup(t *testing.T) { if !errors.Is(err, context.DeadlineExceeded) || result != nil || f.publisherCalled || !f.cleanupContextLive { t.Fatal("timeout cleanup failed") } - if !reflect.DeepEqual(f.calls, []string{"resolve", "start", "stop", "destroy"}) { + if !reflect.DeepEqual(f.calls, []string{"resolve", "start", "abort", "destroy"}) { t.Fatalf("timeout cleanup order: %v", f.calls) } } diff --git a/lib/builds/machine_guest.go b/lib/builds/machine_guest.go index efecb44bc..7cc670608 100644 --- a/lib/builds/machine_guest.go +++ b/lib/builds/machine_guest.go @@ -7,7 +7,6 @@ import ( "path/filepath" "time" - "github.com/kernel/hypeman/lib/forkvm" "github.com/kernel/hypeman/lib/guest" "github.com/kernel/hypeman/lib/hypervisor" "github.com/kernel/hypeman/lib/images" @@ -15,7 +14,8 @@ import ( ) type machineExportManager interface { - StopAndExportMacOSInstance(context.Context, string, string) (instances.MacOSBuildExportReceipt, error) + StopMacOSBuildInstance(context.Context, string) (instances.MacOSBuildExportReceipt, error) + ExportStoppedMacOSBuildInstance(context.Context, string, string) error } // GuestMachineBuildDriver uses ordinary instance lifecycle and the shared system @@ -134,36 +134,50 @@ func (s *guestMachineSession) Sanitize(ctx context.Context) error { return nil } +// Stop is the publication path: only an acknowledged graceful power-off with +// confirmed VMM exit may produce a receipt that permits export. func (s *guestMachineSession) Stop(ctx context.Context) (MachineStopReceipt, error) { if s.receipt.VMMExited { return s.receipt, nil } - // Invalid source can fail before the guest finishes booting. Cleanup must - // await the declared agent rather than firing two early shutdown attempts. - if err := s.exec(ctx, []string{"/usr/bin/true"}, "", nil, 20*time.Second); err != nil { - return s.receipt, err - } - receipt, err := s.exporter.StopAndExportMacOSInstance(ctx, s.ID(), filepath.Join(s.workspace, "bundle")) + receipt, err := s.exporter.StopMacOSBuildInstance(ctx, s.ID()) s.receipt = MachineStopReceipt{Graceful: receipt.Graceful, VMMExited: receipt.VMMExited} return s.receipt, err } +// Abort is failure cleanup only. Nothing from a failed build is exported, so after +// a strict attempt it may use the ordinary forced hypervisor stop. Destroy still +// requires confirmed VMM exit; a non-exited instance remains quarantined. +func (s *guestMachineSession) Abort(ctx context.Context) (MachineStopReceipt, error) { + if s.receipt.VMMExited { + return s.receipt, nil + } + if receipt, err := s.exporter.StopMacOSBuildInstance(ctx, s.ID()); err == nil && receipt.VMMExited { + s.receipt = MachineStopReceipt{Graceful: receipt.Graceful, VMMExited: true} + return s.receipt, nil + } + if inst, err := s.manager.GetInstance(ctx, s.ID()); err == nil && inst.State == instances.StateStopped { + s.receipt = MachineStopReceipt{VMMExited: true} + return s.receipt, nil + } + inst, err := s.manager.StopInstance(ctx, s.ID()) + if err != nil { + return s.receipt, err + } + if inst == nil || inst.State != instances.StateStopped { + return s.receipt, fmt.Errorf("forced stop did not confirm VMM exit") + } + s.receipt = MachineStopReceipt{VMMExited: true} + return s.receipt, nil +} + +// Export copies only from closed storage of a Stopped instance, under the +// instance lock, after a strict graceful receipt and sanitation. func (s *guestMachineSession) Export(ctx context.Context, destination string) error { if !s.receipt.Graceful || !s.receipt.VMMExited || !s.sanitized { return fmt.Errorf("sanitized strict stop receipt required") } - for _, name := range []string{"disk.img", "aux.img", "config.json"} { - if err := ctx.Err(); err != nil { - return err - } - if err := forkvm.CopyRegularFile(filepath.Join(s.workspace, "bundle", name), filepath.Join(destination, name)); err != nil { - return err - } - if err := os.Chmod(filepath.Join(destination, name), 0600); err != nil { - return err - } - } - return nil + return s.exporter.ExportStoppedMacOSBuildInstance(ctx, s.ID(), destination) } func (s *guestMachineSession) Destroy(ctx context.Context) error { diff --git a/lib/builds/machine_manager.go b/lib/builds/machine_manager.go index baf7ade84..a15ad3493 100644 --- a/lib/builds/machine_manager.go +++ b/lib/builds/machine_manager.go @@ -4,6 +4,7 @@ import ( "context" "fmt" "path/filepath" + "strings" "github.com/kernel/hypeman/lib/images" ) @@ -28,7 +29,7 @@ func (m *manager) executeMachineBuild(ctx context.Context, id string, req Create return nil, err } ref, err := images.ParseNormalizedRef(result.Publication.Reference) - if err != nil || ref.Repository() != stripRegistryScheme(m.config.RegistryURL)+"/builds/"+id { + if err != nil || ref.Repository() != strings.TrimSuffix(stripRegistryScheme(m.config.RegistryURL), "/")+"/builds/"+id { return nil, fmt.Errorf("machine publication is outside the authorized job repository") } return &BuildResult{Success: true, ImageDigest: result.Publication.Digest, Provenance: result.Provenance}, nil diff --git a/lib/builds/machine_test.go b/lib/builds/machine_test.go index 4bebe903e..073322c24 100644 --- a/lib/builds/machine_test.go +++ b/lib/builds/machine_test.go @@ -74,6 +74,10 @@ func (f *machineFixture) Stop(ctx context.Context) (MachineStopReceipt, error) { f.cleanupContextLive = ctx.Err() == nil return f.stop, f.record("stop") } +func (f *machineFixture) Abort(ctx context.Context) (MachineStopReceipt, error) { + f.cleanupContextLive = ctx.Err() == nil + return f.stop, f.record("abort") +} func (f *machineFixture) Export(_ context.Context, root string) error { if err := f.record("export"); err != nil { return err @@ -205,7 +209,7 @@ func TestMachineBuildCancellationUsesIndependentCleanup(t *testing.T) { if !errors.Is(err, context.Canceled) || !f.cleanupContextLive || f.publisherCalled { t.Fatal("cancellation did not retain safe cleanup") } - if !reflect.DeepEqual(f.calls, []string{"resolve", "start", "stop", "destroy"}) { + if !reflect.DeepEqual(f.calls, []string{"resolve", "start", "abort", "destroy"}) { t.Fatalf("cancel cleanup: %v", f.calls) } } diff --git a/lib/builds/manager.go b/lib/builds/manager.go index 21a724ab9..e10e3ec38 100644 --- a/lib/builds/manager.go +++ b/lib/builds/manager.go @@ -4,7 +4,9 @@ import ( "bufio" "bytes" "context" + "crypto/sha256" _ "embed" + "encoding/hex" "encoding/json" "fmt" "log/slog" @@ -536,7 +538,7 @@ func (m *manager) CreateBuild(ctx context.Context, req CreateBuildRequest, sourc } // Store source data - hash, err := m.storeSource(ctx, id, sourceData) + hash, err := m.storeSource(ctx, id, sourceData, req.MachineBaseImage != "") if err != nil { deleteBuild(m.paths, id) return nil, fmt.Errorf("store source: %w", err) @@ -607,9 +609,12 @@ func (m *manager) CreateBuild(ctx context.Context, req CreateBuildRequest, sourc // detect mirrored images via checkImageExistsInRegistry dockerfileContent := req.Dockerfile if dockerfileContent == "" { + // Machine recipes are not Dockerfiles; never gunzip untrusted machine source here. tarballPath := m.paths.BuildSourceDir(id) + "/source.tar.gz" - if content, err := ExtractDockerfileFromTarball(tarballPath); err == nil { - dockerfileContent = content + if req.MachineBaseImage == "" { + if content, err := ExtractDockerfileFromTarball(tarballPath); err == nil { + dockerfileContent = content + } } } if dockerfileContent != "" { @@ -722,12 +727,26 @@ func (m *manager) BuilderHasBuilds(builderID string) bool { } // storeSource stores the source tarball for a build -func (m *manager) storeSource(ctx context.Context, buildID string, data []byte) (string, error) { +// storeSource records the SHA256 provenance for every build. Machine builds use +// private exclusive staging; Linux keeps its historical directory/file modes and +// plain write behavior. +func (m *manager) storeSource(ctx context.Context, buildID string, data []byte, private bool) (string, error) { sourceDir := m.paths.BuildSourceDir(buildID) - if err := os.MkdirAll(sourceDir, 0700); err != nil { + sourcePath := filepath.Join(sourceDir, "source.tar.gz") + if private { + if err := os.MkdirAll(sourceDir, 0700); err != nil { + return "", err + } + return stageBuildSource(ctx, bytes.NewReader(data), sourcePath, int64(len(data))) + } + if err := ensureDir(sourceDir); err != nil { + return "", err + } + if err := writeFile(sourcePath, data); err != nil { return "", err } - return stageBuildSource(ctx, bytes.NewReader(data), filepath.Join(sourceDir, "source.tar.gz"), int64(len(data))) + sum := sha256.Sum256(data) + return hex.EncodeToString(sum[:]), nil } // runBuild executes a build in a builder VM diff --git a/lib/instances/macos_build_export_darwin.go b/lib/instances/macos_build_export_darwin.go index ceb4addae..fd9839a91 100644 --- a/lib/instances/macos_build_export_darwin.go +++ b/lib/instances/macos_build_export_darwin.go @@ -14,18 +14,15 @@ import ( "github.com/kernel/hypeman/lib/images" ) -// StopAndExportMacOSInstance is an internal build capability, not an HTTP endpoint. -// The instance lock spans shutdown and both storage clones so Start cannot race export. -// Neither a pre-existing Stopped state nor forced hypervisor shutdown qualifies. -func (m *manager) StopAndExportMacOSInstance(ctx context.Context, id, destination string) (receipt MacOSBuildExportReceipt, err error) { +// StopMacOSBuildInstance performs only the strict graceful stop used by build +// export. It copies no storage and never falls back to a forced hypervisor stop. +// Success requires the shutdown acknowledgement and confirmed VMM exit. +func (m *manager) StopMacOSBuildInstance(ctx context.Context, id string) (receipt MacOSBuildExportReceipt, err error) { if err = ctx.Err(); err != nil { return } if _, bounded := ctx.Deadline(); !bounded { - return receipt, fmt.Errorf("machine export requires a deadline") - } - if !filepath.IsAbs(destination) { - return receipt, fmt.Errorf("export destination must be absolute") + return receipt, fmt.Errorf("machine stop requires a deadline") } lock := m.getInstanceLock(id) lock.Lock() @@ -35,20 +32,8 @@ func (m *manager) StopAndExportMacOSInstance(ctx context.Context, id, destinatio return receipt, err } if current.MacOS == nil || !current.GuestAgentEnabled() || (current.State != StateRunning && current.State != StateInitializing) { - return receipt, fmt.Errorf("export requires an active macOS guest with a system agent") - } - if rel, e := filepath.Rel(current.DataDir, destination); e != nil || filepath.IsLocal(rel) { - return receipt, fmt.Errorf("export must be outside instance storage") + return receipt, fmt.Errorf("strict stop requires an active macOS guest with a system agent") } - if err = os.Mkdir(destination, 0700); err != nil { - return - } - complete := false - defer func() { - if !complete { - _ = os.RemoveAll(destination) - } - }() if err = m.markRestartManualStopLocked(ctx, id); err != nil { return } @@ -58,13 +43,38 @@ func (m *manager) StopAndExportMacOSInstance(ctx context.Context, id, destinatio } receipt = MacOSBuildExportReceipt{Graceful: true, VMMExited: true} m.notifyLifecycleEvent(ctx, LifecycleEventStop, stopped) - if err = writeStoppedMacOSBuildBundle(ctx, stopped, m.paths.InstanceOverlay(id), destination); err != nil { - return receipt, err - } - complete = true return receipt, nil } +// ExportStoppedMacOSBuildInstance copies closed storage for an instance that is +// still Stopped when the copy starts. The instance lock excludes Start during the +// check and copy; a concurrently restarted instance fails export instead. +func (m *manager) ExportStoppedMacOSBuildInstance(ctx context.Context, id, destination string) error { + if err := ctx.Err(); err != nil { + return err + } + if _, bounded := ctx.Deadline(); !bounded { + return fmt.Errorf("machine export requires a deadline") + } + if !filepath.IsAbs(destination) { + return fmt.Errorf("export destination must be absolute") + } + lock := m.getInstanceLock(id) + lock.Lock() + defer lock.Unlock() + current, err := m.currentInstanceWithoutHydration(ctx, id) + if err != nil { + return err + } + if current.State != StateStopped || current.MacOS == nil { + return fmt.Errorf("export requires a stopped macOS instance") + } + if rel, e := filepath.Rel(current.DataDir, destination); e != nil || filepath.IsLocal(rel) { + return fmt.Errorf("export must be outside instance storage") + } + return writeStoppedMacOSBuildBundle(ctx, current, m.paths.InstanceOverlay(id), destination) +} + func writeStoppedMacOSBuildBundle(ctx context.Context, inst *Instance, disk, destination string) error { if inst.State != StateStopped || inst.MacOS == nil { return fmt.Errorf("machine must be stopped") diff --git a/lib/instances/macos_build_export_darwin_test.go b/lib/instances/macos_build_export_darwin_test.go index 19b69fc90..dbe3b073f 100644 --- a/lib/instances/macos_build_export_darwin_test.go +++ b/lib/instances/macos_build_export_darwin_test.go @@ -39,12 +39,14 @@ func TestMacOSBuildBundleRequiresClosedDistinctStorage(t *testing.T) { require.Error(t, writeStoppedMacOSBuildBundle(context.Background(), inst, disk, t.TempDir())) } -func TestMacOSBuildExportRequiresBoundedAbsoluteDestination(t *testing.T) { +func TestMacOSBuildStopAndExportRequireBoundedInputs(t *testing.T) { m := &manager{} - _, e := m.StopAndExportMacOSInstance(context.Background(), "unused", t.TempDir()) + _, e := m.StopMacOSBuildInstance(context.Background(), "unused") require.Error(t, e) ctx, cancel := context.WithCancel(context.Background()) cancel() - _, e = m.StopAndExportMacOSInstance(ctx, "unused", t.TempDir()) + _, e = m.StopMacOSBuildInstance(ctx, "unused") require.ErrorIs(t, e, context.Canceled) + require.Error(t, m.ExportStoppedMacOSBuildInstance(context.Background(), "unused", t.TempDir())) + require.Error(t, m.ExportStoppedMacOSBuildInstance(ctx, "unused", t.TempDir())) } From fa2e336d002f6cbd49406d511c33ed5d74967143 Mon Sep 17 00:00:00 2001 From: chris lee Date: Sun, 11 Oct 2026 00:42:11 -0400 Subject: [PATCH 7/7] Fix macOS build docs tokens and HTTP field comment --- docs/macos-builds.md | 36 ++++++++++++++++++------------------ lib/builds/types.go | 4 ++-- 2 files changed, 20 insertions(+), 20 deletions(-) diff --git a/docs/macos-builds.md b/docs/macos-builds.md index f9f17ea82..88fd12ed5 100644 --- a/docs/macos-builds.md +++ b/docs/macos-builds.md @@ -24,23 +24,23 @@ curl -H "Authorization: Bearer $TOKEN" \ -F "timeout_seconds=1200" "$API/builds" ``` -The installed base must already be Ready. A202 response queues an ordinary job; +The installed base must already be Ready. A 202 response queues an ordinary job; GET /builds/{id}, cancellation and status/provenance use the shared APIs. OpenAPI and generated server request types include machine_base_image; external language SDK releases are separate, not claimed here. Only source, machine_base_image, timeout_seconds, memory_mb, cpus and tags are accepted for machine HTTP builds. -Duplicates/unknown fields, malformed numbers and mutable bases return400. Metadata -is limited64KiB per field, source64MiB and the entire HTTP body65MiB before OpenAPI +Duplicates/unknown fields, malformed numbers and mutable bases return 400. Metadata +is limited to 64 KiB per field, source 64 MiB and the entire HTTP body 65 MiB before OpenAPI validation. These limits do not change Linux source parsing. ## Base and policy The ready installed base must be pinned by canonical SHA256, declare a provisioned -system agent on2222 and be `darwin/arm64`. Driver provisioning additionally checks +system agent on port 2222 and be `darwin/arm64`. Driver provisioning additionally checks actual guest UID0. The driver never installs an agent or falls back to host SSH/sudo. CPU/memory match the base exactly; zero request values inherit them. Build limits, exact MiB precision and bounded deadlines still apply. Timeout -normally defaults600s, capped24h; full machine-image compression/pull can need a +normally defaults to 600 s, capped at 24 hours; full machine-image compression/pull can need a longer explicitly selected deadline. Networking defaults isolated. Optional egress means unrestricted VZ NAT, not @@ -51,22 +51,22 @@ is preserved: concurrent same-identity guests remain forbidden. ## Experimental versioned provisioning recipe Source is the normal build-owned tar.gz, privately staged and SHA256 verified, -including recovery. Compressed machine input is capped64MiB; Linux staging gets -no new size cap. Host extraction into an exclusive0700 temporary directory limits -uncompressed data128MiB, entries1024 and individual regular files64MiB. Only regular +including recovery. Compressed machine input is capped at 64 MiB; Linux staging gets +no new size cap. Host extraction into an exclusive 0700 temporary directory limits +uncompressed data 128 MiB, entries 1024 and individual regular files 64 MiB. Only regular files/directories are accepted; duplicate names, links/devices and path escapes -fail before transfer. Files are0600 or owner-executable0700. Guest tar is not used. +fail before transfer. Files are 0600 or owner-executable 0700. Guest tar is not used. -The archive must contain `hypeman-macos-build.json`, limited64KiB: +The archive must contain `hypeman-macos-build.json`, limited to 64 KiB: ```json {"version":1,"steps":[{"command":["/bin/sh","build.sh"],"env":{"MODE":"release"}}]} ``` -At most32 steps execute as root in one exclusive `/var/root/hypeman-build-{id}` +At at most 32 steps execute as root in one exclusive `/var/root/hypeman-build-{id}` directory. Command executable paths must be absolute; arguments/environment are bounded and reject NUL. Commands run only in the guest, never a host shell. -Unknown JSON fields and extra JSON objects fail. Version1 remains experimental; +Unknown JSON fields and extra JSON objects fail. Version 1 remains experimental; public source/schema/toolchain contracts require review before merge. Sanitation removes and verifies absence of the build-owned guest source directory @@ -112,7 +112,7 @@ The publisher uses a server-configured registry and a token scoped only to trust; plaintext is permitted only on loopback. Redirects are never followed and environment HTTP proxies are disabled for these credential-bearing requests. -A private0600 gzip spool computes compressed digest/size and uncompressed DiffID +A private 0600 gzip spool computes compressed digest/size and uncompressed DiffID with bounded memory. Configuration and layer blobs stream through monolithic POST uploads, not go-containerregistry's large-buffer PATCH path. Each persisted blob is independently GET/hash/size verified. The `latest` manifest is committed @@ -132,18 +132,18 @@ not bootability evidence. Separate isolated Darwin native QA completed a real normal queued build from a manually provisioned, pinned installed base: root provisioning command, source sanitation, acknowledged no-force shutdown, owned VMM exit, matching export, -authenticated streamed28.85GB layer publication, image-readiness gate, fresh +authenticated streamed 28.85GB layer publication, image-readiness gate, fresh HTTP pull/cache destination and cold boot. Root exec found the build marker and -verified absence of the source workspace. The run took483.74s; test-process RSS -peaked36.88MiB, with313.17GiB minimum free disk. The initial attempt correctly +verified absence of the source workspace. The run took 483.74s; test-process RSS +peaked at 36.88MiB, with 313.17GiB minimum free disk. The initial attempt correctly refused publication on lost shutdown acknowledgement, leading to the guest reply scheduling fix. This is not HTTP build activation, automatic clean-base installation or repeat/recovery proof. Final cold-boot test cleanup used the ordinary short stop timeout and forced fallback; no export used that cleanup stop. A subsequent live test used the actual opt-in API server/provider wiring: -unauthenticated/invalid tokens returned401, a read-only token returned403, and -POST /builds returned202. GET reached Ready after the real provision/export/upload +unauthenticated/invalid tokens returned 401, a read-only token returned 403, and +POST /builds returned 202. GET reached Ready after the real provision/export/upload and readiness gate. POST /instances cold-booted the returned image_ref, and the normal authenticated exec WebSocket read the new root marker and verified source absence. No fresh cache claim for this second run (the preceding run separately diff --git a/lib/builds/types.go b/lib/builds/types.go index 7e80d9a45..931ddec6a 100644 --- a/lib/builds/types.go +++ b/lib/builds/types.go @@ -43,8 +43,8 @@ type CreateBuildRequest struct { // The Dockerfile specifies the runtime (e.g., FROM node:20-alpine). Dockerfile string `json:"dockerfile,omitempty"` - // MachineBaseImage selects the internal installed-machine backend. It is not - // exposed by the HTTP schema until recipes and concrete drivers are supported. + // MachineBaseImage selects the installed-machine backend. The HTTP multipart + // field machine_base_image maps here; it is enabled only with macos_builds. MachineBaseImage string `json:"machine_base_image,omitempty"` // BaseImageDigest optionally pins the base image by digest for reproducibility