image-mover

A Go-based platform for migrating container images from a self-hosted Distribution registry into multi-account Amazon ECR. Three products share one engine library:

• image-mover-operator — a Kubebuilder controller that reconciles MigrationRun CRs to drive long-running bulk migrations from inside the cluster (PRD: docs/01-prd-bulk-migration.md). Supports two execution modes: in-pod (default; runs the engine in-process) and scale-out (creates a batch/v1 Job per run with one worker pod per concurrency.workers, each on a distinct node via kubernetes.io/hostname anti-affinity). PRD: docs/04-prd-scale-out.md. The S6 milestone (chart default flip to scale-out) is gated on a 50,000-image staging validation; until then opt in with controller.executionMode=scaleout.
• image-mover-bulk — an operator CLI for plan/apply/status/report workflows against the operator (PRD: same doc).
• image-mover-dashboard — a read-only web UI for MigrationRun visibility: phase pills, per-account throughput, scale-out pre-flight, worker fleet, per-worker throughput, and a live job-state heatmap. (PRD: docs/06-prd-dashboard.md.)

Self-service is the fourth planned product (PRD: docs/02-prd-self-service.md) and is not yet implemented in this tree. The shared engine that all three consume is documented in docs/03-shared-engine.md.

Repo layout

api/v1alpha1/                  CRD types (MigrationRun)
cmd/
  main.go                      operator entrypoint
  bulk/                        bulk CLI
  worker/                      scale-out worker entrypoint (/worker in the image)
internal/
  controller/                  reconciler, planner, runner (in-pod + scaleout), summary upload
  bulkcli/                     plan/apply/status/report logic for the bulk binary
  jobsapi/                     read-only HTTP API for the dashboard's heatmap (port 8082)
pkg/mover/                     shared engine library
  source/ dest/ copy/ referrers/
  filter/ route/ plan/ run/ state/ summary/ metrics/
dashboard/
  backend/                     dashboard HTTP server + SSE hub
  frontend/                    Vite + React SPA
config/                        Kubebuilder-generated CRD manifests, RBAC, samples
deploy/
  helm/
    image-mover-operator/      operator chart (CRD + Deployment + RBAC + jobs-api Service)
    image-mover-dashboard/     dashboard chart
    distribution/              CNCF Distribution registry chart (S3-backed source)
  terraform/
    modules/operator/          operator-account DDB + S3 + IAM
    modules/destination/       per-destination-account ECR role
    modules/distribution-source/  source-side S3 + IAM for Distribution
    examples/single-account/   operator + destination smoke layout
    examples/distribution-source/  region-aware wrapper for distribution-source
docs/                          PRDs and runbooks

Quick starts

Run the controller against a local cluster

# Generate CRDs + run the unit/integration suite (envtest-driven).
make test

# Build the manager binary and run it against your current kubeconfig.
make run

Build the bulk CLI

make build-bulk
./bin/image-mover-bulk help

Subcommands: plan, apply, status, report. Each takes a migration.yaml config; see the bulk PRD §6.1 for the schema.

End-to-end smoke test

The repo doesn't ship an EKS cluster — bring your own. The cluster needs the EKS Pod Identity Agent add-on and node IAM permissions to pull from your ECR registry. The prerequisite tools are terraform, helm, kubectl, docker buildx, the AWS CLI, crane, and htpasswd (macOS preinstalled; Linux: apt-get install apache2-utils / yum install httpd-tools; or docker run --rm httpd:2.4 htpasswd -Bbn ...).

Run the steps below in this order — each one depends on outputs from the previous one.

1. Build and push the operator image

Preferred (fast) — prebuilt path. Cross-compiles the binaries on the host and COPYs them into a distroless image. Avoids the in-container go mod download, which stalls under GOPROXY=direct (this repo's required setting — the network blocks the public proxy), and avoids amd64 QEMU emulation. ~45s.

make docker-prebuilt-ecr ECR_TAG=v0.4.5
make dashboard-prebuilt-ecr DASHBOARD_ECR_TAG=v0.3.14

Fallback — in-container buildx build (slower; can stall on a cold module cache under GOPROXY=direct):

# Defaults: account 820537372947, region us-east-2, repo image-mover-operator,
# platform linux/amd64, GOPROXY=direct.
make docker-buildx-ecr

# Override any of ECR_ACCOUNT / ECR_REGION / ECR_REPO / ECR_TAG / ECR_PLATFORM /
# GOPROXY at the command line:
make docker-buildx-ecr ECR_ACCOUNT=111111111111 ECR_TAG=v0.3.5

The buildx target logs Docker into ECR, creates the repo if missing (idempotent), then runs docker buildx build --push for linux/amd64 only. The operator image bakes both binaries — /manager (default ENTRYPOINT) and /worker (invoked by the K8s Job pod template the reconciler creates in scale-out mode). One image, one tag, both lifecycles. On success the target prints the --set image.repository=... line you'll feed to Helm later.

make docker-buildx-ecr and make dashboard-docker-buildx-ecr are safe to run in parallel: each target uses a per-target buildx builder name (image-mover-operator-buildx vs image-mover-dashboard-buildx) and the shared ecr-login step serializes via a /tmp lockdir to avoid the macOS keychain -25299: item already exists race.

2. Provision the operator-side AWS resources

DynamoDB state table, S3 summary bucket, operator IAM role, destination ECR role.

cd deploy/terraform/examples/single-account
cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars (minimum: summary_bucket_name; uncomment the
# pod_identity block if you want Terraform to wire the EKS association).
terraform init
terraform apply
terraform output -raw helm_values > /tmp/operator-values.yaml

See deploy/terraform/README.md for the multi-account layout.

3. Provision the source registry's AWS resources

S3 bucket and IAM role for the Distribution registry's blob storage. This example wraps modules/distribution-source with a provider "aws" block so the region is set in terraform.tfvars rather than via AWS_REGION.

cd ../distribution-source
cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars (minimum: bucket_name; region defaults to us-east-2).
terraform init
terraform apply
terraform output -raw helm_values > /tmp/distribution-values.yaml

4. Install the Distribution registry

The chart's htpasswd auth requires TLS, so configure ingress with a TLS secret your cluster trusts (cert-manager, AWS Load Balancer Controller + ACM, or a manually-created Secret).

# Generate a bcrypt-hashed entry. Save the password — you'll need it again
# in step 6.
HTPASSWD_LINE=$(htpasswd -Bbn alice s3cret)

helm install distribution deploy/helm/distribution \
  -n registry --create-namespace \
  -f /tmp/distribution-values.yaml \
  --set auth.htpasswd.createSecret=true \
  --set "auth.htpasswd.entries=$HTPASSWD_LINE" \
  --set ingress.enabled=true \
  --set 'ingress.tls[0].secretName=registry-tls' \
  --set 'ingress.tls[0].hosts[0]=registry.example.com' \
  --set 'ingress.hosts[0].host=registry.example.com' \
  --set 'ingress.hosts[0].paths[0].path=/' \
  --set 'ingress.hosts[0].paths[0].pathType=Prefix'

5. Seed an image into Distribution

Distribution comes up empty. Push at least one image so the operator has something to migrate:

crane copy gcr.io/distroless/static:nonroot \
  registry.example.com/infra/static:v1
# Or use kubectl port-forward + a localhost crane copy if you don't have an
# externally-resolvable Ingress yet.

6. Install the operator

helm install image-mover-operator deploy/helm/image-mover-operator \
  -n image-mover --create-namespace \
  -f /tmp/operator-values.yaml \
  --set image.repository=820537372947.dkr.ecr.us-east-2.amazonaws.com/image-mover-operator \
  --set image.tag=v0.4.0 \
  --set controller.scaleout.pushgatewayURL=http://prometheus-pushgateway.prometheus.svc:9091

controller.executionMode defaults to in-pod in the chart; the example above flips it to scaleout, which creates one batch/v1 Job per MigrationRun and one worker pod per spec.concurrency.workers, each on a distinct node (hard pod anti-affinity on kubernetes.io/hostname — see PRD §6.10). Worker metrics are pushed to controller.scaleout.pushgatewayURL; leave it empty if you don't run a Pushgateway and the dashboard's per-worker throughput chart will render an empty-state note.

The Terraform helm_install_hint output prints a pre-filled helm install command if you'd rather copy-paste. See deploy/helm/image-mover-operator/README.md for the full values surface.

7. Submit the sample MigrationRun

The repo ships a working sample under config/samples/ — a MigrationRun plus the source-creds Secret and image-mover-routes ConfigMap it references. Defaults route everything to AWS account 820537372947 in us-east-2:

# Edit before applying:
#   - source-creds-secret.yaml: set username/password to match the htpasswd
#     line you generated in step 4 (alice / s3cret if you used the example).
#   - imagemover_v1alpha1_migrationrun.yaml: adjust source.url if your
#     distribution release is in a different namespace or release name.
#   - routes-configmap.yaml: change account/region if you're not targeting
#     820537372947/us-east-2.
kubectl apply -k config/samples/
kubectl -n image-mover get migrationrun -w

Kustomize applies the Secret + ConfigMap before the MigrationRun so the operator's planner finds them on its first reconcile of the CR.

8. (Optional) Install the dashboard

make dashboard-docker-buildx-ecr
helm install image-mover-dashboard deploy/helm/image-mover-dashboard \
  -n image-mover-dashboard --create-namespace \
  --set image.tag=v0.3.3 \
  --set prometheus.url=http://prometheus-server.prometheus.svc:80 \
  --set summary.region=us-east-2

jobsAPI.url defaults to the in-cluster operator Service URL (http://image-mover-operator.image-mover.svc.cluster.local:8082), which the operator chart now creates by default. The dashboard's RunDetail page renders a worker fleet panel, per-worker throughput chart, and a live job-state heatmap; the runs page renders a scale-out pre-flight tile. See deploy/helm/image-mover-dashboard/README.md.

Make targets

make help lists everything. The most common ones:

Target	What it does
make test	Go tests (envtest-driven for the controller).
make build	Build the operator manager binary.
make build-bulk	Build the image-mover-bulk CLI.
make run	Run the operator from your host against ~/.kube/config.
make manifests / make generate	Regenerate CRDs + DeepCopy.
make helm-lint	Lint every Helm chart (operator + distribution).
make helm-template	Render every chart with default values.
make tf-fmt / make tf-fmt-check	Format Terraform / verify formatting.
make tf-validate	Validate every Terraform module + the example.
make docker-build IMG=...	Build the operator container image (single-arch, current host). Bakes /manager and /worker into the same image.
make docker-prebuilt-ecr	Preferred. Host-compile the operator binaries, COPY into distroless, push to ECR (~45s; no in-container go mod download, no QEMU).
make docker-buildx-ecr	Fallback: in-container buildx build & push of the operator image to ECR (linux/amd64). Slower; can stall on a cold cache under GOPROXY=direct.
make build-dashboard	Build the dashboard backend binary (version stamped from chart appVersion).
make build-dashboard-frontend	Build the dashboard SPA bundle (Vite).
make dashboard-frontend-dev	Run the Vite dev server (proxies /api to localhost:8080).
make dashboard-docker-build DASHBOARD_IMG=...	Build the dashboard image (multi-stage: frontend + backend).
make dashboard-prebuilt-ecr	Preferred. Host-build the dashboard frontend + backend, COPY into distroless, push to ECR (~30s).
make dashboard-docker-buildx-ecr	Fallback: in-container multi-stage build & push of the dashboard image to ECR.
make smoke / make smoke-keep	Run the end-to-end smoke scenarios against the deployed operator (cleanup → seed → bulk → referrers → resumability). --keep skips the trailing cleanup.
make smoke-comprehensive	Cross-sample run: multi-arch + cosign signatures + oras SBOMs + multi-tag/backdated repo, exercised through tag/age/repo/min-pull filters. Needs cosign + oras.
make smoke-longrun	One long, paced migration for watching live dashboard metrics fill in.
make smoke-cleanup	Tear down smoke results (ECR/DDB/S3/CRs). Leaves source images unless WIPE_SOURCE=1.

Documentation

Doc	When to read
This README	First contact: install, build, smoke.
USER_GUIDE.md	Configuring + submitting + monitoring MigrationRun CRs. Multi-account routing, filters, referrers, troubleshooting, scale-out tuning.
ARCHITECTURE.md	How the operator is built — phase machine, engine internals, auth, observability, scale-out runner + worker binary, jobs API.
docs/01-prd-bulk-migration.md	Product requirements + milestone breakdown.
docs/03-shared-engine.md	Shared-engine design: filters, routing, referrers, state.
docs/04-prd-scale-out.md	Scale-out execution PRD: K8s Job + worker pods, work-stealing protocol, dashboard impact. Implementation S1–S4 ships in v0.3.0.
docs/05-m5-smoke-test.md	Manual smoke-test runbook for staging.
docs/06-prd-dashboard.md	Dashboard (web UI) PRD: pages, data model, milestones, v2 roadmap.
hack/smoke/README.md	Automated smoke scenarios + how to run them.
deploy/helm/image-mover-operator/README.md	Operator chart values reference.
deploy/helm/image-mover-dashboard/README.md	Dashboard chart values reference.
deploy/helm/distribution/README.md	Distribution chart values reference.
deploy/terraform/README.md	AWS resource Terraform layout.
dashboard/AIRGAP_SMOKE.md	Air-gap regression check for the dashboard build.

Project status

• Bulk migration (PRD 01): M0–M5 implemented; M6 (production runbook
  • ECR-quota partitioning + IAM security review) is pending. See docs/01-prd-bulk-migration.md §10.
• Scale-out execution (PRD 04): S1–S4 implemented and validated against image-moover-dev smoke scenarios (bulk + referrers + resumability all pass). Operator chart 0.4.0 includes tunable worker pod resources (controller.scaleout.workerResources), resumability short-circuit (no Job created when every planned job is already terminal), migration_id Pushgateway label, per-run accounting (pre-run-Succeeded rows render as Skipped for the new run's totals + heatmap), and multi-arch image preservation (manifest lists / OCI image indexes copy with all child manifests intact instead of flattening to a single platform — previously a silent correctness loss). S5 (50,000-image staging migration) and S6 (default-mode flip + fault-injection drill) are field-test milestones. See docs/04-prd-scale-out.md §10.
• Dashboard (PRD 06): D1–D8 (v1) shipped; v2 panels for scale-out (worker fleet, per-worker throughput, pre-flight, live job-state heatmap) shipped in chart 0.3.0. Dashboard chart 0.3.3 adds scaleout-skipped execution mode + per-run migration_id filtering on RunDetail charts.
• Self-service (PRD 02): not yet started.

Contributing

AGENTS.md documents the conventions enforced in code review (testify-only testing, controller phase machine, no Ginkgo, etc.). Run make test and make helm-lint && make tf-validate before sending a PR.

License

Copyright 2026.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy at http://www.apache.org/licenses/LICENSE-2.0.