diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 52e55a804..ad40ffb58 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -61,11 +61,9 @@ jobs: - name: Test the translation integrity check run: yarn test:translation - # The unit tests above prove the check works; they do not look at a single - # document. On main that job belongs to "Translate and verify", but that - # step only runs on a push -- so until now a pull request could add or - # edit a page and nothing compared it against its English source. The - # branch scope does exactly that, for the pairs this branch touches. + # Check translations changed by this PR and manually translated pages. + # Unchanged automatic translations of English-only edits are regenerated + # and checked by "Translate and verify" after the merge to main. - name: Check the translations this branch changes if: github.event_name == 'pull_request' run: node scripts/check-translation-integrity.mjs --since origin/${{ github.base_ref }} diff --git a/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx b/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx index af843fb0a..11de7fed4 100644 --- a/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx +++ b/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx @@ -9,886 +9,878 @@ ProductsVersion: # ACP Registry v2 Migration Runbook -Migration from the legacy ACP Registry is required when an existing ACP environment moves image workloads from the legacy Registry to **Registry v2** in `image-registry-system`. Registry v2 means the Registry deployed and managed by `cluster-image-registry-operator`. +This runbook migrates one ACP business cluster from the legacy chart Registry to Registry v2, managed by `cluster-image-registry-operator`. The standalone `registry-migrate` tool imports metadata while Registry v2 reuses the same S3 bucket, endpoint, and prefix. -The migration workflow is based on the validated ACP `ac` runbook. No dedicated Registry migration subcommand is required. Use the following `ac` commands: +The commands do not directly delete images or run GC. P5 removes the legacy chart resources and its leftover Ingress; the `ac delete images` calls in P8 are dry runs. Never add `--confirm`. P8 also restores the recorded Pruner suspension state: if that state is `false`, scheduled image cleanup can resume. Confirm that behavior before approving the cutover. -- `ac get images` to export the legacy image list. -- `ac registry login` to write OCI registry credentials. -- `ac image mirror` to copy image blobs when the source and target registries do not share storage. -- `ac image info` to read image digests and media types from the registry selected for metadata inspection. -- `ac create -f` to create `ImageStreamMapping` resources and backfill Registry v2 Image and ImageStream metadata. +## Stage overview -During migration, use a maintenance window when possible. At minimum, pause writes to the legacy Registry. Reads can remain available. If new images are pushed during migration, repeat image copy and metadata backfill before the final cutover. +| Stage | Purpose | Service interruption | Retry after failure | +| --- | --- | --- | --- | +| 0 | Validate tools, credentials, and the business cluster context | No | Yes | +| P0 plan | Generate and review the read-only plan | No | Yes | +| P1-backup | Save resource identities and prepare recovery material | No | Yes | +| P1-credentials | Hand off pull Secrets in place | No | Yes | +| P2-stop-writers | Stop legacy writers (**maintenance window starts**) | **Yes** | Yes | +| P3-final-inventory | Freeze the image inventory and S3 object listing | **Yes** | Yes; a recorded protection choice is immutable | +| P4-prepare-operator | Hold the Operator at zero replicas and prepare read-only Config | **Yes** | Yes | +| P5-remove-legacy-chart | Uninstall the legacy plugin, then remove its leftover Ingress | **Yes** | Retry checks; do not repeat a completed uninstall | +| P6-cold-import | Import metadata and namespace limits | **Yes** | Yes | +| P7-verify | Resume the Operator and verify read-only service | Read access restored | Yes | +| P8-commit | Verify business access and open writes (**maintenance window ends**) | Restored | Inspect the recorded and live state first | -Metadata backfill writes Registry v2 Image API objects only. It does not copy or repair registry storage data such as manifests, config blobs, layers, repository revision links, or tag links. Do not create `ImageStreamMapping` objects until Registry v2 can access the corresponding image data through storage reuse or a completed image copy. If legacy image data is intentionally not preserved, do not run metadata backfill for those images; push the images again to Registry v2 instead. +Use the same run directory for retries. Start at section 0. -## Migration Scenarios +## Prerequisites -| Scenario | Use when | Blob migration | Metadata migration | -| --- | --- | --- | --- | -| Storage reuse | The legacy Registry and Registry v2 run in the same ACP environment and business cluster, and Registry v2 reuses the legacy Registry storage. | Not required. Registry v2 is attached to the existing blob storage. | Required. Read digest and media type from the legacy Registry, then backfill `ImageStream` and `ImageStreamMapping` objects. | -| Cross-registry sync | The legacy Registry and Registry v2 use different storage, different clusters, or different ACP environments. | Required. Use `ac image mirror` to copy images from the legacy Registry to the target Registry. | Required after image copy. Read digest and media type from Registry v2, then backfill `ImageStream` and `ImageStreamMapping` objects. | +- **Tools and recovery procedures**: contact your ACP service provider for approved `registry-migrate` and compatible `ac` binaries, the Registry v2 bundle, and the platform pause and S3 recovery procedures. +- **Namespaces**: the default Operator installation namespace is `image-registry-system`; the default Registry operand namespace is `cpaas-system`. The operand namespace is fixed at installation. Confirm the actual values and use `scope.operandNamespace` in `plan.json` throughout the migration. +- **Storage scope**: stop if the legacy Registry's image-data backend uses PVCs or HostPath, or the migration contains non-image OCI artifacts or a non-default S3 root/prefix, unless the provider has approved another procedure. HostPath mounts used only by node trust or resolver components are not image-data storage. +- **Operator pause**: the Operator must remain at zero replicas during P4–P6. Manual InstallPlan approval does not pause an already running Operator. Establish the provider-approved pause before P4 and before removing the legacy chart. +- **CLI compatibility**: `AC_IMAGE_REGISTRY_MODE=legacy/modern` selects the mode, but an explicit `--registry-url` can keep requests on the Registry HTTP compatibility path. `AC_REGISTRY_COMPAT_OUTPUT` selects the output contract; leave it unset or set it to `4.3` for these checks. -## Required Inputs +## 0. Prepare the terminal and tools -Prepare the following inputs before running the migration: +Use one dedicated **Bash** terminal for all migration steps; do not enable `set -u` in an interactive zsh session. Login-token lifetime depends on the platform identity provider. If requests return `Unauthorized`, refresh the login associated with the kubeconfig used for migration. List the available sessions first: -| Variable | Description | Example | -| --- | --- | --- | -| `MIGRATION_KUBECONFIG` | Optional kubeconfig used only for migration. | `/tmp/registry-migration/kubeconfig` | -| `ACP_PLATFORM_URL` | ACP platform login URL. | `https://acp.example.com` | -| `ACP_SESSION_NAME` | ACP login session name for migration. | `registry-migration` | -| `ACP_USERNAME` | ACP account used for migration. | `admin` | -| `ACP_PASSWORD` | ACP password. Store it only in a private environment file. | `******` | -| `ACP_CLUSTER` | Business cluster selected after login. | `business-1` | -| `ACP_IDP` | Optional identity provider name required by the login entry. | `ldap-test` | -| `ACP_AUTH_TYPE` | Optional authentication type required by the login entry. | `ldap` | -| `LEGACY_REGISTRY_URL` | Legacy Registry URL used by `ac get images --registry-url`. Include `http://` or `https://`. | `http://old-registry.example.com:5000` | -| `LEGACY_REGISTRY_HOST` | Legacy Registry external host used by `ac image mirror`. Do not include a scheme. | `old-registry.example.com:5000` | -| `TARGET_REGISTRY_HOST` | Registry v2 external host used by mirror, image inspect, metadata backfill, and verification. Do not include a scheme. | `image-registry.example.com` | -| `IMAGE_LIST_FILE` | Image list to migrate. Each line is `namespace/name:tag`. | `/tmp/registry-migration/image-list.txt` | -| `IMAGE_DIGEST_FILE` | Optional image and digest list. Each line is `namespace/name:tag sha256:...`. | `/tmp/registry-migration/image-digests.txt` | -| `MAPPING_FILE` | Mirror mapping file for cross-registry sync. Each line is `source=target`. | `/tmp/registry-migration/mirror-map.txt` | -| `METADATA_DIR` | Output directory for `ac image info`, `ImageStreamMapping`, and apply results. | `/tmp/registry-migration/metadata` | -| `METADATA_INSPECT_REGISTRY_HOST` | Registry host used by metadata backfill when inspecting digest and media type. Use the legacy Registry for storage reuse and Registry v2 for cross-registry sync. | `old-registry.example.com:5000` | -| `REGISTRY_INSECURE_FLAG` | Set to `--insecure` when the registry uses HTTP, a self-signed certificate, or a test certificate. Set to an empty string for trusted HTTPS. | `--insecure` | -| `REGISTRY_AUTH_DIR` | OCI registry auth directory. | `/tmp/registry-migration/registry-auth` | -| `REGISTRY_AUTH_FILE` | OCI registry auth file written by `ac registry login --to`. | `/tmp/registry-migration/registry-auth/config.json` | +```bash +ac config get-sessions +``` -Create a migration workspace: +The row marked `*` in `CURRENT` is the active session. Use its `PLATFORM` and `SESSION` values to log in again: ```bash -export MIGRATION_DIR=/tmp/registry-migration -mkdir -p "$MIGRATION_DIR" +printf 'PLATFORM: '; IFS= read -r PLATFORM +printf 'SESSION: '; IFS= read -r SESSION +ac login "$PLATFORM" --name "$SESSION" +``` -export MIGRATION_KUBECONFIG="$MIGRATION_DIR/kubeconfig" -export IMAGE_LIST_FILE="$MIGRATION_DIR/image-list.txt" -export IMAGE_DIGEST_FILE="$MIGRATION_DIR/image-digests.txt" -export MAPPING_FILE="$MIGRATION_DIR/mirror-map.txt" -export METADATA_DIR="$MIGRATION_DIR/metadata" -export REGISTRY_AUTH_DIR="$MIGRATION_DIR/registry-auth" -export REGISTRY_AUTH_FILE="$REGISTRY_AUTH_DIR/config.json" -export REGISTRY_INSECURE_FLAG="${REGISTRY_INSECURE_FLAG:---insecure}" +The block below validates tools and credentials and selects a context. For a separate business kubeconfig, add `--kubeconfig /absolute/path/to/business.kubeconfig` to the login command above and use the same path in the block's `export KCFG=` line. Setup clears earlier `KCFG` values; login to the default file does not refresh a separate copy. -mkdir -p "$REGISTRY_AUTH_DIR" "$METADATA_DIR" +```bash +setup_run() { +umask 077 +unset KCFG # clear a stale path left over from an earlier terminal +export KCFG="$HOME/.kube/config" +test -f "$KCFG" || { echo "ERROR: kubeconfig not found: $KCFG" >&2; echo "Edit the KCFG line above if the business kubeconfig is elsewhere" >&2; return 1; } +for b in registry-migrate ac jq curl; do + command -v "$b" >/dev/null || { echo "ERROR: $b not found in PATH" >&2; return 1; } +done +printf 'Enter the real platform administrator account: ' +IFS= read -r ACTOR +test -n "$ACTOR" || { echo "ERROR: administrator account is required" >&2; return 1; } +export REGISTRY_NS="cpaas-system" # used to create the plan on a first run; overridden by plan.json on re-runs +export OPERATOR_NS="image-registry-system" +CTX_LIST="$(ac --kubeconfig "$KCFG" config get-contexts -o name || true)" +test -n "$CTX_LIST" || { echo "ERROR: no context in $KCFG" >&2; echo "Login first: ac login --platform --name " >&2; return 1; } +CTX="$(ac --kubeconfig "$KCFG" config current-context 2>/dev/null || true)" +test -n "$CTX" || { test "$(echo "$CTX_LIST" | grep -c .)" -eq 1 && CTX="$CTX_LIST" || { + echo "Available contexts:" >&2; echo "$CTX_LIST" >&2; printf 'Enter the context to use: ' >&2; IFS= read -r CTX; }; } +echo "$CTX_LIST" | grep -Fxq "$CTX" || { echo "ERROR: context not found: $CTX" >&2; return 1; } +ac --kubeconfig "$KCFG" --context "$CTX" whoami >/dev/null 2>&1 || { + echo "ERROR: not authenticated for context $CTX (token expired?)" >&2 + echo "Existing sessions:" >&2 + ac --kubeconfig "$KCFG" config get-sessions >&2 || true + echo "Re-login: ac login --platform --name " >&2 + return 1; } +OUT="$PWD/registry-migrate-runs" +if test -n "${RUN_DIR:-}" && ! test -f "$RUN_DIR/plan.json"; then + echo "WARN: ignoring stale RUN_DIR (no plan.json): $RUN_DIR" >&2 + unset RUN_DIR +fi +if test -n "${RUN_DIR:-}"; then + test -f "$RUN_DIR/plan.json" || { echo "ERROR: RUN_DIR is set but plan.json is missing: $RUN_DIR" >&2; echo "Fix: unset RUN_DIR to start a new migration" >&2; return 1; } + RUN_ID="$(jq -er '.runID' "$RUN_DIR/plan.json")" + REGISTRY_NS="$(jq -er '.scope.operandNamespace' "$RUN_DIR/plan.json")" + OPERATOR_NS="$(jq -er '.scope.operatorNamespace' "$RUN_DIR/plan.json")" + REGISTRY_SERVICE="$(jq -er '.scope.serviceName' "$RUN_DIR/plan.json")" + REGISTRY_DEPLOY="$(jq -r '.discovery.deployment // empty' "$RUN_DIR/plan.json")" + GATEWAY_DEPLOY="$(jq -r '.discovery.gatewayDeployment // empty' "$RUN_DIR/plan.json")" + OUT="$(dirname "$RUN_DIR")" +else + RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)-business-$RANDOM" + RUN_DIR="$OUT/$RUN_ID" +fi +mkdir -p "$OUT" +S3_ARGS=() +FORCE_ARGS=() +registry-migrate version +ac version --client +printf 'context=%s\nrun-dir=%s\n' "$CTX" "$RUN_DIR" +} +setup_run || echo "STOP: section 0 aborted; fix the error above and re-run this block. The shell stays open." >&2 ``` -If the legacy Registry uses HTTP, derive `LEGACY_REGISTRY_URL` from `LEGACY_REGISTRY_HOST`. If it uses HTTPS, set the full URL explicitly: +Confirm that `context=` identifies the intended business cluster. Before P0, select it with `ac --kubeconfig "$KCFG" config use-context "$CTX"`: the `ac` helpers use `CTX`, but `registry-migrate` uses the file's `current-context`. + +Setup creates `OUT`; successful P0 creates `$RUN_DIR`. For retries, restore the original absolute `RUN_DIR` (appendix B), then reselect the required S3, force, nested-repository and P3 protection options. Do not discard a valid plan to bypass a failure. + +The next block defines mandatory helpers. An error, timeout, or `STOP` message means the step needs attention even if the shell remains open or a later command prints output. Confirm the stage's success check before entering the next section. ```bash -export LEGACY_REGISTRY_URL="${LEGACY_REGISTRY_URL:-http://$LEGACY_REGISTRY_HOST}" +acn() { + test -n "${CTX:-}" || { echo "ERROR: CTX is empty; run section 0 first" >&2; return 1; } + ac --kubeconfig "${KCFG:-$HOME/.kube/config}" --context "$CTX" "$@" +} +acns() { acn -n "$REGISTRY_NS" "$@"; } +evidence() { + jq --arg actor "$ACTOR" --arg at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --arg field "$2" \ + '{version:"v1",runID:.runID,planDigest:.digest,observedAt:$at,actor:$actor,($field):true}' \ + "$RUN_DIR/plan.json" > "$RUN_DIR/$1" +} +stop_deployment() { + local namespace="$1" name="$2" selector pods + acn -n "$namespace" get deployment "$name" -o wide || return 1 + selector="$(acn -n "$namespace" get deployment "$name" -o json | + jq -er '.spec.selector.matchLabels | to_entries | map("\(.key)=\(.value)") | join(",") | select(length > 0)')" || return 1 + acn -n "$namespace" scale deployment "$name" --replicas=0 || return 1 + pods="$(acn -n "$namespace" get pods -l "$selector" -o name)" || return 1 + if test -n "$pods"; then + acn -n "$namespace" wait --for=delete pod -l "$selector" --timeout=180s || true + fi +} ``` -`REGISTRY_AUTH_DIR` stores an OCI registry auth file. Setting this variable does not start a local container runtime. -If both source and target registries use trusted HTTPS certificates, disable the insecure flag: +## 1. P0: create and review the plan ```bash -export REGISTRY_INSECURE_FLAG="" +registry-migrate plan --output-dir "$OUT" --run-id "$RUN_ID" \ + --operand-namespace "$REGISTRY_NS" --operator-namespace "$OPERATOR_NS" \ + --kubeconfig "$KCFG" && +jq -e '(.discovery.blockers // []) | length == 0' "$RUN_DIR/plan.json" && +REGISTRY_DEPLOY="$(jq -er '.discovery.deployment | select(length > 0)' "$RUN_DIR/plan.json")" && +GATEWAY_DEPLOY="$(jq -er '.discovery.gatewayDeployment | select(length > 0)' "$RUN_DIR/plan.json")" && +REGISTRY_SERVICE="$(jq -er '.scope.serviceName' "$RUN_DIR/plan.json")" && +jq '{scope,storage:.discovery.storage,routes:.discovery.routes,limits:.namespaceLimits,warnings:.discovery.warnings}' "$RUN_DIR/plan.json" && +echo "P0 OK: $RUN_DIR/plan.json" ``` -## Log In and Check Registry Mode - -Log in to ACP and create a migration-only kubeconfig: +`P0 OK` means the plan was generated and the blocker check passed. If it is absent, stay at P0 and inspect the error or blockers: ```bash -LOGIN_ARGS=( - ac login "$ACP_PLATFORM_URL" - --name "$ACP_SESSION_NAME" - --username "$ACP_USERNAME" - --password "$ACP_PASSWORD" - --kubeconfig "$MIGRATION_KUBECONFIG" -) +jq '.discovery.blockers' "$RUN_DIR/plan.json" +``` -[ -z "${ACP_IDP:-}" ] || LOGIN_ARGS+=(--idp "$ACP_IDP") -[ -z "${ACP_AUTH_TYPE:-}" ] || LOGIN_ARGS+=(--auth-type "$ACP_AUTH_TYPE") +Planning is read-only. Compare the scope, S3 bucket/endpoint/prefix, original host/TLS/IngressClass, replica counts, scheduling, and timeouts with the change record and live cluster. Stop on any mismatch. If `scope.operandNamespace` is wrong, correct `REGISTRY_NS` in section 0 before planning again. -"${LOGIN_ARGS[@]}" +Check for pre-existing Registry v2 metadata before stopping writers. Legacy chart test images should be pushed to the legacy Registry; do not manually create backing ImageStream objects as migration test data. Existing metadata is a review condition, not a reason to delete it. Save the following list and inspect any objects in repositories covered by this migration: -export KUBECONFIG="$MIGRATION_KUBECONFIG" -[ -z "${ACP_CLUSTER:-}" ] || ac config use-cluster "$ACP_CLUSTER" +```bash +check_existing_streams() { + local crd + crd="$(acn get crd imagestreams.storage.image.alauda.io --ignore-not-found -o name)" || return 1 + if test -z "$crd"; then + echo "No Registry v2 backing ImageStream CRD exists yet." + return 0 + fi + acn get imagestreams.storage.image.alauda.io -A -o json \ + > "$RUN_DIR/existing-image-streams.json" || return 1 + jq -r '.items[] | [.metadata.namespace, .metadata.name, .metadata.uid, + (.metadata.annotations["registry-migrate.run-id"] // "unowned"), + (.metadata.annotations["registry-migrate.plan-digest"] // "")] | @tsv' \ + "$RUN_DIR/existing-image-streams.json" +} +check_existing_streams_or_stop() { + check_existing_streams || { + echo "STOP: existing ImageStream inventory failed; fix the error before stopping writers or continuing." >&2 + echo "Do not continue to P1 until the inventory file is complete." >&2 + return 1 + } +} +check_existing_streams_or_stop ``` -If the cluster has both legacy and Registry v2 backends, `auto` mode can require an explicit selection. Start migration in legacy mode and confirm that the legacy Registry can be discovered: +## 2. P1-backup: save recovery material + +Complete this stage **before P1-credentials**, which rewrites pull Secret contents in place. The tool creates a redacted Registry-scoped snapshot of resource identities, owners, ServiceAccounts, references, and data digests. It contains **neither Secret data nor S3 image data**. + +Separately preserve chart versions/values, platform objects and protected Secret data through the approved recovery procedure. Keep them private and outside the redacted backup. Recreating a Secret or ServiceAccount cannot restore its UID. Confirm recovery and retention arrangements before chart removal. ```bash -ac config set-registry-mode legacy -ac config get-registry-mode -ac registry info -ac registry info --internal +registry-migrate backup --run-dir "$RUN_DIR" --kubeconfig "$KCFG" --approve ``` -`ac registry info` shows the Registry discovered for the current mode. It might return an in-cluster Service such as `image-registry.cpaas-system`, which is not directly reachable from an administrator workstation. Use `LEGACY_REGISTRY_URL` for image discovery and use `LEGACY_REGISTRY_HOST` and `TARGET_REGISTRY_HOST` for image copy. +Success check: the command exits 0 and writes the resource and Secret summaries under `$RUN_DIR`. On failure, resolve the reported error and retry P1-backup before continuing. -## Prepare the Image List +## 3. P1-credentials: hand off pull Secrets -Export the legacy image list visible to the current ACP user: +Keep the legacy Registry readable. First pause the platform source managing `sa-imagepullsecret-controller` so it cannot restart the controller. If discovery reports multiple candidates, use `acn get deployments -A -o wide | grep -i imagepull` to identify the intended controller before proceeding. ```bash -ac config set-registry-mode legacy -ac get images \ - --registry-url "$LEGACY_REGISTRY_URL" \ - > "$MIGRATION_DIR/legacy-images.txt" +PULL_SECRET_REF="$(acn get deployments -A -o json | jq -er ' + [.items[] | select((.metadata.name | test("sa-imagepullsecret-controller";"i")) or + ((.metadata.labels // {}) | tojson | test("sa-imagepullsecret-controller";"i"))) | + [.metadata.namespace,.metadata.name]] | + if length == 1 then .[0] | @tsv else error("review legacy pull-secret controller candidates") end')" +PULL_SECRET_NS=""; PULL_SECRET_DEPLOY="" +IFS=$'\t' read -r PULL_SECRET_NS PULL_SECRET_DEPLOY <<< "$PULL_SECRET_REF" +if test -z "$PULL_SECRET_DEPLOY"; then + echo "STOP: cannot resolve the legacy pull-secret controller; fix the jq error above and re-run. Stage not recorded." >&2 +else + if test "${#FORCE_ARGS[@]}" -eq 0; then + stop_deployment "$PULL_SECRET_NS" "$PULL_SECRET_DEPLOY" + else + acn -n "$PULL_SECRET_NS" get deployment "$PULL_SECRET_DEPLOY" -o wide + echo "force mode: leaving the legacy pull-secret controller state unchanged" + fi + evidence controller-stopped.json legacyControllerStopped + registry-migrate run --run-dir "$RUN_DIR" --stage P1-credentials \ + --evidence-file "$RUN_DIR/controller-stopped.json" \ + "${FORCE_ARGS[@]}" --kubeconfig "$KCFG" --approve +fi ``` -Generate the image list and the optional digest list from the `ac get images` table output: +Success check: the tool reports the Secret count and `updated/skipped` counters and records the stage in the journal. Existing Secret names/UIDs and their ServiceAccount owner UIDs remain unchanged. In the normal path, verify the old controller is stopped before treating its stop evidence as valid. Do not uninstall the legacy chart yet. -```bash -awk 'NR > 1 && NF >= 2 { print $1 ":" $2 }' \ - "$MIGRATION_DIR/legacy-images.txt" \ - | sort -u > "$IMAGE_LIST_FILE" +## 4. P2-stop-writers: stop the legacy writers (maintenance window starts) -awk 'NR > 1 && NF >= 4 && $4 != "unknown" { print $1 ":" $2 " " $4 }' \ - "$MIGRATION_DIR/legacy-images.txt" \ - | sort -u > "$IMAGE_DIGEST_FILE" -``` +Pause the platform source managing the legacy chart. Stop the Registry and Gateway Deployments, suspend the legacy Pruner CronJob if present, and stop external jobs or metadata writers using the same S3 bucket/prefix. Suspending a CronJob does not stop an already active Job. -You can also write the list manually when only selected images should be migrated: +The block prompts for missing writer-record fields and creates `$RUN_DIR/writer-record.txt`; an existing non-empty `WRITER_RECORD` is reused. ```bash -cat >> "$IMAGE_LIST_FILE" <> "$IMAGE_DIGEST_FILE" < +stop_writers() { +export WRITER_RECORD="${WRITER_RECORD:-$RUN_DIR/writer-record.txt}" +if ! test -s "$WRITER_RECORD"; then + printf 'Maintenance window start (RFC3339): ' + IFS= read -r WINDOW_START || { echo "STOP: maintenance window start is required; nothing was stopped." >&2; return 1; } + test -n "$WINDOW_START" || { echo "STOP: maintenance window start is required; nothing was stopped." >&2; return 1; } + printf 'External writers (or none): ' + IFS= read -r EXTERNAL_WRITERS || { echo "STOP: external writer record is required; nothing was stopped." >&2; return 1; } + test -n "$EXTERNAL_WRITERS" || { echo "STOP: external writer record is required; nothing was stopped." >&2; return 1; } + printf 'Writer stop time (RFC3339): ' + IFS= read -r STOP_TIME || { echo "STOP: writer stop time is required; nothing was stopped." >&2; return 1; } + test -n "$STOP_TIME" || { echo "STOP: writer stop time is required; nothing was stopped." >&2; return 1; } + cat > "$WRITER_RECORD" <&2 + return 1 +fi +if test "${#FORCE_ARGS[@]}" -eq 0; then + stop_deployment "$REGISTRY_NS" "$REGISTRY_DEPLOY" + stop_deployment "$REGISTRY_NS" "$GATEWAY_DEPLOY" +else + acns get deployment "$REGISTRY_DEPLOY" "$GATEWAY_DEPLOY" -o wide + echo "force mode: leaving legacy Registry/Gateway states unchanged" +fi +LEGACY_PRUNER="$(acns get cronjob image-pruner --ignore-not-found -o name)" +if test -n "$LEGACY_PRUNER"; then + if test "${#FORCE_ARGS[@]}" -eq 0; then + acns patch cronjob image-pruner --type=merge -p '{"spec":{"suspend":true}}' + else + echo "force mode: leaving the legacy Pruner state unchanged" + fi +fi +acns get endpoints "$REGISTRY_SERVICE" -o wide +acns get endpointslices.discovery.k8s.io -l "kubernetes.io/service-name=$REGISTRY_SERVICE" -o wide +mkdir -p "$RUN_DIR/verification" +acn get cronjob,job -A -o wide > "$RUN_DIR/verification/legacy-jobs.txt" || + echo "WARN: could not list cluster jobs" >&2 +if test "$WRITER_RECORD" != "$RUN_DIR/writer-record.txt"; then + cp "$WRITER_RECORD" "$RUN_DIR/writer-record.txt" +fi +evidence writers-stopped.json legacyWritersStopped +registry-migrate run --run-dir "$RUN_DIR" --stage P2-stop-writers \ + --evidence-file "$RUN_DIR/writers-stopped.json" \ + "${FORCE_ARGS[@]}" --kubeconfig "$KCFG" --approve +} +stop_writers ``` -Review the generated list before continuing: +Normal-path success check: the old Registry/Gateway Pods and Ready endpoints are gone, the Pruner is suspended, relevant active Jobs have finished, and the stage is recorded. Review `verification/legacy-jobs.txt` for jobs affecting this Registry; unrelated platform jobs are outside this migration. If the listing fails, resolve that failure before P3. -```bash -sed -n '1,20p' "$IMAGE_LIST_FILE" -sed -n '1,20p' "$IMAGE_DIGEST_FILE" -wc -l "$IMAGE_LIST_FILE" -``` +Confirm external writers against `WRITER_RECORD`. Two unchanged S3 listings are only supporting observations: equal object counts and latest `LastModified` values do not prove that all writers are stopped or rule out overwrites. -Remove temporary test images, retired namespaces, and incorrect tags from the list. +If the platform cannot hold old components stopped, use `FORCE_ARGS=(--force)` only after the administrator accepts the consistency risk. Record their actual state and the tool's force warning; do not claim those Pods or endpoints are absent. See appendix B. -## Write Registry Credentials +## 5. P3-final-inventory: freeze the inventory and S3 state -Write credentials for the legacy and target registries. `ac image mirror` and `ac image info` read this file: +The final inventory records repository/tag/digest relationships and manifest/config metadata. `s3-snapshot.json` is an object-listing snapshot with counts, bytes, and a listing digest; **it is not a backup of image contents**. Layers stay in the existing S3 bucket/prefix. -```bash -export DOCKER_CONFIG="$REGISTRY_AUTH_DIR" +For a recovery point, use a storage snapshot or backup that has been tested for restore. Enabling S3 versioning alone is not verified recovery evidence. If no recovery point is available, choose the explicit skip path and retain its warning. + +Once P3 succeeds, its inventory and protection state are frozen. Adding proof later does not change a run that skipped verification. Before starting, choose exactly one of the following paths; running both assigns the last choice, so do not execute both. -ac registry login \ - --registry "$LEGACY_REGISTRY_HOST" \ - --skip-check \ - --to "$REGISTRY_AUTH_FILE" \ - $REGISTRY_INSECURE_FLAG +**Path 1: explicitly skip recovery verification.** The administrator accepts that no S3 recovery point has been established; the tool records a warning and `not-established`. -ac registry login \ - --registry "$TARGET_REGISTRY_HOST" \ - --skip-check \ - --to "$REGISTRY_AUTH_FILE" \ - $REGISTRY_INSECURE_FLAG +```bash +P3_PROTECTION=(--skip-s3-backup-verification) ``` -If the source and target registries belong to different ACP environments, write both credentials into the same auth file with the corresponding kubeconfig: +**Path 2: provide a verified recovery point.** The tool records the verification assertion and reference; it does not perform the storage restore itself. -```bash -KUBECONFIG="$SOURCE_KUBECONFIG" \ - ac registry login --registry "$LEGACY_REGISTRY_HOST" --skip-check --to "$REGISTRY_AUTH_FILE" $REGISTRY_INSECURE_FLAG +`RESTORE_REFERENCE` identifies an actual verified backup/snapshot or its restore-test report, for example `backup-registry-20261104-001` or `/secure/recovery/registry-restore-test.json`. The associated record should identify the source bucket/prefix, recovery time, and restore result. A bucket name or S3 URL is not a recovery-point reference. -KUBECONFIG="$TARGET_KUBECONFIG" \ - ac registry login --registry "$TARGET_REGISTRY_HOST" --skip-check --to "$REGISTRY_AUTH_FILE" $REGISTRY_INSECURE_FLAG +```bash +printf 'Enter the verified S3 recovery-point identifier or report path: ' +IFS= read -r RESTORE_REFERENCE +test -n "$RESTORE_REFERENCE" +jq --arg actor "$ACTOR" --arg at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ + --arg ref "$RESTORE_REFERENCE" \ + '{version:"v1",runID:.runID,planDigest:.digest,observedAt:$at,actor:$actor,restoreVerified:true,restoreReference:$ref}' \ + "$RUN_DIR/plan.json" > "$RUN_DIR/restore-proof.json" +P3_PROTECTION=(--evidence-file "$RUN_DIR/restore-proof.json") ``` -## Prepare Metadata Backfill +Nested repositories are not skipped by default. Use the option below only for administrator-approved historical test residue. Review every affected repository in `legacy-metadata-report.json`; do not use it to exclude live workloads just to pass the stage. + +```bash +NESTED_ARGS=() +# NESTED_ARGS=(--skip-nested-repositories) +``` -Registry v2 needs Image API metadata in addition to image blobs. Use `ImageStreamMapping` to backfill `Image`, `ImageStream`, and tag metadata. +If P3 cannot resolve a cluster-internal S3 endpoint such as `*.svc.cluster.local`, establish the port forward and `S3_ARGS` in appendix A, then return here. -Define the metadata backfill functions in the migration shell session: +This example uses `resume`. In the current tool, `run` and `resume` execute the same selected-stage logic: an incomplete P3 can rescan S3 and metadata, with no pagination checkpoint. After P3 is recorded, retries validate its frozen artifacts rather than scanning the live source again. ```bash -ensure_imagestream() { - namespace="$1" - stream="$2" - stream_file="$METADATA_DIR/${namespace}-${stream}.imagestream.yaml" - stream_output_file="$METADATA_DIR/${namespace}-${stream}.imagestream.apply.txt" +registry-migrate resume --run-dir "$RUN_DIR" --stage P3-final-inventory \ + "${P3_PROTECTION[@]}" "${S3_ARGS[@]}" "${NESTED_ARGS[@]}" \ + "${FORCE_ARGS[@]}" --kubeconfig "$KCFG" --approve - if ac get imagestreams.image.alauda.io "$stream" -n "$namespace" >/dev/null 2>&1; then - return 0 - fi +registry-migrate report --run-dir "$RUN_DIR" - cat > "$stream_file" < "$stream_output_file" 2>&1; then - return 0 - fi +Success check: P3 is recorded and `final-inventory.json` and `s3-snapshot.json` are present. The scan reads tag links, manifests, and configs without copying layers. - echo "[metadata] failed to create ImageStream $namespace/$stream; output=$stream_output_file" >&2 - sed -n '1,20p' "$stream_output_file" >&2 - return 1 -} +Before retrying an incomplete P3 with `NESTED_ARGS=(--skip-nested-repositories)`, approve each affected repository in the report. Excluded tags are not imported. Do not edit a successful P3's inventory to add them; use a newly approved migration run. -write_imagestream_mapping() { - image="$1" - digest="${2:-}" - namespace="${image%%/*}" - rest="${image#*/}" - stream="${rest%%:*}" - tag="${rest##*:}" - inspect_registry_host="${METADATA_INSPECT_REGISTRY_HOST:-$TARGET_REGISTRY_HOST}" - source_ref="$inspect_registry_host/$image" - repo_ref="$TARGET_REGISTRY_HOST/$namespace/$stream" - inspect_ref="$source_ref" - name_prefix="${namespace}-${stream}-${tag}" - info_file="$METADATA_DIR/${name_prefix}.info.txt" - mapping_file="$METADATA_DIR/${name_prefix}.mapping.yaml" - apply_output_file="$METADATA_DIR/${name_prefix}.apply.txt" - BACKFILL_LAST_ACTION="" - - if [ -n "$digest" ]; then - inspect_ref="$inspect_registry_host/$namespace/$stream@$digest" - fi +## 6. P4-prepare-operator: prepare the read-only Registry v2 - if ! ac image info "$inspect_ref" \ - --registry-config "$REGISTRY_AUTH_FILE" \ - $REGISTRY_INSECURE_FLAG > "$info_file" 2>&1; then - BACKFILL_LAST_ACTION="failed" - echo "[metadata] failed to inspect $inspect_ref; output=$info_file" >&2 - sed -n '1,20p' "$info_file" >&2 - return 1 - fi +In the ACP console, open **Administrator → Marketplace → OperatorHub**, select the target business cluster, and install or upgrade `cluster-image-registry-operator` in `image-registry-system`. Set the operand namespace to `cpaas-system` (or the namespace in `plan.json`), approve the InstallPlan, and wait for the CSV to report `Succeeded` and the Operator Deployment to exist. Then keep the Operator at zero replicas during P4–P6 using the verified pause procedure. - [ -n "$digest" ] || digest="$(awk '/^Digest:/ { sub(/^Digest:[[:space:]]*/, ""); print; exit }' "$info_file")" - media_type="$(awk '/^Media Type:/ { sub(/^Media Type:[[:space:]]*/, ""); print; exit }' "$info_file")" +Record the original Operator replica count for P7, separately from the planned Registry count. P8 restores the Pruner baseline; a new ImagePruner records `registry-migrate.original-suspend=false`, allowing scheduled cleanup. To keep cleanup disabled, arrange an approved suspended baseline before P4. Do not change the recorded annotation afterwards. - if [ -z "$digest" ]; then - echo "failed to read digest from $source_ref" >&2 +```bash +prepare_operator() { +acn -n "$OPERATOR_NS" get subscription,installplan,csv -o wide || return 1 +if ! acn -n "$OPERATOR_NS" get deployment cluster-image-registry-operator -o wide >/dev/null; then + echo "STOP: install or upgrade cluster-image-registry-operator before P4" >&2 + return 1 +fi +stop_deployment "$OPERATOR_NS" cluster-image-registry-operator || return 1 +PRUNER="$(acn get imagepruner.imageregistry.operator.alauda.io cluster --ignore-not-found -o name)" || return 1 +if test -n "$PRUNER"; then + acn get "$PRUNER" -o json > "$RUN_DIR/imagepruner-before.json" || return 1 + jq '{annotations:.metadata.annotations,suspend:.spec.suspend}' "$RUN_DIR/imagepruner-before.json" || return 1 + existing_run="$(jq -r '.metadata.annotations["registry-migrate.run-id"] // empty' "$RUN_DIR/imagepruner-before.json")" || return 1 + existing_plan="$(jq -r '.metadata.annotations["registry-migrate.plan-digest"] // empty' "$RUN_DIR/imagepruner-before.json")" || return 1 + current_run="$(jq -er '.runID' "$RUN_DIR/plan.json")" || return 1 + current_plan="$(jq -er '.digest' "$RUN_DIR/plan.json")" || return 1 + if test -n "$existing_run" && test "$existing_run" != "$current_run"; then + echo "STOP: imagepruner belongs to run $existing_run; refusing to reuse it" >&2 return 1 fi - if [ -z "$media_type" ]; then - echo "failed to read media type from $source_ref" >&2 + if test -n "$existing_plan" && test "$existing_plan" != "$current_plan"; then + echo "STOP: imagepruner was prepared by a different plan ($existing_plan)" >&2 return 1 fi - - ensure_imagestream "$namespace" "$stream" || return 1 - - current_line="$( - ac get imagestreamtags "$stream:$tag" -n "$namespace" 2>/dev/null \ - | awk 'NR > 1 && $3 == "current" { print $4 " " $6; exit }' - )" - current_digest="${current_line%% *}" - current_reference="${current_line#* }" - expected_reference="$repo_ref@$digest" - - if [ "$current_digest" = "$digest" ] && [ "$current_reference" = "$expected_reference" ]; then - BACKFILL_LAST_ACTION="skipped" - return 0 - fi - - cat > "$mapping_file" < "$apply_output_file" 2>&1; then - BACKFILL_LAST_ACTION="applied" - return 0 + if test -z "$existing_run"; then + acn annotate "$PRUNER" "registry-migrate.run-id=$current_run" || return 1 fi - - BACKFILL_LAST_ACTION="failed" - echo "[metadata] failed $image; output=$apply_output_file" >&2 - sed -n '1,20p' "$apply_output_file" >&2 - return 1 -} - -backfill_metadata() { - total=0 - success=0 - applied=0 - skipped=0 - failed=0 - progress_interval="${METADATA_PROGRESS_INTERVAL:-100}" - verbose="${METADATA_VERBOSE:-false}" - - if [ -n "${IMAGE_DIGEST_FILE:-}" ] && [ -s "$IMAGE_DIGEST_FILE" ]; then - echo "[metadata] start backfill source=$IMAGE_DIGEST_FILE output_dir=$METADATA_DIR progress_interval=$progress_interval" - while read -r image digest; do - [ -n "$image" ] || continue - total=$((total + 1)) - if write_imagestream_mapping "$image" "$digest"; then - success=$((success + 1)) - [ "$BACKFILL_LAST_ACTION" = "applied" ] && applied=$((applied + 1)) - [ "$BACKFILL_LAST_ACTION" = "skipped" ] && skipped=$((skipped + 1)) - else - failed=$((failed + 1)) - fi - if [ "$verbose" = "true" ]; then - echo "[metadata] [$total] $image action=${BACKFILL_LAST_ACTION:-unknown}" - elif [ "$progress_interval" -gt 0 ] && [ $((total % progress_interval)) -eq 0 ]; then - echo "[metadata] progress: total=$total success=$success applied=$applied skipped=$skipped failed=$failed" - fi - done < "$IMAGE_DIGEST_FILE" - elif [ -n "${IMAGE_LIST_FILE:-}" ] && [ -s "$IMAGE_LIST_FILE" ]; then - echo "[metadata] start backfill source=$IMAGE_LIST_FILE output_dir=$METADATA_DIR progress_interval=$progress_interval" - while read -r image; do - [ -n "$image" ] || continue - total=$((total + 1)) - if write_imagestream_mapping "$image"; then - success=$((success + 1)) - [ "$BACKFILL_LAST_ACTION" = "applied" ] && applied=$((applied + 1)) - [ "$BACKFILL_LAST_ACTION" = "skipped" ] && skipped=$((skipped + 1)) - else - failed=$((failed + 1)) - fi - if [ "$verbose" = "true" ]; then - echo "[metadata] [$total] $image action=${BACKFILL_LAST_ACTION:-unknown}" - elif [ "$progress_interval" -gt 0 ] && [ $((total % progress_interval)) -eq 0 ]; then - echo "[metadata] progress: total=$total success=$success applied=$applied skipped=$skipped failed=$failed" - fi - done < "$IMAGE_LIST_FILE" - else - echo "[metadata] no image list found; check IMAGE_DIGEST_FILE or IMAGE_LIST_FILE" >&2 - return 1 + if test -z "$existing_plan"; then + acn annotate "$PRUNER" "registry-migrate.plan-digest=$current_plan" || return 1 fi - - echo "[metadata] summary: total=$total success=$success applied=$applied skipped=$skipped failed=$failed output_dir=$METADATA_DIR" - [ "$failed" -eq 0 ] +fi +evidence operator-paused.json operatorPaused || return 1 +registry-migrate run --run-dir "$RUN_DIR" --stage P4-prepare-operator \ + --evidence-file "$RUN_DIR/operator-paused.json" \ + "${FORCE_ARGS[@]}" --kubeconfig "$KCFG" --approve || return 1 +acn get config.imageregistry.operator.alauda.io cluster -o json | + jq -e '.spec.readOnly == true and + .spec.imagePullSecret.managementState == "Unmanaged" and + .spec.storage.managementState == "Unmanaged"' || return 1 +acn get imagepruner.imageregistry.operator.alauda.io cluster -o json | + jq -e '.spec.suspend == true' || return 1 } +prepare_operator ``` -Confirm that the function exists in the current shell: +Success check: the report shows `prepared`; Config has the planned S3 settings/routes, `readOnly=true`, and pull Secret/storage management both `Unmanaged`. The Pruner is suspended. `Unmanaged` storage leaves S3 lifecycle management with the administrator; it does not disconnect Registry from S3 or prohibit GC. -```bash -if type backfill_metadata >/dev/null 2>&1; then - echo "backfill_metadata loaded" -fi -``` +Stop if an ImagePruner belongs to another run/plan; resolve ownership with the provider. Preserve `registry-migrate.original-suspend` for P8. After an assertion failure, check the live Config, CRD version and pause state before retrying. + +## 7. P5-remove-legacy-chart: uninstall the legacy chart -For large migrations, the function prints progress every 100 images by default. Adjust the interval if needed: +Uninstalling the chart removes its workloads; the tool cannot reinstall them automatically. First capture the pre-uninstall Secret list below. Keep this original baseline when retrying after an uninstall; capturing it afterwards would hide losses. ```bash -export METADATA_PROGRESS_INTERVAL=500 +date -u > "$RUN_DIR/pre-uninstall-secrets.at" +acns get secret -o name > "$RUN_DIR/pre-uninstall-secrets.txt" ``` -Set `METADATA_VERBOSE=true` only when troubleshooting a small set of images. - -## Scenario A: Reuse Legacy Storage +Before uninstalling, confirm the platform's retention procedure protects the original business pull Secrets/ServiceAccounts, planned Ingress TLS Secrets, and P4 target S3 credentials. The P4 target Secret is `image-registry-private-configuration-user`; retain its UID and data. A differently named legacy source S3 Secret may retire with the chart once the P4 target checkpoint is established. -Use this path when the legacy Registry and Registry v2 run in the same ACP environment and business cluster, and Registry v2 can be configured to access the same blob storage that the legacy Registry uses. +The administrator then uninstalls the legacy Registry/Gateway plugin through the cluster plugin page. Chart-owned Secrets may be deleted with the chart, so check immediately afterwards: -This path does not copy image blobs. However, do not assume that Registry v2 can read old blobs by tag before metadata exists. The correct sequence is: - -1. Pause writes to the legacy Registry. -2. Attach Registry v2 to the legacy blob storage. -3. Read digest and media type from the legacy Registry. -4. Create `ImageStream` and `ImageStreamMapping` objects in Registry v2. -5. Verify reads through Registry v2. +```bash +test -s "$RUN_DIR/pre-uninstall-secrets.txt" || + echo "STOP: pre-uninstall snapshot missing; the diff below proves nothing" >&2 +printf 'baseline captured at: %s\n' "$(cat "$RUN_DIR/pre-uninstall-secrets.at" 2>/dev/null || echo unknown)" +acns get secret -o name > "$RUN_DIR/post-uninstall-secrets.txt" +diff "$RUN_DIR/pre-uninstall-secrets.txt" "$RUN_DIR/post-uninstall-secrets.txt" || + echo "WARN: Secret list changed; confirm the TLS Secret and S3 credentials still exist before continuing" >&2 +``` -Do not run prune or Registry GC against either Registry until migration has been accepted. +Treat the name-only diff as an initial loss check. Compare required UIDs with plan/P1 records and P4's credential checkpoint. Stop if a required Secret/ServiceAccount is missing or recreated, or target S3 credentials changed; use protected recovery material. Recreating a name does not restore its UID. -Set the Registry namespaces: +Once retention checks pass, remove the leftover Ingress and record P5. The next block deletes the plan's `gateway=true` Ingresses after comparing their UIDs; retain the route backup before running it. ```bash -export LEGACY_REGISTRY_NAMESPACE="${LEGACY_REGISTRY_NAMESPACE:-cpaas-system}" -export MODERN_REGISTRY_NAMESPACE="${MODERN_REGISTRY_NAMESPACE:-image-registry-system}" +remove_legacy_chart() { +acn -n "$OPERATOR_NS" get deployment cluster-image-registry-operator -o json | + jq -e '.spec.replicas == 0' >/dev/null || + echo "WARN: operator replicas is not 0; confirm it is still paused before removing the legacy chart" >&2 +ROUTE_COUNT="$(jq -e '[.discovery.routes[] | select(.gateway == true)] | length' "$RUN_DIR/plan.json")" || + { echo "STOP: cannot read routes from plan.json" >&2; return 1; } +test "$ROUTE_COUNT" -gt 0 || + echo "WARN: plan has no gateway route; confirm no legacy Ingress remains before recording the stage" >&2 +FAILED="" +while IFS=$'\t' read -r name expected_uid; do + actual_uid="$(acns get ingress "$name" --ignore-not-found -o jsonpath='{.metadata.uid}')" + if test -z "$actual_uid"; then + echo "SKIP: ingress $name is already absent" + continue + fi + if test "$actual_uid" != "$expected_uid"; then + echo "STOP: ingress $name has a different UID ($actual_uid != $expected_uid); not deleting" >&2 + FAILED=1 + continue + fi + acns delete ingress "$name" --wait=true +done < <(jq -r '.discovery.routes[] | select(.gateway == true) | [.name,.ingressUID] | @tsv' "$RUN_DIR/plan.json") +test -z "$FAILED" || { echo "STOP: legacy Ingress cleanup incomplete; stage not recorded. Shell stays open." >&2; return 1; } +registry-migrate run --run-dir "$RUN_DIR" --stage P5-remove-legacy-chart \ + --kubeconfig "$KCFG" --approve +} +remove_legacy_chart ``` -### Filesystem or PVC Storage Reuse +Success check: P5 is recorded; old Service/Deployments/routes are gone, planned TLS Secrets remain, and business Secret/ServiceAccount identities are unchanged. Stop on a UID mismatch and consult the provider. New Ingresses use `targetName`; old names are cleanup targets only. -For filesystem storage such as NFS or CephFS, make the Registry v2 PVC point to the same legacy Registry data. The exact PV/PVC manifest depends on the storage backend. The example below applies only when the legacy Registry uses an NFS CSI PV. +## 8. P6-cold-import: import metadata and limits -Locate the legacy Registry PVC and PV: +Use the provider-approved `registry-migrate` build with pre-existing ImageStream checks. P6 checks all target streams before its first metadata write. A matching digest alone is insufficient: the repository, source and Registry address must also match the frozen inventory and planned addresses. Equivalent pre-existing objects are reused without changing their UID or ownership. Known internal and external addresses do not require rewriting tag history. -```bash -export LEGACY_REGISTRY_PVC="${LEGACY_REGISTRY_PVC:-image-registry}" -export MODERN_REGISTRY_PVC="${MODERN_REGISTRY_PVC:-image-registry-storage}" -export REUSE_PV_NAME="${REUSE_PV_NAME:-image-registry-legacy-reuse}" - -LEGACY_PV_NAME="$( - ac get pvc "$LEGACY_REGISTRY_PVC" -n "$LEGACY_REGISTRY_NAMESPACE" \ - -o jsonpath='{.spec.volumeName}' -)" -test -n "$LEGACY_PV_NAME" +Keep the new Operator paused. Confirm zero replicas and no Operator Pods, then generate fresh pause evidence and import: -ac get pvc "$LEGACY_REGISTRY_PVC" -n "$LEGACY_REGISTRY_NAMESPACE" -ac get pv "$LEGACY_PV_NAME" +```bash +stop_deployment "$OPERATOR_NS" cluster-image-registry-operator +acn -n "$OPERATOR_NS" get deployment cluster-image-registry-operator -o json | jq -e '.spec.replicas == 0' >/dev/null +evidence operator-paused.json operatorPaused +registry-migrate run --run-dir "$RUN_DIR" --stage P6-cold-import \ + --inventory-file "$RUN_DIR/final-inventory.json" \ + --evidence-file "$RUN_DIR/operator-paused.json" \ + "${FORCE_ARGS[@]}" --kubeconfig "$KCFG" --approve ``` -Confirm that the legacy PV uses the NFS CSI driver and collect the fields needed to reuse it: +Success check: the report shows `imported`, with progress for unique ImageDigest objects, ImageStream/tag relationships, and namespace limits. A shared digest is stored once while preserving its repository/tag references. Retry failures in the same run with the Operator still paused; section 9 checks the result. -```bash -LEGACY_STORAGE_DRIVER="$( - ac get pv "$LEGACY_PV_NAME" -o jsonpath='{.spec.csi.driver}' -)" -test "$LEGACY_STORAGE_DRIVER" = "nfs.csi.k8s.io" +If preflight reports a digest/source conflict or an existing stream that needs changes but is not owned by this migration, stop and resolve it against `final-inventory.json` and the saved ImageStream snapshot. The tool leaves the object unchanged. Do not add `registry-migrate.run-id` or `registry-migrate.plan-digest` manually: annotations written outside the tool journal have no rollback record and can block later runs. A provider-approved journaled adoption procedure must separately validate UID/resourceVersion and the repository/tag/digest mapping. A same-digest object that needs no change is reused automatically. Neither `--force` nor deleting the inventory/journal resolves metadata conflicts. Keep any ImageDigest objects from a partial earlier attempt; P6 reuses matching records. -LEGACY_NFS_SERVER="$( - ac get pv "$LEGACY_PV_NAME" -o jsonpath='{.spec.csi.volumeAttributes.server}' -)" -LEGACY_NFS_SHARE="$( - ac get pv "$LEGACY_PV_NAME" -o jsonpath='{.spec.csi.volumeAttributes.share}' -)" -LEGACY_NFS_SUBDIR="$( - ac get pv "$LEGACY_PV_NAME" -o jsonpath='{.spec.csi.volumeAttributes.subdir}' -)" -LEGACY_NFS_VOLUME_HANDLE="$( - ac get pv "$LEGACY_PV_NAME" -o jsonpath='{.spec.csi.volumeHandle}' -)" -LEGACY_STORAGE_SIZE="$( - ac get pv "$LEGACY_PV_NAME" -o jsonpath='{.spec.capacity.storage}' -)" +After an approved, journaled adoption procedure, rerun the original P6 command with fresh pause evidence. The tool pins eligible DockerImage tag sources to the verified digest and preserves existing history; real digest conflicts remain errors. A UID/resourceVersion test failure means the selected object changed: stop and review a fresh snapshot. Do not continue to P7 until P6 is `imported`. -test -n "$LEGACY_NFS_SERVER" -test -n "$LEGACY_NFS_SHARE" -test -n "$LEGACY_NFS_SUBDIR" -test -n "$LEGACY_NFS_VOLUME_HANDLE" -test -n "$LEGACY_STORAGE_SIZE" -``` +The old Gateway limits become each planned namespace's `registry-migrate-image-limits` LimitRange: image size uses `type: alauda.io/Image` with `max.storage`; tag count uses `type: alauda.io/ImageStream` with `max["alauda.io/repository-tags"]`. Legacy `2GB/2000` becomes `2Gi/2000` (`2Gi=2048Mi`). **2000 is the tag limit for each repository/ImageStream in that namespace, not the namespace-wide total.** The plan selects namespace-specific rules. Migration does not overwrite user-defined limits or apply policy to future namespaces. -If the storage backend is not NFS CSI, do not reuse this manifest. Ask the storage administrator to provide a PV/PVC that mounts the same legacy data and keep the PVC name aligned with `Config/cluster.spec.storage.pvc.claim`. +## 9. P7-verify: resume the Operator and verify read-only service -Before replacing Registry v2 storage, stop the Registry v2 runtime and wait for the data-plane deployments to be removed: +Release the platform pause and restore the Operator replica count from the change record. The command below uses the common value of 1; use the original Operator count if different. Do not confuse it with the Registry or Image API instance replica counts. ```bash -ac patch config.imageregistry.operator.alauda.io cluster \ - --type merge \ - -p '{"spec":{"managementState":"Removed"}}' - -while ac get deployment image-registry -n "$MODERN_REGISTRY_NAMESPACE" >/dev/null 2>&1; do - sleep 5 -done - -while ac get deployment image-api-server -n "$MODERN_REGISTRY_NAMESPACE" >/dev/null 2>&1; do - sleep 5 +acn -n "$OPERATOR_NS" scale deployment cluster-image-registry-operator --replicas=1 +acn -n "$OPERATOR_NS" wait --for=condition=Available \ + deployment/cluster-image-registry-operator --timeout=300s +for deployment in image-registry image-api-server; do + acns wait --for=create "deployment/$deployment" --timeout=600s + acns wait --for=condition=Available "deployment/$deployment" --timeout=600s done +acn wait --for=condition=Available apiservice/v1.image.alauda.io --timeout=300s +acn get apiservice v1.image.alauda.io -o json | + jq -e '.status.conditions // [] | any(.[]; .type == "Available" and .status == "True" and .reason == "Passed")' +registry-migrate run --run-dir "$RUN_DIR" --stage P7-verify --kubeconfig "$KCFG" --approve +registry-migrate verify --run-dir "$RUN_DIR" --kubeconfig "$KCFG" +acn get config.imageregistry.operator.alauda.io cluster -o json | + jq -e '.spec.readOnly == true and + .spec.imagePullSecret.managementState == "Unmanaged" and + .spec.storage.managementState == "Unmanaged"' +acn get imagepruner.imageregistry.operator.alauda.io cluster -o json | + jq -e '.spec.suspend == true' ``` -Create a static PV/PVC that points to the legacy data: - -```bash -cat > "$MIGRATION_DIR/modern-registry-reuse-legacy-storage.yaml" </dev/null 2>&1; do - sleep 5 -done +mkdir -p "$RUN_DIR/verification" +acn get imagedigests.image.alauda.io -o yaml > "$RUN_DIR/verification/imagedigests.yaml" || + echo "WARN: imagedigests not captured" >&2 +acn get imagestreams.image.alauda.io -A -o yaml > "$RUN_DIR/verification/imagestreams.yaml" || + echo "WARN: imagestreams not captured" >&2 +acn get limitrange -A --field-selector metadata.name=registry-migrate-image-limits -o yaml \ + > "$RUN_DIR/verification/migration-limitranges.yaml" || + echo "WARN: limitranges not captured" >&2 +``` -while ! ac get deployment image-api-server -n "$MODERN_REGISTRY_NAMESPACE" >/dev/null 2>&1; do - sleep 5 -done +Compare digest sets rather than repository/tag counts. These commands query the **public** `image.alauda.io` API, which exposes canonical digest names. The internal names of backing `storage.image.alauda.io` CRDs are not directly comparable. -ac rollout status deployment/image-registry -n "$MODERN_REGISTRY_NAMESPACE" --timeout=300s -ac rollout status deployment/image-api-server -n "$MODERN_REGISTRY_NAMESPACE" --timeout=300s +```bash +jq -r '.items[].digest | select(. != null)' "$RUN_DIR/final-inventory.json" | sort -u \ + > "$RUN_DIR/verification/inventory-digests.txt" +acn get imagedigests.image.alauda.io -o json | + jq -r '.items[].metadata.name' | sort -u > "$RUN_DIR/verification/cluster-digests.txt" +wc -l < "$RUN_DIR/verification/inventory-digests.txt" +comm -23 "$RUN_DIR/verification/inventory-digests.txt" "$RUN_DIR/verification/cluster-digests.txt" ``` -### Object Storage Reuse +`wc -l` prints the inventory's unique digest count. `comm -23` lists inventory digests missing from the cluster; **its output must be empty**. Investigate any missing entry. Additional cluster digests may belong to historical images; record the reason. Digest-set equality does not replace the tool's repository/tag, credential-identity and limit checks. -For object storage such as S3, OSS, GCS, Azure, Swift, or IBM COS, configure Registry v2 to use the same bucket or container, endpoint, region, path prefix, credentials, CA, and access parameters that the legacy Registry used. +Imported `LimitRange` objects are static settings for the namespaces covered by the plan, not a global policy for namespaces created later. -If the legacy Registry uses a path prefix and Registry v2 cannot configure the same prefix, use Scenario B instead. Also use Scenario B if the Registry v2 Operator does not support the legacy object storage type or a required access parameter. +## 10. P8-commit: verify business access, then open writes -If Registry v2 already uses another storage backend, stop the runtime before changing object storage: +Run each code block separately in the **same Bash session** used for P0-P7, then press **Enter**. +Stop on an error before running the next block. If the terminal remains at a `>` prompt, press +**Ctrl+C** and paste that complete block again, without the Markdown fence markers. -```bash -ac patch config.imageregistry.operator.alauda.io cluster \ - --type merge \ - -p '{"spec":{"managementState":"Removed"}}' +### Prepare the Endpoint and Test Image -while ac get deployment image-registry -n "$MODERN_REGISTRY_NAMESPACE" >/dev/null 2>&1; do - sleep 5 -done +`REGISTRY_URL` must be the new Registry's external base URL, including its scheme and any external +port. The default uses the planned scheme and hostname without an external port. When the actual +entry point differs, set `REGISTRY_URL_OVERRIDE`, for example `https://registry.example.com:11780`, +before running this block. Confirm that the address serves this Registry. -while ac get deployment image-api-server -n "$MODERN_REGISTRY_NAMESPACE" >/dev/null 2>&1; do - sleep 5 -done +```bash +P8_ENDPOINT_OK=0 +P8_LEGACY_QUERY_OK=0 +P8_MODERN_QUERY_OK=0 +unset REGISTRY_HOST REGISTRY_SCHEME REGISTRY_URL +REGISTRY_HOST="$(jq -er ' + ([.discovery.routes[] | select(.gateway == true)][0] // + .discovery.routes[0]).hostname | strings | select(length > 0) +' "$RUN_DIR/plan.json")" && +REGISTRY_SCHEME="$(jq -er ' + ([.discovery.routes[] | select(.gateway == true)][0] // + .discovery.routes[0]).scheme // "http" +' "$RUN_DIR/plan.json")" && +REGISTRY_URL="${REGISTRY_URL_OVERRIDE:-$REGISTRY_SCHEME://$REGISTRY_HOST}" && +REGISTRY_URL="${REGISTRY_URL%/}" && +mkdir -p "$RUN_DIR/verification" && +printf 'Planned host=%s\nREGISTRY_URL=%s\n' "$REGISTRY_HOST" "$REGISTRY_URL" ``` -Confirm the supported storage fields in the current Operator version: +Expected result: the planned host and approved external URL are printed. Resolve any error or +address mismatch before continuing. + +For HTTPS, the issuer must be trusted and the certificate SAN must match the host. For a private CA, +set `REGISTRY_CA_BUNDLE` to its PEM file; the following block supplies it to curl. +`REGISTRY_TLS_INSECURE=1` is diagnostic only; repeat the probe with certificate verification enabled +before accepting TLS. ```bash -ac explain configs.imageregistry.operator.alauda.io.spec.storage --recursive +CURL_TLS_ARGS=() +if test -n "${REGISTRY_CA_BUNDLE:-}"; then + CURL_TLS_ARGS+=(--cacert "$REGISTRY_CA_BUNDLE") +fi +if test "${REGISTRY_TLS_INSECURE:-0}" = 1; then + CURL_TLS_ARGS+=(--insecure) +fi ``` -Then inspect the legacy Registry configuration and secrets: +Probe the endpoint and save its response headers in one request: ```bash -ac get deployment image-registry -n "$LEGACY_REGISTRY_NAMESPACE" -o yaml -ac get configmap image-registry-config -n "$LEGACY_REGISTRY_NAMESPACE" -o yaml -ac get secret -n "$LEGACY_REGISTRY_NAMESPACE" +P8_ENDPOINT_OK=0 +if test -n "${REGISTRY_URL:-}" && + HTTP_STATUS="$(curl -sS "${CURL_TLS_ARGS[@]}" \ + --connect-timeout 10 --max-time 20 -o /dev/null -w '%{http_code}' \ + -D "$RUN_DIR/verification/registry-v2-headers.txt" "$REGISTRY_URL/v2/")"; then + case "$HTTP_STATUS" in + 200|401|403) P8_ENDPOINT_OK=1; echo "Registry reachable: HTTP $HTTP_STATUS" ;; + *) echo "STOP: unexpected HTTP $HTTP_STATUS from $REGISTRY_URL/v2/" >&2 ;; + esac +else + echo "STOP: endpoint request failed; check the URL, network, and TLS error." >&2 +fi +test "$P8_ENDPOINT_OK" = 1 ``` -The following S3 structure is an example only. Use `ac explain` and the actual legacy Registry configuration as the source of truth, replace bucket, region, endpoint, CA, and related access settings, and remove fields that the current environment does not use: - -```yaml -apiVersion: imageregistry.operator.alauda.io/v1 -kind: Config -metadata: - name: cluster -spec: - managementState: Managed - storage: - managementState: Unmanaged - s3: - bucket: - region: - regionEndpoint: - virtualHostedStyle: false - trustedCA: - name: -``` +Expected result: `Registry reachable` is printed and the block exits successfully. An anonymous +`200`, `401`, or `403` response proves reachability only. A connection or TLS error fails this check; +`502` or `504` usually indicates that a proxy or Ingress cannot reach the backend. -Save the storage configuration and apply it: +Inspect the authentication challenge, if present: ```bash -ac apply -f "$MIGRATION_DIR/modern-registry-object-storage.yaml" - -while ! ac get deployment image-registry -n "$MODERN_REGISTRY_NAMESPACE" >/dev/null 2>&1; do - sleep 5 -done - -while ! ac get deployment image-api-server -n "$MODERN_REGISTRY_NAMESPACE" >/dev/null 2>&1; do - sleep 5 -done +grep -i '^WWW-Authenticate:' "$RUN_DIR/verification/registry-v2-headers.txt" || true ``` -Do not store object storage credentials in the migration environment file. Create or reuse the Secret required by the Registry v2 Operator, then verify that the Registry pod has picked up the storage settings: - -```bash -ac get deployment image-registry -n "$MODERN_REGISTRY_NAMESPACE" -o yaml -ac logs -n "$MODERN_REGISTRY_NAMESPACE" deployment/image-registry -c registry --tail=80 - -ac rollout status deployment/image-registry -n "$MODERN_REGISTRY_NAMESPACE" --timeout=300s -ac rollout status deployment/image-api-server -n "$MODERN_REGISTRY_NAMESPACE" --timeout=300s -``` +Confirm that the challenge's realm host and port are reachable through the approved entry point. +See appendix A if they differ. No output means the response did not include this header. -After storage reuse is configured, backfill metadata by inspecting the legacy Registry: +Set `TEST_NS` and `TEST_REPO` to an approved migrated test image, or review the first inventory entry +selected by default. `TEST_REPO` uses `namespace/repository:tag` format. ```bash -export METADATA_INSPECT_REGISTRY_HOST="$LEGACY_REGISTRY_HOST" -ac config set-registry-mode modern -backfill_metadata +TEST_NS="${TEST_NS:-$(jq -er ' + .items[0].namespace | strings | select(length > 0) +' "$RUN_DIR/final-inventory.json")}" && +TEST_REPO="${TEST_REPO:-$(jq -er ' + .items[0] | select(.namespace and .repository and .tag) | + "\(.namespace)/\(.repository):\(.tag)" +' "$RUN_DIR/final-inventory.json")}" && +export TEST_NS TEST_REPO && +printf 'TEST_NS=%s\nTEST_REPO=%s\n' "$TEST_NS" "$TEST_REPO" ``` -If `ac image info "$TARGET_REGISTRY_HOST/$image"` fails before metadata backfill, that does not necessarily mean storage reuse is broken. Registry v2 can return `NAME_UNKNOWN` when Image and ImageStream metadata do not exist yet. Use the `backfill_metadata` result and the verification steps below as the decision point. +Expected result: both values identify the approved migrated test image. If selection fails or the +default is unsuitable, correct the values before continuing. -## Scenario B: Copy Images Between Registries +### Check CLI Compatibility -Use this path when the source and target registries do not share storage. +The compatibility client used by these `ac` commands currently skips Registry certificate +verification internally. Successful queries do not validate TLS trust. Use the verified curl probe +and successful OCI-client and kubelet pulls as TLS acceptance evidence. -Generate a mirror mapping file. This preserves namespace, repository, and tag names and only replaces the Registry host: +Query the namespace inventory in legacy mode: ```bash -while read -r image; do - [ -n "$image" ] || continue - printf '%s/%s=%s/%s\n' \ - "$LEGACY_REGISTRY_HOST" "$image" \ - "$TARGET_REGISTRY_HOST" "$image" -done < "$IMAGE_LIST_FILE" > "$MAPPING_FILE" - -sed -n '1,20p' "$MAPPING_FILE" -wc -l "$MAPPING_FILE" +P8_LEGACY_QUERY_OK=0 +if test "${P8_ENDPOINT_OK:-0}" = 1 && test -n "${TEST_NS:-}" && + env AC_IMAGE_REGISTRY_MODE=legacy \ + ac --kubeconfig "$KCFG" --context "$CTX" \ + get images -n "$TEST_NS" -o json --registry-url "$REGISTRY_URL" \ + > "$RUN_DIR/verification/legacy-images.json" && + jq -se 'length == 1 and (.[0] | type == "array")' \ + "$RUN_DIR/verification/legacy-images.json" >/dev/null; then + P8_LEGACY_QUERY_OK=1 + jq -r '"Legacy query succeeded: \(length) items"' \ + "$RUN_DIR/verification/legacy-images.json" +else + echo "STOP: endpoint or legacy inventory check failed." >&2 + false +fi ``` -Run a dry run before copying: +Query the cluster inventory in modern mode: ```bash -MIRROR_DRY_RUN_LOG="$MIGRATION_DIR/mirror-dry-run.log" - -ac image mirror \ - -f "$MAPPING_FILE" \ - --registry-config "$REGISTRY_AUTH_FILE" \ - --dry-run \ - --keep-manifest-list \ - $REGISTRY_INSECURE_FLAG \ - > "$MIRROR_DRY_RUN_LOG" 2>&1 - -echo "mirror dry-run log: $MIRROR_DRY_RUN_LOG" -sed -n '1,20p' "$MIRROR_DRY_RUN_LOG" -wc -l "$MIRROR_DRY_RUN_LOG" +P8_MODERN_QUERY_OK=0 +if test "${P8_ENDPOINT_OK:-0}" = 1 && + env AC_IMAGE_REGISTRY_MODE=modern \ + ac --kubeconfig "$KCFG" --context "$CTX" \ + get images -A -o json --registry-url "$REGISTRY_URL" \ + > "$RUN_DIR/verification/modern-images.json" && + jq -se 'length == 1 and (.[0] | type == "array")' \ + "$RUN_DIR/verification/modern-images.json" >/dev/null; then + P8_MODERN_QUERY_OK=1 + jq -r '"Modern query succeeded: \(length) items"' \ + "$RUN_DIR/verification/modern-images.json" +else + echo "STOP: endpoint or modern inventory check failed." >&2 + false +fi ``` -After reviewing the plan, copy the images. `--continue-on-error` continues with later mappings after a failed image, but the command still returns a non-zero exit code if any mapping failed. Capture that exit code so that log summarization still runs even in a shell that uses `set -e`: - -```bash -MIRROR_COPY_LOG="$MIGRATION_DIR/mirror-copy.log" -mirror_rc=0 - -ac image mirror \ - -f "$MAPPING_FILE" \ - --registry-config "$REGISTRY_AUTH_FILE" \ - --continue-on-error \ - --keep-manifest-list \ - --max-per-registry 4 \ - $REGISTRY_INSECURE_FLAG \ - > "$MIRROR_COPY_LOG" 2>&1 || mirror_rc=$? +Expected result: each block prints its success message and saves one JSON array. The queries have +different namespace scopes, so their counts need not match. Counts alone do not prove completeness. -echo "mirror copy summary: rc=$mirror_rc log=$MIRROR_COPY_LOG" -grep -Ein 'error|failed|denied|unauthorized|forbidden' "$MIRROR_COPY_LOG" | sed -n '1,20p' || true +Run the delete previews only after both queries succeed. **Never add `--confirm`, `--yes`, or `-f`.** -if [ "$mirror_rc" -ne 0 ]; then - echo "mirror copy had failed mappings; fix the failures before metadata backfill or cutover" >&2 +```bash +if test "${P8_LEGACY_QUERY_OK:-0}" = 1 && + test "${P8_MODERN_QUERY_OK:-0}" = 1 && test -n "${TEST_REPO:-}"; then + env AC_IMAGE_REGISTRY_MODE=legacy \ + ac --kubeconfig "$KCFG" --context "$CTX" delete images \ + --repo "$TEST_REPO" --registry-url "$REGISTRY_URL" && + env AC_IMAGE_REGISTRY_MODE=modern \ + ac --kubeconfig "$KCFG" --context "$CTX" delete images \ + --repo "$TEST_REPO" --registry-url "$REGISTRY_URL" +else + echo "STOP: both inventory queries and the test image selection must succeed." >&2 false fi ``` -Keep `--keep-manifest-list` during migration to preserve manifest lists and multi-architecture images. +Expected result: both previews identify the approved tag without deleting it; dry-run wording varies +by `ac` version. Both modes pass `--registry-url`, so these are HTTP compatibility checks. P7 verifies +the public Image API separately. A failed check remains unverified even if a later block succeeds. -After the image copy finishes, switch to modern mode and backfill metadata by inspecting Registry v2: +The next snippet is a **limited credential-format check**, not a complete audit of every namespace. It examines `.dockerconfigjson` Secrets only; legacy chart pull Secrets commonly use `.dockercfg` and must be checked separately against the P1 handoff records and original SA references. Other registries' credentials need not contain this Registry host. ```bash -export METADATA_INSPECT_REGISTRY_HOST="$TARGET_REGISTRY_HOST" -ac config set-registry-mode modern -backfill_metadata +acn get secret -A -o json | + jq -r --arg h "$REGISTRY_HOST" ' + .items[] | select(.type == "kubernetes.io/dockerconfigjson") | + (.data[".dockerconfigjson"] | @base64d | fromjson? | (.auths // {}) | keys) as $k | + select(($k | any(test($h))) | not) | + "REVIEW \(.metadata.namespace)/\(.metadata.name): no auth entry for \($h)"' ``` -## Rerun Safely +No output does not prove all pull Secrets were checked or that registry addresses match exactly: the snippet uses a regex, not an exact host/port comparison. Cross-check the P1 scope, legacy `.dockercfg` objects, each original image's registry address/port, and SA references. A `REVIEW` entry for another registry is not automatically a migration failure. -Migration steps are designed to be rerunnable. If the current `ImageStreamTag` already points to the expected digest and target Registry reference, `backfill_metadata` skips creating a new `ImageStreamMapping` and counts the item as `skipped`. +Before P8, pull Secret management is `Unmanaged`, so the Operator **does not rotate or repair these credentials automatically**. Resolve missing entries or invalid tokens before approving the handoff; do not wait for a background refresh. -For storage reuse, rerun metadata backfill: +Before committing, create a disposable Pod with `imagePullPolicy=Always`, using an approved migrated test image's **original address, ServiceAccount and pull Secret**. Verify actual kubelet pulls on the relevant business nodes and through each address/port used by the workload. Record the Pod, digest and pull result, then remove the test Pod. API queries or existing running Pods alone do not prove new pulls work. -```bash -export METADATA_INSPECT_REGISTRY_HOST="$LEGACY_REGISTRY_HOST" -ac config set-registry-mode modern -backfill_metadata -``` +Do not run `ac adm prune images` or GC during this procedure. They require platform-level authority and can remove data. Confirm the recorded Pruner suspension value and whether scheduled cleanup may resume after P8. -For cross-registry sync, rerun image copy and metadata backfill: +### Commit and Verify the Cutover -```bash -MIRROR_COPY_LOG="$MIGRATION_DIR/mirror-rerun.log" -mirror_rc=0 +Continue only after the business and credential checks pass and an administrator approves opening +writes with a controlled recovery plan. Prepare the approval evidence. Its verification fields +record the completed checks; creating the file does not perform them. -ac image mirror \ - -f "$MAPPING_FILE" \ - --registry-config "$REGISTRY_AUTH_FILE" \ - --continue-on-error \ - --keep-manifest-list \ - --max-per-registry 4 \ - $REGISTRY_INSECURE_FLAG \ - > "$MIRROR_COPY_LOG" 2>&1 || mirror_rc=$? +```bash +( + test -n "${ACTOR:-}" || { + echo "STOP: set ACTOR to the platform administrator account." >&2 + exit 1 + } + jq --arg actor "$ACTOR" --arg at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" ' + {version:"v1",runID:.runID,planDigest:.digest,observedAt:$at,actor:$actor, + commitApproved:true,businessProbeVerified:true,pullSecretHandoffVerified:true} + ' "$RUN_DIR/plan.json" > "$RUN_DIR/commit-approval.json" && + jq -e '.actor | length > 0' "$RUN_DIR/commit-approval.json" >/dev/null +) +``` -echo "mirror rerun summary: rc=$mirror_rc log=$MIRROR_COPY_LOG" -grep -Ein 'error|failed|denied|unauthorized|forbidden' "$MIRROR_COPY_LOG" | sed -n '1,20p' || true +Expected result: the block exits successfully and creates `commit-approval.json`. The parentheses +keep an error from closing the migration terminal. After reviewing the evidence, commit the cutover: -if [ "$mirror_rc" -ne 0 ]; then - echo "mirror rerun had failed mappings; inspect $MIRROR_COPY_LOG before metadata backfill" >&2 - false -fi - -export METADATA_INSPECT_REGISTRY_HOST="$TARGET_REGISTRY_HOST" -backfill_metadata +```bash +registry-migrate run --run-dir "$RUN_DIR" --stage P8-commit \ + --evidence-file "$RUN_DIR/commit-approval.json" --kubeconfig "$KCFG" --approve ``` -After a rerun, confirm that the same tag still points to the expected digest and target Registry reference. +Expected result: P8 is recorded as `committed`. If the command fails, inspect +`registry-migrate report --run-dir "$RUN_DIR"` and the live state before retrying or recovering; +writes might already be open. -## Verify Migration - -Verify important business images and random samples. The following example selects the first image from the migration list: +Verify the Registry settings: ```bash -VERIFY_IMAGE="$(awk 'NF > 0 { print $1; exit }' "$IMAGE_LIST_FILE")" -test -n "$VERIFY_IMAGE" - -VERIFY_NAMESPACE="${VERIFY_IMAGE%%/*}" -VERIFY_REST="${VERIFY_IMAGE#*/}" -VERIFY_STREAM="${VERIFY_REST%%:*}" -VERIFY_TAG="${VERIFY_REST##*:}" - -echo "verify image: $VERIFY_IMAGE" +( + set -o pipefail + acn get config.imageregistry.operator.alauda.io cluster -o json | + jq -e '.spec.readOnly == false and + .spec.imagePullSecret.managementState == "Managed" and + .spec.storage.managementState == "Unmanaged"' +) +``` -ac get imagestreamtags "$VERIFY_STREAM:$VERIFY_TAG" -n "$VERIFY_NAMESPACE" +Expected result: `true`. Verify the restored Pruner suspension state separately: -VERIFY_REF="$( - ac get imagestreamtags "$VERIFY_STREAM:$VERIFY_TAG" -n "$VERIFY_NAMESPACE" \ - | awk 'NR > 1 && $3 == "current" { print $6; exit }' -)" +```bash +( + set -o pipefail + acn get imagepruner.imageregistry.operator.alauda.io cluster -o json | + jq -e ' + (.metadata.annotations["registry-migrate.original-suspend"] // "false") as $original | + .spec.suspend == ($original == "true")' +) +``` -ac image info "$VERIFY_REF" \ - --registry-config "$REGISTRY_AUTH_FILE" \ - $REGISTRY_INSECURE_FLAG +Expected result: `true`. If it is `false`, compare `imagepruner-before.json` from P4 with the +recorded annotation before considering recovery. A restored value of `false` permits scheduled +cleanup; `Unmanaged` storage does not prevent it. -VERIFY_DIGEST="${VERIFY_REF##*@}" -VERIFY_IMAGE_NAME="$(printf '%s\n' "$VERIFY_DIGEST" | sed 's/:/-/g')" -VERIFY_IMAGES_OUTPUT="$MIGRATION_DIR/verify-modern-images.txt" +Run final verification: -ac get images > "$VERIFY_IMAGES_OUTPUT" -awk -v digest="$VERIFY_DIGEST" -v safe="$VERIFY_IMAGE_NAME" ' - NR > 1 && ($1 == digest || $1 == safe || index($2, digest) > 0) { found = 1 } - END { exit found ? 0 : 1 } -' "$VERIFY_IMAGES_OUTPUT" +```bash +registry-migrate verify --run-dir "$RUN_DIR" --kubeconfig "$KCFG" ``` -The example above verifies one sample only. Before cutover, run a full storage and metadata consistency gate against the migration list: +Expected result: verification exits 0. Then retain the final report: ```bash -VERIFY_SOURCE_FILE="$IMAGE_LIST_FILE" -VERIFY_SOURCE_HAS_DIGEST=false -if [ -n "${IMAGE_DIGEST_FILE:-}" ] && [ -s "$IMAGE_DIGEST_FILE" ]; then - VERIFY_SOURCE_FILE="$IMAGE_DIGEST_FILE" - VERIFY_SOURCE_HAS_DIGEST=true -fi +registry-migrate report --run-dir "$RUN_DIR" +``` -verify_failed=0 +A failing post-commit check does not mean that writes stayed closed. Account for new images and +credential rotation when planning recovery. Finally, use an approved OCI client to push and pull one +disposable repository and retain the output. Resolve authentication-realm or TLS failures and repeat +the client test; API or curl reads do not replace successful OCI and kubelet pulls. -while read -r image digest _; do - [ -n "$image" ] || continue - namespace="${image%%/*}" - rest="${image#*/}" - stream="${rest%%:*}" - tag="${rest##*:}" +## Appendix A: known environment limitations - tag_output_file="$MIGRATION_DIR/verify-${namespace}-${stream}-${tag}.imagestreamtag.txt" - tag_info_file="$MIGRATION_DIR/verify-${namespace}-${stream}-${tag}.tag.info.txt" - digest_info_file="$MIGRATION_DIR/verify-${namespace}-${stream}-${tag}.digest.info.txt" +### S3 reachable only from inside the cluster - if ! ac get imagestreamtags "$stream:$tag" -n "$namespace" > "$tag_output_file" 2>&1; then - echo "[verify] ImageStreamTag missing: $namespace/$stream:$tag output=$tag_output_file" >&2 - verify_failed=$((verify_failed + 1)) - continue - fi +If the plan's S3 endpoint is cluster-internal, this machine may fail to resolve it and P3 can stop with `dial tcp: lookup ... no such host`. Establish a port forward in **a second terminal**, where it remains in the foreground. - ref="$( - awk 'NR > 1 && $3 == "current" { print $6; exit }' "$tag_output_file" - )" - if [ -z "$ref" ]; then - echo "[verify] failed to read current reference for $namespace/$stream:$tag output=$tag_output_file" >&2 - verify_failed=$((verify_failed + 1)) - continue - fi - - if ! ac image info "$ref" \ - --registry-config "$REGISTRY_AUTH_FILE" \ - $REGISTRY_INSECURE_FLAG > "$tag_info_file" 2>&1; then - echo "[verify] target registry cannot inspect ImageStreamTag reference $ref output=$tag_info_file" >&2 - verify_failed=$((verify_failed + 1)) - fi - - if [ "$VERIFY_SOURCE_HAS_DIGEST" = "true" ] && [ -n "$digest" ]; then - digest_ref="$TARGET_REGISTRY_HOST/$namespace/$stream@$digest" - if ! ac image info "$digest_ref" \ - --registry-config "$REGISTRY_AUTH_FILE" \ - $REGISTRY_INSECURE_FLAG > "$digest_info_file" 2>&1; then - echo "[verify] target registry cannot inspect expected digest $digest_ref output=$digest_info_file" >&2 - verify_failed=$((verify_failed + 1)) - fi - fi -done < "$VERIFY_SOURCE_FILE" +Set `KCFG` and `CTX` in that terminal to the exact values used by the migration terminal. Its environment is independent, and the default/current context may identify another cluster. If the kubeconfig has no `current-context`, explicitly supply the selected `CTX`. Replace `minio-system/service/minio` and the port with the actual S3 Service details: -if [ "$verify_failed" -ne 0 ]; then - echo "[verify] migration storage/metadata consistency gate failed: $verify_failed images failed" >&2 - false -fi +```bash +: "${KCFG:=$HOME/.kube/config}" +: "${CTX:=$(ac --kubeconfig "$KCFG" config current-context)}" +ac --kubeconfig "$KCFG" --context "$CTX" -n minio-system port-forward service/minio 19000:9000 ``` -If this gate fails, at least one Image API metadata entry does not match readable Registry v2 storage data. Do not switch traffic, prune images, or run registry garbage collection. Fix storage reuse or image copy first, or remove stale backfilled metadata after explicitly accepting that legacy image data will not be preserved. +Back in the **migration terminal**, confirm that the tunnel works and set the override: -Acceptance criteria: +```bash +curl --fail --connect-timeout 5 --max-time 10 http://127.0.0.1:19000/minio/health/live +S3_ARGS=(--s3-endpoint http://127.0.0.1:19000) +``` -- Registry v2 can inspect or pull the current digest reference from `ImageStreamTag`. -- `ImageStreamMapping` creation succeeded. -- Reruns do not produce unexplained failures. -- `ImageStreamTag` points to the expected digest and Registry v2 address. -- `ac get images` in modern mode shows the migrated Image resources through the ACP Image API. -- The full storage and metadata consistency gate passes for every migrated image in the migration list. -- Sample images in critical business namespaces can be read. +This override changes only P3's transport address; the endpoint recorded in the plan/snapshot and target Config remains the source endpoint. Port forwarding does not convert HTTP to HTTPS. Reestablish a dropped tunnel and retry the same run. -## Switch Registry Mode +### Authentication realm and ingress port disagree -After verification, switch the current kubeconfig context to modern Registry mode: +The 401 response from `/v2/` may advertise a `WWW-Authenticate` realm with a port different from the externally reachable Ingress port, such as 443 instead of 11780. Registry data can remain readable while OCI authentication fails. Use the `REGISTRY_URL` defined in section 10 to inspect the challenge: ```bash -ac config set-registry-mode modern -ac config get-registry-mode +curl -sS -k -D - -o /dev/null "$REGISTRY_URL/v2/" | grep -i '^www-authenticate:' ``` -`ac config set-registry-mode` changes only the current context's cluster by default. Use `--all-clusters` only after every cluster in the ACP session has completed Registry migration. +Replace diagnostic `-k` with system trust or `--cacert /path/to/ca.pem` for verified TLS. If the realm is unreachable, diagnose it with curl, correct the entry-point configuration, and repeat OCI/kubelet pull checks. Record an entry-point failure separately from metadata integrity; a successful curl read does not establish full business acceptance. + +## Appendix B: retries, exceptions, and recovery -Run dry-run image management commands to confirm that the backend has switched to Registry v2: +### Continue the same run after the terminal exits ```bash -ac get images -ac delete images --repo "$VERIFY_IMAGE" -ac adm prune images +export RUN_DIR="/absolute/path/to/registry-migrate-runs/original-run-id" +# Re-set the arrays you need, as described in section 5 (skip the ones you do not use): +# S3_ARGS=(--s3-endpoint http://127.0.0.1:19000) +# FORCE_ARGS=(--force) +# NESTED_ARGS=(--skip-nested-repositories) ``` -`ac delete images` and `ac adm prune images` are dry-run by default. Add `--confirm` only after reviewing the output. +Restore the original absolute `RUN_DIR`, run section 0 again, then reselect the required S3, force, nested-repository and P3 protection options; section 0 resets some arrays. `P3_PROTECTION` must match the recorded choice. Do not repeat a completed manual uninstall or capture a new post-uninstall baseline as if it were the original. -## Roll Back +### Escape hatches -If workloads cannot read business images after cutover, switch the affected context back to legacy mode: +| Switch | When to use it | Effect | +| --- | --- | --- | +| `FORCE_ARGS=(--force)` | The administrator accepts relevant legacy components still running | Bypasses only the legacy writer fence; API, UID, S3, metadata, stage and P4/P6 Operator-pause checks remain. Record the force warning and actual running state. | +| `P3_PROTECTION=(--skip-s3-backup-verification)` | The administrator accepts no established S3 recovery point | Records a warning and `not-established`; proof added later cannot change a successful P3 in this run. | +| `NESTED_ARGS=(--skip-nested-repositories)` | Affected nested repositories are approved historical test residue | Records exclusions; excluded tags are not imported. Review them individually. | + +The original 4.3 `ac` has no `set-registry-mode`. Use its explicit Registry URL, kubeconfig and context for legacy queries. The separate compatibility acceptance record covers its full permission/pagination matrix; do not treat a different client's success as proof for the original binary. + +### Rollback + +`rollback` records `rollback-requested` in the journal and prevents later stages from advancing in that run. **It does not perform a restore**: it does not reinstall the chart, recreate original UIDs, remove Registry v2 resources or restore S3 objects. Use the report and the following checklist for administrator-owned recovery: ```bash -ac config set-registry-mode legacy -ac config get-registry-mode +registry-migrate rollback --run-dir "$RUN_DIR" --kubeconfig "$KCFG" --approve +registry-migrate report --run-dir "$RUN_DIR" ``` -Keep the following artifacts for troubleshooting and reruns: - -- Image lists. -- Mirror mapping files. -- `ac image mirror` output. -- `ac image info` output. -- Generated `ImageStreamMapping` YAML files. -- `ac create -f` output. -- Migration verification output. +- **Before chart removal**: restore original replicas, platform/Pruner state and chart configuration using the change record; use the separately protected material for credential data. P1 summaries provide identity evidence, not Secret contents. +- **After chart removal**: stop new Registry writes and reconciliation first, then restore the chart/platform configuration, credentials and S3 access through the approved recovery procedure. +- **After P8**: also account for new images and token rotation. Do not reactivate the legacy S3 writer without coordinating the recovery boundary. +- **After recovery**: create a new run ID. Do not edit the old plan/journal to bypass identity drift. -After the issue is fixed, continue from the image list. You do not need to rebuild the migration scope from the beginning. +The tool is idempotent within one run: completed work is skipped or revalidated. Preserve the plan, journal, backup summaries, inventory, snapshot, evidence, verification output and separate recovery material. diff --git a/scripts/check-translation-integrity.mjs b/scripts/check-translation-integrity.mjs index aa8898871..82dc7ed11 100644 --- a/scripts/check-translation-integrity.mjs +++ b/scripts/check-translation-integrity.mjs @@ -28,8 +28,9 @@ * untracked -- i.e. the ones translate just (re)wrote * --all every file under docs/ * --only just this file (repeatable); overrides every other scope - * --since every pair this branch touches since -- the pull - * request scope, where nothing is modified in the worktree + * --since translations changed since , plus hand-translated + * pairs whose source changed; automatic translations of + * English-only edits are generated and checked on main * --docs check a docs tree outside this repo (implies a full scan) * --failures write the failing documents to , one path per line, for * translate-verified.mjs to hand back to the translator @@ -116,14 +117,9 @@ const changedTargetFiles = () => { } /** - * Target files of every pair this branch touches since . - * - * The default scope reads the working tree, which is right on main -- translate - * has just rewritten those files. On a pull request the tree is clean, so that - * scope selects nothing and the documents the branch actually changed go - * unchecked; the only thing running on a pull request was the check's own unit - * tests. A source-language edit counts too: it can leave an existing - * translation short of a section that was only just added. + * Pairs touched since , and which translations the branch changed. + * Unchanged automatic translations are regenerated on main; hand-translated + * pages and translations changed by the branch still need checking here. */ const changedSince = (ref) => { const stdout = execFileSync( @@ -137,18 +133,21 @@ const changedSince = (ref) => { ) const targets = new Set() const sources = new Set() + const changedTargets = new Set() for (const line of stdout.split('\n')) { if (!line.trim()) continue const abs = path.resolve(docsDir, line.trim()) if (!DOC_EXTENSIONS.has(path.extname(abs))) continue // Either language names the same pair; the scan below is driven by targets. const fromSource = abs.startsWith(sourceDir + path.sep) + if (!fromSource && !abs.startsWith(targetDir + path.sep)) continue const source = fromSource ? abs : path.join(sourceDir, path.relative(targetDir, abs)) const target = fromSource ? path.join(targetDir, path.relative(sourceDir, abs)) : abs sources.add(source) if (fs.existsSync(target)) targets.add(target) + if (!fromSource) changedTargets.add(target) } - return { targets: [...targets], sources } + return { targets: [...targets], sources, changedTargets } } // Computed once: the missing-translation audit below needs the source side of @@ -825,7 +824,15 @@ const selectTargets = () => { }) } if (scanAll) return walk(targetDir) - if (branchScope) return branchScope.targets + if (branchScope) { + return branchScope.targets.filter((file) => { + const sourceFile = path.join(sourceDir, path.relative(targetDir, file)) + if (branchScope.changedTargets.has(file) || !fs.existsSync(sourceFile)) return true + if (disablesAutoTranslation(fs.readFileSync(sourceFile, 'utf8'))) return true + console.log(`PENDING ${relative(file)} -- regenerated and checked from ${relative(sourceFile)} on main`) + return false + }) + } return changedTargetFiles() } const targetFiles = selectTargets().filter((file) => diff --git a/scripts/check-translation-integrity.test.mjs b/scripts/check-translation-integrity.test.mjs index ce14ce37f..f50426444 100644 --- a/scripts/check-translation-integrity.test.mjs +++ b/scripts/check-translation-integrity.test.mjs @@ -596,6 +596,69 @@ a fs.rmSync(root, { recursive: true, force: true }) } +// --------------------------------------------------------------------------- +// English-only edits are translated on main. An unchanged generated translation +// is pending regeneration, not a broken translation introduced by the branch. +// The named-file scope must still catch that mismatch after translation runs. +// --------------------------------------------------------------------------- +{ + const root = makeGitTree({ + 'en/a.mdx': `${FRONTMATTER}# T\n\nProse.\n`, + 'zh/a.mdx': `${FRONTMATTER}# T\n\n正文。\n`, + }) + fs.appendFileSync(path.join(root, 'en', 'a.mdx'), '\n## New section\n\n```bash\necho ready\n```\n') + commitAll(root, 'update English only') + const before = read(root, 'zh', 'a.mdx') + + const scoped = check(root, '--since', 'HEAD~1') + ok(scoped.status === 0, 'an English-only edit defers its unchanged automatic translation', scoped.out) + ok(scoped.out.includes('PENDING') && scoped.out.includes('zh/a.mdx'), 'the pending translation is named', scoped.out) + ok(read(root, 'zh', 'a.mdx') === before, 'branch checks leave the pending translation untouched') + + const strict = check(root, '--only', path.join(root, 'zh', 'a.mdx')) + ok(strict.status === 1, 'post-translation checks still reject the outdated translation', strict.out) + const full = check(root) + ok(full.status === 1, 'the full translation scan still rejects the outdated translation', full.out) + fs.rmSync(root, { recursive: true, force: true }) +} + +// --------------------------------------------------------------------------- +// If the branch changes both languages, it owns the new translation and must +// provide all the new English content rather than deferring the damaged pair. +// --------------------------------------------------------------------------- +{ + const root = makeGitTree({ + 'en/a.md': `${FRONTMATTER}# T\n\nProse.\n`, + 'zh/a.md': `${FRONTMATTER}# T\n\n正文。\n`, + }) + fs.appendFileSync(path.join(root, 'en', 'a.md'), '\n## New section\n\n```bash\necho ready\n```\n') + fs.appendFileSync(path.join(root, 'zh', 'a.md'), '\n更新了正文。\n') + commitAll(root, 'change both languages with an incomplete translation') + + const { status, out } = check(root, '--since', 'HEAD~1') + ok(status === 1, 'an incomplete translation changed with its source still fails the branch', out) + ok(!out.includes('PENDING'), 'a changed translation is never deferred', out) + fs.rmSync(root, { recursive: true, force: true }) +} + +// --------------------------------------------------------------------------- +// An English-only edit to a hand-translated page cannot be deferred to main: +// the automatic translator is explicitly disabled for that page. +// --------------------------------------------------------------------------- +for (const i18n of ['i18n:\n disableAutoTranslation: true\n', 'i18n: { disableAutoTranslation: true }\n']) { + const root = makeGitTree({ + 'en/a.md': `---\nid: KB1\n${i18n}---\n# T\n\nProse.\n`, + 'zh/a.md': `${FRONTMATTER}# T\n\n正文。\n`, + }) + fs.appendFileSync(path.join(root, 'en', 'a.md'), '\n## New section\n\nProse.\n') + commitAll(root, 'update a hand-translated source only') + + const { status, out } = check(root, '--since', 'HEAD~1') + ok(status === 1, 'an outdated hand translation fails an English-only branch', out) + ok(!out.includes('PENDING'), 'hand translations are never deferred', out) + fs.rmSync(root, { recursive: true, force: true }) +} + // --------------------------------------------------------------------------- // A new English page on a branch has no translation yet and is not supposed to // -- translate writes it on main. Failing the branch for that would block every