From 8f6b5e12c636a2dc0abaf7a5dbbb042896b23106 Mon Sep 17 00:00:00 2001 From: huizhang Date: Thu, 29 Oct 2026 19:10:46 +0800 Subject: [PATCH 01/12] docs: update Registry v2 migration runbook --- .../ACP_Registry_v2_Migration_Runbook.mdx | 1037 +++++------------ 1 file changed, 261 insertions(+), 776 deletions(-) diff --git a/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx b/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx index af843fb0a..01e1f9a39 100644 --- a/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx +++ b/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx @@ -9,886 +9,371 @@ ProductsVersion: # ACP Registry v2 Migration Runbook -Migration from the legacy ACP Registry is required when an existing ACP environment moves image workloads from the legacy Registry to **Registry v2** in `image-registry-system`. Registry v2 means the Registry deployed and managed by `cluster-image-registry-operator`. - -The migration workflow is based on the validated ACP `ac` runbook. No dedicated Registry migration subcommand is required. Use the following `ac` commands: - -- `ac get images` to export the legacy image list. -- `ac registry login` to write OCI registry credentials. -- `ac image mirror` to copy image blobs when the source and target registries do not share storage. -- `ac image info` to read image digests and media types from the registry selected for metadata inspection. -- `ac create -f` to create `ImageStreamMapping` resources and backfill Registry v2 Image and ImageStream metadata. - -During migration, use a maintenance window when possible. At minimum, pause writes to the legacy Registry. Reads can remain available. If new images are pushed during migration, repeat image copy and metadata backfill before the final cutover. - -Metadata backfill writes Registry v2 Image API objects only. It does not copy or repair registry storage data such as manifests, config blobs, layers, repository revision links, or tag links. Do not create `ImageStreamMapping` objects until Registry v2 can access the corresponding image data through storage reuse or a completed image copy. If legacy image data is intentionally not preserved, do not run metadata backfill for those images; push the images again to Registry v2 instead. - -## Migration Scenarios - -| Scenario | Use when | Blob migration | Metadata migration | -| --- | --- | --- | --- | -| Storage reuse | The legacy Registry and Registry v2 run in the same ACP environment and business cluster, and Registry v2 reuses the legacy Registry storage. | Not required. Registry v2 is attached to the existing blob storage. | Required. Read digest and media type from the legacy Registry, then backfill `ImageStream` and `ImageStreamMapping` objects. | -| Cross-registry sync | The legacy Registry and Registry v2 use different storage, different clusters, or different ACP environments. | Required. Use `ac image mirror` to copy images from the legacy Registry to the target Registry. | Required after image copy. Read digest and media type from Registry v2, then backfill `ImageStream` and `ImageStreamMapping` objects. | - -## Required Inputs - -Prepare the following inputs before running the migration: - -| Variable | Description | Example | -| --- | --- | --- | -| `MIGRATION_KUBECONFIG` | Optional kubeconfig used only for migration. | `/tmp/registry-migration/kubeconfig` | -| `ACP_PLATFORM_URL` | ACP platform login URL. | `https://acp.example.com` | -| `ACP_SESSION_NAME` | ACP login session name for migration. | `registry-migration` | -| `ACP_USERNAME` | ACP account used for migration. | `admin` | -| `ACP_PASSWORD` | ACP password. Store it only in a private environment file. | `******` | -| `ACP_CLUSTER` | Business cluster selected after login. | `business-1` | -| `ACP_IDP` | Optional identity provider name required by the login entry. | `ldap-test` | -| `ACP_AUTH_TYPE` | Optional authentication type required by the login entry. | `ldap` | -| `LEGACY_REGISTRY_URL` | Legacy Registry URL used by `ac get images --registry-url`. Include `http://` or `https://`. | `http://old-registry.example.com:5000` | -| `LEGACY_REGISTRY_HOST` | Legacy Registry external host used by `ac image mirror`. Do not include a scheme. | `old-registry.example.com:5000` | -| `TARGET_REGISTRY_HOST` | Registry v2 external host used by mirror, image inspect, metadata backfill, and verification. Do not include a scheme. | `image-registry.example.com` | -| `IMAGE_LIST_FILE` | Image list to migrate. Each line is `namespace/name:tag`. | `/tmp/registry-migration/image-list.txt` | -| `IMAGE_DIGEST_FILE` | Optional image and digest list. Each line is `namespace/name:tag sha256:...`. | `/tmp/registry-migration/image-digests.txt` | -| `MAPPING_FILE` | Mirror mapping file for cross-registry sync. Each line is `source=target`. | `/tmp/registry-migration/mirror-map.txt` | -| `METADATA_DIR` | Output directory for `ac image info`, `ImageStreamMapping`, and apply results. | `/tmp/registry-migration/metadata` | -| `METADATA_INSPECT_REGISTRY_HOST` | Registry host used by metadata backfill when inspecting digest and media type. Use the legacy Registry for storage reuse and Registry v2 for cross-registry sync. | `old-registry.example.com:5000` | -| `REGISTRY_INSECURE_FLAG` | Set to `--insecure` when the registry uses HTTP, a self-signed certificate, or a test certificate. Set to an empty string for trusted HTTPS. | `--insecure` | -| `REGISTRY_AUTH_DIR` | OCI registry auth directory. | `/tmp/registry-migration/registry-auth` | -| `REGISTRY_AUTH_FILE` | OCI registry auth file written by `ac registry login --to`. | `/tmp/registry-migration/registry-auth/config.json` | - -Create a migration workspace: - -```bash -export MIGRATION_DIR=/tmp/registry-migration -mkdir -p "$MIGRATION_DIR" - -export MIGRATION_KUBECONFIG="$MIGRATION_DIR/kubeconfig" -export IMAGE_LIST_FILE="$MIGRATION_DIR/image-list.txt" -export IMAGE_DIGEST_FILE="$MIGRATION_DIR/image-digests.txt" -export MAPPING_FILE="$MIGRATION_DIR/mirror-map.txt" -export METADATA_DIR="$MIGRATION_DIR/metadata" -export REGISTRY_AUTH_DIR="$MIGRATION_DIR/registry-auth" -export REGISTRY_AUTH_FILE="$REGISTRY_AUTH_DIR/config.json" -export REGISTRY_INSECURE_FLAG="${REGISTRY_INSECURE_FLAG:---insecure}" - -mkdir -p "$REGISTRY_AUTH_DIR" "$METADATA_DIR" +This runbook upgrades one ACP business cluster from the legacy chart Registry to the Registry managed by `cluster-image-registry-operator`. The migration is performed by the independent `registry-migrate` tool; it is not an automatic takeover feature of the Operator. + +Before the maintenance window, contact your ACP platform service provider and obtain the provider-approved `registry-migrate` binary, compatible `ac`, the Registry v2 Operator bundle/CRD, the ACP pause/hold procedure, and the S3 backup/restore procedure. This runbook was checked against `registry-migrate` v4.5.0 (commit `6edeb747`). The migration tool version is independent of the ACP release. Do not build binaries during the window. + +This procedure covers one ACP cluster, one legacy Registry, the same business cluster, legacy S3 storage, and a Gateway Ingress as the external entry point. Registry v2 reuses the same S3 bucket, endpoint, and prefix. A legacy `image-registry` Service of type `NodePort` is acceptable when clients use the Gateway Ingress address. + +PVC/filesystem migration, cross-cluster mirroring, `ac image mirror`, manual `ImageStreamMapping` backfill, and automatic Operator takeover are outside this runbook. Migrate one cluster at a time. P2 starts the maintenance window; P7 restores read-only service; P8 opens writes. + +## 0. Prepare the terminal and tools + +Use a dedicated Bash terminal. `cpaas-system` and `image-registry-system` are common ACP namespaces; confirm them. + +```bash +set -e -o pipefail +umask 077 +export KCFG="/path/to/business.kubeconfig" +printf 'Enter the real platform administrator account: ' +IFS= read -r ACTOR +test -n "$ACTOR" +export REGISTRY_NS="cpaas-system" +export OPERATOR_NS="image-registry-system" +CTX="$(ac --kubeconfig "$KCFG" config current-context)" +test -n "$CTX" +OUT="$PWD/registry-migrate-runs" + +if test -n "$RUN_DIR"; then + test -f "$RUN_DIR/plan.json" + RUN_ID="$(jq -er '.runID' "$RUN_DIR/plan.json")" + REGISTRY_NS="$(jq -er '.scope.operandNamespace' "$RUN_DIR/plan.json")" + OPERATOR_NS="$(jq -er '.scope.operatorNamespace' "$RUN_DIR/plan.json")" + REGISTRY_DEPLOY="$(jq -r '.discovery.deployment // empty' "$RUN_DIR/plan.json")" + GATEWAY_DEPLOY="$(jq -r '.discovery.gatewayDeployment // empty' "$RUN_DIR/plan.json")" + OUT="$(dirname "$RUN_DIR")" +else + RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)-business-$RANDOM" + RUN_DIR="$OUT/$RUN_ID" +fi +mkdir -p "$OUT" +S3_ARGS=() +FORCE_ARGS=() +registry-migrate version +ac version --client +printf 'context=%s\nrun-dir=%s\n' "$CTX" "$RUN_DIR" + +acn() { ac --kubeconfig "$KCFG" --context "$CTX" "$@"; } +acns() { acn -n "$REGISTRY_NS" "$@"; } +evidence() { + jq --arg actor "$ACTOR" --arg at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --arg field "$2" \ + '{version:"v1",runID:.runID,planDigest:.digest,observedAt:$at,actor:$actor,($field):true}' \ + "$RUN_DIR/plan.json" > "$RUN_DIR/$1" +} +stop_deployment() { + local namespace="$1" name="$2" selector + acn -n "$namespace" get deployment "$name" -o wide + selector="$(acn -n "$namespace" get deployment "$name" -o json | + jq -er '.spec.selector.matchLabels | to_entries | map("\(.key)=\(.value)") | join(",") | select(length > 0)')" + acn -n "$namespace" scale deployment "$name" --replicas=0 + pods="$(acn -n "$namespace" get pods -l "$selector" -o name)" + if test -n "$pods"; then + acn -n "$namespace" wait --for=delete pod -l "$selector" --timeout=180s + fi +} ``` -If the legacy Registry uses HTTP, derive `LEGACY_REGISTRY_URL` from `LEGACY_REGISTRY_HOST`. If it uses HTTPS, set the full URL explicitly: +For a new migration, unset `RUN_DIR` before this section. To recover after the terminal exits, export the original absolute `RUN_DIR` and run this section again; it restores the run ID, namespaces, and Deployment names from `plan.json`. Do not create a new run for a retry. Re-select S3 and force arrays after reload. -```bash -export LEGACY_REGISTRY_URL="${LEGACY_REGISTRY_URL:-http://$LEGACY_REGISTRY_HOST}" -``` - -`REGISTRY_AUTH_DIR` stores an OCI registry auth file. Setting this variable does not start a local container runtime. -If both source and target registries use trusted HTTPS certificates, disable the insecure flag: +## 1. P0: create and review the plan ```bash -export REGISTRY_INSECURE_FLAG="" +registry-migrate plan --output-dir "$OUT" --run-id "$RUN_ID" \ + --operand-namespace "$REGISTRY_NS" --operator-namespace "$OPERATOR_NS" \ + --kubeconfig "$KCFG" +jq -e '(.discovery.blockers // []) | length == 0' "$RUN_DIR/plan.json" +REGISTRY_DEPLOY="$(jq -er '.discovery.deployment | select(length > 0)' "$RUN_DIR/plan.json")" +GATEWAY_DEPLOY="$(jq -er '.discovery.gatewayDeployment | select(length > 0)' "$RUN_DIR/plan.json")" +jq '{scope,storage:.discovery.storage,routes:.discovery.routes,limits:.namespaceLimits,warnings:.discovery.warnings}' "$RUN_DIR/plan.json" ``` -## Log In and Check Registry Mode +The plan is read-only. Review every blocker, S3 bucket/endpoint/prefix, old host/TLS/IngressClass, replicas, scheduling fields, and timeout values. Stop if any blocker is present or the plan does not match the cluster. -Log in to ACP and create a migration-only kubeconfig: - -```bash -LOGIN_ARGS=( - ac login "$ACP_PLATFORM_URL" - --name "$ACP_SESSION_NAME" - --username "$ACP_USERNAME" - --password "$ACP_PASSWORD" - --kubeconfig "$MIGRATION_KUBECONFIG" -) +## 2. P1-backup: save recovery material -[ -z "${ACP_IDP:-}" ] || LOGIN_ARGS+=(--idp "$ACP_IDP") -[ -z "${ACP_AUTH_TYPE:-}" ] || LOGIN_ARGS+=(--auth-type "$ACP_AUTH_TYPE") - -"${LOGIN_ARGS[@]}" - -export KUBECONFIG="$MIGRATION_KUBECONFIG" -[ -z "${ACP_CLUSTER:-}" ] || ac config use-cluster "$ACP_CLUSTER" -``` - -If the cluster has both legacy and Registry v2 backends, `auto` mode can require an explicit selection. Start migration in legacy mode and confirm that the legacy Registry can be discovered: +Save old chart values, platform management objects, protected credentials, and the provider’s S3 recovery record. The tool backup is a redacted Registry-scoped snapshot; it does not contain Secret data or restore S3 image data by itself. ```bash -ac config set-registry-mode legacy -ac config get-registry-mode -ac registry info -ac registry info --internal +registry-migrate backup --run-dir "$RUN_DIR" --kubeconfig "$KCFG" --approve ``` -`ac registry info` shows the Registry discovered for the current mode. It might return an in-cluster Service such as `image-registry.cpaas-system`, which is not directly reachable from an administrator workstation. Use `LEGACY_REGISTRY_URL` for image discovery and use `LEGACY_REGISTRY_HOST` and `TARGET_REGISTRY_HOST` for image copy. +Expected result: a recorded stage and relevant resource/Secret summaries containing UID, owner, ServiceAccount, and reference information. This is not a backup of every object in the namespace. -## Prepare the Image List +## 3. P1-credentials: hand off pull Secrets -Export the legacy image list visible to the current ACP user: +The legacy Registry must still be readable. First pause the AppRelease/ClusterPlugin or other platform source that manages `sa-imagepullsecret-controller`. ```bash -ac config set-registry-mode legacy -ac get images \ - --registry-url "$LEGACY_REGISTRY_URL" \ - > "$MIGRATION_DIR/legacy-images.txt" +PULL_SECRET_REF="$(acn get deployments -A -o json | jq -er ' + [.items[] | select((.metadata.name | test("sa-imagepullsecret-controller";"i")) or + ((.metadata.labels // {}) | tojson | test("sa-imagepullsecret-controller";"i"))) | + [.metadata.namespace,.metadata.name]] | + if length == 1 then .[0] | @tsv else error("review legacy pull-secret controller candidates") end')" +IFS=$'\t' read -r PULL_SECRET_NS PULL_SECRET_DEPLOY <<< "$PULL_SECRET_REF" +stop_deployment "$PULL_SECRET_NS" "$PULL_SECRET_DEPLOY" +evidence controller-stopped.json legacyControllerStopped +registry-migrate run --run-dir "$RUN_DIR" --stage P1-credentials \ + --evidence-file "$RUN_DIR/controller-stopped.json" \ + "${FORCE_ARGS[@]}" --kubeconfig "$KCFG" --approve ``` -Generate the image list and the optional digest list from the `ac get images` table output: +Expected result: Secret totals, progress, `updated/skipped`, and a recorded stage. Existing Secret names/UIDs and ServiceAccount owner UIDs remain unchanged. Do not recreate same-named Secrets. Do not uninstall the old chart until this phase succeeds. -```bash -awk 'NR > 1 && NF >= 2 { print $1 ":" $2 }' \ - "$MIGRATION_DIR/legacy-images.txt" \ - | sort -u > "$IMAGE_LIST_FILE" +## 4. P2-stop-writers: stop legacy writers -awk 'NR > 1 && NF >= 4 && $4 != "unknown" { print $1 ":" $2 " " $4 }' \ - "$MIGRATION_DIR/legacy-images.txt" \ - | sort -u > "$IMAGE_DIGEST_FILE" -``` +Pause the legacy chart management source. Stop the legacy Registry and Gateway Deployments, pause the legacy Pruner when present, and stop external jobs writing the same S3 bucket/prefix. An old Image API or another metadata writer must be stopped and recorded by an administrator. -You can also write the list manually when only selected images should be migrated: +Before running the command, set `WRITER_RECORD` to the maintenance/change record that lists the external S3 and metadata writers, the stop time, and the administrator approval. The file is copied into the run directory for audit. ```bash -cat >> "$IMAGE_LIST_FILE" <> "$IMAGE_DIGEST_FILE" < -EOF +export WRITER_RECORD="/path/to/maintenance-writer-record.txt" +stop_deployment "$REGISTRY_NS" "$REGISTRY_DEPLOY" +stop_deployment "$REGISTRY_NS" "$GATEWAY_DEPLOY" +LEGACY_PRUNER="$(acns get cronjob image-pruner --ignore-not-found -o name)" +if test -n "$LEGACY_PRUNER"; then + acns patch cronjob image-pruner --type=merge -p '{"spec":{"suspend":true}}' +fi +acns get endpoints image-registry -o wide +acns get endpointslices.discovery.k8s.io -l kubernetes.io/service-name=image-registry -o wide +test -n "$WRITER_RECORD" +test -s "$WRITER_RECORD" +if test "$WRITER_RECORD" != "$RUN_DIR/writer-record.txt"; then + cp "$WRITER_RECORD" "$RUN_DIR/writer-record.txt" +fi +evidence writers-stopped.json legacyWritersStopped +registry-migrate run --run-dir "$RUN_DIR" --stage P2-stop-writers \ + --evidence-file "$RUN_DIR/writers-stopped.json" \ + "${FORCE_ARGS[@]}" --kubeconfig "$KCFG" --approve ``` -Review the generated list before continuing: +Expected result: old Registry/Gateway Pods and Ready endpoints are gone, the old Pruner is suspended, active jobs have finished, and the stage is recorded. If the platform cannot keep old components stopped, set `FORCE_ARGS=(--force)` only after an administrator accepts the consistency risk. Force does not bypass API, UID, S3, metadata, stage, or new Operator pause checks. -```bash -sed -n '1,20p' "$IMAGE_LIST_FILE" -sed -n '1,20p' "$IMAGE_DIGEST_FILE" -wc -l "$IMAGE_LIST_FILE" -``` +## 5. P3-final-inventory: freeze the inventory and S3 state -Remove temporary test images, retired namespaces, and incorrect tags from the list. +The final inventory records repository/tag/digest relationships and manifest/config metadata. The S3 snapshot records object count, bytes, and listing digest. It does not copy layers or provide data recovery. -## Write Registry Credentials +Choose exactly one path. -Write credentials for the legacy and target registries. `ac image mirror` and `ac image info` read this file: +**Verified recovery point:** ```bash -export DOCKER_CONFIG="$REGISTRY_AUTH_DIR" - -ac registry login \ - --registry "$LEGACY_REGISTRY_HOST" \ - --skip-check \ - --to "$REGISTRY_AUTH_FILE" \ - $REGISTRY_INSECURE_FLAG - -ac registry login \ - --registry "$TARGET_REGISTRY_HOST" \ - --skip-check \ - --to "$REGISTRY_AUTH_FILE" \ - $REGISTRY_INSECURE_FLAG +printf 'Enter the verified S3 recovery-point identifier or report path: ' +IFS= read -r RESTORE_REFERENCE +test -n "$RESTORE_REFERENCE" +jq --arg actor "$ACTOR" --arg at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ + --arg ref "$RESTORE_REFERENCE" \ + '{version:"v1",runID:.runID,planDigest:.digest,observedAt:$at,actor:$actor,restoreVerified:true,restoreReference:$ref}' \ + "$RUN_DIR/plan.json" > "$RUN_DIR/restore-proof.json" +P3_PROTECTION=(--evidence-file "$RUN_DIR/restore-proof.json") ``` -If the source and target registries belong to different ACP environments, write both credentials into the same auth file with the corresponding kubeconfig: +**Explicitly skipped recovery verification:** an administrator may accept a run with no S3 recovery point. The tool records a warning and `not-established` state. ```bash -KUBECONFIG="$SOURCE_KUBECONFIG" \ - ac registry login --registry "$LEGACY_REGISTRY_HOST" --skip-check --to "$REGISTRY_AUTH_FILE" $REGISTRY_INSECURE_FLAG - -KUBECONFIG="$TARGET_KUBECONFIG" \ - ac registry login --registry "$TARGET_REGISTRY_HOST" --skip-check --to "$REGISTRY_AUTH_FILE" $REGISTRY_INSECURE_FLAG +P3_PROTECTION=(--skip-s3-backup-verification) ``` -## Prepare Metadata Backfill - -Registry v2 needs Image API metadata in addition to image blobs. Use `ImageStreamMapping` to backfill `Image`, `ImageStream`, and tag metadata. - -Define the metadata backfill functions in the migration shell session: +Do not use both paths. ```bash -ensure_imagestream() { - namespace="$1" - stream="$2" - stream_file="$METADATA_DIR/${namespace}-${stream}.imagestream.yaml" - stream_output_file="$METADATA_DIR/${namespace}-${stream}.imagestream.apply.txt" - - if ac get imagestreams.image.alauda.io "$stream" -n "$namespace" >/dev/null 2>&1; then - return 0 - fi - - cat > "$stream_file" < "$stream_output_file" 2>&1; then - return 0 - fi - - echo "[metadata] failed to create ImageStream $namespace/$stream; output=$stream_output_file" >&2 - sed -n '1,20p' "$stream_output_file" >&2 - return 1 -} - -write_imagestream_mapping() { - image="$1" - digest="${2:-}" - namespace="${image%%/*}" - rest="${image#*/}" - stream="${rest%%:*}" - tag="${rest##*:}" - inspect_registry_host="${METADATA_INSPECT_REGISTRY_HOST:-$TARGET_REGISTRY_HOST}" - source_ref="$inspect_registry_host/$image" - repo_ref="$TARGET_REGISTRY_HOST/$namespace/$stream" - inspect_ref="$source_ref" - name_prefix="${namespace}-${stream}-${tag}" - info_file="$METADATA_DIR/${name_prefix}.info.txt" - mapping_file="$METADATA_DIR/${name_prefix}.mapping.yaml" - apply_output_file="$METADATA_DIR/${name_prefix}.apply.txt" - BACKFILL_LAST_ACTION="" - - if [ -n "$digest" ]; then - inspect_ref="$inspect_registry_host/$namespace/$stream@$digest" - fi - - if ! ac image info "$inspect_ref" \ - --registry-config "$REGISTRY_AUTH_FILE" \ - $REGISTRY_INSECURE_FLAG > "$info_file" 2>&1; then - BACKFILL_LAST_ACTION="failed" - echo "[metadata] failed to inspect $inspect_ref; output=$info_file" >&2 - sed -n '1,20p' "$info_file" >&2 - return 1 - fi - - [ -n "$digest" ] || digest="$(awk '/^Digest:/ { sub(/^Digest:[[:space:]]*/, ""); print; exit }' "$info_file")" - media_type="$(awk '/^Media Type:/ { sub(/^Media Type:[[:space:]]*/, ""); print; exit }' "$info_file")" +registry-migrate resume --run-dir "$RUN_DIR" --stage P3-final-inventory \ + "${P3_PROTECTION[@]}" "${S3_ARGS[@]}" \ + "${FORCE_ARGS[@]}" --kubeconfig "$KCFG" --approve +registry-migrate report --run-dir "$RUN_DIR" +jq '{items:(.items|length),digest,sample:(.items[0] | {namespace,repository,tag,digest})}' "$RUN_DIR/final-inventory.json" +``` - if [ -z "$digest" ]; then - echo "failed to read digest from $source_ref" >&2 - return 1 - fi - if [ -z "$media_type" ]; then - echo "failed to read media type from $source_ref" >&2 - return 1 - fi +Expected result: recorded stage, `final-inventory.json`, and `s3-snapshot.json`. Scanning reads tag links, manifests, and configs; it does not move layers. Approved test residue excluded with `--skip-nested-repositories` must be reviewed in `legacy-metadata-report.json` and not counted as migrated. - ensure_imagestream "$namespace" "$stream" || return 1 - current_line="$( - ac get imagestreamtags "$stream:$tag" -n "$namespace" 2>/dev/null \ - | awk 'NR > 1 && $3 == "current" { print $4 " " $6; exit }' - )" - current_digest="${current_line%% *}" - current_reference="${current_line#* }" - expected_reference="$repo_ref@$digest" +## 6. P4-prepare-operator: prepare Registry v2 read-only - if [ "$current_digest" = "$digest" ] && [ "$current_reference" = "$expected_reference" ]; then - BACKFILL_LAST_ACTION="skipped" - return 0 - fi +Install or upgrade the provider-approved Operator bundle/CRD. Establish the ACP-specific pause/hold that keeps the current Operator Deployment at zero during P4–P6. A Manual Subscription controls InstallPlan approval; it does not pause an already running Operator. - cat > "$mapping_file" < "$apply_output_file" 2>&1; then - BACKFILL_LAST_ACTION="applied" - return 0 +```bash +acn -n "$OPERATOR_NS" get subscription,installplan,csv -o wide +stop_deployment "$OPERATOR_NS" cluster-image-registry-operator +PRUNER="$(acn get imagepruner.imageregistry.operator.alauda.io cluster --ignore-not-found -o name)" +if test -n "$PRUNER"; then + acn get "$PRUNER" -o json > "$RUN_DIR/imagepruner-before.json" + jq '{annotations:.metadata.annotations,suspend:.spec.suspend}' "$RUN_DIR/imagepruner-before.json" + existing_run="$(jq -r '.metadata.annotations["registry-migrate.run-id"] // empty' "$RUN_DIR/imagepruner-before.json")" + existing_plan="$(jq -r '.metadata.annotations["registry-migrate.plan-digest"] // empty' "$RUN_DIR/imagepruner-before.json")" + current_run="$(jq -er '.runID' "$RUN_DIR/plan.json")" + current_plan="$(jq -er '.digest' "$RUN_DIR/plan.json")" + test -z "$existing_run" -o "$existing_run" = "$current_run" + test -z "$existing_plan" -o "$existing_plan" = "$current_plan" + if test -z "$existing_run"; then + acn annotate "$PRUNER" "registry-migrate.run-id=$current_run" fi - - BACKFILL_LAST_ACTION="failed" - echo "[metadata] failed $image; output=$apply_output_file" >&2 - sed -n '1,20p' "$apply_output_file" >&2 - return 1 -} - -backfill_metadata() { - total=0 - success=0 - applied=0 - skipped=0 - failed=0 - progress_interval="${METADATA_PROGRESS_INTERVAL:-100}" - verbose="${METADATA_VERBOSE:-false}" - - if [ -n "${IMAGE_DIGEST_FILE:-}" ] && [ -s "$IMAGE_DIGEST_FILE" ]; then - echo "[metadata] start backfill source=$IMAGE_DIGEST_FILE output_dir=$METADATA_DIR progress_interval=$progress_interval" - while read -r image digest; do - [ -n "$image" ] || continue - total=$((total + 1)) - if write_imagestream_mapping "$image" "$digest"; then - success=$((success + 1)) - [ "$BACKFILL_LAST_ACTION" = "applied" ] && applied=$((applied + 1)) - [ "$BACKFILL_LAST_ACTION" = "skipped" ] && skipped=$((skipped + 1)) - else - failed=$((failed + 1)) - fi - if [ "$verbose" = "true" ]; then - echo "[metadata] [$total] $image action=${BACKFILL_LAST_ACTION:-unknown}" - elif [ "$progress_interval" -gt 0 ] && [ $((total % progress_interval)) -eq 0 ]; then - echo "[metadata] progress: total=$total success=$success applied=$applied skipped=$skipped failed=$failed" - fi - done < "$IMAGE_DIGEST_FILE" - elif [ -n "${IMAGE_LIST_FILE:-}" ] && [ -s "$IMAGE_LIST_FILE" ]; then - echo "[metadata] start backfill source=$IMAGE_LIST_FILE output_dir=$METADATA_DIR progress_interval=$progress_interval" - while read -r image; do - [ -n "$image" ] || continue - total=$((total + 1)) - if write_imagestream_mapping "$image"; then - success=$((success + 1)) - [ "$BACKFILL_LAST_ACTION" = "applied" ] && applied=$((applied + 1)) - [ "$BACKFILL_LAST_ACTION" = "skipped" ] && skipped=$((skipped + 1)) - else - failed=$((failed + 1)) - fi - if [ "$verbose" = "true" ]; then - echo "[metadata] [$total] $image action=${BACKFILL_LAST_ACTION:-unknown}" - elif [ "$progress_interval" -gt 0 ] && [ $((total % progress_interval)) -eq 0 ]; then - echo "[metadata] progress: total=$total success=$success applied=$applied skipped=$skipped failed=$failed" - fi - done < "$IMAGE_LIST_FILE" - else - echo "[metadata] no image list found; check IMAGE_DIGEST_FILE or IMAGE_LIST_FILE" >&2 - return 1 + if test -z "$existing_plan"; then + acn annotate "$PRUNER" "registry-migrate.plan-digest=$current_plan" fi - - echo "[metadata] summary: total=$total success=$success applied=$applied skipped=$skipped failed=$failed output_dir=$METADATA_DIR" - [ "$failed" -eq 0 ] -} -``` - -Confirm that the function exists in the current shell: - -```bash -if type backfill_metadata >/dev/null 2>&1; then - echo "backfill_metadata loaded" fi +evidence operator-paused.json operatorPaused +registry-migrate run --run-dir "$RUN_DIR" --stage P4-prepare-operator \ + --evidence-file "$RUN_DIR/operator-paused.json" \ + "${FORCE_ARGS[@]}" --kubeconfig "$KCFG" --approve +acn get config.imageregistry.operator.alauda.io cluster -o json | + jq -e '.spec.readOnly == true and .spec.imagePullSecret.managementState == "Unmanaged"' +acn get imagepruner.imageregistry.operator.alauda.io cluster -o json | + jq -e '.spec.suspend == true' ``` -For large migrations, the function prints progress every 100 images by default. Adjust the interval if needed: - -```bash -export METADATA_PROGRESS_INTERVAL=500 -``` +Expected result: `state=prepared`; Config reuses planned S3/routes, is read-only, and leaves pull-Secret management Unmanaged. The Pruner is suspended. If it did not exist, the tool creates it and records `registry-migrate.original-suspend=false`. An annotation conflict is a stop condition; do not use `--overwrite`. -Set `METADATA_VERBOSE=true` only when troubleshooting a small set of images. +## 7. P5-remove-legacy-chart: uninstall the old chart -## Scenario A: Reuse Legacy Storage - -Use this path when the legacy Registry and Registry v2 run in the same ACP environment and business cluster, and Registry v2 can be configured to access the same blob storage that the legacy Registry uses. - -This path does not copy image blobs. However, do not assume that Registry v2 can read old blobs by tag before metadata exists. The correct sequence is: - -1. Pause writes to the legacy Registry. -2. Attach Registry v2 to the legacy blob storage. -3. Read digest and media type from the legacy Registry. -4. Create `ImageStream` and `ImageStreamMapping` objects in Registry v2. -5. Verify reads through Registry v2. - -Do not run prune or Registry GC against either Registry until migration has been accepted. - -Set the Registry namespaces: +The platform administrator uninstalls the legacy Registry/Gateway chart through the actual AppRelease/ClusterPlugin process. The tool does not perform this uninstall. Keep business pull Secrets, ServiceAccounts, S3 data, and TLS Secrets. ```bash -export LEGACY_REGISTRY_NAMESPACE="${LEGACY_REGISTRY_NAMESPACE:-cpaas-system}" -export MODERN_REGISTRY_NAMESPACE="${MODERN_REGISTRY_NAMESPACE:-image-registry-system}" -``` - -### Filesystem or PVC Storage Reuse - -For filesystem storage such as NFS or CephFS, make the Registry v2 PVC point to the same legacy Registry data. The exact PV/PVC manifest depends on the storage backend. The example below applies only when the legacy Registry uses an NFS CSI PV. - -Locate the legacy Registry PVC and PV: - -```bash -export LEGACY_REGISTRY_PVC="${LEGACY_REGISTRY_PVC:-image-registry}" -export MODERN_REGISTRY_PVC="${MODERN_REGISTRY_PVC:-image-registry-storage}" -export REUSE_PV_NAME="${REUSE_PV_NAME:-image-registry-legacy-reuse}" - -LEGACY_PV_NAME="$( - ac get pvc "$LEGACY_REGISTRY_PVC" -n "$LEGACY_REGISTRY_NAMESPACE" \ - -o jsonpath='{.spec.volumeName}' -)" -test -n "$LEGACY_PV_NAME" - -ac get pvc "$LEGACY_REGISTRY_PVC" -n "$LEGACY_REGISTRY_NAMESPACE" -ac get pv "$LEGACY_PV_NAME" -``` - -Confirm that the legacy PV uses the NFS CSI driver and collect the fields needed to reuse it: - -```bash -LEGACY_STORAGE_DRIVER="$( - ac get pv "$LEGACY_PV_NAME" -o jsonpath='{.spec.csi.driver}' -)" -test "$LEGACY_STORAGE_DRIVER" = "nfs.csi.k8s.io" - -LEGACY_NFS_SERVER="$( - ac get pv "$LEGACY_PV_NAME" -o jsonpath='{.spec.csi.volumeAttributes.server}' -)" -LEGACY_NFS_SHARE="$( - ac get pv "$LEGACY_PV_NAME" -o jsonpath='{.spec.csi.volumeAttributes.share}' -)" -LEGACY_NFS_SUBDIR="$( - ac get pv "$LEGACY_PV_NAME" -o jsonpath='{.spec.csi.volumeAttributes.subdir}' -)" -LEGACY_NFS_VOLUME_HANDLE="$( - ac get pv "$LEGACY_PV_NAME" -o jsonpath='{.spec.csi.volumeHandle}' -)" -LEGACY_STORAGE_SIZE="$( - ac get pv "$LEGACY_PV_NAME" -o jsonpath='{.spec.capacity.storage}' -)" - -test -n "$LEGACY_NFS_SERVER" -test -n "$LEGACY_NFS_SHARE" -test -n "$LEGACY_NFS_SUBDIR" -test -n "$LEGACY_NFS_VOLUME_HANDLE" -test -n "$LEGACY_STORAGE_SIZE" -``` - -If the storage backend is not NFS CSI, do not reuse this manifest. Ask the storage administrator to provide a PV/PVC that mounts the same legacy data and keep the PVC name aligned with `Config/cluster.spec.storage.pvc.claim`. - -Before replacing Registry v2 storage, stop the Registry v2 runtime and wait for the data-plane deployments to be removed: - -```bash -ac patch config.imageregistry.operator.alauda.io cluster \ - --type merge \ - -p '{"spec":{"managementState":"Removed"}}' - -while ac get deployment image-registry -n "$MODERN_REGISTRY_NAMESPACE" >/dev/null 2>&1; do - sleep 5 -done - -while ac get deployment image-api-server -n "$MODERN_REGISTRY_NAMESPACE" >/dev/null 2>&1; do - sleep 5 -done -``` - -Create a static PV/PVC that points to the legacy data: - -```bash -cat > "$MIGRATION_DIR/modern-registry-reuse-legacy-storage.yaml" </dev/null 2>&1; do - sleep 5 -done - -while ! ac get deployment image-api-server -n "$MODERN_REGISTRY_NAMESPACE" >/dev/null 2>&1; do - sleep 5 +jq -r '.discovery.routes[] | select(.gateway == true) | [.name,.ingressUID] | @tsv' "$RUN_DIR/plan.json" | +while IFS=$'\t' read -r name expected_uid; do + actual_uid="$(acns get ingress "$name" --ignore-not-found -o jsonpath='{.metadata.uid}')" + if test -n "$actual_uid"; then + test "$actual_uid" = "$expected_uid" + acns delete ingress "$name" --wait=true + fi done - -ac rollout status deployment/image-registry -n "$MODERN_REGISTRY_NAMESPACE" --timeout=300s -ac rollout status deployment/image-api-server -n "$MODERN_REGISTRY_NAMESPACE" --timeout=300s +registry-migrate run --run-dir "$RUN_DIR" --stage P5-remove-legacy-chart \ + --kubeconfig "$KCFG" --approve ``` -### Object Storage Reuse - -For object storage such as S3, OSS, GCS, Azure, Swift, or IBM COS, configure Registry v2 to use the same bucket or container, endpoint, region, path prefix, credentials, CA, and access parameters that the legacy Registry used. +Expected result: recorded stage. The tool confirms old Service/Deployments/routes are gone, TLS Secret remains, and business Secret/ServiceAccount identities have not changed. New Ingress uses planned `targetName` values; the old Ingress name is only a deletion target. -If the legacy Registry uses a path prefix and Registry v2 cannot configure the same prefix, use Scenario B instead. Also use Scenario B if the Registry v2 Operator does not support the legacy object storage type or a required access parameter. +## 8. P6-cold-import: import metadata and limits -If Registry v2 already uses another storage backend, stop the runtime before changing object storage: +The new Operator must still be paused. Re-check the pause and generate a fresh evidence file before importing. ```bash -ac patch config.imageregistry.operator.alauda.io cluster \ - --type merge \ - -p '{"spec":{"managementState":"Removed"}}' - -while ac get deployment image-registry -n "$MODERN_REGISTRY_NAMESPACE" >/dev/null 2>&1; do - sleep 5 -done - -while ac get deployment image-api-server -n "$MODERN_REGISTRY_NAMESPACE" >/dev/null 2>&1; do - sleep 5 -done -``` - -Confirm the supported storage fields in the current Operator version: - -```bash -ac explain configs.imageregistry.operator.alauda.io.spec.storage --recursive +stop_deployment "$OPERATOR_NS" cluster-image-registry-operator +evidence operator-paused.json operatorPaused +registry-migrate run --run-dir "$RUN_DIR" --stage P6-cold-import \ + --inventory-file "$RUN_DIR/final-inventory.json" \ + --evidence-file "$RUN_DIR/operator-paused.json" \ + "${FORCE_ARGS[@]}" --kubeconfig "$KCFG" --approve ``` -Then inspect the legacy Registry configuration and secrets: +Expected result: `state=imported`, with progress for unique ImageDigest objects, ImageStream/tag relationships, and namespace limits. Shared digests are stored once while repository/tag references remain. No blobs or layers are uploaded and no ordinary `ImageStreamMapping` bypass is used. -```bash -ac get deployment image-registry -n "$LEGACY_REGISTRY_NAMESPACE" -o yaml -ac get configmap image-registry-config -n "$LEGACY_REGISTRY_NAMESPACE" -o yaml -ac get secret -n "$LEGACY_REGISTRY_NAMESPACE" -``` +The legacy Gateway limits are projected into existing namespace `registry-migrate-image-limits` LimitRange: `type: alauda.io/Image` with `max.storage` for image size, and `type: alauda.io/ImageStream` with `max["alauda.io/repository-tags"]` for repository-tag count. A legacy `2GB/2000` value is represented as `2Gi/2000`. Namespace-specific rules are selected by the approved plan; review `namespaceLimits` when rules overlap. The migration does not create ResourceQuota, overwrite existing user limits, or configure future namespaces. -The following S3 structure is an example only. Use `ac explain` and the actual legacy Registry configuration as the source of truth, replace bucket, region, endpoint, CA, and related access settings, and remove fields that the current environment does not use: - -```yaml -apiVersion: imageregistry.operator.alauda.io/v1 -kind: Config -metadata: - name: cluster -spec: - managementState: Managed - storage: - managementState: Unmanaged - s3: - bucket: - region: - regionEndpoint: - virtualHostedStyle: false - trustedCA: - name: -``` +## 9. P7-verify: resume the Operator and verify read-only service -Save the storage configuration and apply it: +Release the ACP pause/hold through the platform procedure. Then restore the Operator and wait for the data plane. ```bash -ac apply -f "$MIGRATION_DIR/modern-registry-object-storage.yaml" - -while ! ac get deployment image-registry -n "$MODERN_REGISTRY_NAMESPACE" >/dev/null 2>&1; do - sleep 5 +acn -n "$OPERATOR_NS" scale deployment cluster-image-registry-operator --replicas=1 +acn -n "$OPERATOR_NS" wait --for=condition=Available \ + deployment/cluster-image-registry-operator --timeout=300s +for deployment in image-registry image-api-server; do + acns wait --for=create "deployment/$deployment" --timeout=600s + acns wait --for=condition=Available "deployment/$deployment" --timeout=600s done - -while ! ac get deployment image-api-server -n "$MODERN_REGISTRY_NAMESPACE" >/dev/null 2>&1; do - sleep 5 -done -``` - -Do not store object storage credentials in the migration environment file. Create or reuse the Secret required by the Registry v2 Operator, then verify that the Registry pod has picked up the storage settings: - -```bash -ac get deployment image-registry -n "$MODERN_REGISTRY_NAMESPACE" -o yaml -ac logs -n "$MODERN_REGISTRY_NAMESPACE" deployment/image-registry -c registry --tail=80 - -ac rollout status deployment/image-registry -n "$MODERN_REGISTRY_NAMESPACE" --timeout=300s -ac rollout status deployment/image-api-server -n "$MODERN_REGISTRY_NAMESPACE" --timeout=300s -``` - -After storage reuse is configured, backfill metadata by inspecting the legacy Registry: - -```bash -export METADATA_INSPECT_REGISTRY_HOST="$LEGACY_REGISTRY_HOST" -ac config set-registry-mode modern -backfill_metadata +acn wait --for=condition=Available apiservice/v1.image.alauda.io --timeout=300s +acn get apiservice v1.image.alauda.io -o json | + jq -e '.status.conditions // [] | any(.[]; .type == "Available" and .status == "True" and .reason == "Passed")' +registry-migrate run --run-dir "$RUN_DIR" --stage P7-verify --kubeconfig "$KCFG" --approve +registry-migrate verify --run-dir "$RUN_DIR" --kubeconfig "$KCFG" ``` -If `ac image info "$TARGET_REGISTRY_HOST/$image"` fails before metadata backfill, that does not necessarily mean storage reuse is broken. Registry v2 can return `NAME_UNKNOWN` when Image and ImageStream metadata do not exist yet. Use the `backfill_metadata` result and the verification steps below as the decision point. - -## Scenario B: Copy Images Between Registries +Expected result: Operator, Registry, Image API, and APIService are ready; P7 is `verified` and `verify` exits zero. Confirm Service ports `80→8080`, `443→5000`, and `5000→5000`, Ready endpoints, the new Ingress backend, Config, metadata, and limits. Config remains read-only, pull-Secret management remains Unmanaged, and the Pruner remains suspended. Stop on failure. -Use this path when the source and target registries do not share storage. +## 10. P8-commit: verify business access, then open writes -Generate a mirror mapping file. This preserves namespace, repository, and tag names and only replaces the Registry host: +Use the actual external URL for this cluster, including protocol and port. Do not copy a test environment port to another cluster. A trusted system CA or provider PEM CA is required for TLS acceptance; `curl -k` is only a diagnostic. ```bash -while read -r image; do - [ -n "$image" ] || continue - printf '%s/%s=%s/%s\n' \ - "$LEGACY_REGISTRY_HOST" "$image" \ - "$TARGET_REGISTRY_HOST" "$image" -done < "$IMAGE_LIST_FILE" > "$MAPPING_FILE" - -sed -n '1,20p' "$MAPPING_FILE" -wc -l "$MAPPING_FILE" -``` +test -n "$REGISTRY_URL" +REGISTRY_HOST="$(jq -er '.discovery.routes[0].hostname' "$RUN_DIR/plan.json")" +jq -ne --arg url "$REGISTRY_URL" --arg host "$REGISTRY_HOST" \ + '($url | capture("^https?://(?[^/:]+)(:[0-9]+)?/?$").host) == $host' +CURL_TLS_ARGS=() +if test -n "$REGISTRY_CA_BUNDLE"; then CURL_TLS_ARGS=(--cacert "$REGISTRY_CA_BUNDLE"); fi +http_code="$(curl -sS "${CURL_TLS_ARGS[@]}" --connect-timeout 10 --max-time 20 \ + -o /dev/null -w '%{http_code}' "$REGISTRY_URL/v2/")" +case "$http_code" in 200|401|403) echo "Registry reachable: HTTP $http_code" ;; *) echo "unexpected HTTP $http_code"; exit 1 ;; esac +test -n "$TEST_NS" +test -n "$TEST_REPO" +mkdir -p "$RUN_DIR/verification" -Run a dry run before copying: - -```bash -MIRROR_DRY_RUN_LOG="$MIGRATION_DIR/mirror-dry-run.log" - -ac image mirror \ - -f "$MAPPING_FILE" \ - --registry-config "$REGISTRY_AUTH_FILE" \ - --dry-run \ - --keep-manifest-list \ - $REGISTRY_INSECURE_FLAG \ - > "$MIRROR_DRY_RUN_LOG" 2>&1 - -echo "mirror dry-run log: $MIRROR_DRY_RUN_LOG" -sed -n '1,20p' "$MIRROR_DRY_RUN_LOG" -wc -l "$MIRROR_DRY_RUN_LOG" +env AC_IMAGE_REGISTRY_MODE=legacy \ + ac --kubeconfig "$KCFG" --context "$CTX" get images -n "$TEST_NS" -o json \ + --registry-url "$REGISTRY_URL" | tee "$RUN_DIR/verification/legacy-images.json" +env AC_IMAGE_REGISTRY_MODE=modern \ + ac --kubeconfig "$KCFG" --context "$CTX" get images -o json \ + | tee "$RUN_DIR/verification/modern-images.json" +env AC_IMAGE_REGISTRY_MODE=legacy \ + ac --kubeconfig "$KCFG" --context "$CTX" delete images \ + --repo "$TEST_REPO" --registry-url "$REGISTRY_URL" +env AC_IMAGE_REGISTRY_MODE=modern \ + ac --kubeconfig "$KCFG" --context "$CTX" delete images --repo "$TEST_REPO" ``` -After reviewing the plan, copy the images. `--continue-on-error` continues with later mappings after a failed image, but the command still returns a non-zero exit code if any mapping failed. Capture that exit code so that log summarization still runs even in a shell that uses `set -e`: +Expected result: anonymous `/v2/` 401/403 proves only entry-point reachability; authenticated legacy and modern queries succeed, and both delete commands remain dry-run. The legacy view lists repository/tag data; the modern view uses ImageDigest/ImageStream data. Do not compare row counts as proof of completeness. -```bash -MIRROR_COPY_LOG="$MIGRATION_DIR/mirror-copy.log" -mirror_rc=0 - -ac image mirror \ - -f "$MAPPING_FILE" \ - --registry-config "$REGISTRY_AUTH_FILE" \ - --continue-on-error \ - --keep-manifest-list \ - --max-per-registry 4 \ - $REGISTRY_INSECURE_FLAG \ - > "$MIRROR_COPY_LOG" 2>&1 || mirror_rc=$? - -echo "mirror copy summary: rc=$mirror_rc log=$MIRROR_COPY_LOG" -grep -Ein 'error|failed|denied|unauthorized|forbidden' "$MIRROR_COPY_LOG" | sed -n '1,20p' || true - -if [ "$mirror_rc" -ne 0 ]; then - echo "mirror copy had failed mappings; fix the failures before metadata backfill or cutover" >&2 - false -fi -``` +Create a disposable test Pod with `imagePullPolicy=Always` using the original image reference, ServiceAccount and Secret. Verify pulls through the original `:80`, `:443`, and `:5000` paths and the external host as applicable. Record the Pod, digest, query output, and pull result. Do not perform a positive push before P8. -Keep `--keep-manifest-list` during migration to preserve manifest lists and multi-architecture images. - -After the image copy finishes, switch to modern mode and backfill metadata by inspecting Registry v2: +After the business checks pass and an administrator approves the cutover: ```bash -export METADATA_INSPECT_REGISTRY_HOST="$TARGET_REGISTRY_HOST" -ac config set-registry-mode modern -backfill_metadata +jq --arg actor "$ACTOR" --arg at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ + '{version:"v1",runID:.runID,planDigest:.digest,observedAt:$at,actor:$actor,commitApproved:true,businessProbeVerified:true,pullSecretHandoffVerified:true}' \ + "$RUN_DIR/plan.json" > "$RUN_DIR/commit-approval.json" +registry-migrate run --run-dir "$RUN_DIR" --stage P8-commit \ + --evidence-file "$RUN_DIR/commit-approval.json" --kubeconfig "$KCFG" --approve +acn get config.imageregistry.operator.alauda.io cluster -o json | + jq -e '.spec.readOnly == false and .spec.imagePullSecret.managementState == "Managed"' +acn get imagepruner.imageregistry.operator.alauda.io cluster -o json | jq -e ' + .metadata.annotations["registry-migrate.original-suspend"] as $original | + ($original == "true" or $original == "false") and (.spec.suspend == ($original == "true"))' +registry-migrate verify --run-dir "$RUN_DIR" --kubeconfig "$KCFG" +registry-migrate report --run-dir "$RUN_DIR" ``` -## Rerun Safely +Expected result: `state=committed`; Config is writable, pull-Secret management is Managed, and the Pruner has its recorded original suspension value. After Operator reconciliation, use an approved OCI client to push and pull one disposable repository and retain that output. If a failure occurs now, the write transition has already been committed; do not assume read-only state. -Migration steps are designed to be rerunnable. If the current `ImageStreamTag` already points to the expected digest and target Registry reference, `backfill_metadata` skips creating a new `ImageStreamMapping` and counts the item as `skipped`. +## Appendix: S3 access, retries, exceptions, and recovery -For storage reuse, rerun metadata backfill: +### S3 available only inside the cluster ```bash -export METADATA_INSPECT_REGISTRY_HOST="$LEGACY_REGISTRY_HOST" -ac config set-registry-mode modern -backfill_metadata +ac --kubeconfig "/path/to/business.kubeconfig" --context "actual-context" \ + -n minio-system port-forward service/minio 19000:9000 ``` -For cross-registry sync, rerun image copy and metadata backfill: +In the original terminal: ```bash -MIRROR_COPY_LOG="$MIGRATION_DIR/mirror-rerun.log" -mirror_rc=0 - -ac image mirror \ - -f "$MAPPING_FILE" \ - --registry-config "$REGISTRY_AUTH_FILE" \ - --continue-on-error \ - --keep-manifest-list \ - --max-per-registry 4 \ - $REGISTRY_INSECURE_FLAG \ - > "$MIRROR_COPY_LOG" 2>&1 || mirror_rc=$? - -echo "mirror rerun summary: rc=$mirror_rc log=$MIRROR_COPY_LOG" -grep -Ein 'error|failed|denied|unauthorized|forbidden' "$MIRROR_COPY_LOG" | sed -n '1,20p' || true - -if [ "$mirror_rc" -ne 0 ]; then - echo "mirror rerun had failed mappings; inspect $MIRROR_COPY_LOG before metadata backfill" >&2 - false -fi - -export METADATA_INSPECT_REGISTRY_HOST="$TARGET_REGISTRY_HOST" -backfill_metadata -``` - -After a rerun, confirm that the same tag still points to the expected digest and target Registry reference. - -## Verify Migration - -Verify important business images and random samples. The following example selects the first image from the migration list: - -```bash -VERIFY_IMAGE="$(awk 'NF > 0 { print $1; exit }' "$IMAGE_LIST_FILE")" -test -n "$VERIFY_IMAGE" - -VERIFY_NAMESPACE="${VERIFY_IMAGE%%/*}" -VERIFY_REST="${VERIFY_IMAGE#*/}" -VERIFY_STREAM="${VERIFY_REST%%:*}" -VERIFY_TAG="${VERIFY_REST##*:}" - -echo "verify image: $VERIFY_IMAGE" - -ac get imagestreamtags "$VERIFY_STREAM:$VERIFY_TAG" -n "$VERIFY_NAMESPACE" - -VERIFY_REF="$( - ac get imagestreamtags "$VERIFY_STREAM:$VERIFY_TAG" -n "$VERIFY_NAMESPACE" \ - | awk 'NR > 1 && $3 == "current" { print $6; exit }' -)" - -ac image info "$VERIFY_REF" \ - --registry-config "$REGISTRY_AUTH_FILE" \ - $REGISTRY_INSECURE_FLAG - -VERIFY_DIGEST="${VERIFY_REF##*@}" -VERIFY_IMAGE_NAME="$(printf '%s\n' "$VERIFY_DIGEST" | sed 's/:/-/g')" -VERIFY_IMAGES_OUTPUT="$MIGRATION_DIR/verify-modern-images.txt" - -ac get images > "$VERIFY_IMAGES_OUTPUT" -awk -v digest="$VERIFY_DIGEST" -v safe="$VERIFY_IMAGE_NAME" ' - NR > 1 && ($1 == digest || $1 == safe || index($2, digest) > 0) { found = 1 } - END { exit found ? 0 : 1 } -' "$VERIFY_IMAGES_OUTPUT" -``` - -The example above verifies one sample only. Before cutover, run a full storage and metadata consistency gate against the migration list: - -```bash -VERIFY_SOURCE_FILE="$IMAGE_LIST_FILE" -VERIFY_SOURCE_HAS_DIGEST=false -if [ -n "${IMAGE_DIGEST_FILE:-}" ] && [ -s "$IMAGE_DIGEST_FILE" ]; then - VERIFY_SOURCE_FILE="$IMAGE_DIGEST_FILE" - VERIFY_SOURCE_HAS_DIGEST=true -fi - -verify_failed=0 - -while read -r image digest _; do - [ -n "$image" ] || continue - namespace="${image%%/*}" - rest="${image#*/}" - stream="${rest%%:*}" - tag="${rest##*:}" - - tag_output_file="$MIGRATION_DIR/verify-${namespace}-${stream}-${tag}.imagestreamtag.txt" - tag_info_file="$MIGRATION_DIR/verify-${namespace}-${stream}-${tag}.tag.info.txt" - digest_info_file="$MIGRATION_DIR/verify-${namespace}-${stream}-${tag}.digest.info.txt" - - if ! ac get imagestreamtags "$stream:$tag" -n "$namespace" > "$tag_output_file" 2>&1; then - echo "[verify] ImageStreamTag missing: $namespace/$stream:$tag output=$tag_output_file" >&2 - verify_failed=$((verify_failed + 1)) - continue - fi - - ref="$( - awk 'NR > 1 && $3 == "current" { print $6; exit }' "$tag_output_file" - )" - if [ -z "$ref" ]; then - echo "[verify] failed to read current reference for $namespace/$stream:$tag output=$tag_output_file" >&2 - verify_failed=$((verify_failed + 1)) - continue - fi - - if ! ac image info "$ref" \ - --registry-config "$REGISTRY_AUTH_FILE" \ - $REGISTRY_INSECURE_FLAG > "$tag_info_file" 2>&1; then - echo "[verify] target registry cannot inspect ImageStreamTag reference $ref output=$tag_info_file" >&2 - verify_failed=$((verify_failed + 1)) - fi - - if [ "$VERIFY_SOURCE_HAS_DIGEST" = "true" ] && [ -n "$digest" ]; then - digest_ref="$TARGET_REGISTRY_HOST/$namespace/$stream@$digest" - if ! ac image info "$digest_ref" \ - --registry-config "$REGISTRY_AUTH_FILE" \ - $REGISTRY_INSECURE_FLAG > "$digest_info_file" 2>&1; then - echo "[verify] target registry cannot inspect expected digest $digest_ref output=$digest_info_file" >&2 - verify_failed=$((verify_failed + 1)) - fi - fi -done < "$VERIFY_SOURCE_FILE" - -if [ "$verify_failed" -ne 0 ]; then - echo "[verify] migration storage/metadata consistency gate failed: $verify_failed images failed" >&2 - false -fi +curl --fail --connect-timeout 5 --max-time 10 \ + http://127.0.0.1:19000/minio/health/live +S3_ARGS=(--s3-endpoint http://127.0.0.1:19000) ``` -If this gate fails, at least one Image API metadata entry does not match readable Registry v2 storage data. Do not switch traffic, prune images, or run registry garbage collection. Fix storage reuse or image copy first, or remove stale backfilled metadata after explicitly accepting that legacy image data will not be preserved. - -Acceptance criteria: - -- Registry v2 can inspect or pull the current digest reference from `ImageStreamTag`. -- `ImageStreamMapping` creation succeeded. -- Reruns do not produce unexplained failures. -- `ImageStreamTag` points to the expected digest and Registry v2 address. -- `ac get images` in modern mode shows the migrated Image resources through the ACP Image API. -- The full storage and metadata consistency gate passes for every migrated image in the migration list. -- Sample images in critical business namespaces can be read. - -## Switch Registry Mode +The override changes only the P3 transport address; the source endpoint remains in the plan/snapshot and is not replaced in the target Config. Rebuild a broken tunnel and retry the same run. An HTTPS tunnel still uses `https://`; port forwarding does not convert the protocol. -After verification, switch the current kubeconfig context to modern Registry mode: +### Exceptions and recovery -```bash -ac config set-registry-mode modern -ac config get-registry-mode -``` - -`ac config set-registry-mode` changes only the current context's cluster by default. Use `--all-clusters` only after every cluster in the ACP session has completed Registry migration. +- **Force writer fence:** after an administrator accepts the consistency risk, set `FORCE_ARGS=(--force)` before P1/P2/P3/P4/P6. It cannot bypass API, UID, S3, metadata, or stage checks, and never bypasses the new Operator pause in P4/P6. +- **Skip S3 verification:** set `P3_PROTECTION=(--skip-s3-backup-verification)` only when the administrator accepts that no S3 recovery point was established. +- **Nested test residue:** add `--skip-nested-repositories` to P3 only for approved historical test residue. Review `legacy-metadata-report.json`. +- **Original 4.3 ac:** it has no `set-registry-mode`; use `--registry-url`, `--kubeconfig`, and `--context` for a legacy query. The full permission/pagination matrix belongs to the platform’s separate compatibility acceptance record, not to the production migration window. -Run dry-run image management commands to confirm that the backend has switched to Registry v2: +If the Bash terminal exits, export the original absolute run directory and run Section 0 again: ```bash -ac get images -ac delete images --repo "$VERIFY_IMAGE" -ac adm prune images +export RUN_DIR="/absolute/path/to/registry-migrate-runs/original-run-id" ``` -`ac delete images` and `ac adm prune images` are dry-run by default. Add `--confirm` only after reviewing the output. +Section 0 restores the original run ID, scope, and discovered Deployments. Re-select S3, force, and P3 protection arrays. Do not create a new run or repeat completed manual uninstall/Ingress deletion steps. -## Roll Back +The tool is idempotent within one run: completed Secrets, metadata, limits, and checks are skipped or revalidated. Preserve plan, journal, backups, inventory, snapshot, evidence, and verification output. P3’s protection state is immutable; adding proof later does not change a run that originally skipped verification. -If workloads cannot read business images after cutover, switch the affected context back to legacy mode: +For recovery: ```bash -ac config set-registry-mode legacy -ac config get-registry-mode +registry-migrate rollback --run-dir "$RUN_DIR" --kubeconfig "$KCFG" --approve +registry-migrate report --run-dir "$RUN_DIR" ``` -Keep the following artifacts for troubleshooting and reruns: - -- Image lists. -- Mirror mapping files. -- `ac image mirror` output. -- `ac image info` output. -- Generated `ImageStreamMapping` YAML files. -- `ac create -f` output. -- Migration verification output. - -After the issue is fixed, continue from the image list. You do not need to rebuild the migration scope from the beginning. +`rollback` records a manual recovery request and protects source data. It does not reinstall the old chart, recreate an original-UID Secret, delete Registry v2 resources, or restore S3 objects. Before chart removal, restore old replicas, Pruner/platform state, credentials, and chart configuration from the saved materials. After chart removal, stop new Registry writes and reconciliation first, then manually restore chart/platform configuration, credentials, and S3 access. After P8, account for new images and credential rotation; do not point the same S3 writer back at the legacy Registry without a controlled recovery plan. After a real recovery, start a new run ID and do not edit plan/journal to bypass identity drift. From d9d96dfc9580aff6776511cdb76235e5fc7534f2 Mon Sep 17 00:00:00 2001 From: huizhang Date: Fri, 30 Oct 2026 00:01:00 +0800 Subject: [PATCH 02/12] docs: refine Registry v2 migration runbook --- .../ACP_Registry_v2_Migration_Runbook.mdx | 42 +++++++++++++------ 1 file changed, 30 insertions(+), 12 deletions(-) diff --git a/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx b/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx index 01e1f9a39..b9d767d01 100644 --- a/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx +++ b/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx @@ -39,6 +39,7 @@ if test -n "$RUN_DIR"; then RUN_ID="$(jq -er '.runID' "$RUN_DIR/plan.json")" REGISTRY_NS="$(jq -er '.scope.operandNamespace' "$RUN_DIR/plan.json")" OPERATOR_NS="$(jq -er '.scope.operatorNamespace' "$RUN_DIR/plan.json")" + REGISTRY_SERVICE="$(jq -er '.scope.serviceName' "$RUN_DIR/plan.json")" REGISTRY_DEPLOY="$(jq -r '.discovery.deployment // empty' "$RUN_DIR/plan.json")" GATEWAY_DEPLOY="$(jq -r '.discovery.gatewayDeployment // empty' "$RUN_DIR/plan.json")" OUT="$(dirname "$RUN_DIR")" @@ -49,6 +50,7 @@ fi mkdir -p "$OUT" S3_ARGS=() FORCE_ARGS=() +NESTED_ARGS=() registry-migrate version ac version --client printf 'context=%s\nrun-dir=%s\n' "$CTX" "$RUN_DIR" @@ -84,6 +86,7 @@ registry-migrate plan --output-dir "$OUT" --run-id "$RUN_ID" \ jq -e '(.discovery.blockers // []) | length == 0' "$RUN_DIR/plan.json" REGISTRY_DEPLOY="$(jq -er '.discovery.deployment | select(length > 0)' "$RUN_DIR/plan.json")" GATEWAY_DEPLOY="$(jq -er '.discovery.gatewayDeployment | select(length > 0)' "$RUN_DIR/plan.json")" +REGISTRY_SERVICE="$(jq -er '.scope.serviceName' "$RUN_DIR/plan.json")" jq '{scope,storage:.discovery.storage,routes:.discovery.routes,limits:.namespaceLimits,warnings:.discovery.warnings}' "$RUN_DIR/plan.json" ``` @@ -110,7 +113,12 @@ PULL_SECRET_REF="$(acn get deployments -A -o json | jq -er ' [.metadata.namespace,.metadata.name]] | if length == 1 then .[0] | @tsv else error("review legacy pull-secret controller candidates") end')" IFS=$'\t' read -r PULL_SECRET_NS PULL_SECRET_DEPLOY <<< "$PULL_SECRET_REF" -stop_deployment "$PULL_SECRET_NS" "$PULL_SECRET_DEPLOY" +if test "${#FORCE_ARGS[@]}" -eq 0; then + stop_deployment "$PULL_SECRET_NS" "$PULL_SECRET_DEPLOY" +else + acn -n "$PULL_SECRET_NS" get deployment "$PULL_SECRET_DEPLOY" -o wide + echo "force mode: leaving the legacy pull-secret controller state unchanged" +fi evidence controller-stopped.json legacyControllerStopped registry-migrate run --run-dir "$RUN_DIR" --stage P1-credentials \ --evidence-file "$RUN_DIR/controller-stopped.json" \ @@ -127,14 +135,23 @@ Before running the command, set `WRITER_RECORD` to the maintenance/change record ```bash export WRITER_RECORD="/path/to/maintenance-writer-record.txt" -stop_deployment "$REGISTRY_NS" "$REGISTRY_DEPLOY" -stop_deployment "$REGISTRY_NS" "$GATEWAY_DEPLOY" +if test "${#FORCE_ARGS[@]}" -eq 0; then + stop_deployment "$REGISTRY_NS" "$REGISTRY_DEPLOY" + stop_deployment "$REGISTRY_NS" "$GATEWAY_DEPLOY" +else + acns get deployment "$REGISTRY_DEPLOY" "$GATEWAY_DEPLOY" -o wide + echo "force mode: leaving legacy Registry/Gateway states unchanged" +fi LEGACY_PRUNER="$(acns get cronjob image-pruner --ignore-not-found -o name)" if test -n "$LEGACY_PRUNER"; then - acns patch cronjob image-pruner --type=merge -p '{"spec":{"suspend":true}}' + if test "${#FORCE_ARGS[@]}" -eq 0; then + acns patch cronjob image-pruner --type=merge -p '{"spec":{"suspend":true}}' + else + echo "force mode: leaving the legacy Pruner state unchanged" + fi fi -acns get endpoints image-registry -o wide -acns get endpointslices.discovery.k8s.io -l kubernetes.io/service-name=image-registry -o wide +acns get endpoints "$REGISTRY_SERVICE" -o wide +acns get endpointslices.discovery.k8s.io -l "kubernetes.io/service-name=$REGISTRY_SERVICE" -o wide test -n "$WRITER_RECORD" test -s "$WRITER_RECORD" if test "$WRITER_RECORD" != "$RUN_DIR/writer-record.txt"; then @@ -146,7 +163,7 @@ registry-migrate run --run-dir "$RUN_DIR" --stage P2-stop-writers \ "${FORCE_ARGS[@]}" --kubeconfig "$KCFG" --approve ``` -Expected result: old Registry/Gateway Pods and Ready endpoints are gone, the old Pruner is suspended, active jobs have finished, and the stage is recorded. If the platform cannot keep old components stopped, set `FORCE_ARGS=(--force)` only after an administrator accepts the consistency risk. Force does not bypass API, UID, S3, metadata, stage, or new Operator pause checks. +Expected result: old Registry/Gateway Pods and Ready endpoints are gone, the old Pruner is suspended, active jobs have finished, and the stage is recorded. The copied `WRITER_RECORD` is a mandatory administrator gate for external writers; the tool separately checks live cluster writers. If the platform cannot keep old components stopped, set `FORCE_ARGS=(--force)` only after an administrator accepts the consistency risk. Force does not bypass API, UID, S3, metadata, stage, or new Operator pause checks. ## 5. P3-final-inventory: freeze the inventory and S3 state @@ -178,12 +195,13 @@ Do not use both paths. ```bash registry-migrate resume --run-dir "$RUN_DIR" --stage P3-final-inventory \ "${P3_PROTECTION[@]}" "${S3_ARGS[@]}" \ + "${NESTED_ARGS[@]}" \ "${FORCE_ARGS[@]}" --kubeconfig "$KCFG" --approve registry-migrate report --run-dir "$RUN_DIR" jq '{items:(.items|length),digest,sample:(.items[0] | {namespace,repository,tag,digest})}' "$RUN_DIR/final-inventory.json" ``` -Expected result: recorded stage, `final-inventory.json`, and `s3-snapshot.json`. Scanning reads tag links, manifests, and configs; it does not move layers. Approved test residue excluded with `--skip-nested-repositories` must be reviewed in `legacy-metadata-report.json` and not counted as migrated. +Expected result: recorded stage, `final-inventory.json`, and `s3-snapshot.json`. Scanning reads tag links, manifests, and configs; it does not move layers. Approved test residue excluded with `--skip-nested-repositories` must be reviewed in `legacy-metadata-report.json` and not counted as migrated. Set `NESTED_ARGS=(--skip-nested-repositories)` before P3 only for that approved residue; the normal value is `NESTED_ARGS=()`. ## 6. P4-prepare-operator: prepare Registry v2 read-only @@ -281,10 +299,10 @@ Expected result: Operator, Registry, Image API, and APIService are ready; P7 is ## 10. P8-commit: verify business access, then open writes -Use the actual external URL for this cluster, including protocol and port. Do not copy a test environment port to another cluster. A trusted system CA or provider PEM CA is required for TLS acceptance; `curl -k` is only a diagnostic. +Use the actual external URL for this cluster, including protocol and port. Do not copy a test environment port to another cluster. A trusted system CA or provider PEM CA is required for TLS acceptance; `curl -k` is only a diagnostic. Set the three test inputs for this cluster before running the probe block. ```bash -test -n "$REGISTRY_URL" +export REGISTRY_URL="${REGISTRY_URL:?set the actual external Registry URL, including scheme and port}" REGISTRY_HOST="$(jq -er '.discovery.routes[0].hostname' "$RUN_DIR/plan.json")" jq -ne --arg url "$REGISTRY_URL" --arg host "$REGISTRY_HOST" \ '($url | capture("^https?://(?[^/:]+)(:[0-9]+)?/?$").host) == $host' @@ -293,8 +311,8 @@ if test -n "$REGISTRY_CA_BUNDLE"; then CURL_TLS_ARGS=(--cacert "$REGISTRY_CA_BUN http_code="$(curl -sS "${CURL_TLS_ARGS[@]}" --connect-timeout 10 --max-time 20 \ -o /dev/null -w '%{http_code}' "$REGISTRY_URL/v2/")" case "$http_code" in 200|401|403) echo "Registry reachable: HTTP $http_code" ;; *) echo "unexpected HTTP $http_code"; exit 1 ;; esac -test -n "$TEST_NS" -test -n "$TEST_REPO" +export TEST_NS="${TEST_NS:?set a namespace containing a migrated test image}" +export TEST_REPO="${TEST_REPO:?set an isolated namespace/repository:tag for dry-run deletion}" mkdir -p "$RUN_DIR/verification" env AC_IMAGE_REGISTRY_MODE=legacy \ From 47b5cb1d8c91f32d25565be4e10b8efb88c418f3 Mon Sep 17 00:00:00 2001 From: huizhang Date: Fri, 30 Oct 2026 10:09:52 +0800 Subject: [PATCH 03/12] docs: finish migration runbook review fixes --- docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx b/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx index b9d767d01..5b6cea8ce 100644 --- a/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx +++ b/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx @@ -163,7 +163,7 @@ registry-migrate run --run-dir "$RUN_DIR" --stage P2-stop-writers \ "${FORCE_ARGS[@]}" --kubeconfig "$KCFG" --approve ``` -Expected result: old Registry/Gateway Pods and Ready endpoints are gone, the old Pruner is suspended, active jobs have finished, and the stage is recorded. The copied `WRITER_RECORD` is a mandatory administrator gate for external writers; the tool separately checks live cluster writers. If the platform cannot keep old components stopped, set `FORCE_ARGS=(--force)` only after an administrator accepts the consistency risk. Force does not bypass API, UID, S3, metadata, stage, or new Operator pause checks. +Expected result in the normal path: old Registry/Gateway Pods and Ready endpoints are gone, the old Pruner is suspended, active jobs have finished, and the stage is recorded. The copied `WRITER_RECORD` is a mandatory administrator gate for external writers; the tool separately checks live cluster writers. If the platform cannot keep old components stopped, set `FORCE_ARGS=(--force)` only after an administrator accepts the consistency risk; in that path, record the actual running state and expect the tool’s force-fence warning instead of claiming that the old Pods or endpoints are gone. Force does not bypass API, UID, S3, metadata, stage, or new Operator pause checks. ## 5. P3-final-inventory: freeze the inventory and S3 state From aae6e2113b801e8c0780fc63073dc08a5bb0516d Mon Sep 17 00:00:00 2001 From: huizhang Date: Mon, 2 Nov 2026 13:03:25 +0800 Subject: [PATCH 04/12] docs: add release 4.5 registry compatibility gates --- .../ACP_Registry_v2_Migration_Runbook.mdx | 50 +++++++++++++++++-- 1 file changed, 46 insertions(+), 4 deletions(-) diff --git a/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx b/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx index 5b6cea8ce..93eca272f 100644 --- a/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx +++ b/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx @@ -17,6 +17,18 @@ This procedure covers one ACP cluster, one legacy Registry, the same business cl PVC/filesystem migration, cross-cluster mirroring, `ac image mirror`, manual `ImageStreamMapping` backfill, and automatic Operator takeover are outside this runbook. Migrate one cluster at a time. P2 starts the maintenance window; P7 restores read-only service; P8 opens writes. +## Release 4.5 Compatibility Gates + +Release 4.5 separates the Operator installation namespace from the Registry operand namespace. The default installation namespace is `image-registry-system`; the default operand namespace is `cpaas-system`. The operand namespace is persisted for the installation, so do not infer it from the Operator namespace or from a later environment variable. Use the namespace recorded in `plan.json` as the source of truth. + +The Operator provides steady-state Registry, Image API, pull-secret rotation, namespace limits, Ingress, storage and Pruner behavior. It does not automatically take over the legacy chart. The migration tool and the platform administrator own legacy writer shutdown, credential handoff, S3/inventory protection, old chart removal and metadata import. A PVC, HostPath, non-image OCI artifact, or non-default S3 root/prefix must stop the plan unless the provider has approved a separate migration path. + +The Operator must be held at zero replicas before P4 and P6. Manual InstallPlan approval only controls whether OLM installs a bundle; it does not pause an Operator that is already running. If the platform cannot provide a verified pause/hold procedure, stop before removing the legacy chart. Otherwise the Operator can create a default Config, storage, or Pruner while the import boundary is still being prepared. + +Use the release-4.5 compatible `ac` for the compatibility checks. Its default `ac get images` and `ac delete images` contract remains the 4.3 repository/tag interface. Leave `AC_REGISTRY_COMPAT_OUTPUT` unset or set it to `4.3`; use `native` only for a separate modern API diagnostic. An explicit `--registry-url` follows the legacy Registry HTTP path, while auto mode can select the modern backend. The original 4.3 `ac` has no Registry mode setting; test it with an explicit Registry URL and a dedicated kubeconfig. + +Modern verification uses `imagedigests.image.alauda.io`, `imagestreams` and `imagestreamtags`. Do not treat the row count of `ac get images` as an Image API completeness check. `ac adm prune images` is a platform-level operation; namespace users may manage their permitted tags but must not be used as proof that cluster-wide prune or GC is authorized. + ## 0. Prepare the terminal and tools Use a dedicated Bash terminal. `cpaas-system` and `image-registry-system` are common ACP namespaces; confirm them. @@ -233,7 +245,9 @@ registry-migrate run --run-dir "$RUN_DIR" --stage P4-prepare-operator \ --evidence-file "$RUN_DIR/operator-paused.json" \ "${FORCE_ARGS[@]}" --kubeconfig "$KCFG" --approve acn get config.imageregistry.operator.alauda.io cluster -o json | - jq -e '.spec.readOnly == true and .spec.imagePullSecret.managementState == "Unmanaged"' + jq -e '.spec.readOnly == true and + .spec.imagePullSecret.managementState == "Unmanaged" and + .spec.storage.managementState == "Unmanaged"' acn get imagepruner.imageregistry.operator.alauda.io cluster -o json | jq -e '.spec.suspend == true' ``` @@ -295,7 +309,18 @@ registry-migrate run --run-dir "$RUN_DIR" --stage P7-verify --kubeconfig "$KCFG" registry-migrate verify --run-dir "$RUN_DIR" --kubeconfig "$KCFG" ``` -Expected result: Operator, Registry, Image API, and APIService are ready; P7 is `verified` and `verify` exits zero. Confirm Service ports `80→8080`, `443→5000`, and `5000→5000`, Ready endpoints, the new Ingress backend, Config, metadata, and limits. Config remains read-only, pull-Secret management remains Unmanaged, and the Pruner remains suspended. Stop on failure. +Expected result: Operator, Registry, Image API, and APIService are ready; P7 is `verified` and `verify` exits zero. Confirm Service ports `80→8080`, `443→5000`, and `5000→5000`, Ready endpoints, the new Ingress backend, Config, metadata, and limits. Config remains read-only, storage and pull-Secret management remain Unmanaged, and the Pruner remains suspended. + +Confirm the release-4.5 metadata resources and the namespace limit scope: + +```bash +acn get imagedigests.image.alauda.io -o yaml > "$RUN_DIR/verification/imagedigests.yaml" +acn get imagestreams.image.alauda.io -A -o yaml > "$RUN_DIR/verification/imagestreams.yaml" +acn get limitrange -A -l "registry-migrate.run-id=$RUN_ID" -o yaml \ + > "$RUN_DIR/verification/migration-limitranges.yaml" +``` + +The imported `LimitRange` objects are a static snapshot for namespaces already covered by the plan. They do not become a global policy and do not automatically apply to namespaces created later. ## 10. P8-commit: verify business access, then open writes @@ -306,14 +331,19 @@ export REGISTRY_URL="${REGISTRY_URL:?set the actual external Registry URL, inclu REGISTRY_HOST="$(jq -er '.discovery.routes[0].hostname' "$RUN_DIR/plan.json")" jq -ne --arg url "$REGISTRY_URL" --arg host "$REGISTRY_HOST" \ '($url | capture("^https?://(?[^/:]+)(:[0-9]+)?/?$").host) == $host' +mkdir -p "$RUN_DIR/verification" CURL_TLS_ARGS=() if test -n "$REGISTRY_CA_BUNDLE"; then CURL_TLS_ARGS=(--cacert "$REGISTRY_CA_BUNDLE"); fi http_code="$(curl -sS "${CURL_TLS_ARGS[@]}" --connect-timeout 10 --max-time 20 \ -o /dev/null -w '%{http_code}' "$REGISTRY_URL/v2/")" case "$http_code" in 200|401|403) echo "Registry reachable: HTTP $http_code" ;; *) echo "unexpected HTTP $http_code"; exit 1 ;; esac +curl -sS "${CURL_TLS_ARGS[@]}" --connect-timeout 10 --max-time 20 \ + -D "$RUN_DIR/verification/registry-v2-headers.txt" -o /dev/null "$REGISTRY_URL/v2/" +if grep -qi '^WWW-Authenticate:' "$RUN_DIR/verification/registry-v2-headers.txt"; then + echo "Review the WWW-Authenticate realm and confirm its host and port are reachable from the same client." +fi export TEST_NS="${TEST_NS:?set a namespace containing a migrated test image}" export TEST_REPO="${TEST_REPO:?set an isolated namespace/repository:tag for dry-run deletion}" -mkdir -p "$RUN_DIR/verification" env AC_IMAGE_REGISTRY_MODE=legacy \ ac --kubeconfig "$KCFG" --context "$CTX" get images -n "$TEST_NS" -o json \ @@ -330,6 +360,18 @@ env AC_IMAGE_REGISTRY_MODE=modern \ Expected result: anonymous `/v2/` 401/403 proves only entry-point reachability; authenticated legacy and modern queries succeed, and both delete commands remain dry-run. The legacy view lists repository/tag data; the modern view uses ImageDigest/ImageStream data. Do not compare row counts as proof of completeness. +The commands above use the release-4.5 compatible `ac`. When the original 4.3 client is part of the customer contract, run the same read and dry-run checks separately with `AC43_BIN`, an explicit `--registry-url "$REGISTRY_URL"`, and an isolated kubeconfig. Do not call `ac config set-registry-mode` with the original 4.3 binary. The compatibility binary keeps the 4.3 output shape by default; setting `AC_REGISTRY_COMPAT_OUTPUT=native` changes the output contract and is not valid evidence for old scripts. + +For modern-native diagnostics only, query the Image API resource directly: + +```bash +env AC_IMAGE_REGISTRY_MODE=modern AC_REGISTRY_COMPAT_OUTPUT=native \ + ac --kubeconfig "$KCFG" --context "$CTX" get images -o json \ + > "$RUN_DIR/verification/modern-native-images.json" +``` + +Do not use that native output to compare with the original 4.3 pipeline contract. Before enabling prune or GC, verify that the caller has platform-admin access to the Registry Config; a namespace-level tag administrator is not sufficient. + Create a disposable test Pod with `imagePullPolicy=Always` using the original image reference, ServiceAccount and Secret. Verify pulls through the original `:80`, `:443`, and `:5000` paths and the external host as applicable. Record the Pod, digest, query output, and pull result. Do not perform a positive push before P8. After the business checks pass and an administrator approves the cutover: @@ -349,7 +391,7 @@ registry-migrate verify --run-dir "$RUN_DIR" --kubeconfig "$KCFG" registry-migrate report --run-dir "$RUN_DIR" ``` -Expected result: `state=committed`; Config is writable, pull-Secret management is Managed, and the Pruner has its recorded original suspension value. After Operator reconciliation, use an approved OCI client to push and pull one disposable repository and retain that output. If a failure occurs now, the write transition has already been committed; do not assume read-only state. +Expected result: `state=committed`; Config is writable, storage and pull-Secret management are Managed, and the Pruner has its recorded original suspension value. After Operator reconciliation, use an approved OCI client to push and pull one disposable repository and retain that output. If a failure occurs now, the write transition has already been committed; do not assume read-only state. ## Appendix: S3 access, retries, exceptions, and recovery From 29e8b5ae8b70d535e1e5031937619acb3eab4001 Mon Sep 17 00:00:00 2001 From: huizhang Date: Tue, 3 Nov 2026 22:20:42 +0800 Subject: [PATCH 05/12] docs: rewrite the Registry v2 migration runbook Restructure the runbook around a stage overview and add the checks that were missing where a mistake cannot be undone. - add a stage table with interruption and re-run columns - fail closed when an Ingress UID drifts in P5 instead of deleting it - snapshot the Secret list before the chart uninstall in P5 - refuse to record P2 when the maintenance record is missing - pass the CA bundle to ac through SSL_CERT_FILE in P8 - assert the storage management state in P7 and P8 - compare digest sets with comm instead of counting rows - split the appendix into known limitations and retries/recovery --- .../ACP_Registry_v2_Migration_Runbook.mdx | 534 +++++++++++++----- 1 file changed, 389 insertions(+), 145 deletions(-) diff --git a/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx b/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx index 93eca272f..2db10d6f0 100644 --- a/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx +++ b/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx @@ -9,45 +9,87 @@ ProductsVersion: # ACP Registry v2 Migration Runbook -This runbook upgrades one ACP business cluster from the legacy chart Registry to the Registry managed by `cluster-image-registry-operator`. The migration is performed by the independent `registry-migrate` tool; it is not an automatic takeover feature of the Operator. +This runbook migrates the Registry in one ACP business cluster from the legacy chart release to the Registry v2 instance managed by `cluster-image-registry-operator`. The migration is performed by the standalone `registry-migrate` tool. Only one cluster is involved, and Registry v2 reuses the same S3 bucket, endpoint, and prefix as the legacy Registry. -Before the maintenance window, contact your ACP platform service provider and obtain the provider-approved `registry-migrate` binary, compatible `ac`, the Registry v2 Operator bundle/CRD, the ACP pause/hold procedure, and the S3 backup/restore procedure. This runbook was checked against `registry-migrate` v4.5.0 (commit `6edeb747`). The migration tool version is independent of the ACP release. Do not build binaries during the window. +**This procedure deletes no images.** On the resource side there are only two delete operations: uninstalling the legacy chart in P5 (performed by the platform administrator in the plugin page) and deleting the leftover Ingress. `ac delete images` in P8 is a dry run; never add `--confirm`. -This procedure covers one ACP cluster, one legacy Registry, the same business cluster, legacy S3 storage, and a Gateway Ingress as the external entry point. Registry v2 reuses the same S3 bucket, endpoint, and prefix. A legacy `image-registry` Service of type `NodePort` is acceptable when clients use the Gateway Ingress address. +## Stage overview -PVC/filesystem migration, cross-cluster mirroring, `ac image mirror`, manual `ImageStreamMapping` backfill, and automatic Operator takeover are outside this runbook. Migrate one cluster at a time. P2 starts the maintenance window; P7 restores read-only service; P8 opens writes. +| Stage | Purpose | Service interruption | Rerunnable on failure | +| --- | --- | --- | --- | +| 0 | Validate the tools, the login session, and the cluster context | No | Yes | +| P0 plan | Generate the read-only plan and check the blockers | No | Yes | +| P1-backup | Save the recovery material | No | Yes | +| P1-credentials | Hand over the pull Secret controller | No | Yes | +| P2-stop-writers | Stop the legacy writers (**maintenance window starts**) | **Yes** | Yes | +| P3-final-inventory | Freeze the image inventory and the S3 snapshot | **Yes** | Yes, but the protection state is immutable | +| P4-prepare-operator | Scale the Operator to zero and create the read-only Registry v2 | **Yes** | Yes | +| P5-remove-legacy-chart | Uninstall the legacy plugin by hand; the tool deletes the leftover Ingress | **Yes** | The commands can be re-run; the uninstall must be completed by hand | +| P6-cold-import | Import the metadata and the namespace limits | **Yes** | Yes | +| P7-verify | Restore the Operator and verify the read-only service | Partially restored | Yes | +| P8-commit | Verify business access and open writes (**maintenance window ends**) | Restored | Yes | -## Release 4.5 Compatibility Gates +The table above is the route map for the whole procedure. Start at section 0. -Release 4.5 separates the Operator installation namespace from the Registry operand namespace. The default installation namespace is `image-registry-system`; the default operand namespace is `cpaas-system`. The operand namespace is persisted for the installation, so do not infer it from the Operator namespace or from a later environment variable. Use the namespace recorded in `plan.json` as the source of truth. +## Prerequisites -The Operator provides steady-state Registry, Image API, pull-secret rotation, namespace limits, Ingress, storage and Pruner behavior. It does not automatically take over the legacy chart. The migration tool and the platform administrator own legacy writer shutdown, credential handoff, S3/inventory protection, old chart removal and metadata import. A PVC, HostPath, non-image OCI artifact, or non-default S3 root/prefix must stop the plan unless the provider has approved a separate migration path. +- **Tool versions**: obtain the provider-approved `registry-migrate` binary, a compatible `ac` binary, the Registry v2 bundle, and the S3 backup/restore procedure from your ACP service provider. The `registry-migrate` version is independent of the ACP release. +- **Namespaces**: the Operator is installed in `image-registry-system`, and the Registry instance (the operand namespace) runs in `cpaas-system`. The operand namespace is fixed at installation time and cannot be inferred from the Operator namespace or from environment variables; always use `scope.operandNamespace` in `plan.json`. +- **Stop immediately if any of the following is present**, unless the provider has approved a separate migration path: PVCs, HostPath volumes, non-image OCI artifacts, or a non-default S3 root/prefix. +- **The Operator must stay at zero replicas between P4 and P6.** Approving an InstallPlan by hand only controls whether OLM installs the bundle; it does not pause an Operator that is already running. If the platform cannot provide a verified pause procedure, abort the migration before P4; otherwise the Operator creates its own default Config, storage, and Pruner and breaks the import boundary. +- **`ac` compatibility**: `ac get images` and `ac delete images` still use the legacy chart Registry repository/tag contract by default, controlled by `AC_REGISTRY_COMPAT_OUTPUT` (leave it unset or set it to `4.3`). P8 uses `AC_IMAGE_REGISTRY_MODE=legacy/modern` to select a view; the two settings do not replace each other. -The Operator must be held at zero replicas before P4 and P6. Manual InstallPlan approval only controls whether OLM installs a bundle; it does not pause an Operator that is already running. If the platform cannot provide a verified pause/hold procedure, stop before removing the legacy chart. Otherwise the Operator can create a default Config, storage, or Pruner while the import boundary is still being prepared. +## 0. Prepare the terminal and tools + +Use a dedicated terminal and **run every step in it**. You must have completed `ac login` before this section: an ACP login token usually expires after 24 hours, and once it does every cluster request returns `Unauthorized`. When unsure, list the sessions first: -Use the release-4.5 compatible `ac` for the compatibility checks. Its default `ac get images` and `ac delete images` contract remains the 4.3 repository/tag interface. Leave `AC_REGISTRY_COMPAT_OUTPUT` unset or set it to `4.3`; use `native` only for a separate modern API diagnostic. An explicit `--registry-url` follows the legacy Registry HTTP path, while auto mode can select the modern backend. The original 4.3 `ac` has no Registry mode setting; test it with an explicit Registry URL and a dedicated kubeconfig. +```bash +ac config get-sessions +``` -Modern verification uses `imagedigests.image.alauda.io`, `imagestreams` and `imagestreamtags`. Do not treat the row count of `ac get images` as an Image API completeness check. `ac adm prune images` is a platform-level operation; namespace users may manage their permitted tags but must not be used as proof that cluster-wide prune or GC is authorized. +The row marked with `*` in the `CURRENT` column is the active session. Use its `PLATFORM` and `SESSION` columns to log in again: -## 0. Prepare the terminal and tools +```bash +printf 'PLATFORM: '; IFS= read -r PLATFORM +printf 'SESSION: '; IFS= read -r SESSION +ac login "$PLATFORM" --name "$SESSION" +``` -Use a dedicated Bash terminal. `cpaas-system` and `image-registry-system` are common ACP namespaces; confirm them. +Then run the block below. It validates the tools, confirms the authentication state, selects a context, and defines the variables and functions that every later step uses. If the kubeconfig is not at `~/.kube/config`, change only the `export KCFG=` line. ```bash -set -e -o pipefail +setup_run() { umask 077 -export KCFG="/path/to/business.kubeconfig" +unset KCFG # clear a stale path left over from an earlier terminal +export KCFG="$HOME/.kube/config" +test -f "$KCFG" || { echo "ERROR: kubeconfig not found: $KCFG" >&2; echo "Edit the KCFG line above if the business kubeconfig is elsewhere" >&2; return 1; } +for b in registry-migrate ac jq curl; do + command -v "$b" >/dev/null || { echo "ERROR: $b not found in PATH" >&2; return 1; } +done printf 'Enter the real platform administrator account: ' IFS= read -r ACTOR -test -n "$ACTOR" -export REGISTRY_NS="cpaas-system" +test -n "$ACTOR" || { echo "ERROR: administrator account is required" >&2; return 1; } +export REGISTRY_NS="cpaas-system" # used to create the plan on a first run; overridden by plan.json on re-runs export OPERATOR_NS="image-registry-system" -CTX="$(ac --kubeconfig "$KCFG" config current-context)" -test -n "$CTX" +CTX_LIST="$(ac --kubeconfig "$KCFG" config get-contexts -o name || true)" +test -n "$CTX_LIST" || { echo "ERROR: no context in $KCFG" >&2; echo "Login first: ac login --platform --name " >&2; return 1; } +CTX="$(ac --kubeconfig "$KCFG" config current-context 2>/dev/null || true)" +test -n "$CTX" || { test "$(echo "$CTX_LIST" | grep -c .)" -eq 1 && CTX="$CTX_LIST" || { + echo "Available contexts:" >&2; echo "$CTX_LIST" >&2; printf 'Enter the context to use: ' >&2; IFS= read -r CTX; }; } +echo "$CTX_LIST" | grep -Fxq "$CTX" || { echo "ERROR: context not found: $CTX" >&2; return 1; } +ac --kubeconfig "$KCFG" --context "$CTX" whoami >/dev/null 2>&1 || { + echo "ERROR: not authenticated for context $CTX (token expired?)" >&2 + echo "Existing sessions:" >&2 + ac --kubeconfig "$KCFG" config get-sessions >&2 || true + echo "Re-login: ac login --platform --name " >&2 + return 1; } OUT="$PWD/registry-migrate-runs" - -if test -n "$RUN_DIR"; then - test -f "$RUN_DIR/plan.json" +if test -n "${RUN_DIR:-}" && ! test -f "$RUN_DIR/plan.json"; then + echo "WARN: ignoring stale RUN_DIR (no plan.json): $RUN_DIR" >&2 + unset RUN_DIR +fi +if test -n "${RUN_DIR:-}"; then + test -f "$RUN_DIR/plan.json" || { echo "ERROR: RUN_DIR is set but plan.json is missing: $RUN_DIR" >&2; echo "Fix: unset RUN_DIR to start a new migration" >&2; return 1; } RUN_ID="$(jq -er '.runID' "$RUN_DIR/plan.json")" REGISTRY_NS="$(jq -er '.scope.operandNamespace' "$RUN_DIR/plan.json")" OPERATOR_NS="$(jq -er '.scope.operatorNamespace' "$RUN_DIR/plan.json")" @@ -62,12 +104,24 @@ fi mkdir -p "$OUT" S3_ARGS=() FORCE_ARGS=() -NESTED_ARGS=() registry-migrate version ac version --client printf 'context=%s\nrun-dir=%s\n' "$CTX" "$RUN_DIR" +} +setup_run || echo "STOP: section 0 aborted; fix the error above and re-run this block. The shell stays open." >&2 +``` + +Every later step depends on the variables and functions defined here, and any failure only prints `STOP`; the terminal stays open. When the block finishes, confirm that the `context=` line it prints is the business cluster for this migration. Fix the cause and re-run the block if needed (a re-run resets `S3_ARGS` and `FORCE_ARGS`). + +If the terminal closes unexpectedly, export the original `RUN_DIR` as described in appendix B and run this section again; it restores the run ID, the namespaces, and the Deployment names from `plan.json`. Do not start a new run. -acn() { ac --kubeconfig "$KCFG" --context "$CTX" "$@"; } +The next block is mandatory. It defines `acn`, `acns`, `evidence`, and `stop_deployment`, which the later sections call. + +```bash +acn() { + test -n "${CTX:-}" || { echo "ERROR: CTX is empty; run section 0 first" >&2; return 1; } + ac --kubeconfig "${KCFG:-$HOME/.kube/config}" --context "$CTX" "$@" +} acns() { acn -n "$REGISTRY_NS" "$@"; } evidence() { jq --arg actor "$ACTOR" --arg at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --arg field "$2" \ @@ -75,48 +129,55 @@ evidence() { "$RUN_DIR/plan.json" > "$RUN_DIR/$1" } stop_deployment() { - local namespace="$1" name="$2" selector + local namespace="$1" name="$2" selector pods acn -n "$namespace" get deployment "$name" -o wide selector="$(acn -n "$namespace" get deployment "$name" -o json | jq -er '.spec.selector.matchLabels | to_entries | map("\(.key)=\(.value)") | join(",") | select(length > 0)')" acn -n "$namespace" scale deployment "$name" --replicas=0 pods="$(acn -n "$namespace" get pods -l "$selector" -o name)" if test -n "$pods"; then - acn -n "$namespace" wait --for=delete pod -l "$selector" --timeout=180s + acn -n "$namespace" wait --for=delete pod -l "$selector" --timeout=180s || true fi } ``` -For a new migration, unset `RUN_DIR` before this section. To recover after the terminal exits, export the original absolute `RUN_DIR` and run this section again; it restores the run ID, namespaces, and Deployment names from `plan.json`. Do not create a new run for a retry. Re-select S3 and force arrays after reload. - ## 1. P0: create and review the plan ```bash registry-migrate plan --output-dir "$OUT" --run-id "$RUN_ID" \ --operand-namespace "$REGISTRY_NS" --operator-namespace "$OPERATOR_NS" \ - --kubeconfig "$KCFG" -jq -e '(.discovery.blockers // []) | length == 0' "$RUN_DIR/plan.json" -REGISTRY_DEPLOY="$(jq -er '.discovery.deployment | select(length > 0)' "$RUN_DIR/plan.json")" -GATEWAY_DEPLOY="$(jq -er '.discovery.gatewayDeployment | select(length > 0)' "$RUN_DIR/plan.json")" -REGISTRY_SERVICE="$(jq -er '.scope.serviceName' "$RUN_DIR/plan.json")" -jq '{scope,storage:.discovery.storage,routes:.discovery.routes,limits:.namespaceLimits,warnings:.discovery.warnings}' "$RUN_DIR/plan.json" + --kubeconfig "$KCFG" && +jq -e '(.discovery.blockers // []) | length == 0' "$RUN_DIR/plan.json" && +REGISTRY_DEPLOY="$(jq -er '.discovery.deployment | select(length > 0)' "$RUN_DIR/plan.json")" && +GATEWAY_DEPLOY="$(jq -er '.discovery.gatewayDeployment | select(length > 0)' "$RUN_DIR/plan.json")" && +REGISTRY_SERVICE="$(jq -er '.scope.serviceName' "$RUN_DIR/plan.json")" && +jq '{scope,storage:.discovery.storage,routes:.discovery.routes,limits:.namespaceLimits,warnings:.discovery.warnings}' "$RUN_DIR/plan.json" && +echo "P0 OK: $RUN_DIR/plan.json" +``` + +Seeing `P0 OK` means the plan was generated and passed the blocker check; if the line is missing, the step failed. A failing `&&` chain only returns a non-zero exit code and does not close the terminal. When blockers are present, inspect them first: + +```bash +jq '.discovery.blockers' "$RUN_DIR/plan.json" ``` -The plan is read-only. Review every blocker, S3 bucket/endpoint/prefix, old host/TLS/IngressClass, replicas, scheduling fields, and timeout values. Stop if any blocker is present or the plan does not match the cluster. +Planning is a read-only operation. Check every blocker, the S3 bucket/endpoint/prefix, the legacy host/TLS/IngressClass, the replica counts, and the timeouts, taking the expected values from the change record and the current cluster state; stop if the plan disagrees with the cluster. The `scope.operandNamespace` it prints must be the namespace that actually holds the Registry; if it is not, correct `REGISTRY_NS` in section 0 and re-run P0. ## 2. P1-backup: save recovery material -Save old chart values, platform management objects, protected credentials, and the provider’s S3 recovery record. The tool backup is a redacted Registry-scoped snapshot; it does not contain Secret data or restore S3 image data by itself. +**This stage must complete before P1-credentials**: the credentials stage rewrites pull Secret contents, and reversing the order leaves the backup snapshot without recovery value. + +The backup is a redacted, Registry-scoped snapshot that records UIDs, owners, ServiceAccounts, and references; it contains no image layer data. Run: ```bash registry-migrate backup --run-dir "$RUN_DIR" --kubeconfig "$KCFG" --approve ``` -Expected result: a recorded stage and relevant resource/Secret summaries containing UID, owner, ServiceAccount, and reference information. This is not a backup of every object in the namespace. +Success check: the command exits 0 and adds a resource/Secret summary file under `$RUN_DIR`. If it exits non-zero, do not continue to P1-credentials; check RBAC and re-run this section. ## 3. P1-credentials: hand off pull Secrets -The legacy Registry must still be readable. First pause the AppRelease/ClusterPlugin or other platform source that manages `sa-imagepullsecret-controller`. +The legacy Registry must still be readable. First suspend the AppRelease/ClusterPlugin or other platform source that manages `sa-imagepullsecret-controller`, then run the block below. If jq reports several candidates, identify the single target by hand with `acn get deployments -A -o wide | grep -i imagepull` and re-run this section. ```bash PULL_SECRET_REF="$(acn get deployments -A -o json | jq -er ' @@ -124,29 +185,40 @@ PULL_SECRET_REF="$(acn get deployments -A -o json | jq -er ' ((.metadata.labels // {}) | tojson | test("sa-imagepullsecret-controller";"i"))) | [.metadata.namespace,.metadata.name]] | if length == 1 then .[0] | @tsv else error("review legacy pull-secret controller candidates") end')" +PULL_SECRET_NS=""; PULL_SECRET_DEPLOY="" IFS=$'\t' read -r PULL_SECRET_NS PULL_SECRET_DEPLOY <<< "$PULL_SECRET_REF" -if test "${#FORCE_ARGS[@]}" -eq 0; then - stop_deployment "$PULL_SECRET_NS" "$PULL_SECRET_DEPLOY" +if test -z "$PULL_SECRET_DEPLOY"; then + echo "STOP: cannot resolve the legacy pull-secret controller; fix the jq error above and re-run. Stage not recorded." >&2 else - acn -n "$PULL_SECRET_NS" get deployment "$PULL_SECRET_DEPLOY" -o wide - echo "force mode: leaving the legacy pull-secret controller state unchanged" + if test "${#FORCE_ARGS[@]}" -eq 0; then + stop_deployment "$PULL_SECRET_NS" "$PULL_SECRET_DEPLOY" + else + acn -n "$PULL_SECRET_NS" get deployment "$PULL_SECRET_DEPLOY" -o wide + echo "force mode: leaving the legacy pull-secret controller state unchanged" + fi + evidence controller-stopped.json legacyControllerStopped + registry-migrate run --run-dir "$RUN_DIR" --stage P1-credentials \ + --evidence-file "$RUN_DIR/controller-stopped.json" \ + "${FORCE_ARGS[@]}" --kubeconfig "$KCFG" --approve fi -evidence controller-stopped.json legacyControllerStopped -registry-migrate run --run-dir "$RUN_DIR" --stage P1-credentials \ - --evidence-file "$RUN_DIR/controller-stopped.json" \ - "${FORCE_ARGS[@]}" --kubeconfig "$KCFG" --approve ``` -Expected result: Secret totals, progress, `updated/skipped`, and a recorded stage. Existing Secret names/UIDs and ServiceAccount owner UIDs remain unchanged. Do not recreate same-named Secrets. Do not uninstall the old chart until this phase succeeds. +Success check: the output reports the Secret count and the `updated/skipped` counters, and the stage state is written to the journal; the name and UID of existing Secrets and the ServiceAccount owner UID stay unchanged. Do not uninstall the legacy chart before this stage succeeds. -## 4. P2-stop-writers: stop legacy writers +## 4. P2-stop-writers: stop the legacy writers (maintenance window starts) -Pause the legacy chart management source. Stop the legacy Registry and Gateway Deployments, pause the legacy Pruner when present, and stop external jobs writing the same S3 bucket/prefix. An old Image API or another metadata writer must be stopped and recorded by an administrator. +First suspend the platform source that manages the legacy chart. Scale the legacy Registry and Gateway Deployments to zero; if a legacy Pruner CronJob exists, set `suspend=true`; and stop every external Job that writes to the same bucket/prefix. -Before running the command, set `WRITER_RECORD` to the maintenance/change record that lists the external S3 and metadata writers, the stop time, and the administrator approval. The file is copied into the run directory for audit. +Before you run anything, point `WRITER_RECORD` at the maintenance or change record. It must name the external S3 and metadata writers, the stop time, and the administrator's approval. The file is copied into the run directory for auditing; if it is missing, the block below stops nothing. ```bash -export WRITER_RECORD="/path/to/maintenance-writer-record.txt" +stop_writers() { +export WRITER_RECORD="${WRITER_RECORD:-/tmp/maintenance-writer-record.txt}" +if ! test -s "$WRITER_RECORD"; then + echo "STOP: WRITER_RECORD missing or empty: $WRITER_RECORD; nothing was stopped." >&2 + echo "Create it first: window time, external writers, stop time, approver. The shell stays open." >&2 + return 1 +fi if test "${#FORCE_ARGS[@]}" -eq 0; then stop_deployment "$REGISTRY_NS" "$REGISTRY_DEPLOY" stop_deployment "$REGISTRY_NS" "$GATEWAY_DEPLOY" @@ -164,8 +236,9 @@ if test -n "$LEGACY_PRUNER"; then fi acns get endpoints "$REGISTRY_SERVICE" -o wide acns get endpointslices.discovery.k8s.io -l "kubernetes.io/service-name=$REGISTRY_SERVICE" -o wide -test -n "$WRITER_RECORD" -test -s "$WRITER_RECORD" +mkdir -p "$RUN_DIR/verification" +acn get cronjob,job -A -o wide > "$RUN_DIR/verification/legacy-jobs.txt" || + echo "WARN: could not list cluster jobs" >&2 if test "$WRITER_RECORD" != "$RUN_DIR/writer-record.txt"; then cp "$WRITER_RECORD" "$RUN_DIR/writer-record.txt" fi @@ -173,17 +246,31 @@ evidence writers-stopped.json legacyWritersStopped registry-migrate run --run-dir "$RUN_DIR" --stage P2-stop-writers \ --evidence-file "$RUN_DIR/writers-stopped.json" \ "${FORCE_ARGS[@]}" --kubeconfig "$KCFG" --approve +} +stop_writers ``` -Expected result in the normal path: old Registry/Gateway Pods and Ready endpoints are gone, the old Pruner is suspended, active jobs have finished, and the stage is recorded. The copied `WRITER_RECORD` is a mandatory administrator gate for external writers; the tool separately checks live cluster writers. If the platform cannot keep old components stopped, set `FORCE_ARGS=(--force)` only after an administrator accepts the consistency risk; in that path, record the actual running state and expect the tool’s force-fence warning instead of claiming that the old Pods or endpoints are gone. Force does not bypass API, UID, S3, metadata, stage, or new Operator pause checks. +Success check: the legacy Registry/Gateway Pods and their Ready endpoints are gone, the legacy Pruner is suspended, the active Jobs have finished, and the stage state is written to the journal. + +**Confirm that no other writer remains before entering P3**: check `verification/legacy-jobs.txt` for CronJobs or Jobs that still target the legacy Registry, and if S3 is reachable, list it twice a few minutes apart — the object count and the newest `LastModified` must be identical. + +If the platform cannot keep the legacy components stopped, set `FORCE_ARGS=(--force)` only after the administrator accepts the consistency risk. That path records the actual runtime state and prints a force-fence warning. See the escape hatches in appendix B. ## 5. P3-final-inventory: freeze the inventory and S3 state -The final inventory records repository/tag/digest relationships and manifest/config metadata. The S3 snapshot records object count, bytes, and listing digest. It does not copy layers or provide data recovery. +The final inventory records the repository/tag/digest relationships together with the manifest and config metadata; the S3 snapshot records the object count, the byte count, and the listing digest. Registry v2 reuses the same bucket and prefix, so **the layers stay where they are: this stage copies nothing and deletes nothing**. It also creates no data copy — if you need a recovery point, the administrator must take an S3 snapshot or enable versioning before P3. + +**Once recorded, the P3 protection state cannot be changed**: adding proof afterwards has no effect, and the only remedy is a new run from P0. Before you start, confirm that a recovery point exists if you choose path 2, and always re-run with the same path you chose the first time. -Choose exactly one path. +Choose exactly one path. Run only one of the two blocks; running both makes path 2 overwrite path 1. -**Verified recovery point:** +**Path 1: skip the restore verification explicitly** (the administrator accepts running without an S3 recovery point; the tool records a warning and the `not-established` state) + +```bash +P3_PROTECTION=(--skip-s3-backup-verification) +``` + +**Path 2: provide a verified recovery point** (the tool records `restoreVerified` and the reference) ```bash printf 'Enter the verified S3 recovery-point identifier or report path: ' @@ -196,31 +283,37 @@ jq --arg actor "$ACTOR" --arg at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ P3_PROTECTION=(--evidence-file "$RUN_DIR/restore-proof.json") ``` -**Explicitly skipped recovery verification:** an administrator may accept a run with no S3 recovery point. The tool records a warning and `not-established` state. +Nested repositories are not skipped by default. Turn the switch on only after confirming that every nested repository in `legacy-metadata-report.json` is an approved historical test leftover: ```bash -P3_PROTECTION=(--skip-s3-backup-verification) +NESTED_ARGS=() +# NESTED_ARGS=(--skip-nested-repositories) ``` -Do not use both paths. +If the S3 endpoint in `plan.json` is a cluster-internal address (of the form `*.svc.cluster.local`), this host cannot resolve it and the stage stops with `dial tcp: lookup ... no such host`. Go to appendix A now, set up the port forward and `S3_ARGS`, then come back and run the block below. + +P3 uses `resume` rather than `run`: after an interruption, `resume` reuses the completed scan, while every other stage uses `run` (idempotent, completed items are skipped). ```bash registry-migrate resume --run-dir "$RUN_DIR" --stage P3-final-inventory \ - "${P3_PROTECTION[@]}" "${S3_ARGS[@]}" \ - "${NESTED_ARGS[@]}" \ + "${P3_PROTECTION[@]}" "${S3_ARGS[@]}" "${NESTED_ARGS[@]}" \ "${FORCE_ARGS[@]}" --kubeconfig "$KCFG" --approve + registry-migrate report --run-dir "$RUN_DIR" -jq '{items:(.items|length),digest,sample:(.items[0] | {namespace,repository,tag,digest})}' "$RUN_DIR/final-inventory.json" + +jq '{items:(.items|length),digest,sample:(.items[0] // {} | {namespace,repository,tag})}' "$RUN_DIR/final-inventory.json" ``` -Expected result: recorded stage, `final-inventory.json`, and `s3-snapshot.json`. Scanning reads tag links, manifests, and configs; it does not move layers. Approved test residue excluded with `--skip-nested-repositories` must be reviewed in `legacy-metadata-report.json` and not counted as migrated. Set `NESTED_ARGS=(--skip-nested-repositories)` before P3 only for that approved residue; the normal value is `NESTED_ARGS=()`. +Success check: the stage state is written to the journal, and `final-inventory.json` and `s3-snapshot.json` are created; the scan only reads tag links, manifests, and configs and moves no layers. +If the stage fails on nested repositories, read `legacy-metadata-report.json` first and set `NESTED_ARGS=(--skip-nested-repositories)` only when every entry is an approved historical test leftover. Excluded repositories never reach `final-inventory.json`, are not imported in P6, and cannot be added later, so review them one by one. -## 6. P4-prepare-operator: prepare Registry v2 read-only +## 6. P4-prepare-operator: prepare the read-only Registry v2 -Install or upgrade the provider-approved Operator bundle/CRD. Establish the ACP-specific pause/hold that keeps the current Operator Deployment at zero during P4–P6. A Manual Subscription controls InstallPlan approval; it does not pause an already running Operator. +Install or upgrade the provider-approved Operator bundle/CRD (in the platform plugin page; the tool does not do this) and establish the ACP pause procedure so that the Operator Deployment stays at zero replicas from P4 through P6. A Manual Subscription only controls InstallPlan approval; it does not pause an Operator that is already running. ```bash +prepare_operator() { acn -n "$OPERATOR_NS" get subscription,installplan,csv -o wide stop_deployment "$OPERATOR_NS" cluster-image-registry-operator PRUNER="$(acn get imagepruner.imageregistry.operator.alauda.io cluster --ignore-not-found -o name)" @@ -231,8 +324,14 @@ if test -n "$PRUNER"; then existing_plan="$(jq -r '.metadata.annotations["registry-migrate.plan-digest"] // empty' "$RUN_DIR/imagepruner-before.json")" current_run="$(jq -er '.runID' "$RUN_DIR/plan.json")" current_plan="$(jq -er '.digest' "$RUN_DIR/plan.json")" - test -z "$existing_run" -o "$existing_run" = "$current_run" - test -z "$existing_plan" -o "$existing_plan" = "$current_plan" + if test -n "$existing_run" && test "$existing_run" != "$current_run"; then + echo "STOP: imagepruner belongs to run $existing_run; refusing to reuse it" >&2 + return 1 + fi + if test -n "$existing_plan" && test "$existing_plan" != "$current_plan"; then + echo "STOP: imagepruner was prepared by a different plan ($existing_plan)" >&2 + return 1 + fi if test -z "$existing_run"; then acn annotate "$PRUNER" "registry-migrate.run-id=$current_run" fi @@ -250,35 +349,77 @@ acn get config.imageregistry.operator.alauda.io cluster -o json | .spec.storage.managementState == "Unmanaged"' acn get imagepruner.imageregistry.operator.alauda.io cluster -o json | jq -e '.spec.suspend == true' +} +prepare_operator +``` + +Success check: `state` in `registry-migrate report` is `prepared`; the Config reuses the planned S3 settings and routes, is read-only, and manages both the pull Secret and storage as `Unmanaged`; the Pruner is suspended. If no Pruner existed, the tool creates one and records `registry-migrate.original-suspend`. + +If the ImagePruner already carries a `run-id` or `plan-digest` annotation from another run that disagrees with the current plan, the command aborts without overwriting the value (at that point the Operator is already at zero replicas, but the stage is not recorded). Once you are sure the old run is abandoned, you may delete those two annotations with provider approval and re-run this section; **do not delete `registry-migrate.original-suspend`**, because P8 depends on it. A failing assertion means the Operator reconciled the Config again; confirm that the pause procedure is in effect before re-running this section. + +## 7. P5-remove-legacy-chart: uninstall the legacy chart + +This step cannot be undone. **First** record the current Secret list as the baseline for the comparison after the uninstall (it must run before the uninstall, otherwise the diff below is always empty): + +```bash +date -u > "$RUN_DIR/pre-uninstall-secrets.at" +acns get secret -o name > "$RUN_DIR/pre-uninstall-secrets.txt" ``` -Expected result: `state=prepared`; Config reuses planned S3/routes, is read-only, and leaves pull-Secret management Unmanaged. The Pruner is suspended. If it did not exist, the tool creates it and records `registry-migrate.original-suspend=false`. An annotation conflict is a stop condition; do not use `--overwrite`. +The platform administrator uninstalls the legacy Registry/Gateway plugin in the cluster plugin page; the tool does not uninstall anything. The uninstall deletes the resources in the chart manifest, so a TLS Secret or an S3 credential Secret generated by that chart disappears with it. Verify immediately afterwards: + +```bash +test -s "$RUN_DIR/pre-uninstall-secrets.txt" || + echo "STOP: pre-uninstall snapshot missing; the diff below proves nothing" >&2 +printf 'baseline captured at: %s\n' "$(cat "$RUN_DIR/pre-uninstall-secrets.at" 2>/dev/null || echo unknown)" +acns get secret -o name > "$RUN_DIR/post-uninstall-secrets.txt" +diff "$RUN_DIR/pre-uninstall-secrets.txt" "$RUN_DIR/post-uninstall-secrets.txt" || + echo "WARN: Secret list changed; confirm the TLS Secret and S3 credentials still exist before continuing" >&2 +``` -## 7. P5-remove-legacy-chart: uninstall the old chart +The baseline timestamp must be earlier than the uninstall; otherwise the diff proves nothing and you have to compare against the P1 backup by hand. Keep the business pull Secrets, the ServiceAccounts, the S3 data, and the TLS Secret. If a TLS or S3 credential Secret shows up in the diff, restore it before continuing. -The platform administrator uninstalls the legacy Registry/Gateway chart through the actual AppRelease/ClusterPlugin process. The tool does not perform this uninstall. Keep business pull Secrets, ServiceAccounts, S3 data, and TLS Secrets. +Then delete the leftover Ingress and record the stage. **The next block permanently deletes the Ingresses marked `gateway=true` in the plan**, so confirm the routes are backed up first: ```bash -jq -r '.discovery.routes[] | select(.gateway == true) | [.name,.ingressUID] | @tsv' "$RUN_DIR/plan.json" | +remove_legacy_chart() { +acn -n "$OPERATOR_NS" get deployment cluster-image-registry-operator -o json | + jq -e '.spec.replicas == 0' >/dev/null || + echo "WARN: operator replicas is not 0; confirm it is still paused before removing the legacy chart" >&2 +ROUTE_COUNT="$(jq -e '[.discovery.routes[] | select(.gateway == true)] | length' "$RUN_DIR/plan.json")" || + { echo "STOP: cannot read routes from plan.json" >&2; return 1; } +test "$ROUTE_COUNT" -gt 0 || + echo "WARN: plan has no gateway route; confirm no legacy Ingress remains before recording the stage" >&2 +FAILED="" while IFS=$'\t' read -r name expected_uid; do actual_uid="$(acns get ingress "$name" --ignore-not-found -o jsonpath='{.metadata.uid}')" - if test -n "$actual_uid"; then - test "$actual_uid" = "$expected_uid" - acns delete ingress "$name" --wait=true + if test -z "$actual_uid"; then + echo "SKIP: ingress $name is already absent" + continue fi -done + if test "$actual_uid" != "$expected_uid"; then + echo "STOP: ingress $name has a different UID ($actual_uid != $expected_uid); not deleting" >&2 + FAILED=1 + continue + fi + acns delete ingress "$name" --wait=true +done < <(jq -r '.discovery.routes[] | select(.gateway == true) | [.name,.ingressUID] | @tsv' "$RUN_DIR/plan.json") +test -z "$FAILED" || { echo "STOP: legacy Ingress cleanup incomplete; stage not recorded. Shell stays open." >&2; return 1; } registry-migrate run --run-dir "$RUN_DIR" --stage P5-remove-legacy-chart \ --kubeconfig "$KCFG" --approve +} +remove_legacy_chart ``` -Expected result: recorded stage. The tool confirms old Service/Deployments/routes are gone, TLS Secret remains, and business Secret/ServiceAccount identities have not changed. New Ingress uses planned `targetName` values; the old Ingress name is only a deletion target. +Success check: the stage state is written to the journal; the tool confirms that the legacy Service/Deployment/route are gone, the TLS Secret still exists, and the business Secret/ServiceAccount identities are unchanged. The new Ingress uses the planned `targetName`; the legacy Ingress names are used only as deletion targets. A UID mismatch means the Ingress was recreated: stop and contact your service provider instead of forcing the deletion. ## 8. P6-cold-import: import metadata and limits -The new Operator must still be paused. Re-check the pause and generate a fresh evidence file before importing. +The Operator must still be at zero replicas. Confirm it again and generate a fresh evidence file: ```bash stop_deployment "$OPERATOR_NS" cluster-image-registry-operator +acn -n "$OPERATOR_NS" get deployment cluster-image-registry-operator -o json | jq -e '.spec.replicas == 0' >/dev/null evidence operator-paused.json operatorPaused registry-migrate run --run-dir "$RUN_DIR" --stage P6-cold-import \ --inventory-file "$RUN_DIR/final-inventory.json" \ @@ -286,13 +427,13 @@ registry-migrate run --run-dir "$RUN_DIR" --stage P6-cold-import \ "${FORCE_ARGS[@]}" --kubeconfig "$KCFG" --approve ``` -Expected result: `state=imported`, with progress for unique ImageDigest objects, ImageStream/tag relationships, and namespace limits. Shared digests are stored once while repository/tag references remain. No blobs or layers are uploaded and no ordinary `ImageStreamMapping` bypass is used. +Success check: `state` in `registry-migrate report` is `imported`, and the output reports the unique ImageDigest objects, the ImageStream/tag relationships, and the namespace limit progress. A shared digest is stored once and keeps all of its repository/tag references. If the stage fails, confirm that the Operator is still at zero replicas and re-run it; the import counts are checked in section 9. -The legacy Gateway limits are projected into existing namespace `registry-migrate-image-limits` LimitRange: `type: alauda.io/Image` with `max.storage` for image size, and `type: alauda.io/ImageStream` with `max["alauda.io/repository-tags"]` for repository-tag count. A legacy `2GB/2000` value is represented as `2Gi/2000`. Namespace-specific rules are selected by the approved plan; review `namespaceLimits` when rules overlap. The migration does not create ResourceQuota, overwrite existing user limits, or configure future namespaces. +The legacy Gateway limits are projected onto the `registry-migrate-image-limits` LimitRange of each existing namespace: image size uses `type: alauda.io/Image` with `max.storage`, and the repository-tag count uses `type: alauda.io/ImageStream` with `max["alauda.io/repository-tags"]`. The legacy `2GB/2000` becomes `2Gi/2000` in the new limits (`2Gi=2048Mi`; `2000` means the namespace may hold at most 2000 repository tags). The approved plan selects the per-namespace rules. The migration never overwrites limits that users already set and never configures namespaces created later. ## 9. P7-verify: resume the Operator and verify read-only service -Release the ACP pause/hold through the platform procedure. Then restore the Operator and wait for the data plane. +Release the ACP pause through the platform procedure, then restore the Operator and wait for the data plane. `--replicas=1` is the default; if the change record lists a different original replica count, change that line. ```bash acn -n "$OPERATOR_NS" scale deployment cluster-image-registry-operator --replicas=1 @@ -307,133 +448,236 @@ acn get apiservice v1.image.alauda.io -o json | jq -e '.status.conditions // [] | any(.[]; .type == "Available" and .status == "True" and .reason == "Passed")' registry-migrate run --run-dir "$RUN_DIR" --stage P7-verify --kubeconfig "$KCFG" --approve registry-migrate verify --run-dir "$RUN_DIR" --kubeconfig "$KCFG" +acn get config.imageregistry.operator.alauda.io cluster -o json | + jq -e '.spec.readOnly == true and + .spec.imagePullSecret.managementState == "Unmanaged" and + .spec.storage.managementState == "Unmanaged"' +acn get imagepruner.imageregistry.operator.alauda.io cluster -o json | + jq -e '.spec.suspend == true' ``` -Expected result: Operator, Registry, Image API, and APIService are ready; P7 is `verified` and `verify` exits zero. Confirm Service ports `80→8080`, `443→5000`, and `5000→5000`, Ready endpoints, the new Ingress backend, Config, metadata, and limits. Config remains read-only, storage and pull-Secret management remain Unmanaged, and the Pruner remains suspended. +Success check: the Operator, Registry, Image API, and APIService are all ready; the P7 state is `verified` and `verify` exits 0. The Service ports are `80→8080`, `443→5000`, and `5000→5000`, the endpoints are Ready, and the new Ingress backend, Config, metadata, and limits are correct; the Config is still read-only, storage and the pull Secret are still `Unmanaged`, and the Pruner is still suspended. + +**This is the most important checkpoint in the maintenance window: business pulls should work again, and pushes must be rejected (403/405).** If pulls fail, check whether the pull Secret contains the new ingress host (see section 10), the Service ports, and the Ingress backend first. Conversely, if a push succeeds at this stage, `readOnly` is not in effect: stop and do not enter P8. -Confirm the release-4.5 metadata resources and the namespace limit scope: +Collect the verification evidence: ```bash -acn get imagedigests.image.alauda.io -o yaml > "$RUN_DIR/verification/imagedigests.yaml" -acn get imagestreams.image.alauda.io -A -o yaml > "$RUN_DIR/verification/imagestreams.yaml" -acn get limitrange -A -l "registry-migrate.run-id=$RUN_ID" -o yaml \ - > "$RUN_DIR/verification/migration-limitranges.yaml" +mkdir -p "$RUN_DIR/verification" +acn get imagedigests.image.alauda.io -o yaml > "$RUN_DIR/verification/imagedigests.yaml" || + echo "WARN: imagedigests not captured" >&2 +acn get imagestreams.image.alauda.io -A -o yaml > "$RUN_DIR/verification/imagestreams.yaml" || + echo "WARN: imagestreams not captured" >&2 +acn get limitrange -A --field-selector metadata.name=registry-migrate-image-limits -o yaml \ + > "$RUN_DIR/verification/migration-limitranges.yaml" || + echo "WARN: limitranges not captured" >&2 ``` -The imported `LimitRange` objects are a static snapshot for namespaces already covered by the plan. They do not become a global policy and do not automatically apply to namespaces created later. +Compare the import result. A shared digest is stored once, so compare digest sets rather than repository/tag counts: + +```bash +jq -r '.items[].digest | select(. != null)' "$RUN_DIR/final-inventory.json" | sort -u \ + > "$RUN_DIR/verification/inventory-digests.txt" +acn get imagedigests.image.alauda.io -o json | + jq -r '.items[].metadata.name' | sort -u > "$RUN_DIR/verification/cluster-digests.txt" +wc -l < "$RUN_DIR/verification/inventory-digests.txt" +comm -23 "$RUN_DIR/verification/inventory-digests.txt" "$RUN_DIR/verification/cluster-digests.txt" +``` + +The first command prints the number of digests in the inventory. `comm -23` lists the digests that are in the inventory but missing from the cluster, and **the output must be empty**; if it is not, find out whether those images were skipped or failed to import. The opposite direction (present in the cluster but not in the inventory) is allowed, because those objects may be historical or created by the Operator, but record the reason in the migration notes. + +`LimitRange` is a static snapshot of the namespaces the plan already covers. It is not a global policy and is not applied automatically to namespaces created later. ## 10. P8-commit: verify business access, then open writes -Use the actual external URL for this cluster, including protocol and port. Do not copy a test environment port to another cluster. A trusted system CA or provider PEM CA is required for TLS acceptance; `curl -k` is only a diagnostic. Set the three test inputs for this cluster before running the probe block. +`REGISTRY_URL` must point at **the Ingress address that the new Registry exposes**, that is, the entry point used to reach the Registry from outside the cluster. It defaults to the planned route (`scheme` + `hostname`) and needs no manual input; when the real entry point is a different domain or VIP, override it with `REGISTRY_URL_OVERRIDE` instead of editing this block. The port comes from this cluster's plan, so do not carry over a port from another environment. + +Certificates fall into two cases: + +- The ingress certificate is issued by a trusted CA: `export REGISTRY_CA_BUNDLE=/path/to/ca.pem`. curl uses it for verification, and `ac` inherits the same CA through `SSL_CERT_FILE`. +- The ingress certificate is issued by an internal CA whose SAN does not include this host (for example an auto-generated `*.nip.io` domain): TLS verification is bound to fail, so `export REGISTRY_TLS_INSECURE=1` to skip it and note this environment limitation in the migration record. ```bash -export REGISTRY_URL="${REGISTRY_URL:?set the actual external Registry URL, including scheme and port}" -REGISTRY_HOST="$(jq -er '.discovery.routes[0].hostname' "$RUN_DIR/plan.json")" +export REGISTRY_HOST="$(jq -er '([.discovery.routes[] | select(.gateway == true)][0] // .discovery.routes[0]).hostname' "$RUN_DIR/plan.json")" +REGISTRY_SCHEME="$(jq -er '([.discovery.routes[] | select(.gateway == true)][0] // .discovery.routes[0]).scheme // "http"' "$RUN_DIR/plan.json")" +unset REGISTRY_URL +export REGISTRY_URL="${REGISTRY_URL_OVERRIDE:-$REGISTRY_SCHEME://$REGISTRY_HOST}" +printf 'REGISTRY_URL=%s\n' "$REGISTRY_URL" jq -ne --arg url "$REGISTRY_URL" --arg host "$REGISTRY_HOST" \ - '($url | capture("^https?://(?[^/:]+)(:[0-9]+)?/?$").host) == $host' + '($url | capture("^https?://(?[^/:]+)(:[0-9]+)?/?$").host) == $host' >/dev/null 2>&1 || + echo "NOTE: REGISTRY_URL is not the bare planned ingress host ($REGISTRY_HOST); confirm it is the external address of the new Registry." mkdir -p "$RUN_DIR/verification" CURL_TLS_ARGS=() -if test -n "$REGISTRY_CA_BUNDLE"; then CURL_TLS_ARGS=(--cacert "$REGISTRY_CA_BUNDLE"); fi -http_code="$(curl -sS "${CURL_TLS_ARGS[@]}" --connect-timeout 10 --max-time 20 \ - -o /dev/null -w '%{http_code}' "$REGISTRY_URL/v2/")" -case "$http_code" in 200|401|403) echo "Registry reachable: HTTP $http_code" ;; *) echo "unexpected HTTP $http_code"; exit 1 ;; esac -curl -sS "${CURL_TLS_ARGS[@]}" --connect-timeout 10 --max-time 20 \ - -D "$RUN_DIR/verification/registry-v2-headers.txt" -o /dev/null "$REGISTRY_URL/v2/" -if grep -qi '^WWW-Authenticate:' "$RUN_DIR/verification/registry-v2-headers.txt"; then - echo "Review the WWW-Authenticate realm and confirm its host and port are reachable from the same client." +AC_TLS_ARGS=() +if test -n "${REGISTRY_CA_BUNDLE:-}"; then + CURL_TLS_ARGS+=(--cacert "$REGISTRY_CA_BUNDLE") + export SSL_CERT_FILE="$REGISTRY_CA_BUNDLE" fi -export TEST_NS="${TEST_NS:?set a namespace containing a migrated test image}" -export TEST_REPO="${TEST_REPO:?set an isolated namespace/repository:tag for dry-run deletion}" - -env AC_IMAGE_REGISTRY_MODE=legacy \ - ac --kubeconfig "$KCFG" --context "$CTX" get images -n "$TEST_NS" -o json \ - --registry-url "$REGISTRY_URL" | tee "$RUN_DIR/verification/legacy-images.json" -env AC_IMAGE_REGISTRY_MODE=modern \ - ac --kubeconfig "$KCFG" --context "$CTX" get images -o json \ - | tee "$RUN_DIR/verification/modern-images.json" -env AC_IMAGE_REGISTRY_MODE=legacy \ - ac --kubeconfig "$KCFG" --context "$CTX" delete images \ - --repo "$TEST_REPO" --registry-url "$REGISTRY_URL" -env AC_IMAGE_REGISTRY_MODE=modern \ - ac --kubeconfig "$KCFG" --context "$CTX" delete images --repo "$TEST_REPO" -``` +if test "${REGISTRY_TLS_INSECURE:-0}" = 1; then + CURL_TLS_ARGS+=(--insecure) + AC_TLS_ARGS+=(--insecure-skip-tls-verify) +fi +probe_and_export() { + http_code="$(curl -sS "${CURL_TLS_ARGS[@]}" --connect-timeout 10 --max-time 20 \ + -o /dev/null -w '%{http_code}' "$REGISTRY_URL/v2/")" + case "$http_code" in + 200|401|403) echo "Registry reachable: HTTP $http_code" ;; + *) echo "ERROR: unexpected HTTP ${http_code:-000} from $REGISTRY_URL/v2/" >&2 + echo "000 = this machine cannot reach the host (route/VPN/DNS); 502/504 = a proxy or the ingress cannot reach the backend." >&2 + echo "The shell stays open. Fix the cause, then re-run this block." >&2 + return 1 ;; + esac + curl -sS "${CURL_TLS_ARGS[@]}" --connect-timeout 10 --max-time 20 \ + -D "$RUN_DIR/verification/registry-v2-headers.txt" -o /dev/null "$REGISTRY_URL/v2/" + if grep -qi '^WWW-Authenticate:' "$RUN_DIR/verification/registry-v2-headers.txt"; then + echo "Check the host and port of the WWW-Authenticate realm; see appendix A if they differ from the ingress." >&2 + fi -Expected result: anonymous `/v2/` 401/403 proves only entry-point reachability; authenticated legacy and modern queries succeed, and both delete commands remain dry-run. The legacy view lists repository/tag data; the modern view uses ImageDigest/ImageStream data. Do not compare row counts as proof of completeness. + TEST_NS="${TEST_NS:-$(jq -er '.items[0].namespace' "$RUN_DIR/final-inventory.json")}" || + { echo "ERROR: final-inventory.json has no items; P3 produced no inventory" >&2; return 1; } + TEST_REPO="${TEST_REPO:-$(jq -r '.items[0] | "\(.namespace)/\(.repository):\(.tag)"' "$RUN_DIR/final-inventory.json")}" + export TEST_NS TEST_REPO + + env AC_IMAGE_REGISTRY_MODE=legacy \ + ac --kubeconfig "$KCFG" --context "$CTX" "${AC_TLS_ARGS[@]}" get images -n "$TEST_NS" -o json \ + --registry-url "$REGISTRY_URL" > "$RUN_DIR/verification/legacy-images.json" || + { echo "ERROR: legacy view failed; see the error above. The shell stays open." >&2; return 1; } + env AC_IMAGE_REGISTRY_MODE=modern \ + ac --kubeconfig "$KCFG" --context "$CTX" "${AC_TLS_ARGS[@]}" get images -A -o json \ + --registry-url "$REGISTRY_URL" > "$RUN_DIR/verification/modern-images.json" || + { echo "ERROR: modern view failed; see the error above. The shell stays open." >&2; return 1; } + printf 'legacy=%s modern=%s\n' \ + "$(jq '.items | length' "$RUN_DIR/verification/legacy-images.json" 2>/dev/null || echo n/a)" \ + "$(jq '.items | length' "$RUN_DIR/verification/modern-images.json" 2>/dev/null || echo n/a)" + + # Keep these as dry runs: never add --confirm / --yes / -f, or the tag is really deleted. + env AC_IMAGE_REGISTRY_MODE=legacy \ + ac --kubeconfig "$KCFG" --context "$CTX" "${AC_TLS_ARGS[@]}" delete images \ + --repo "$TEST_REPO" --registry-url "$REGISTRY_URL" + env AC_IMAGE_REGISTRY_MODE=modern \ + ac --kubeconfig "$KCFG" --context "$CTX" "${AC_TLS_ARGS[@]}" delete images \ + --repo "$TEST_REPO" --registry-url "$REGISTRY_URL" +} +probe_and_export +``` -The commands above use the release-4.5 compatible `ac`. When the original 4.3 client is part of the customer contract, run the same read and dry-run checks separately with `AC43_BIN`, an explicit `--registry-url "$REGISTRY_URL"`, and an isolated kubeconfig. Do not call `ac config set-registry-mode` with the original 4.3 binary. The compatibility binary keeps the 4.3 output shape by default; setting `AC_REGISTRY_COMPAT_OUTPUT=native` changes the output contract and is not valid evidence for old scripts. +Success check: an anonymous 401/403 from `/v2/` only proves that the entry point is reachable; it says nothing about authentication or read/write access. The stage passes only when the authenticated legacy and modern queries both succeed and both `delete images` calls remain dry runs (they print `To delete, run with --confirm flag`). The legacy view lists repository/tag data; the modern view uses ImageDigest/ImageStream data. -For modern-native diagnostics only, query the Image API resource directly: +Next, check that every namespace's pull Secret contains the new ingress host: ```bash -env AC_IMAGE_REGISTRY_MODE=modern AC_REGISTRY_COMPAT_OUTPUT=native \ - ac --kubeconfig "$KCFG" --context "$CTX" get images -o json \ - > "$RUN_DIR/verification/modern-native-images.json" +acn get secret -A -o json | + jq -r --arg h "$REGISTRY_HOST" ' + .items[] | select(.type == "kubernetes.io/dockerconfigjson") | + (.data[".dockerconfigjson"] | @base64d | fromjson? | (.auths // {}) | keys) as $k | + select(($k | any(test($h))) | not) | + "REVIEW \(.metadata.namespace)/\(.metadata.name): no auth entry for \($h)"' ``` -Do not use that native output to compare with the original 4.3 pipeline contract. Before enabling prune or GC, verify that the caller has platform-admin access to the Registry Config; a namespace-level tag administrator is not sufficient. +No output means every pull Secret already points at the new entry point. When a `REVIEW` line appears, namespaces managed by the Operator refresh on their own, so wait a minute or two and run the check again; Secrets left over from before the migration that the Operator does not manage are never rewritten, so the administrator must confirm that Pods in those namespaces can still pull. -Create a disposable test Pod with `imagePullPolicy=Always` using the original image reference, ServiceAccount and Secret. Verify pulls through the original `:80`, `:443`, and `:5000` paths and the external host as applicable. Record the Pod, digest, query output, and pull result. Do not perform a positive push before P8. +If you plan to enable the Pruner or GC after the migration, first confirm that the caller has platform-admin rights on the Registry Config. `ac adm prune images` is a platform-level operation that acts on the whole cluster's Registry, so **do not run it as part of this procedure**; adding and removing tags is a namespace permission, and the dry runs above work with the current login identity. -After the business checks pass and an administrator approves the cutover: +Run the block below only after the business checks pass and the administrator approves the switch. It is the irreversible point where writes are opened: rolling back afterwards also means handling new images and credential rotation, so make sure there is no way back before you continue. The evidence it writes sets `businessProbeVerified` and `pullSecretHandoffVerified` directly, so **complete the queries above and the pull Secret check first**. ```bash +commit_run() { +test -n "${ACTOR:-}" || { echo "ERROR: ACTOR is empty; export ACTOR= and re-run this block" >&2; return 1; } jq --arg actor "$ACTOR" --arg at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ '{version:"v1",runID:.runID,planDigest:.digest,observedAt:$at,actor:$actor,commitApproved:true,businessProbeVerified:true,pullSecretHandoffVerified:true}' \ "$RUN_DIR/plan.json" > "$RUN_DIR/commit-approval.json" +jq -e '.actor | length > 0' "$RUN_DIR/commit-approval.json" >/dev/null || + { echo "ERROR: commit-approval.json has an empty actor; the tool rejects it" >&2; return 1; } registry-migrate run --run-dir "$RUN_DIR" --stage P8-commit \ --evidence-file "$RUN_DIR/commit-approval.json" --kubeconfig "$KCFG" --approve acn get config.imageregistry.operator.alauda.io cluster -o json | - jq -e '.spec.readOnly == false and .spec.imagePullSecret.managementState == "Managed"' + jq -e '.spec.readOnly == false and + .spec.imagePullSecret.managementState == "Managed" and + .spec.storage.managementState == "Unmanaged"' || + echo "NOTE: the commit itself may have succeeded and only this Config assertion failed; check the state in registry-migrate report before considering rollback" >&2 acn get imagepruner.imageregistry.operator.alauda.io cluster -o json | jq -e ' - .metadata.annotations["registry-migrate.original-suspend"] as $original | - ($original == "true" or $original == "false") and (.spec.suspend == ($original == "true"))' + (.metadata.annotations["registry-migrate.original-suspend"] // "false") as $original | + .spec.suspend == ($original == "true")' || + echo "NOTE: if the actual suspend value disagrees with the annotation, compare imagepruner-before.json from P4 before considering rollback" >&2 registry-migrate verify --run-dir "$RUN_DIR" --kubeconfig "$KCFG" registry-migrate report --run-dir "$RUN_DIR" +} +commit_run ``` -Expected result: `state=committed`; Config is writable, storage and pull-Secret management are Managed, and the Pruner has its recorded original suspension value. After Operator reconciliation, use an approved OCI client to push and pull one disposable repository and retain that output. If a failure occurs now, the write transition has already been committed; do not assume read-only state. +Success check: `state` in `registry-migrate report` is `committed`; the Config is writable (`readOnly=false`), pull Secret management becomes `Managed`, storage stays `Unmanaged` (the administrator keeps managing S3), and `spec.suspend` on the Pruner returns to the value recorded before the migration (`registry-migrate.original-suspend`; a pre-migration value of `false` returns to `false`). A failing assertion does not mean the commit failed: check `state` before deciding to roll back. + +Finally, push to and pull from a throwaway repository with an approved OCI client and keep the output. When the Registry authentication realm disagrees with the ingress port (appendix A), `skopeo`, `docker`, and `containerd` fail; in that case verify the manifest and blob `sha256` values through the Registry v2 API with `curl` instead. -## Appendix: S3 access, retries, exceptions, and recovery +## Appendix A: known environment limitations -### S3 available only inside the cluster +### S3 reachable only from inside the cluster + +If the S3 endpoint in `plan.json` is a cluster-internal address (of the form `*.svc.cluster.local`), this host cannot resolve it and P3 stops while listing objects with `dial tcp: lookup ... no such host`. Set up a port forward first. + +Run this in **a second terminal** (it blocks in the foreground, so do not use the migration terminal). Replace `minio-system/service/minio` with the real S3 Service name and port; if the migration terminal uses a context other than `current-context`, export `CTX` in the new terminal first: ```bash -ac --kubeconfig "/path/to/business.kubeconfig" --context "actual-context" \ - -n minio-system port-forward service/minio 19000:9000 +: "${KCFG:=$HOME/.kube/config}" +: "${CTX:=$(ac --kubeconfig "$KCFG" config current-context)}" +ac --kubeconfig "$KCFG" --context "$CTX" -n minio-system port-forward service/minio 19000:9000 ``` -In the original terminal: +Back in the **migration terminal**, confirm that the tunnel works and set the override: ```bash -curl --fail --connect-timeout 5 --max-time 10 \ - http://127.0.0.1:19000/minio/health/live +curl --fail --connect-timeout 5 --max-time 10 http://127.0.0.1:19000/minio/health/live S3_ARGS=(--s3-endpoint http://127.0.0.1:19000) ``` -The override changes only the P3 transport address; the source endpoint remains in the plan/snapshot and is not replaced in the target Config. Rebuild a broken tunnel and retry the same run. An HTTPS tunnel still uses `https://`; port forwarding does not convert the protocol. +The override only changes the transport address used by P3; the endpoint in the plan, in the snapshot, and in the target Config stays the same. An HTTPS tunnel still uses `https://`, because a port forward does not change the protocol. Rebuild the tunnel if it drops, then retry the same run. + +### Authentication realm and ingress port disagree -### Exceptions and recovery +The `WWW-Authenticate: realm=...` address in the 401 response from `/v2/` may use a different port from the ingress (for example the realm says 443 while the ingress only exposes 11780). Authentication in `skopeo`, `docker`, and `containerd` then fails even though the Registry itself is healthy. The `REGISTRY_URL` used here requires the definition block in section 10. + +```bash +curl -sS -k -D - -o /dev/null "$REGISTRY_URL/v2/" | grep -i '^www-authenticate:' +``` -- **Force writer fence:** after an administrator accepts the consistency risk, set `FORCE_ARGS=(--force)` before P1/P2/P3/P4/P6. It cannot bypass API, UID, S3, metadata, or stage checks, and never bypasses the new Operator pause in P4/P6. -- **Skip S3 verification:** set `P3_PROTECTION=(--skip-s3-backup-verification)` only when the administrator accepts that no S3 recovery point was established. -- **Nested test residue:** add `--skip-nested-repositories` to P3 only for approved historical test residue. Review `legacy-metadata-report.json`. -- **Original 4.3 ac:** it has no `set-registry-mode`; use `--registry-url`, `--kubeconfig`, and `--context` for a legacy query. The full permission/pagination matrix belongs to the platform’s separate compatibility acceptance record, not to the production migration window. +Replace `-k` with `--cacert /path/to/ca.pem` when the ingress certificate is trusted. If the realm is unreachable, verify through the Registry v2 API with `curl`; do not conclude that the migration failed, because this is an entry-point configuration issue that has to be fixed separately. -If the Bash terminal exits, export the original absolute run directory and run Section 0 again: +## Appendix B: retries, exceptions, and recovery + +### Continue the same run after the terminal exits ```bash export RUN_DIR="/absolute/path/to/registry-migrate-runs/original-run-id" +# Re-set the arrays you need, as described in section 5 (skip the ones you do not use): +# S3_ARGS=(--s3-endpoint http://127.0.0.1:19000) +# FORCE_ARGS=(--force) +# NESTED_ARGS=(--skip-nested-repositories) ``` -Section 0 restores the original run ID, scope, and discovered Deployments. Re-select S3, force, and P3 protection arrays. Do not create a new run or repeat completed manual uninstall/Ingress deletion steps. +Then run section 0 again; it restores the run ID, the namespaces, and the Deployment names from `plan.json`. `P3_PROTECTION` must match the first choice. Do not start a new run, and do not repeat the manual uninstall or the Ingress deletion you already completed. + +### Escape hatches + +| Switch | When to use it | Cost | +| --- | --- | --- | +| `FORCE_ARGS=(--force)` | The platform cannot keep the legacy components stopped and the administrator accepts the consistency risk | It does not bypass the API, UID, S3, metadata, or stage checks, and it never bypasses the P4/P6 Operator pause check; the output carries a force-fence warning, and you must not claim that the legacy Pods or endpoints are gone | +| `P3_PROTECTION=(--skip-s3-backup-verification)` | The administrator accepts continuing without an S3 recovery point | Records a warning and the `not-established` state, and cannot be repaired afterwards | +| `NESTED_ARGS=(--skip-nested-repositories)` | Every nested repository in `legacy-metadata-report.json` is an approved historical test leftover | Excluded repositories do not count towards the migration result and must be reviewed one by one | -The tool is idempotent within one run: completed Secrets, metadata, limits, and checks are skipped or revalidated. Preserve plan, journal, backups, inventory, snapshot, evidence, and verification output. P3’s protection state is immutable; adding proof later does not change a run that originally skipped verification. +`ac` 4.3 has no `set-registry-mode`; legacy queries use `--registry-url`, `--kubeconfig`, and `--context`. The full permission and pagination matrix belongs to the platform's separate compatibility acceptance record, not to the production migration window. -For recovery: +### Rollback + +`rollback` **is not an automatic restore**. It does three things only: it records the rollback request in the journal, locks the run so that no later stage can advance, and prints the manual recovery checklist. It does not reinstall the legacy chart, does not recreate Secrets with their original UIDs, does not delete Registry v2 resources, and does not restore S3 objects. The actual recovery is the manual work below. ```bash registry-migrate rollback --run-dir "$RUN_DIR" --kubeconfig "$KCFG" --approve registry-migrate report --run-dir "$RUN_DIR" ``` -`rollback` records a manual recovery request and protects source data. It does not reinstall the old chart, recreate an original-UID Secret, delete Registry v2 resources, or restore S3 objects. Before chart removal, restore old replicas, Pruner/platform state, credentials, and chart configuration from the saved materials. After chart removal, stop new Registry writes and reconciliation first, then manually restore chart/platform configuration, credentials, and S3 access. After P8, account for new images and credential rotation; do not point the same S3 writer back at the legacy Registry without a controlled recovery plan. After a real recovery, start a new run ID and do not edit plan/journal to bypass identity drift. +- **Before uninstalling the chart**: restore the original replica counts, the Pruner and platform state, the credentials, and the chart configuration from the change record and the P1 backup. +- **After uninstalling the chart**: stop writes to and reconciliation of the new Registry first, then restore the chart/platform configuration, the credentials, and S3 access by hand. +- **After P8**: new images and credential rotation also need handling. Without a controlled recovery plan, do not point the same S3 writer back at the legacy Registry. +- **Start a new run ID** after the recovery, and do not bypass identity drift by editing the plan or the journal. + +The tool is idempotent within a run: completed Secrets, metadata, limits, and checks are skipped or re-verified. Keep the plan, the journal, the backup, the inventory, the snapshot, the evidence, and the verification output. From 24057b1505d177deefbae0bef58c22be891d6077 Mon Sep 17 00:00:00 2001 From: huizhang Date: Wed, 4 Nov 2026 10:49:45 +0800 Subject: [PATCH 06/12] docs: clarify Registry migration checkpoints and recovery --- .../ACP_Registry_v2_Migration_Runbook.mdx | 199 +++++++++--------- 1 file changed, 104 insertions(+), 95 deletions(-) diff --git a/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx b/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx index 2db10d6f0..f95865f04 100644 --- a/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx +++ b/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx @@ -9,45 +9,45 @@ ProductsVersion: # ACP Registry v2 Migration Runbook -This runbook migrates the Registry in one ACP business cluster from the legacy chart release to the Registry v2 instance managed by `cluster-image-registry-operator`. The migration is performed by the standalone `registry-migrate` tool. Only one cluster is involved, and Registry v2 reuses the same S3 bucket, endpoint, and prefix as the legacy Registry. +This runbook migrates one ACP business cluster from the legacy chart Registry to Registry v2, managed by `cluster-image-registry-operator`. The standalone `registry-migrate` tool imports metadata while Registry v2 reuses the same S3 bucket, endpoint, and prefix. -**This procedure deletes no images.** On the resource side there are only two delete operations: uninstalling the legacy chart in P5 (performed by the platform administrator in the plugin page) and deleting the leftover Ingress. `ac delete images` in P8 is a dry run; never add `--confirm`. +The commands do not directly delete images or run GC. P5 removes the legacy chart resources and its leftover Ingress; the `ac delete images` calls in P8 are dry runs. Never add `--confirm`. P8 also restores the recorded Pruner suspension state: if that state is `false`, scheduled image cleanup can resume. Confirm that behavior before approving the cutover. ## Stage overview -| Stage | Purpose | Service interruption | Rerunnable on failure | +| Stage | Purpose | Service interruption | Retry after failure | | --- | --- | --- | --- | -| 0 | Validate the tools, the login session, and the cluster context | No | Yes | -| P0 plan | Generate the read-only plan and check the blockers | No | Yes | -| P1-backup | Save the recovery material | No | Yes | -| P1-credentials | Hand over the pull Secret controller | No | Yes | -| P2-stop-writers | Stop the legacy writers (**maintenance window starts**) | **Yes** | Yes | -| P3-final-inventory | Freeze the image inventory and the S3 snapshot | **Yes** | Yes, but the protection state is immutable | -| P4-prepare-operator | Scale the Operator to zero and create the read-only Registry v2 | **Yes** | Yes | -| P5-remove-legacy-chart | Uninstall the legacy plugin by hand; the tool deletes the leftover Ingress | **Yes** | The commands can be re-run; the uninstall must be completed by hand | -| P6-cold-import | Import the metadata and the namespace limits | **Yes** | Yes | -| P7-verify | Restore the Operator and verify the read-only service | Partially restored | Yes | -| P8-commit | Verify business access and open writes (**maintenance window ends**) | Restored | Yes | - -The table above is the route map for the whole procedure. Start at section 0. +| 0 | Validate tools, credentials, and the business cluster context | No | Yes | +| P0 plan | Generate and review the read-only plan | No | Yes | +| P1-backup | Save resource identities and prepare recovery material | No | Yes | +| P1-credentials | Hand off pull Secrets in place | No | Yes | +| P2-stop-writers | Stop legacy writers (**maintenance window starts**) | **Yes** | Yes | +| P3-final-inventory | Freeze the image inventory and S3 object listing | **Yes** | Yes; a recorded protection choice is immutable | +| P4-prepare-operator | Hold the Operator at zero replicas and prepare read-only Config | **Yes** | Yes | +| P5-remove-legacy-chart | Uninstall the legacy plugin, then remove its leftover Ingress | **Yes** | Retry checks; do not repeat a completed uninstall | +| P6-cold-import | Import metadata and namespace limits | **Yes** | Yes | +| P7-verify | Resume the Operator and verify read-only service | Read access restored | Yes | +| P8-commit | Verify business access and open writes (**maintenance window ends**) | Restored | Inspect the recorded and live state first | + +Use the same run directory for retries. Start at section 0. ## Prerequisites -- **Tool versions**: obtain the provider-approved `registry-migrate` binary, a compatible `ac` binary, the Registry v2 bundle, and the S3 backup/restore procedure from your ACP service provider. The `registry-migrate` version is independent of the ACP release. -- **Namespaces**: the Operator is installed in `image-registry-system`, and the Registry instance (the operand namespace) runs in `cpaas-system`. The operand namespace is fixed at installation time and cannot be inferred from the Operator namespace or from environment variables; always use `scope.operandNamespace` in `plan.json`. -- **Stop immediately if any of the following is present**, unless the provider has approved a separate migration path: PVCs, HostPath volumes, non-image OCI artifacts, or a non-default S3 root/prefix. -- **The Operator must stay at zero replicas between P4 and P6.** Approving an InstallPlan by hand only controls whether OLM installs the bundle; it does not pause an Operator that is already running. If the platform cannot provide a verified pause procedure, abort the migration before P4; otherwise the Operator creates its own default Config, storage, and Pruner and breaks the import boundary. -- **`ac` compatibility**: `ac get images` and `ac delete images` still use the legacy chart Registry repository/tag contract by default, controlled by `AC_REGISTRY_COMPAT_OUTPUT` (leave it unset or set it to `4.3`). P8 uses `AC_IMAGE_REGISTRY_MODE=legacy/modern` to select a view; the two settings do not replace each other. +- **Tools and recovery procedures**: contact your ACP service provider for approved `registry-migrate` and compatible `ac` binaries, the Registry v2 bundle, and the platform pause and S3 recovery procedures. The migration tool version is independent of the ACP release. +- **Namespaces**: the default Operator installation namespace is `image-registry-system`; the default Registry operand namespace is `cpaas-system`. The operand namespace is fixed at installation. Confirm the actual values and use `scope.operandNamespace` in `plan.json` throughout the migration. +- **Storage scope**: stop if the legacy Registry's image-data backend uses PVCs or HostPath, or the migration contains non-image OCI artifacts or a non-default S3 root/prefix, unless the provider has approved another procedure. HostPath mounts used only by node trust or resolver components are not image-data storage. +- **Operator pause**: the Operator must remain at zero replicas during P4–P6. Manual InstallPlan approval does not pause an already running Operator. Establish the provider-approved pause before P4 and before removing the legacy chart. +- **CLI compatibility**: `AC_IMAGE_REGISTRY_MODE=legacy/modern` selects the mode, but an explicit `--registry-url` can keep requests on the Registry HTTP compatibility path. `AC_REGISTRY_COMPAT_OUTPUT` selects the output contract; leave it unset or set it to `4.3` for these checks. ## 0. Prepare the terminal and tools -Use a dedicated terminal and **run every step in it**. You must have completed `ac login` before this section: an ACP login token usually expires after 24 hours, and once it does every cluster request returns `Unauthorized`. When unsure, list the sessions first: +Use one dedicated **Bash** terminal for all migration steps; do not enable `set -u` in an interactive zsh session. Login-token lifetime depends on the platform identity provider. If requests return `Unauthorized`, refresh the login associated with the kubeconfig used for migration. List the available sessions first: ```bash ac config get-sessions ``` -The row marked with `*` in the `CURRENT` column is the active session. Use its `PLATFORM` and `SESSION` columns to log in again: +The row marked `*` in `CURRENT` is the active session. Use its `PLATFORM` and `SESSION` values to log in again: ```bash printf 'PLATFORM: '; IFS= read -r PLATFORM @@ -55,7 +55,7 @@ printf 'SESSION: '; IFS= read -r SESSION ac login "$PLATFORM" --name "$SESSION" ``` -Then run the block below. It validates the tools, confirms the authentication state, selects a context, and defines the variables and functions that every later step uses. If the kubeconfig is not at `~/.kube/config`, change only the `export KCFG=` line. +The block below validates tools and credentials and selects a context. For a separate business kubeconfig, add `--kubeconfig /absolute/path/to/business.kubeconfig` to the login command above and use the same path in the block's `export KCFG=` line. Setup clears earlier `KCFG` values; login to the default file does not refresh a separate copy. ```bash setup_run() { @@ -111,11 +111,11 @@ printf 'context=%s\nrun-dir=%s\n' "$CTX" "$RUN_DIR" setup_run || echo "STOP: section 0 aborted; fix the error above and re-run this block. The shell stays open." >&2 ``` -Every later step depends on the variables and functions defined here, and any failure only prints `STOP`; the terminal stays open. When the block finishes, confirm that the `context=` line it prints is the business cluster for this migration. Fix the cause and re-run the block if needed (a re-run resets `S3_ARGS` and `FORCE_ARGS`). +Confirm that `context=` identifies the intended business cluster. Before P0, select it with `ac --kubeconfig "$KCFG" config use-context "$CTX"`: the `ac` helpers use `CTX`, but `registry-migrate` uses the file's `current-context`. -If the terminal closes unexpectedly, export the original `RUN_DIR` as described in appendix B and run this section again; it restores the run ID, the namespaces, and the Deployment names from `plan.json`. Do not start a new run. +Setup creates `OUT`; successful P0 creates `$RUN_DIR`. For retries, restore the original absolute `RUN_DIR` (appendix B), then reselect the required S3, force, nested-repository and P3 protection options. Do not discard a valid plan to bypass a failure. -The next block is mandatory. It defines `acn`, `acns`, `evidence`, and `stop_deployment`, which the later sections call. +The next block defines mandatory helpers. An error, timeout, or `STOP` message means the step needs attention even if the shell remains open or a later command prints output. Confirm the stage's success check before entering the next section. ```bash acn() { @@ -155,29 +155,29 @@ jq '{scope,storage:.discovery.storage,routes:.discovery.routes,limits:.namespace echo "P0 OK: $RUN_DIR/plan.json" ``` -Seeing `P0 OK` means the plan was generated and passed the blocker check; if the line is missing, the step failed. A failing `&&` chain only returns a non-zero exit code and does not close the terminal. When blockers are present, inspect them first: +`P0 OK` means the plan was generated and the blocker check passed. If it is absent, stay at P0 and inspect the error or blockers: ```bash jq '.discovery.blockers' "$RUN_DIR/plan.json" ``` -Planning is a read-only operation. Check every blocker, the S3 bucket/endpoint/prefix, the legacy host/TLS/IngressClass, the replica counts, and the timeouts, taking the expected values from the change record and the current cluster state; stop if the plan disagrees with the cluster. The `scope.operandNamespace` it prints must be the namespace that actually holds the Registry; if it is not, correct `REGISTRY_NS` in section 0 and re-run P0. +Planning is read-only. Compare the scope, S3 bucket/endpoint/prefix, original host/TLS/IngressClass, replica counts, scheduling, and timeouts with the change record and live cluster. Stop on any mismatch. If `scope.operandNamespace` is wrong, correct `REGISTRY_NS` in section 0 before planning again. ## 2. P1-backup: save recovery material -**This stage must complete before P1-credentials**: the credentials stage rewrites pull Secret contents, and reversing the order leaves the backup snapshot without recovery value. +Complete this stage **before P1-credentials**, which rewrites pull Secret contents in place. The tool creates a redacted Registry-scoped snapshot of resource identities, owners, ServiceAccounts, references, and data digests. It contains **neither Secret data nor S3 image data**. -The backup is a redacted, Registry-scoped snapshot that records UIDs, owners, ServiceAccounts, and references; it contains no image layer data. Run: +Separately preserve chart versions/values, platform objects and protected Secret data through the approved recovery procedure. Keep them private and outside the redacted backup. Recreating a Secret or ServiceAccount cannot restore its UID. Confirm recovery and retention arrangements before chart removal. ```bash registry-migrate backup --run-dir "$RUN_DIR" --kubeconfig "$KCFG" --approve ``` -Success check: the command exits 0 and adds a resource/Secret summary file under `$RUN_DIR`. If it exits non-zero, do not continue to P1-credentials; check RBAC and re-run this section. +Success check: the command exits 0 and writes the resource and Secret summaries under `$RUN_DIR`. On failure, resolve the reported error and retry P1-backup before continuing. ## 3. P1-credentials: hand off pull Secrets -The legacy Registry must still be readable. First suspend the AppRelease/ClusterPlugin or other platform source that manages `sa-imagepullsecret-controller`, then run the block below. If jq reports several candidates, identify the single target by hand with `acn get deployments -A -o wide | grep -i imagepull` and re-run this section. +Keep the legacy Registry readable. First pause the platform source managing `sa-imagepullsecret-controller` so it cannot restart the controller. If discovery reports multiple candidates, use `acn get deployments -A -o wide | grep -i imagepull` to identify the intended controller before proceeding. ```bash PULL_SECRET_REF="$(acn get deployments -A -o json | jq -er ' @@ -203,13 +203,13 @@ else fi ``` -Success check: the output reports the Secret count and the `updated/skipped` counters, and the stage state is written to the journal; the name and UID of existing Secrets and the ServiceAccount owner UID stay unchanged. Do not uninstall the legacy chart before this stage succeeds. +Success check: the tool reports the Secret count and `updated/skipped` counters and records the stage in the journal. Existing Secret names/UIDs and their ServiceAccount owner UIDs remain unchanged. In the normal path, verify the old controller is stopped before treating its stop evidence as valid. Do not uninstall the legacy chart yet. ## 4. P2-stop-writers: stop the legacy writers (maintenance window starts) -First suspend the platform source that manages the legacy chart. Scale the legacy Registry and Gateway Deployments to zero; if a legacy Pruner CronJob exists, set `suspend=true`; and stop every external Job that writes to the same bucket/prefix. +Pause the platform source managing the legacy chart. Stop the Registry and Gateway Deployments, suspend the legacy Pruner CronJob if present, and stop external jobs or metadata writers using the same S3 bucket/prefix. Suspending a CronJob does not stop an already active Job. -Before you run anything, point `WRITER_RECORD` at the maintenance or change record. It must name the external S3 and metadata writers, the stop time, and the administrator's approval. The file is copied into the run directory for auditing; if it is missing, the block below stops nothing. +Before running the block, set `WRITER_RECORD` to the maintenance/change record naming those writers, their stop times, and administrator approval. If the file is absent or empty, this block returns before stopping components. ```bash stop_writers() { @@ -250,27 +250,29 @@ registry-migrate run --run-dir "$RUN_DIR" --stage P2-stop-writers \ stop_writers ``` -Success check: the legacy Registry/Gateway Pods and their Ready endpoints are gone, the legacy Pruner is suspended, the active Jobs have finished, and the stage state is written to the journal. +Normal-path success check: the old Registry/Gateway Pods and Ready endpoints are gone, the Pruner is suspended, relevant active Jobs have finished, and the stage is recorded. Review `verification/legacy-jobs.txt` for jobs affecting this Registry; unrelated platform jobs are outside this migration. If the listing fails, resolve that failure before P3. -**Confirm that no other writer remains before entering P3**: check `verification/legacy-jobs.txt` for CronJobs or Jobs that still target the legacy Registry, and if S3 is reachable, list it twice a few minutes apart — the object count and the newest `LastModified` must be identical. +Confirm external writers against `WRITER_RECORD`. Two unchanged S3 listings are only supporting observations: equal object counts and latest `LastModified` values do not prove that all writers are stopped or rule out overwrites. -If the platform cannot keep the legacy components stopped, set `FORCE_ARGS=(--force)` only after the administrator accepts the consistency risk. That path records the actual runtime state and prints a force-fence warning. See the escape hatches in appendix B. +If the platform cannot hold old components stopped, use `FORCE_ARGS=(--force)` only after the administrator accepts the consistency risk. Record their actual state and the tool's force warning; do not claim those Pods or endpoints are absent. See appendix B. ## 5. P3-final-inventory: freeze the inventory and S3 state -The final inventory records the repository/tag/digest relationships together with the manifest and config metadata; the S3 snapshot records the object count, the byte count, and the listing digest. Registry v2 reuses the same bucket and prefix, so **the layers stay where they are: this stage copies nothing and deletes nothing**. It also creates no data copy — if you need a recovery point, the administrator must take an S3 snapshot or enable versioning before P3. +The final inventory records repository/tag/digest relationships and manifest/config metadata. `s3-snapshot.json` is an object-listing snapshot with counts, bytes, and a listing digest; **it is not a backup of image contents**. Layers stay in the existing S3 bucket/prefix. -**Once recorded, the P3 protection state cannot be changed**: adding proof afterwards has no effect, and the only remedy is a new run from P0. Before you start, confirm that a recovery point exists if you choose path 2, and always re-run with the same path you chose the first time. +For a recovery point, use a storage snapshot or backup that has been tested for restore. Enabling S3 versioning alone is not verified recovery evidence. If no recovery point is available, choose the explicit skip path and retain its warning. -Choose exactly one path. Run only one of the two blocks; running both makes path 2 overwrite path 1. +Once P3 succeeds, its inventory and protection state are frozen. Adding proof later does not change a run that skipped verification. Before starting, choose exactly one of the following paths; running both assigns the last choice, so do not execute both. -**Path 1: skip the restore verification explicitly** (the administrator accepts running without an S3 recovery point; the tool records a warning and the `not-established` state) +**Path 1: explicitly skip recovery verification.** The administrator accepts that no S3 recovery point has been established; the tool records a warning and `not-established`. ```bash P3_PROTECTION=(--skip-s3-backup-verification) ``` -**Path 2: provide a verified recovery point** (the tool records `restoreVerified` and the reference) +**Path 2: provide a verified recovery point.** The tool records the verification assertion and reference; it does not perform the storage restore itself. + +`RESTORE_REFERENCE` identifies an actual verified backup/snapshot or its restore-test report, for example `backup-registry-20261104-001` or `/secure/recovery/registry-restore-test.json`. The associated record should identify the source bucket/prefix, recovery time, and restore result. A bucket name or S3 URL is not a recovery-point reference. ```bash printf 'Enter the verified S3 recovery-point identifier or report path: ' @@ -283,16 +285,16 @@ jq --arg actor "$ACTOR" --arg at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ P3_PROTECTION=(--evidence-file "$RUN_DIR/restore-proof.json") ``` -Nested repositories are not skipped by default. Turn the switch on only after confirming that every nested repository in `legacy-metadata-report.json` is an approved historical test leftover: +Nested repositories are not skipped by default. Use the option below only for administrator-approved historical test residue. Review every affected repository in `legacy-metadata-report.json`; do not use it to exclude live workloads just to pass the stage. ```bash NESTED_ARGS=() # NESTED_ARGS=(--skip-nested-repositories) ``` -If the S3 endpoint in `plan.json` is a cluster-internal address (of the form `*.svc.cluster.local`), this host cannot resolve it and the stage stops with `dial tcp: lookup ... no such host`. Go to appendix A now, set up the port forward and `S3_ARGS`, then come back and run the block below. +If P3 cannot resolve a cluster-internal S3 endpoint such as `*.svc.cluster.local`, establish the port forward and `S3_ARGS` in appendix A, then return here. -P3 uses `resume` rather than `run`: after an interruption, `resume` reuses the completed scan, while every other stage uses `run` (idempotent, completed items are skipped). +This example uses `resume`. In the current tool, `run` and `resume` execute the same selected-stage logic: an incomplete P3 can rescan S3 and metadata, with no pagination checkpoint. After P3 is recorded, retries validate its frozen artifacts rather than scanning the live source again. ```bash registry-migrate resume --run-dir "$RUN_DIR" --stage P3-final-inventory \ @@ -304,13 +306,15 @@ registry-migrate report --run-dir "$RUN_DIR" jq '{items:(.items|length),digest,sample:(.items[0] // {} | {namespace,repository,tag})}' "$RUN_DIR/final-inventory.json" ``` -Success check: the stage state is written to the journal, and `final-inventory.json` and `s3-snapshot.json` are created; the scan only reads tag links, manifests, and configs and moves no layers. +Success check: P3 is recorded and `final-inventory.json` and `s3-snapshot.json` are present. The scan reads tag links, manifests, and configs without copying layers. -If the stage fails on nested repositories, read `legacy-metadata-report.json` first and set `NESTED_ARGS=(--skip-nested-repositories)` only when every entry is an approved historical test leftover. Excluded repositories never reach `final-inventory.json`, are not imported in P6, and cannot be added later, so review them one by one. +Before retrying an incomplete P3 with `NESTED_ARGS=(--skip-nested-repositories)`, approve each affected repository in the report. Excluded tags are not imported. Do not edit a successful P3's inventory to add them; use a newly approved migration run. ## 6. P4-prepare-operator: prepare the read-only Registry v2 -Install or upgrade the provider-approved Operator bundle/CRD (in the platform plugin page; the tool does not do this) and establish the ACP pause procedure so that the Operator Deployment stays at zero replicas from P4 through P6. A Manual Subscription only controls InstallPlan approval; it does not pause an Operator that is already running. +Install or upgrade the provider-approved bundle/CRD through the ACP Operator installation workflow. Keep the Operator at zero replicas during P4–P6 using the verified pause procedure; a Manual Subscription alone is insufficient. The tool prepares Config and credentials here; the Registry workloads are reconciled after resuming the Operator in P7. + +Record the original Operator replica count for P7, separately from the planned Registry count. P8 restores the Pruner baseline; a new ImagePruner records `registry-migrate.original-suspend=false`, allowing scheduled cleanup. To keep cleanup disabled, arrange an approved suspended baseline before P4. Do not change the recorded annotation afterwards. ```bash prepare_operator() { @@ -353,20 +357,22 @@ acn get imagepruner.imageregistry.operator.alauda.io cluster -o json | prepare_operator ``` -Success check: `state` in `registry-migrate report` is `prepared`; the Config reuses the planned S3 settings and routes, is read-only, and manages both the pull Secret and storage as `Unmanaged`; the Pruner is suspended. If no Pruner existed, the tool creates one and records `registry-migrate.original-suspend`. +Success check: the report shows `prepared`; Config has the planned S3 settings/routes, `readOnly=true`, and pull Secret/storage management both `Unmanaged`. The Pruner is suspended. `Unmanaged` storage leaves S3 lifecycle management with the administrator; it does not disconnect Registry from S3 or prohibit GC. -If the ImagePruner already carries a `run-id` or `plan-digest` annotation from another run that disagrees with the current plan, the command aborts without overwriting the value (at that point the Operator is already at zero replicas, but the stage is not recorded). Once you are sure the old run is abandoned, you may delete those two annotations with provider approval and re-run this section; **do not delete `registry-migrate.original-suspend`**, because P8 depends on it. A failing assertion means the Operator reconciled the Config again; confirm that the pause procedure is in effect before re-running this section. +Stop if an ImagePruner belongs to another run/plan; resolve ownership with the provider. Preserve `registry-migrate.original-suspend` for P8. After an assertion failure, check the live Config, CRD version and pause state before retrying. ## 7. P5-remove-legacy-chart: uninstall the legacy chart -This step cannot be undone. **First** record the current Secret list as the baseline for the comparison after the uninstall (it must run before the uninstall, otherwise the diff below is always empty): +Uninstalling the chart removes its workloads; the tool cannot reinstall them automatically. First capture the pre-uninstall Secret list below. Keep this original baseline when retrying after an uninstall; capturing it afterwards would hide losses. ```bash date -u > "$RUN_DIR/pre-uninstall-secrets.at" acns get secret -o name > "$RUN_DIR/pre-uninstall-secrets.txt" ``` -The platform administrator uninstalls the legacy Registry/Gateway plugin in the cluster plugin page; the tool does not uninstall anything. The uninstall deletes the resources in the chart manifest, so a TLS Secret or an S3 credential Secret generated by that chart disappears with it. Verify immediately afterwards: +Before uninstalling, confirm the platform's retention procedure protects the original business pull Secrets/ServiceAccounts, planned Ingress TLS Secrets, and P4 target S3 credentials. The P4 target Secret is `image-registry-private-configuration-user`; retain its UID and data. A differently named legacy source S3 Secret may retire with the chart once the P4 target checkpoint is established. + +The administrator then uninstalls the legacy Registry/Gateway plugin through the cluster plugin page. Chart-owned Secrets may be deleted with the chart, so check immediately afterwards: ```bash test -s "$RUN_DIR/pre-uninstall-secrets.txt" || @@ -377,9 +383,9 @@ diff "$RUN_DIR/pre-uninstall-secrets.txt" "$RUN_DIR/post-uninstall-secrets.txt" echo "WARN: Secret list changed; confirm the TLS Secret and S3 credentials still exist before continuing" >&2 ``` -The baseline timestamp must be earlier than the uninstall; otherwise the diff proves nothing and you have to compare against the P1 backup by hand. Keep the business pull Secrets, the ServiceAccounts, the S3 data, and the TLS Secret. If a TLS or S3 credential Secret shows up in the diff, restore it before continuing. +Treat the name-only diff as an initial loss check. Compare required UIDs with plan/P1 records and P4's credential checkpoint. Stop if a required Secret/ServiceAccount is missing or recreated, or target S3 credentials changed; use protected recovery material. Recreating a name does not restore its UID. -Then delete the leftover Ingress and record the stage. **The next block permanently deletes the Ingresses marked `gateway=true` in the plan**, so confirm the routes are backed up first: +Once retention checks pass, remove the leftover Ingress and record P5. The next block deletes the plan's `gateway=true` Ingresses after comparing their UIDs; retain the route backup before running it. ```bash remove_legacy_chart() { @@ -411,11 +417,11 @@ registry-migrate run --run-dir "$RUN_DIR" --stage P5-remove-legacy-chart \ remove_legacy_chart ``` -Success check: the stage state is written to the journal; the tool confirms that the legacy Service/Deployment/route are gone, the TLS Secret still exists, and the business Secret/ServiceAccount identities are unchanged. The new Ingress uses the planned `targetName`; the legacy Ingress names are used only as deletion targets. A UID mismatch means the Ingress was recreated: stop and contact your service provider instead of forcing the deletion. +Success check: P5 is recorded; old Service/Deployments/routes are gone, planned TLS Secrets remain, and business Secret/ServiceAccount identities are unchanged. Stop on a UID mismatch and consult the provider. New Ingresses use `targetName`; old names are cleanup targets only. ## 8. P6-cold-import: import metadata and limits -The Operator must still be at zero replicas. Confirm it again and generate a fresh evidence file: +Keep the new Operator paused. Confirm zero replicas and no Operator Pods, then generate fresh pause evidence and import: ```bash stop_deployment "$OPERATOR_NS" cluster-image-registry-operator @@ -427,13 +433,13 @@ registry-migrate run --run-dir "$RUN_DIR" --stage P6-cold-import \ "${FORCE_ARGS[@]}" --kubeconfig "$KCFG" --approve ``` -Success check: `state` in `registry-migrate report` is `imported`, and the output reports the unique ImageDigest objects, the ImageStream/tag relationships, and the namespace limit progress. A shared digest is stored once and keeps all of its repository/tag references. If the stage fails, confirm that the Operator is still at zero replicas and re-run it; the import counts are checked in section 9. +Success check: the report shows `imported`, with progress for unique ImageDigest objects, ImageStream/tag relationships, and namespace limits. A shared digest is stored once while preserving its repository/tag references. Retry failures in the same run with the Operator still paused; section 9 checks the result. -The legacy Gateway limits are projected onto the `registry-migrate-image-limits` LimitRange of each existing namespace: image size uses `type: alauda.io/Image` with `max.storage`, and the repository-tag count uses `type: alauda.io/ImageStream` with `max["alauda.io/repository-tags"]`. The legacy `2GB/2000` becomes `2Gi/2000` in the new limits (`2Gi=2048Mi`; `2000` means the namespace may hold at most 2000 repository tags). The approved plan selects the per-namespace rules. The migration never overwrites limits that users already set and never configures namespaces created later. +The old Gateway limits become each planned namespace's `registry-migrate-image-limits` LimitRange: image size uses `type: alauda.io/Image` with `max.storage`; tag count uses `type: alauda.io/ImageStream` with `max["alauda.io/repository-tags"]`. Legacy `2GB/2000` becomes `2Gi/2000` (`2Gi=2048Mi`). **2000 is the tag limit for each repository/ImageStream in that namespace, not the namespace-wide total.** The plan selects namespace-specific rules. Migration does not overwrite user-defined limits or apply policy to future namespaces. ## 9. P7-verify: resume the Operator and verify read-only service -Release the ACP pause through the platform procedure, then restore the Operator and wait for the data plane. `--replicas=1` is the default; if the change record lists a different original replica count, change that line. +Release the platform pause and restore the Operator replica count from the change record. The command below uses the common value of 1; use the original Operator count if different. Do not confuse it with the Registry or Image API instance replica counts. ```bash acn -n "$OPERATOR_NS" scale deployment cluster-image-registry-operator --replicas=1 @@ -456,11 +462,11 @@ acn get imagepruner.imageregistry.operator.alauda.io cluster -o json | jq -e '.spec.suspend == true' ``` -Success check: the Operator, Registry, Image API, and APIService are all ready; the P7 state is `verified` and `verify` exits 0. The Service ports are `80→8080`, `443→5000`, and `5000→5000`, the endpoints are Ready, and the new Ingress backend, Config, metadata, and limits are correct; the Config is still read-only, storage and the pull Secret are still `Unmanaged`, and the Pruner is still suspended. +Success check: P7 is `verified` and `verify` exits 0. Confirm the Operator, Registry, Image API and APIService are ready, with Ready endpoints and Service ports `80→8080`, `443→5000`, and `5000→5000`. Check the new Ingress backend, Config, metadata and limits. Config remains read-only; storage and pull Secret management remain `Unmanaged`; the Pruner stays suspended. -**This is the most important checkpoint in the maintenance window: business pulls should work again, and pushes must be rejected (403/405).** If pulls fail, check whether the pull Secret contains the new ingress host (see section 10), the Service ports, and the Ingress backend first. Conversely, if a push succeeds at this stage, `readOnly` is not in effect: stop and do not enter P8. +Read access is restored here, while business writes must remain disabled. Check `readOnly` and the tool's result; an authentication failure alone does not prove read-only enforcement. If an authorized write unexpectedly succeeds, stop before P8. For pull failures, check the original image address, SA/Secret reference, Service ports and Ingress backend. -Collect the verification evidence: +Collect the verification evidence. A warning means that file was not captured successfully; resolve it before accepting the checkpoint: ```bash mkdir -p "$RUN_DIR/verification" @@ -473,7 +479,7 @@ acn get limitrange -A --field-selector metadata.name=registry-migrate-image-limi echo "WARN: limitranges not captured" >&2 ``` -Compare the import result. A shared digest is stored once, so compare digest sets rather than repository/tag counts: +Compare digest sets rather than repository/tag counts. These commands query the **public** `image.alauda.io` API, which exposes canonical digest names. The internal names of backing `storage.image.alauda.io` CRDs are not directly comparable. ```bash jq -r '.items[].digest | select(. != null)' "$RUN_DIR/final-inventory.json" | sort -u \ @@ -484,18 +490,17 @@ wc -l < "$RUN_DIR/verification/inventory-digests.txt" comm -23 "$RUN_DIR/verification/inventory-digests.txt" "$RUN_DIR/verification/cluster-digests.txt" ``` -The first command prints the number of digests in the inventory. `comm -23` lists the digests that are in the inventory but missing from the cluster, and **the output must be empty**; if it is not, find out whether those images were skipped or failed to import. The opposite direction (present in the cluster but not in the inventory) is allowed, because those objects may be historical or created by the Operator, but record the reason in the migration notes. +`wc -l` prints the inventory's unique digest count. `comm -23` lists inventory digests missing from the cluster; **its output must be empty**. Investigate any missing entry. Additional cluster digests may belong to historical images; record the reason. Digest-set equality does not replace the tool's repository/tag, credential-identity and limit checks. -`LimitRange` is a static snapshot of the namespaces the plan already covers. It is not a global policy and is not applied automatically to namespaces created later. +Imported `LimitRange` objects are static settings for the namespaces covered by the plan, not a global policy for namespaces created later. ## 10. P8-commit: verify business access, then open writes -`REGISTRY_URL` must point at **the Ingress address that the new Registry exposes**, that is, the entry point used to reach the Registry from outside the cluster. It defaults to the planned route (`scheme` + `hostname`) and needs no manual input; when the real entry point is a different domain or VIP, override it with `REGISTRY_URL_OVERRIDE` instead of editing this block. The port comes from this cluster's plan, so do not carry over a port from another environment. +`REGISTRY_URL` must reach the new Registry through the original external entry point. The default below combines the plan's `scheme` and `hostname`; it **does not supply an external port** and may not reflect upstream TLS termination. Before running the block, set `REGISTRY_URL_OVERRIDE` to the actual full URL when needed, for example `https://registry.example.com:11780`. Confirm any alternate domain/VIP serves this Registry; do not reuse another environment's port. -Certificates fall into two cases: +For TLS, the issuer must be trusted and the SAN must match the host. For a private CA, set `REGISTRY_CA_BUNDLE` to its PEM file; the block supplies it to curl and exports `SSL_CERT_FILE` for `ac`. Correct any certificate/address mismatch. `REGISTRY_TLS_INSECURE=1` is diagnostic only and does not establish TLS acceptance. -- The ingress certificate is issued by a trusted CA: `export REGISTRY_CA_BUNDLE=/path/to/ca.pem`. curl uses it for verification, and `ac` inherits the same CA through `SSL_CERT_FILE`. -- The ingress certificate is issued by an internal CA whose SAN does not include this host (for example an auto-generated `*.nip.io` domain): TLS verification is bound to fail, so `export REGISTRY_TLS_INSECURE=1` to skip it and note this environment limitation in the migration record. +`TEST_NS` and `TEST_REPO` default to the first inventory entry, which is not necessarily a disposable test target. Set them to an approved migrated test image before the block. Both delete calls must remain dry runs; never add confirmation flags. ```bash export REGISTRY_HOST="$(jq -er '([.discovery.routes[] | select(.gateway == true)][0] // .discovery.routes[0]).hostname' "$RUN_DIR/plan.json")" @@ -561,9 +566,9 @@ probe_and_export() { probe_and_export ``` -Success check: an anonymous 401/403 from `/v2/` only proves that the entry point is reachable; it says nothing about authentication or read/write access. The stage passes only when the authenticated legacy and modern queries both succeed and both `delete images` calls remain dry runs (they print `To delete, run with --confirm flag`). The legacy view lists repository/tag data; the modern view uses ImageDigest/ImageStream data. +Success check: queries in both modes succeed, and both delete calls identify the intended tag without deleting it; dry-run wording varies by `ac` version. Both modes pass `--registry-url`, so these are HTTP compatibility checks, not independent public Image API checks; P7 covers that API. Anonymous `/v2/` 200/401/403 proves reachability only, and row counts do not prove completeness. A failed or timed-out query remains unverified even if a later command succeeds. -Next, check that every namespace's pull Secret contains the new ingress host: +The next snippet is a **limited credential-format check**, not a complete audit of every namespace. It examines `.dockerconfigjson` Secrets only; legacy chart pull Secrets commonly use `.dockercfg` and must be checked separately against the P1 handoff records and original SA references. Other registries' credentials need not contain this Registry host. ```bash acn get secret -A -o json | @@ -574,11 +579,15 @@ acn get secret -A -o json | "REVIEW \(.metadata.namespace)/\(.metadata.name): no auth entry for \($h)"' ``` -No output means every pull Secret already points at the new entry point. When a `REVIEW` line appears, namespaces managed by the Operator refresh on their own, so wait a minute or two and run the check again; Secrets left over from before the migration that the Operator does not manage are never rewritten, so the administrator must confirm that Pods in those namespaces can still pull. +No output does not prove all pull Secrets were checked or that registry addresses match exactly: the snippet uses a regex, not an exact host/port comparison. Cross-check the P1 scope, legacy `.dockercfg` objects, each original image's registry address/port, and SA references. A `REVIEW` entry for another registry is not automatically a migration failure. + +Before P8, pull Secret management is `Unmanaged`, so the Operator **does not rotate or repair these credentials automatically**. Resolve missing entries or invalid tokens before approving the handoff; do not wait for a background refresh. + +Before committing, create a disposable Pod with `imagePullPolicy=Always`, using an approved migrated test image's **original address, ServiceAccount and pull Secret**. Verify actual kubelet pulls on the relevant business nodes and through each address/port used by the workload. Record the Pod, digest and pull result, then remove the test Pod. API queries or existing running Pods alone do not prove new pulls work. -If you plan to enable the Pruner or GC after the migration, first confirm that the caller has platform-admin rights on the Registry Config. `ac adm prune images` is a platform-level operation that acts on the whole cluster's Registry, so **do not run it as part of this procedure**; adding and removing tags is a namespace permission, and the dry runs above work with the current login identity. +Do not run `ac adm prune images` or GC during this procedure. They require platform-level authority and can remove data. Confirm the recorded Pruner suspension value and whether scheduled cleanup may resume after P8. -Run the block below only after the business checks pass and the administrator approves the switch. It is the irreversible point where writes are opened: rolling back afterwards also means handling new images and credential rotation, so make sure there is no way back before you continue. The evidence it writes sets `businessProbeVerified` and `pullSecretHandoffVerified` directly, so **complete the queries above and the pull Secret check first**. +Run the next block only after those business and credential checks pass and an administrator approves opening writes with a controlled recovery plan. It directly asserts `businessProbeVerified` and `pullSecretHandoffVerified`; creating that evidence file is not the verification itself. ```bash commit_run() { @@ -605,17 +614,17 @@ registry-migrate report --run-dir "$RUN_DIR" commit_run ``` -Success check: `state` in `registry-migrate report` is `committed`; the Config is writable (`readOnly=false`), pull Secret management becomes `Managed`, storage stays `Unmanaged` (the administrator keeps managing S3), and `spec.suspend` on the Pruner returns to the value recorded before the migration (`registry-migrate.original-suspend`; a pre-migration value of `false` returns to `false`). A failing assertion does not mean the commit failed: check `state` before deciding to roll back. +Success check: the report shows `committed`; Config has `readOnly=false`, pull Secret management is `Managed`, and storage stays `Unmanaged`. The Pruner returns to `registry-migrate.original-suspend`; `false` permits scheduled cleanup to resume. Storage being `Unmanaged` does not prevent that cleanup. -Finally, push to and pull from a throwaway repository with an approved OCI client and keep the output. When the Registry authentication realm disagrees with the ingress port (appendix A), `skopeo`, `docker`, and `containerd` fail; in that case verify the manifest and blob `sha256` values through the Registry v2 API with `curl` instead. +A failing post-commit assertion does not establish that writes stayed closed. Check the journal and live state before recovery; account for new images and credential rotation after cutover. Finally, use an approved OCI client to push and pull one disposable repository and retain the output. Fix authentication-realm or TLS failures and repeat the client test; curl manifest/blob reads are diagnostic evidence, not a replacement for successful OCI or kubelet pulls. ## Appendix A: known environment limitations ### S3 reachable only from inside the cluster -If the S3 endpoint in `plan.json` is a cluster-internal address (of the form `*.svc.cluster.local`), this host cannot resolve it and P3 stops while listing objects with `dial tcp: lookup ... no such host`. Set up a port forward first. +If the plan's S3 endpoint is cluster-internal, this machine may fail to resolve it and P3 can stop with `dial tcp: lookup ... no such host`. Establish a port forward in **a second terminal**, where it remains in the foreground. -Run this in **a second terminal** (it blocks in the foreground, so do not use the migration terminal). Replace `minio-system/service/minio` with the real S3 Service name and port; if the migration terminal uses a context other than `current-context`, export `CTX` in the new terminal first: +Set `KCFG` and `CTX` in that terminal to the exact values used by the migration terminal. Its environment is independent, and the default/current context may identify another cluster. If the kubeconfig has no `current-context`, explicitly supply the selected `CTX`. Replace `minio-system/service/minio` and the port with the actual S3 Service details: ```bash : "${KCFG:=$HOME/.kube/config}" @@ -630,17 +639,17 @@ curl --fail --connect-timeout 5 --max-time 10 http://127.0.0.1:19000/minio/healt S3_ARGS=(--s3-endpoint http://127.0.0.1:19000) ``` -The override only changes the transport address used by P3; the endpoint in the plan, in the snapshot, and in the target Config stays the same. An HTTPS tunnel still uses `https://`, because a port forward does not change the protocol. Rebuild the tunnel if it drops, then retry the same run. +This override changes only P3's transport address; the endpoint recorded in the plan/snapshot and target Config remains the source endpoint. Port forwarding does not convert HTTP to HTTPS. Reestablish a dropped tunnel and retry the same run. ### Authentication realm and ingress port disagree -The `WWW-Authenticate: realm=...` address in the 401 response from `/v2/` may use a different port from the ingress (for example the realm says 443 while the ingress only exposes 11780). Authentication in `skopeo`, `docker`, and `containerd` then fails even though the Registry itself is healthy. The `REGISTRY_URL` used here requires the definition block in section 10. +The 401 response from `/v2/` may advertise a `WWW-Authenticate` realm with a port different from the externally reachable Ingress port, such as 443 instead of 11780. Registry data can remain readable while OCI authentication fails. Use the `REGISTRY_URL` defined in section 10 to inspect the challenge: ```bash curl -sS -k -D - -o /dev/null "$REGISTRY_URL/v2/" | grep -i '^www-authenticate:' ``` -Replace `-k` with `--cacert /path/to/ca.pem` when the ingress certificate is trusted. If the realm is unreachable, verify through the Registry v2 API with `curl`; do not conclude that the migration failed, because this is an entry-point configuration issue that has to be fixed separately. +Replace diagnostic `-k` with system trust or `--cacert /path/to/ca.pem` for verified TLS. If the realm is unreachable, diagnose it with curl, correct the entry-point configuration, and repeat OCI/kubelet pull checks. Record an entry-point failure separately from metadata integrity; a successful curl read does not establish full business acceptance. ## Appendix B: retries, exceptions, and recovery @@ -654,30 +663,30 @@ export RUN_DIR="/absolute/path/to/registry-migrate-runs/original-run-id" # NESTED_ARGS=(--skip-nested-repositories) ``` -Then run section 0 again; it restores the run ID, the namespaces, and the Deployment names from `plan.json`. `P3_PROTECTION` must match the first choice. Do not start a new run, and do not repeat the manual uninstall or the Ingress deletion you already completed. +Restore the original absolute `RUN_DIR`, run section 0 again, then reselect the required S3, force, nested-repository and P3 protection options; section 0 resets some arrays. `P3_PROTECTION` must match the recorded choice. Do not repeat a completed manual uninstall or capture a new post-uninstall baseline as if it were the original. ### Escape hatches -| Switch | When to use it | Cost | +| Switch | When to use it | Effect | | --- | --- | --- | -| `FORCE_ARGS=(--force)` | The platform cannot keep the legacy components stopped and the administrator accepts the consistency risk | It does not bypass the API, UID, S3, metadata, or stage checks, and it never bypasses the P4/P6 Operator pause check; the output carries a force-fence warning, and you must not claim that the legacy Pods or endpoints are gone | -| `P3_PROTECTION=(--skip-s3-backup-verification)` | The administrator accepts continuing without an S3 recovery point | Records a warning and the `not-established` state, and cannot be repaired afterwards | -| `NESTED_ARGS=(--skip-nested-repositories)` | Every nested repository in `legacy-metadata-report.json` is an approved historical test leftover | Excluded repositories do not count towards the migration result and must be reviewed one by one | +| `FORCE_ARGS=(--force)` | The administrator accepts relevant legacy components still running | Bypasses only the legacy writer fence; API, UID, S3, metadata, stage and P4/P6 Operator-pause checks remain. Record the force warning and actual running state. | +| `P3_PROTECTION=(--skip-s3-backup-verification)` | The administrator accepts no established S3 recovery point | Records a warning and `not-established`; proof added later cannot change a successful P3 in this run. | +| `NESTED_ARGS=(--skip-nested-repositories)` | Affected nested repositories are approved historical test residue | Records exclusions; excluded tags are not imported. Review them individually. | -`ac` 4.3 has no `set-registry-mode`; legacy queries use `--registry-url`, `--kubeconfig`, and `--context`. The full permission and pagination matrix belongs to the platform's separate compatibility acceptance record, not to the production migration window. +The original 4.3 `ac` has no `set-registry-mode`. Use its explicit Registry URL, kubeconfig and context for legacy queries. The separate compatibility acceptance record covers its full permission/pagination matrix; do not treat a different client's success as proof for the original binary. ### Rollback -`rollback` **is not an automatic restore**. It does three things only: it records the rollback request in the journal, locks the run so that no later stage can advance, and prints the manual recovery checklist. It does not reinstall the legacy chart, does not recreate Secrets with their original UIDs, does not delete Registry v2 resources, and does not restore S3 objects. The actual recovery is the manual work below. +`rollback` records `rollback-requested` in the journal and prevents later stages from advancing in that run. **It does not perform a restore**: it does not reinstall the chart, recreate original UIDs, remove Registry v2 resources or restore S3 objects. Use the report and the following checklist for administrator-owned recovery: ```bash registry-migrate rollback --run-dir "$RUN_DIR" --kubeconfig "$KCFG" --approve registry-migrate report --run-dir "$RUN_DIR" ``` -- **Before uninstalling the chart**: restore the original replica counts, the Pruner and platform state, the credentials, and the chart configuration from the change record and the P1 backup. -- **After uninstalling the chart**: stop writes to and reconciliation of the new Registry first, then restore the chart/platform configuration, the credentials, and S3 access by hand. -- **After P8**: new images and credential rotation also need handling. Without a controlled recovery plan, do not point the same S3 writer back at the legacy Registry. -- **Start a new run ID** after the recovery, and do not bypass identity drift by editing the plan or the journal. +- **Before chart removal**: restore original replicas, platform/Pruner state and chart configuration using the change record; use the separately protected material for credential data. P1 summaries provide identity evidence, not Secret contents. +- **After chart removal**: stop new Registry writes and reconciliation first, then restore the chart/platform configuration, credentials and S3 access through the approved recovery procedure. +- **After P8**: also account for new images and token rotation. Do not reactivate the legacy S3 writer without coordinating the recovery boundary. +- **After recovery**: create a new run ID. Do not edit the old plan/journal to bypass identity drift. -The tool is idempotent within a run: completed Secrets, metadata, limits, and checks are skipped or re-verified. Keep the plan, the journal, the backup, the inventory, the snapshot, the evidence, and the verification output. +The tool is idempotent within one run: completed work is skipped or revalidated. Preserve the plan, journal, backup summaries, inventory, snapshot, evidence, verification output and separate recovery material. From 576200c2c2b7bf5cf3cec23cbcdc9105f32060ac Mon Sep 17 00:00:00 2001 From: huizhang Date: Mon, 9 Nov 2026 18:59:54 +0800 Subject: [PATCH 07/12] docs: document pre-existing Registry v2 metadata checks --- .../ACP_Registry_v2_Migration_Runbook.mdx | 35 +++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx b/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx index f95865f04..a115f5d0a 100644 --- a/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx +++ b/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx @@ -163,6 +163,33 @@ jq '.discovery.blockers' "$RUN_DIR/plan.json" Planning is read-only. Compare the scope, S3 bucket/endpoint/prefix, original host/TLS/IngressClass, replica counts, scheduling, and timeouts with the change record and live cluster. Stop on any mismatch. If `scope.operandNamespace` is wrong, correct `REGISTRY_NS` in section 0 before planning again. +Check for pre-existing Registry v2 metadata before stopping writers. Legacy chart test images should be pushed to the legacy Registry; do not manually create backing ImageStream objects as migration test data. Existing metadata is a review condition, not a reason to delete it. Save the following list and inspect any objects in repositories covered by this migration: + +```bash +check_existing_streams() { + local crd + crd="$(acn get crd imagestreams.storage.image.alauda.io --ignore-not-found -o name)" || return 1 + if test -z "$crd"; then + echo "No Registry v2 backing ImageStream CRD exists yet." + return 0 + fi + acn get imagestreams.storage.image.alauda.io -A -o json \ + > "$RUN_DIR/existing-image-streams.json" || return 1 + jq -r '.items[] | [.metadata.namespace, .metadata.name, .metadata.uid, + (.metadata.annotations["registry-migrate.run-id"] // "unowned"), + (.metadata.annotations["registry-migrate.plan-digest"] // "")] | @tsv' \ + "$RUN_DIR/existing-image-streams.json" +} +check_existing_streams_or_stop() { + check_existing_streams || { + echo "STOP: existing ImageStream inventory failed; fix the error before stopping writers or continuing." >&2 + echo "Do not continue to P1 until the inventory file is complete." >&2 + return 1 + } +} +check_existing_streams_or_stop +``` + ## 2. P1-backup: save recovery material Complete this stage **before P1-credentials**, which rewrites pull Secret contents in place. The tool creates a redacted Registry-scoped snapshot of resource identities, owners, ServiceAccounts, references, and data digests. It contains **neither Secret data nor S3 image data**. @@ -421,6 +448,8 @@ Success check: P5 is recorded; old Service/Deployments/routes are gone, planned ## 8. P6-cold-import: import metadata and limits +Use `registry-migrate` v4.5.1 or a provider-approved later build for existing ImageStreams. P6 checks all target streams before its first metadata write. A matching digest alone is insufficient: the repository, source and Registry address must also match the frozen inventory and planned addresses. Equivalent pre-existing objects are reused without changing their UID or ownership. Known internal and external addresses do not require rewriting tag history. + Keep the new Operator paused. Confirm zero replicas and no Operator Pods, then generate fresh pause evidence and import: ```bash @@ -435,6 +464,12 @@ registry-migrate run --run-dir "$RUN_DIR" --stage P6-cold-import \ Success check: the report shows `imported`, with progress for unique ImageDigest objects, ImageStream/tag relationships, and namespace limits. A shared digest is stored once while preserving its repository/tag references. Retry failures in the same run with the Operator still paused; section 9 checks the result. +If preflight reports a digest/source conflict, stop and resolve the difference against `final-inventory.json`. If it reports that an existing stream needs changes but is not owned by this migration, the tool has left it unchanged. Neither `--force` nor deleting the inventory/journal resolves metadata conflicts. Keep any ImageDigest objects from a partial earlier attempt; P6 reuses matching records. + +If preflight reports that an existing stream needs changes but is not owned by this migration, stop. Do not add `registry-migrate.run-id` or `registry-migrate.plan-digest` manually: annotations written outside the tool journal have no rollback record and can block later runs. Preserve and compare the ImageStream's complete snapshot with the frozen inventory; a provider-approved journaled adoption procedure must separately validate its UID/resourceVersion and the repository/tag/digest mapping. A same-digest object that needs no change is reused automatically. + +After an approved, journaled adoption procedure, rerun the original P6 command with fresh pause evidence. The tool pins eligible DockerImage tag sources to the verified digest and preserves existing history; real digest conflicts remain errors. A UID/resourceVersion test failure means the selected object changed: stop and review a fresh snapshot. Do not continue to P7 until P6 is `imported`. + The old Gateway limits become each planned namespace's `registry-migrate-image-limits` LimitRange: image size uses `type: alauda.io/Image` with `max.storage`; tag count uses `type: alauda.io/ImageStream` with `max["alauda.io/repository-tags"]`. Legacy `2GB/2000` becomes `2Gi/2000` (`2Gi=2048Mi`). **2000 is the tag limit for each repository/ImageStream in that namespace, not the namespace-wide total.** The plan selects namespace-specific rules. Migration does not overwrite user-defined limits or apply policy to future namespaces. ## 9. P7-verify: resume the Operator and verify read-only service From 41ac4f819235508b69ac697ceeb825a7ae7db686 Mon Sep 17 00:00:00 2001 From: huizhang Date: Mon, 9 Nov 2026 20:15:17 +0800 Subject: [PATCH 08/12] docs: simplify Registry migration runbook prose --- docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx b/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx index a115f5d0a..b60560453 100644 --- a/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx +++ b/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx @@ -33,7 +33,7 @@ Use the same run directory for retries. Start at section 0. ## Prerequisites -- **Tools and recovery procedures**: contact your ACP service provider for approved `registry-migrate` and compatible `ac` binaries, the Registry v2 bundle, and the platform pause and S3 recovery procedures. The migration tool version is independent of the ACP release. +- **Tools and recovery procedures**: contact your ACP service provider for approved `registry-migrate` and compatible `ac` binaries, the Registry v2 bundle, and the platform pause and S3 recovery procedures. - **Namespaces**: the default Operator installation namespace is `image-registry-system`; the default Registry operand namespace is `cpaas-system`. The operand namespace is fixed at installation. Confirm the actual values and use `scope.operandNamespace` in `plan.json` throughout the migration. - **Storage scope**: stop if the legacy Registry's image-data backend uses PVCs or HostPath, or the migration contains non-image OCI artifacts or a non-default S3 root/prefix, unless the provider has approved another procedure. HostPath mounts used only by node trust or resolver components are not image-data storage. - **Operator pause**: the Operator must remain at zero replicas during P4–P6. Manual InstallPlan approval does not pause an already running Operator. Establish the provider-approved pause before P4 and before removing the legacy chart. @@ -448,7 +448,7 @@ Success check: P5 is recorded; old Service/Deployments/routes are gone, planned ## 8. P6-cold-import: import metadata and limits -Use `registry-migrate` v4.5.1 or a provider-approved later build for existing ImageStreams. P6 checks all target streams before its first metadata write. A matching digest alone is insufficient: the repository, source and Registry address must also match the frozen inventory and planned addresses. Equivalent pre-existing objects are reused without changing their UID or ownership. Known internal and external addresses do not require rewriting tag history. +Use the provider-approved `registry-migrate` build with pre-existing ImageStream checks. P6 checks all target streams before its first metadata write. A matching digest alone is insufficient: the repository, source and Registry address must also match the frozen inventory and planned addresses. Equivalent pre-existing objects are reused without changing their UID or ownership. Known internal and external addresses do not require rewriting tag history. Keep the new Operator paused. Confirm zero replicas and no Operator Pods, then generate fresh pause evidence and import: @@ -464,9 +464,7 @@ registry-migrate run --run-dir "$RUN_DIR" --stage P6-cold-import \ Success check: the report shows `imported`, with progress for unique ImageDigest objects, ImageStream/tag relationships, and namespace limits. A shared digest is stored once while preserving its repository/tag references. Retry failures in the same run with the Operator still paused; section 9 checks the result. -If preflight reports a digest/source conflict, stop and resolve the difference against `final-inventory.json`. If it reports that an existing stream needs changes but is not owned by this migration, the tool has left it unchanged. Neither `--force` nor deleting the inventory/journal resolves metadata conflicts. Keep any ImageDigest objects from a partial earlier attempt; P6 reuses matching records. - -If preflight reports that an existing stream needs changes but is not owned by this migration, stop. Do not add `registry-migrate.run-id` or `registry-migrate.plan-digest` manually: annotations written outside the tool journal have no rollback record and can block later runs. Preserve and compare the ImageStream's complete snapshot with the frozen inventory; a provider-approved journaled adoption procedure must separately validate its UID/resourceVersion and the repository/tag/digest mapping. A same-digest object that needs no change is reused automatically. +If preflight reports a digest/source conflict or an existing stream that needs changes but is not owned by this migration, stop and resolve it against `final-inventory.json` and the saved ImageStream snapshot. The tool leaves the object unchanged. Do not add `registry-migrate.run-id` or `registry-migrate.plan-digest` manually: annotations written outside the tool journal have no rollback record and can block later runs. A provider-approved journaled adoption procedure must separately validate UID/resourceVersion and the repository/tag/digest mapping. A same-digest object that needs no change is reused automatically. Neither `--force` nor deleting the inventory/journal resolves metadata conflicts. Keep any ImageDigest objects from a partial earlier attempt; P6 reuses matching records. After an approved, journaled adoption procedure, rerun the original P6 command with fresh pause evidence. The tool pins eligible DockerImage tag sources to the verified digest and preserves existing history; real digest conflicts remain errors. A UID/resourceVersion test failure means the selected object changed: stop and review a fresh snapshot. Do not continue to P7 until P6 is `imported`. From f5417251ceb6a9dc8325d14d76d88466117ee1c0 Mon Sep 17 00:00:00 2001 From: huizhang Date: Tue, 10 Nov 2026 15:01:07 +0800 Subject: [PATCH 09/12] docs: make P2 writer record setup runnable --- .../ACP_Registry_v2_Migration_Runbook.mdx | 22 ++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx b/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx index b60560453..4ab9d417a 100644 --- a/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx +++ b/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx @@ -236,14 +236,30 @@ Success check: the tool reports the Secret count and `updated/skipped` counters Pause the platform source managing the legacy chart. Stop the Registry and Gateway Deployments, suspend the legacy Pruner CronJob if present, and stop external jobs or metadata writers using the same S3 bucket/prefix. Suspending a CronJob does not stop an already active Job. -Before running the block, set `WRITER_RECORD` to the maintenance/change record naming those writers, their stop times, and administrator approval. If the file is absent or empty, this block returns before stopping components. +The block prompts for missing writer-record fields and creates `$RUN_DIR/writer-record.txt`; an existing non-empty `WRITER_RECORD` is reused. ```bash stop_writers() { -export WRITER_RECORD="${WRITER_RECORD:-/tmp/maintenance-writer-record.txt}" +export WRITER_RECORD="${WRITER_RECORD:-$RUN_DIR/writer-record.txt}" +if ! test -s "$WRITER_RECORD"; then + printf 'Maintenance window start (RFC3339): ' + IFS= read -r WINDOW_START || { echo "STOP: maintenance window start is required; nothing was stopped." >&2; return 1; } + test -n "$WINDOW_START" || { echo "STOP: maintenance window start is required; nothing was stopped." >&2; return 1; } + printf 'External writers (or none): ' + IFS= read -r EXTERNAL_WRITERS || { echo "STOP: external writer record is required; nothing was stopped." >&2; return 1; } + test -n "$EXTERNAL_WRITERS" || { echo "STOP: external writer record is required; nothing was stopped." >&2; return 1; } + printf 'Writer stop time (RFC3339): ' + IFS= read -r STOP_TIME || { echo "STOP: writer stop time is required; nothing was stopped." >&2; return 1; } + test -n "$STOP_TIME" || { echo "STOP: writer stop time is required; nothing was stopped." >&2; return 1; } + cat > "$WRITER_RECORD" <&2 - echo "Create it first: window time, external writers, stop time, approver. The shell stays open." >&2 return 1 fi if test "${#FORCE_ARGS[@]}" -eq 0; then From 05d7ab10260e6e623b5277d3bef3a01896deb8bc Mon Sep 17 00:00:00 2001 From: huizhang Date: Tue, 10 Nov 2026 16:58:39 +0800 Subject: [PATCH 10/12] docs: clarify P4 operator installation --- .../ACP_Registry_v2_Migration_Runbook.mdx | 44 ++++++++++--------- 1 file changed, 24 insertions(+), 20 deletions(-) diff --git a/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx b/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx index 4ab9d417a..41812f75c 100644 --- a/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx +++ b/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx @@ -130,11 +130,11 @@ evidence() { } stop_deployment() { local namespace="$1" name="$2" selector pods - acn -n "$namespace" get deployment "$name" -o wide + acn -n "$namespace" get deployment "$name" -o wide || return 1 selector="$(acn -n "$namespace" get deployment "$name" -o json | - jq -er '.spec.selector.matchLabels | to_entries | map("\(.key)=\(.value)") | join(",") | select(length > 0)')" - acn -n "$namespace" scale deployment "$name" --replicas=0 - pods="$(acn -n "$namespace" get pods -l "$selector" -o name)" + jq -er '.spec.selector.matchLabels | to_entries | map("\(.key)=\(.value)") | join(",") | select(length > 0)')" || return 1 + acn -n "$namespace" scale deployment "$name" --replicas=0 || return 1 + pods="$(acn -n "$namespace" get pods -l "$selector" -o name)" || return 1 if test -n "$pods"; then acn -n "$namespace" wait --for=delete pod -l "$selector" --timeout=180s || true fi @@ -355,22 +355,26 @@ Before retrying an incomplete P3 with `NESTED_ARGS=(--skip-nested-repositories)` ## 6. P4-prepare-operator: prepare the read-only Registry v2 -Install or upgrade the provider-approved bundle/CRD through the ACP Operator installation workflow. Keep the Operator at zero replicas during P4–P6 using the verified pause procedure; a Manual Subscription alone is insufficient. The tool prepares Config and credentials here; the Registry workloads are reconciled after resuming the Operator in P7. +In the ACP console, open **Administrator → Marketplace → OperatorHub**, select the target business cluster, and install or upgrade `cluster-image-registry-operator` in `image-registry-system`. Set the operand namespace to `cpaas-system` (or the namespace in `plan.json`), approve the InstallPlan, and wait for the CSV to report `Succeeded` and the Operator Deployment to exist. Then keep the Operator at zero replicas during P4–P6 using the verified pause procedure. Record the original Operator replica count for P7, separately from the planned Registry count. P8 restores the Pruner baseline; a new ImagePruner records `registry-migrate.original-suspend=false`, allowing scheduled cleanup. To keep cleanup disabled, arrange an approved suspended baseline before P4. Do not change the recorded annotation afterwards. ```bash prepare_operator() { -acn -n "$OPERATOR_NS" get subscription,installplan,csv -o wide -stop_deployment "$OPERATOR_NS" cluster-image-registry-operator -PRUNER="$(acn get imagepruner.imageregistry.operator.alauda.io cluster --ignore-not-found -o name)" +acn -n "$OPERATOR_NS" get subscription,installplan,csv -o wide || return 1 +if ! acn -n "$OPERATOR_NS" get deployment cluster-image-registry-operator -o wide >/dev/null; then + echo "STOP: install or upgrade cluster-image-registry-operator before P4" >&2 + return 1 +fi +stop_deployment "$OPERATOR_NS" cluster-image-registry-operator || return 1 +PRUNER="$(acn get imagepruner.imageregistry.operator.alauda.io cluster --ignore-not-found -o name)" || return 1 if test -n "$PRUNER"; then - acn get "$PRUNER" -o json > "$RUN_DIR/imagepruner-before.json" - jq '{annotations:.metadata.annotations,suspend:.spec.suspend}' "$RUN_DIR/imagepruner-before.json" - existing_run="$(jq -r '.metadata.annotations["registry-migrate.run-id"] // empty' "$RUN_DIR/imagepruner-before.json")" - existing_plan="$(jq -r '.metadata.annotations["registry-migrate.plan-digest"] // empty' "$RUN_DIR/imagepruner-before.json")" - current_run="$(jq -er '.runID' "$RUN_DIR/plan.json")" - current_plan="$(jq -er '.digest' "$RUN_DIR/plan.json")" + acn get "$PRUNER" -o json > "$RUN_DIR/imagepruner-before.json" || return 1 + jq '{annotations:.metadata.annotations,suspend:.spec.suspend}' "$RUN_DIR/imagepruner-before.json" || return 1 + existing_run="$(jq -r '.metadata.annotations["registry-migrate.run-id"] // empty' "$RUN_DIR/imagepruner-before.json")" || return 1 + existing_plan="$(jq -r '.metadata.annotations["registry-migrate.plan-digest"] // empty' "$RUN_DIR/imagepruner-before.json")" || return 1 + current_run="$(jq -er '.runID' "$RUN_DIR/plan.json")" || return 1 + current_plan="$(jq -er '.digest' "$RUN_DIR/plan.json")" || return 1 if test -n "$existing_run" && test "$existing_run" != "$current_run"; then echo "STOP: imagepruner belongs to run $existing_run; refusing to reuse it" >&2 return 1 @@ -380,22 +384,22 @@ if test -n "$PRUNER"; then return 1 fi if test -z "$existing_run"; then - acn annotate "$PRUNER" "registry-migrate.run-id=$current_run" + acn annotate "$PRUNER" "registry-migrate.run-id=$current_run" || return 1 fi if test -z "$existing_plan"; then - acn annotate "$PRUNER" "registry-migrate.plan-digest=$current_plan" + acn annotate "$PRUNER" "registry-migrate.plan-digest=$current_plan" || return 1 fi fi -evidence operator-paused.json operatorPaused +evidence operator-paused.json operatorPaused || return 1 registry-migrate run --run-dir "$RUN_DIR" --stage P4-prepare-operator \ --evidence-file "$RUN_DIR/operator-paused.json" \ - "${FORCE_ARGS[@]}" --kubeconfig "$KCFG" --approve + "${FORCE_ARGS[@]}" --kubeconfig "$KCFG" --approve || return 1 acn get config.imageregistry.operator.alauda.io cluster -o json | jq -e '.spec.readOnly == true and .spec.imagePullSecret.managementState == "Unmanaged" and - .spec.storage.managementState == "Unmanaged"' + .spec.storage.managementState == "Unmanaged"' || return 1 acn get imagepruner.imageregistry.operator.alauda.io cluster -o json | - jq -e '.spec.suspend == true' + jq -e '.spec.suspend == true' || return 1 } prepare_operator ``` From ceecf0add9b1d7edd051048bbfd9955f619aaee2 Mon Sep 17 00:00:00 2001 From: huizhang Date: Wed, 11 Nov 2026 16:10:40 +0800 Subject: [PATCH 11/12] docs: make Registry migration P8 commands easier to run Split long interactive functions into short Bash blocks with explicit results and failure checks. Validate compatibility inventory output as exactly one JSON array and prevent delete previews after failed or incomplete queries. Clarify the compatibility client's Registry TLS verification limitation and retain verified HTTPS, OCI-client, and kubelet checks for acceptance. --- .../ACP_Registry_v2_Migration_Runbook.mdx | 289 +++++++++++++----- 1 file changed, 215 insertions(+), 74 deletions(-) diff --git a/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx b/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx index 41812f75c..11de7fed4 100644 --- a/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx +++ b/docs/en/solutions/ACP_Registry_v2_Migration_Runbook.mdx @@ -549,77 +549,172 @@ Imported `LimitRange` objects are static settings for the namespaces covered by ## 10. P8-commit: verify business access, then open writes -`REGISTRY_URL` must reach the new Registry through the original external entry point. The default below combines the plan's `scheme` and `hostname`; it **does not supply an external port** and may not reflect upstream TLS termination. Before running the block, set `REGISTRY_URL_OVERRIDE` to the actual full URL when needed, for example `https://registry.example.com:11780`. Confirm any alternate domain/VIP serves this Registry; do not reuse another environment's port. +Run each code block separately in the **same Bash session** used for P0-P7, then press **Enter**. +Stop on an error before running the next block. If the terminal remains at a `>` prompt, press +**Ctrl+C** and paste that complete block again, without the Markdown fence markers. -For TLS, the issuer must be trusted and the SAN must match the host. For a private CA, set `REGISTRY_CA_BUNDLE` to its PEM file; the block supplies it to curl and exports `SSL_CERT_FILE` for `ac`. Correct any certificate/address mismatch. `REGISTRY_TLS_INSECURE=1` is diagnostic only and does not establish TLS acceptance. +### Prepare the Endpoint and Test Image -`TEST_NS` and `TEST_REPO` default to the first inventory entry, which is not necessarily a disposable test target. Set them to an approved migrated test image before the block. Both delete calls must remain dry runs; never add confirmation flags. +`REGISTRY_URL` must be the new Registry's external base URL, including its scheme and any external +port. The default uses the planned scheme and hostname without an external port. When the actual +entry point differs, set `REGISTRY_URL_OVERRIDE`, for example `https://registry.example.com:11780`, +before running this block. Confirm that the address serves this Registry. + +```bash +P8_ENDPOINT_OK=0 +P8_LEGACY_QUERY_OK=0 +P8_MODERN_QUERY_OK=0 +unset REGISTRY_HOST REGISTRY_SCHEME REGISTRY_URL +REGISTRY_HOST="$(jq -er ' + ([.discovery.routes[] | select(.gateway == true)][0] // + .discovery.routes[0]).hostname | strings | select(length > 0) +' "$RUN_DIR/plan.json")" && +REGISTRY_SCHEME="$(jq -er ' + ([.discovery.routes[] | select(.gateway == true)][0] // + .discovery.routes[0]).scheme // "http" +' "$RUN_DIR/plan.json")" && +REGISTRY_URL="${REGISTRY_URL_OVERRIDE:-$REGISTRY_SCHEME://$REGISTRY_HOST}" && +REGISTRY_URL="${REGISTRY_URL%/}" && +mkdir -p "$RUN_DIR/verification" && +printf 'Planned host=%s\nREGISTRY_URL=%s\n' "$REGISTRY_HOST" "$REGISTRY_URL" +``` + +Expected result: the planned host and approved external URL are printed. Resolve any error or +address mismatch before continuing. + +For HTTPS, the issuer must be trusted and the certificate SAN must match the host. For a private CA, +set `REGISTRY_CA_BUNDLE` to its PEM file; the following block supplies it to curl. +`REGISTRY_TLS_INSECURE=1` is diagnostic only; repeat the probe with certificate verification enabled +before accepting TLS. ```bash -export REGISTRY_HOST="$(jq -er '([.discovery.routes[] | select(.gateway == true)][0] // .discovery.routes[0]).hostname' "$RUN_DIR/plan.json")" -REGISTRY_SCHEME="$(jq -er '([.discovery.routes[] | select(.gateway == true)][0] // .discovery.routes[0]).scheme // "http"' "$RUN_DIR/plan.json")" -unset REGISTRY_URL -export REGISTRY_URL="${REGISTRY_URL_OVERRIDE:-$REGISTRY_SCHEME://$REGISTRY_HOST}" -printf 'REGISTRY_URL=%s\n' "$REGISTRY_URL" -jq -ne --arg url "$REGISTRY_URL" --arg host "$REGISTRY_HOST" \ - '($url | capture("^https?://(?[^/:]+)(:[0-9]+)?/?$").host) == $host' >/dev/null 2>&1 || - echo "NOTE: REGISTRY_URL is not the bare planned ingress host ($REGISTRY_HOST); confirm it is the external address of the new Registry." -mkdir -p "$RUN_DIR/verification" CURL_TLS_ARGS=() -AC_TLS_ARGS=() if test -n "${REGISTRY_CA_BUNDLE:-}"; then CURL_TLS_ARGS+=(--cacert "$REGISTRY_CA_BUNDLE") - export SSL_CERT_FILE="$REGISTRY_CA_BUNDLE" fi if test "${REGISTRY_TLS_INSECURE:-0}" = 1; then CURL_TLS_ARGS+=(--insecure) - AC_TLS_ARGS+=(--insecure-skip-tls-verify) fi -probe_and_export() { - http_code="$(curl -sS "${CURL_TLS_ARGS[@]}" --connect-timeout 10 --max-time 20 \ - -o /dev/null -w '%{http_code}' "$REGISTRY_URL/v2/")" - case "$http_code" in - 200|401|403) echo "Registry reachable: HTTP $http_code" ;; - *) echo "ERROR: unexpected HTTP ${http_code:-000} from $REGISTRY_URL/v2/" >&2 - echo "000 = this machine cannot reach the host (route/VPN/DNS); 502/504 = a proxy or the ingress cannot reach the backend." >&2 - echo "The shell stays open. Fix the cause, then re-run this block." >&2 - return 1 ;; +``` + +Probe the endpoint and save its response headers in one request: + +```bash +P8_ENDPOINT_OK=0 +if test -n "${REGISTRY_URL:-}" && + HTTP_STATUS="$(curl -sS "${CURL_TLS_ARGS[@]}" \ + --connect-timeout 10 --max-time 20 -o /dev/null -w '%{http_code}' \ + -D "$RUN_DIR/verification/registry-v2-headers.txt" "$REGISTRY_URL/v2/")"; then + case "$HTTP_STATUS" in + 200|401|403) P8_ENDPOINT_OK=1; echo "Registry reachable: HTTP $HTTP_STATUS" ;; + *) echo "STOP: unexpected HTTP $HTTP_STATUS from $REGISTRY_URL/v2/" >&2 ;; esac - curl -sS "${CURL_TLS_ARGS[@]}" --connect-timeout 10 --max-time 20 \ - -D "$RUN_DIR/verification/registry-v2-headers.txt" -o /dev/null "$REGISTRY_URL/v2/" - if grep -qi '^WWW-Authenticate:' "$RUN_DIR/verification/registry-v2-headers.txt"; then - echo "Check the host and port of the WWW-Authenticate realm; see appendix A if they differ from the ingress." >&2 - fi +else + echo "STOP: endpoint request failed; check the URL, network, and TLS error." >&2 +fi +test "$P8_ENDPOINT_OK" = 1 +``` - TEST_NS="${TEST_NS:-$(jq -er '.items[0].namespace' "$RUN_DIR/final-inventory.json")}" || - { echo "ERROR: final-inventory.json has no items; P3 produced no inventory" >&2; return 1; } - TEST_REPO="${TEST_REPO:-$(jq -r '.items[0] | "\(.namespace)/\(.repository):\(.tag)"' "$RUN_DIR/final-inventory.json")}" - export TEST_NS TEST_REPO +Expected result: `Registry reachable` is printed and the block exits successfully. An anonymous +`200`, `401`, or `403` response proves reachability only. A connection or TLS error fails this check; +`502` or `504` usually indicates that a proxy or Ingress cannot reach the backend. +Inspect the authentication challenge, if present: + +```bash +grep -i '^WWW-Authenticate:' "$RUN_DIR/verification/registry-v2-headers.txt" || true +``` + +Confirm that the challenge's realm host and port are reachable through the approved entry point. +See appendix A if they differ. No output means the response did not include this header. + +Set `TEST_NS` and `TEST_REPO` to an approved migrated test image, or review the first inventory entry +selected by default. `TEST_REPO` uses `namespace/repository:tag` format. + +```bash +TEST_NS="${TEST_NS:-$(jq -er ' + .items[0].namespace | strings | select(length > 0) +' "$RUN_DIR/final-inventory.json")}" && +TEST_REPO="${TEST_REPO:-$(jq -er ' + .items[0] | select(.namespace and .repository and .tag) | + "\(.namespace)/\(.repository):\(.tag)" +' "$RUN_DIR/final-inventory.json")}" && +export TEST_NS TEST_REPO && +printf 'TEST_NS=%s\nTEST_REPO=%s\n' "$TEST_NS" "$TEST_REPO" +``` + +Expected result: both values identify the approved migrated test image. If selection fails or the +default is unsuitable, correct the values before continuing. + +### Check CLI Compatibility + +The compatibility client used by these `ac` commands currently skips Registry certificate +verification internally. Successful queries do not validate TLS trust. Use the verified curl probe +and successful OCI-client and kubelet pulls as TLS acceptance evidence. + +Query the namespace inventory in legacy mode: + +```bash +P8_LEGACY_QUERY_OK=0 +if test "${P8_ENDPOINT_OK:-0}" = 1 && test -n "${TEST_NS:-}" && + env AC_IMAGE_REGISTRY_MODE=legacy \ + ac --kubeconfig "$KCFG" --context "$CTX" \ + get images -n "$TEST_NS" -o json --registry-url "$REGISTRY_URL" \ + > "$RUN_DIR/verification/legacy-images.json" && + jq -se 'length == 1 and (.[0] | type == "array")' \ + "$RUN_DIR/verification/legacy-images.json" >/dev/null; then + P8_LEGACY_QUERY_OK=1 + jq -r '"Legacy query succeeded: \(length) items"' \ + "$RUN_DIR/verification/legacy-images.json" +else + echo "STOP: endpoint or legacy inventory check failed." >&2 + false +fi +``` + +Query the cluster inventory in modern mode: + +```bash +P8_MODERN_QUERY_OK=0 +if test "${P8_ENDPOINT_OK:-0}" = 1 && + env AC_IMAGE_REGISTRY_MODE=modern \ + ac --kubeconfig "$KCFG" --context "$CTX" \ + get images -A -o json --registry-url "$REGISTRY_URL" \ + > "$RUN_DIR/verification/modern-images.json" && + jq -se 'length == 1 and (.[0] | type == "array")' \ + "$RUN_DIR/verification/modern-images.json" >/dev/null; then + P8_MODERN_QUERY_OK=1 + jq -r '"Modern query succeeded: \(length) items"' \ + "$RUN_DIR/verification/modern-images.json" +else + echo "STOP: endpoint or modern inventory check failed." >&2 + false +fi +``` + +Expected result: each block prints its success message and saves one JSON array. The queries have +different namespace scopes, so their counts need not match. Counts alone do not prove completeness. + +Run the delete previews only after both queries succeed. **Never add `--confirm`, `--yes`, or `-f`.** + +```bash +if test "${P8_LEGACY_QUERY_OK:-0}" = 1 && + test "${P8_MODERN_QUERY_OK:-0}" = 1 && test -n "${TEST_REPO:-}"; then env AC_IMAGE_REGISTRY_MODE=legacy \ - ac --kubeconfig "$KCFG" --context "$CTX" "${AC_TLS_ARGS[@]}" get images -n "$TEST_NS" -o json \ - --registry-url "$REGISTRY_URL" > "$RUN_DIR/verification/legacy-images.json" || - { echo "ERROR: legacy view failed; see the error above. The shell stays open." >&2; return 1; } - env AC_IMAGE_REGISTRY_MODE=modern \ - ac --kubeconfig "$KCFG" --context "$CTX" "${AC_TLS_ARGS[@]}" get images -A -o json \ - --registry-url "$REGISTRY_URL" > "$RUN_DIR/verification/modern-images.json" || - { echo "ERROR: modern view failed; see the error above. The shell stays open." >&2; return 1; } - printf 'legacy=%s modern=%s\n' \ - "$(jq '.items | length' "$RUN_DIR/verification/legacy-images.json" 2>/dev/null || echo n/a)" \ - "$(jq '.items | length' "$RUN_DIR/verification/modern-images.json" 2>/dev/null || echo n/a)" - - # Keep these as dry runs: never add --confirm / --yes / -f, or the tag is really deleted. - env AC_IMAGE_REGISTRY_MODE=legacy \ - ac --kubeconfig "$KCFG" --context "$CTX" "${AC_TLS_ARGS[@]}" delete images \ - --repo "$TEST_REPO" --registry-url "$REGISTRY_URL" + ac --kubeconfig "$KCFG" --context "$CTX" delete images \ + --repo "$TEST_REPO" --registry-url "$REGISTRY_URL" && env AC_IMAGE_REGISTRY_MODE=modern \ - ac --kubeconfig "$KCFG" --context "$CTX" "${AC_TLS_ARGS[@]}" delete images \ + ac --kubeconfig "$KCFG" --context "$CTX" delete images \ --repo "$TEST_REPO" --registry-url "$REGISTRY_URL" -} -probe_and_export +else + echo "STOP: both inventory queries and the test image selection must succeed." >&2 + false +fi ``` -Success check: queries in both modes succeed, and both delete calls identify the intended tag without deleting it; dry-run wording varies by `ac` version. Both modes pass `--registry-url`, so these are HTTP compatibility checks, not independent public Image API checks; P7 covers that API. Anonymous `/v2/` 200/401/403 proves reachability only, and row counts do not prove completeness. A failed or timed-out query remains unverified even if a later command succeeds. +Expected result: both previews identify the approved tag without deleting it; dry-run wording varies +by `ac` version. Both modes pass `--registry-url`, so these are HTTP compatibility checks. P7 verifies +the public Image API separately. A failed check remains unverified even if a later block succeeds. The next snippet is a **limited credential-format check**, not a complete audit of every namespace. It examines `.dockerconfigjson` Secrets only; legacy chart pull Secrets commonly use `.dockercfg` and must be checked separately against the P1 handoff records and original SA references. Other registries' credentials need not contain this Registry host. @@ -640,36 +735,82 @@ Before committing, create a disposable Pod with `imagePullPolicy=Always`, using Do not run `ac adm prune images` or GC during this procedure. They require platform-level authority and can remove data. Confirm the recorded Pruner suspension value and whether scheduled cleanup may resume after P8. -Run the next block only after those business and credential checks pass and an administrator approves opening writes with a controlled recovery plan. It directly asserts `businessProbeVerified` and `pullSecretHandoffVerified`; creating that evidence file is not the verification itself. +### Commit and Verify the Cutover + +Continue only after the business and credential checks pass and an administrator approves opening +writes with a controlled recovery plan. Prepare the approval evidence. Its verification fields +record the completed checks; creating the file does not perform them. + +```bash +( + test -n "${ACTOR:-}" || { + echo "STOP: set ACTOR to the platform administrator account." >&2 + exit 1 + } + jq --arg actor "$ACTOR" --arg at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" ' + {version:"v1",runID:.runID,planDigest:.digest,observedAt:$at,actor:$actor, + commitApproved:true,businessProbeVerified:true,pullSecretHandoffVerified:true} + ' "$RUN_DIR/plan.json" > "$RUN_DIR/commit-approval.json" && + jq -e '.actor | length > 0' "$RUN_DIR/commit-approval.json" >/dev/null +) +``` + +Expected result: the block exits successfully and creates `commit-approval.json`. The parentheses +keep an error from closing the migration terminal. After reviewing the evidence, commit the cutover: ```bash -commit_run() { -test -n "${ACTOR:-}" || { echo "ERROR: ACTOR is empty; export ACTOR= and re-run this block" >&2; return 1; } -jq --arg actor "$ACTOR" --arg at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ - '{version:"v1",runID:.runID,planDigest:.digest,observedAt:$at,actor:$actor,commitApproved:true,businessProbeVerified:true,pullSecretHandoffVerified:true}' \ - "$RUN_DIR/plan.json" > "$RUN_DIR/commit-approval.json" -jq -e '.actor | length > 0' "$RUN_DIR/commit-approval.json" >/dev/null || - { echo "ERROR: commit-approval.json has an empty actor; the tool rejects it" >&2; return 1; } registry-migrate run --run-dir "$RUN_DIR" --stage P8-commit \ --evidence-file "$RUN_DIR/commit-approval.json" --kubeconfig "$KCFG" --approve -acn get config.imageregistry.operator.alauda.io cluster -o json | - jq -e '.spec.readOnly == false and - .spec.imagePullSecret.managementState == "Managed" and - .spec.storage.managementState == "Unmanaged"' || - echo "NOTE: the commit itself may have succeeded and only this Config assertion failed; check the state in registry-migrate report before considering rollback" >&2 -acn get imagepruner.imageregistry.operator.alauda.io cluster -o json | jq -e ' - (.metadata.annotations["registry-migrate.original-suspend"] // "false") as $original | - .spec.suspend == ($original == "true")' || - echo "NOTE: if the actual suspend value disagrees with the annotation, compare imagepruner-before.json from P4 before considering rollback" >&2 +``` + +Expected result: P8 is recorded as `committed`. If the command fails, inspect +`registry-migrate report --run-dir "$RUN_DIR"` and the live state before retrying or recovering; +writes might already be open. + +Verify the Registry settings: + +```bash +( + set -o pipefail + acn get config.imageregistry.operator.alauda.io cluster -o json | + jq -e '.spec.readOnly == false and + .spec.imagePullSecret.managementState == "Managed" and + .spec.storage.managementState == "Unmanaged"' +) +``` + +Expected result: `true`. Verify the restored Pruner suspension state separately: + +```bash +( + set -o pipefail + acn get imagepruner.imageregistry.operator.alauda.io cluster -o json | + jq -e ' + (.metadata.annotations["registry-migrate.original-suspend"] // "false") as $original | + .spec.suspend == ($original == "true")' +) +``` + +Expected result: `true`. If it is `false`, compare `imagepruner-before.json` from P4 with the +recorded annotation before considering recovery. A restored value of `false` permits scheduled +cleanup; `Unmanaged` storage does not prevent it. + +Run final verification: + +```bash registry-migrate verify --run-dir "$RUN_DIR" --kubeconfig "$KCFG" -registry-migrate report --run-dir "$RUN_DIR" -} -commit_run ``` -Success check: the report shows `committed`; Config has `readOnly=false`, pull Secret management is `Managed`, and storage stays `Unmanaged`. The Pruner returns to `registry-migrate.original-suspend`; `false` permits scheduled cleanup to resume. Storage being `Unmanaged` does not prevent that cleanup. +Expected result: verification exits 0. Then retain the final report: + +```bash +registry-migrate report --run-dir "$RUN_DIR" +``` -A failing post-commit assertion does not establish that writes stayed closed. Check the journal and live state before recovery; account for new images and credential rotation after cutover. Finally, use an approved OCI client to push and pull one disposable repository and retain the output. Fix authentication-realm or TLS failures and repeat the client test; curl manifest/blob reads are diagnostic evidence, not a replacement for successful OCI or kubelet pulls. +A failing post-commit check does not mean that writes stayed closed. Account for new images and +credential rotation when planning recovery. Finally, use an approved OCI client to push and pull one +disposable repository and retain the output. Resolve authentication-realm or TLS failures and repeat +the client test; API or curl reads do not replace successful OCI and kubelet pulls. ## Appendix A: known environment limitations From ba4bdafd5d17b35c28bdbf15d03c0e8bfcd5bc23 Mon Sep 17 00:00:00 2001 From: huizhang Date: Thu, 12 Nov 2026 16:38:40 +0800 Subject: [PATCH 12/12] fix(ci): defer unchanged automatic translations on pull requests PR checks compared untouched generated translations with updated English before the main-only translation step could regenerate them. Check changed translations and manual translations on PRs, while keeping post-translation validation strict. Add regression tests for source-only edits, changed pairs, and manual-translation opt-outs. --- .github/workflows/main.yml | 8 +-- scripts/check-translation-integrity.mjs | 31 ++++++---- scripts/check-translation-integrity.test.mjs | 63 ++++++++++++++++++++ 3 files changed, 85 insertions(+), 17 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 52e55a804..ad40ffb58 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -61,11 +61,9 @@ jobs: - name: Test the translation integrity check run: yarn test:translation - # The unit tests above prove the check works; they do not look at a single - # document. On main that job belongs to "Translate and verify", but that - # step only runs on a push -- so until now a pull request could add or - # edit a page and nothing compared it against its English source. The - # branch scope does exactly that, for the pairs this branch touches. + # Check translations changed by this PR and manually translated pages. + # Unchanged automatic translations of English-only edits are regenerated + # and checked by "Translate and verify" after the merge to main. - name: Check the translations this branch changes if: github.event_name == 'pull_request' run: node scripts/check-translation-integrity.mjs --since origin/${{ github.base_ref }} diff --git a/scripts/check-translation-integrity.mjs b/scripts/check-translation-integrity.mjs index aa8898871..82dc7ed11 100644 --- a/scripts/check-translation-integrity.mjs +++ b/scripts/check-translation-integrity.mjs @@ -28,8 +28,9 @@ * untracked -- i.e. the ones translate just (re)wrote * --all every file under docs/ * --only just this file (repeatable); overrides every other scope - * --since every pair this branch touches since -- the pull - * request scope, where nothing is modified in the worktree + * --since translations changed since , plus hand-translated + * pairs whose source changed; automatic translations of + * English-only edits are generated and checked on main * --docs check a docs tree outside this repo (implies a full scan) * --failures write the failing documents to , one path per line, for * translate-verified.mjs to hand back to the translator @@ -116,14 +117,9 @@ const changedTargetFiles = () => { } /** - * Target files of every pair this branch touches since . - * - * The default scope reads the working tree, which is right on main -- translate - * has just rewritten those files. On a pull request the tree is clean, so that - * scope selects nothing and the documents the branch actually changed go - * unchecked; the only thing running on a pull request was the check's own unit - * tests. A source-language edit counts too: it can leave an existing - * translation short of a section that was only just added. + * Pairs touched since , and which translations the branch changed. + * Unchanged automatic translations are regenerated on main; hand-translated + * pages and translations changed by the branch still need checking here. */ const changedSince = (ref) => { const stdout = execFileSync( @@ -137,18 +133,21 @@ const changedSince = (ref) => { ) const targets = new Set() const sources = new Set() + const changedTargets = new Set() for (const line of stdout.split('\n')) { if (!line.trim()) continue const abs = path.resolve(docsDir, line.trim()) if (!DOC_EXTENSIONS.has(path.extname(abs))) continue // Either language names the same pair; the scan below is driven by targets. const fromSource = abs.startsWith(sourceDir + path.sep) + if (!fromSource && !abs.startsWith(targetDir + path.sep)) continue const source = fromSource ? abs : path.join(sourceDir, path.relative(targetDir, abs)) const target = fromSource ? path.join(targetDir, path.relative(sourceDir, abs)) : abs sources.add(source) if (fs.existsSync(target)) targets.add(target) + if (!fromSource) changedTargets.add(target) } - return { targets: [...targets], sources } + return { targets: [...targets], sources, changedTargets } } // Computed once: the missing-translation audit below needs the source side of @@ -825,7 +824,15 @@ const selectTargets = () => { }) } if (scanAll) return walk(targetDir) - if (branchScope) return branchScope.targets + if (branchScope) { + return branchScope.targets.filter((file) => { + const sourceFile = path.join(sourceDir, path.relative(targetDir, file)) + if (branchScope.changedTargets.has(file) || !fs.existsSync(sourceFile)) return true + if (disablesAutoTranslation(fs.readFileSync(sourceFile, 'utf8'))) return true + console.log(`PENDING ${relative(file)} -- regenerated and checked from ${relative(sourceFile)} on main`) + return false + }) + } return changedTargetFiles() } const targetFiles = selectTargets().filter((file) => diff --git a/scripts/check-translation-integrity.test.mjs b/scripts/check-translation-integrity.test.mjs index ce14ce37f..f50426444 100644 --- a/scripts/check-translation-integrity.test.mjs +++ b/scripts/check-translation-integrity.test.mjs @@ -596,6 +596,69 @@ a fs.rmSync(root, { recursive: true, force: true }) } +// --------------------------------------------------------------------------- +// English-only edits are translated on main. An unchanged generated translation +// is pending regeneration, not a broken translation introduced by the branch. +// The named-file scope must still catch that mismatch after translation runs. +// --------------------------------------------------------------------------- +{ + const root = makeGitTree({ + 'en/a.mdx': `${FRONTMATTER}# T\n\nProse.\n`, + 'zh/a.mdx': `${FRONTMATTER}# T\n\n正文。\n`, + }) + fs.appendFileSync(path.join(root, 'en', 'a.mdx'), '\n## New section\n\n```bash\necho ready\n```\n') + commitAll(root, 'update English only') + const before = read(root, 'zh', 'a.mdx') + + const scoped = check(root, '--since', 'HEAD~1') + ok(scoped.status === 0, 'an English-only edit defers its unchanged automatic translation', scoped.out) + ok(scoped.out.includes('PENDING') && scoped.out.includes('zh/a.mdx'), 'the pending translation is named', scoped.out) + ok(read(root, 'zh', 'a.mdx') === before, 'branch checks leave the pending translation untouched') + + const strict = check(root, '--only', path.join(root, 'zh', 'a.mdx')) + ok(strict.status === 1, 'post-translation checks still reject the outdated translation', strict.out) + const full = check(root) + ok(full.status === 1, 'the full translation scan still rejects the outdated translation', full.out) + fs.rmSync(root, { recursive: true, force: true }) +} + +// --------------------------------------------------------------------------- +// If the branch changes both languages, it owns the new translation and must +// provide all the new English content rather than deferring the damaged pair. +// --------------------------------------------------------------------------- +{ + const root = makeGitTree({ + 'en/a.md': `${FRONTMATTER}# T\n\nProse.\n`, + 'zh/a.md': `${FRONTMATTER}# T\n\n正文。\n`, + }) + fs.appendFileSync(path.join(root, 'en', 'a.md'), '\n## New section\n\n```bash\necho ready\n```\n') + fs.appendFileSync(path.join(root, 'zh', 'a.md'), '\n更新了正文。\n') + commitAll(root, 'change both languages with an incomplete translation') + + const { status, out } = check(root, '--since', 'HEAD~1') + ok(status === 1, 'an incomplete translation changed with its source still fails the branch', out) + ok(!out.includes('PENDING'), 'a changed translation is never deferred', out) + fs.rmSync(root, { recursive: true, force: true }) +} + +// --------------------------------------------------------------------------- +// An English-only edit to a hand-translated page cannot be deferred to main: +// the automatic translator is explicitly disabled for that page. +// --------------------------------------------------------------------------- +for (const i18n of ['i18n:\n disableAutoTranslation: true\n', 'i18n: { disableAutoTranslation: true }\n']) { + const root = makeGitTree({ + 'en/a.md': `---\nid: KB1\n${i18n}---\n# T\n\nProse.\n`, + 'zh/a.md': `${FRONTMATTER}# T\n\n正文。\n`, + }) + fs.appendFileSync(path.join(root, 'en', 'a.md'), '\n## New section\n\nProse.\n') + commitAll(root, 'update a hand-translated source only') + + const { status, out } = check(root, '--since', 'HEAD~1') + ok(status === 1, 'an outdated hand translation fails an English-only branch', out) + ok(!out.includes('PENDING'), 'hand translations are never deferred', out) + fs.rmSync(root, { recursive: true, force: true }) +} + // --------------------------------------------------------------------------- // A new English page on a branch has no translation yet and is not supposed to // -- translate writes it on main. Failing the branch for that would block every