From 23daee2d4e5d911c4bf4e24299270319539aae59 Mon Sep 17 00:00:00 2001 From: hfxia Date: Fri, 6 Nov 2026 02:40:33 -0500 Subject: [PATCH 1/2] docs(redis): update the cross-cluster migration guide for ACP 4.x and Redis 7.2 The guide covered only platform 3.8 to 3.16. Rewrite it from tests on ACP 4.4 with redis-operator 5.1 and Redis 6.0 and 7.2, and keep the 3.x content in an appendix. --- .../How_to_Migrate_Redis_Across_Clusters.md | 298 +++++++++--------- 1 file changed, 142 insertions(+), 156 deletions(-) diff --git a/docs/en/solutions/ecosystem/redis/How_to_Migrate_Redis_Across_Clusters.md b/docs/en/solutions/ecosystem/redis/How_to_Migrate_Redis_Across_Clusters.md index 28999e1d1..0eb8e4cea 100644 --- a/docs/en/solutions/ecosystem/redis/How_to_Migrate_Redis_Across_Clusters.md +++ b/docs/en/solutions/ecosystem/redis/How_to_Migrate_Redis_Across_Clusters.md @@ -5,18 +5,20 @@ kind: - Solution ProductsVersion: - 3.x + - 4.x id: KB260500095 --- # Migrate Redis Data Across Kubernetes Clusters :::info Applicable Versions -- Platform: **3.8.x – 3.16.x** (image table below stops at 3.16.2) +- Platform **4.x** with **redis-operator 5.1** or later. Redis 7.x sources need a redis-operator release later than **5.1.1**; see [Choose a Migration Mode](#choose-a-migration-mode). +- Platform **3.8.x – 3.16.x**: the procedure is the same, but you must set the image yourself and only Redis 5.x and 6.x are supported. See [Appendix: Platform 3.x](#appendix-platform-3x). ::: ## Introduction -This guide explains how to migrate data between Redis instances running in different Kubernetes clusters using the platform's built-in **RedisShake**-based migration feature. The procedure supports the following source/target architecture combinations: +This guide explains how to copy data from one Redis instance to another, including instances in different Kubernetes clusters, with the platform's built-in migration based on **RedisShake**. You describe a migration with one `RedisShake` custom resource, and redis-operator runs it. The source and the target can each be any of these architectures: | Architecture | Description | | --- | --- | @@ -24,213 +26,200 @@ This guide explains how to migrate data between Redis instances running in diffe | `sentinel` | Sentinel-based primary/replica instance | | `cluster` | Sharded Redis Cluster | -RedisShake performs continuous replication: as long as the source keeps accepting writes, those writes are propagated to the target. Synchronization does not stop on its own. +A `RedisShake` resource runs in one of two modes: -## Glossary +- **`sync`**: copies the full dataset, then keeps replicating every write made on the source. It never stops on its own; you end it by deleting the resource. Use it when the source must stay in service until cutover. +- **`rump`**: copies the keys once, by scanning the source, and then finishes. Writes made after a key was copied are not carried over. Use it when you can stop writes on the source for the whole copy. -| Term | Meaning | -| --- | --- | -| Source | The Redis instance whose data is being migrated. | -| Target | The Redis instance that receives the migrated data. | +## Choose a Migration Mode -## Prerequisites +| Source Redis | Target Redis | Mode | +| --- | --- | --- | +| 5.x or 6.x | 6.x or 7.2 | `sync` | +| 7.0 or 7.2 | 7.2 | `sync` on redis-operator later than 5.1.1; `rump` on earlier releases | +| 7.x | 5.x or 6.x | Not supported | +| 7.4 or later | Any | Not supported | -1. **Capacity check.** Ensure the target has enough memory to hold the source dataset: - - Target is sentinel: target memory should be at least **5/4 of source dataset size**. - - Target is cluster, source is standalone: target memory should be at least **5/4 of source dataset size**. - - Target is cluster, source is cluster: target memory should be at least **5/4 of the largest source shard's dataset size**. - - For acceptable performance, allocate **2-8 vCPU** per RedisShake pod (4 vCPU recommended). -2. **Reduce or pause writes** on the source to lower replication pressure. -3. **Pick the correct RedisShake image** for your platform version (see the [appendix](#appendix-redisshake-image-by-platform-version)). -4. **Network connectivity.** The source and target instances must be reachable from the RedisShake pod over a stable IPv4 network. -5. **Redis version.** The current RedisShake image **does not support Redis 7.x**. Use this method only with Redis 5.x or 6.x source/target instances. +RedisShake added support for the RDB format of Redis 7.0 and 7.2 after redis-operator 5.1.1. What matters is the redis-operator in the cluster where you create the `RedisShake` resource, because the migration pod uses the RedisShake image delivered with that operator. With an earlier release there, a `sync` migration from a Redis 7.x source fails at once, the target stays empty, and the `redis-shake` container log shows: -### Known Issues +```text +[PANIC] parse rdb header error +[error]: verify version, invalid RDB version number 11, 9 +``` -- On platform versions **3.16.0** and **3.16.1**, when the source is a Sentinel instance, the source instance status may be incorrectly reported as `Processing` even though the instance is healthy. The status reporting is fixed in **3.16.2**. +If you see this, either upgrade redis-operator or delete the resource and recreate it with `modelType: rump`. -## Migration Procedure +## Prerequisites -For every scenario below, you create a single `RedisShake` custom resource. Update the image, source/target addresses, and password Secrets to match your environment, then apply the resource using `kubectl` or the platform UI. To change a configuration after deployment, delete the existing resource and create a new one — in-place edits are not supported. +1. **Capacity.** The target must have enough memory for the source dataset: + - Sentinel target: at least **5/4 of the source dataset size**. + - Cluster target with a standalone or Sentinel source: at least **5/4 of the source dataset size**. + - Cluster target with a cluster source: at least **5/4 of the dataset size of the largest source shard**. +2. **Where the migration runs.** Create the `RedisShake` resource in a cluster where redis-operator is installed, usually the target cluster. The migration pod must reach both instances over a stable IPv4 network. +3. **External access.** An instance in a different cluster from the `RedisShake` resource must have NodePort or LoadBalancer access enabled: + - Cluster instance: RedisShake connects to every shard at the address the shard announces, so in-cluster addresses do not work from another cluster. + - Sentinel instance: enable external access for both the data nodes (`spec.expose.type`) and the Sentinel nodes (`spec.sentinel.expose.type`). RedisShake asks Sentinel for the primary's address and then connects to that primary. +4. **Databases.** A Redis Cluster has only database 0. When the target is a cluster, keys in other databases of the source are **skipped without any error**. Run `INFO keyspace` on the source and make sure every key is in `db0` before you migrate to a cluster. +5. **Redis functions.** Function libraries (Redis 7 `FUNCTION LOAD`) are not migrated. Load them on the target separately, for example with `FUNCTION DUMP` on the source and `FUNCTION RESTORE` on the target. +6. **Writes.** Reduce writes on the source to lower replication pressure. With `rump`, stop writes for the whole copy. -### 1. Create Password Secrets +## Migration Procedure + +### 1. Create the Password Secrets -If either side is password-protected, create a Kubernetes Secret per side. The Secret must use the data key `password`. +If a side is password-protected, create one Secret for it, with the data key `password`, in the namespace where you create the `RedisShake` resource: ```bash kubectl -n create secret generic \ --from-literal=password= ``` -### 2. Standalone to Sentinel +### 2. Find the Addresses -```yaml -apiVersion: middle.alauda.cn/v1alpha1 -kind: RedisShake -metadata: - name: standalone-to-sentinel -spec: - image: # See appendix for the right image for your version - keyExists: rewrite - modelType: sync - replicas: 1 - source: - address: - - ":" - passwordSecret: - type: standalone - target: - address: - - "mymaster@:" - passwordSecret: - type: sentinel -``` +Find the access endpoints in **Data Services > Instance Detail > Access Method > In-cluster Access / External Access**, then write each address in the format for its architecture and role: -:::warning -When the **source or target is a Sentinel instance**, the address must be prefixed with `mymaster@`, and the value should be the Sentinel access endpoint. Find it in **Data Services > Instance Detail > Access Method > In-cluster Access / External Access**. -::: +| Architecture | As source | As target | +| --- | --- | --- | +| `standalone` | `:` | `:` | +| `sentinel` | `mymaster@:` | `mymaster@:` | +| `cluster` | `master@:` | `:` | -### 3. Sentinel to Sentinel +- For a Sentinel instance, use the **Sentinel** endpoint. `mymaster` is the primary name the platform gives every Sentinel instance. +- For a cluster instance, any one node endpoint is enough: RedisShake discovers the other shards. Only a cluster **source** takes the `master@` prefix. -```yaml -apiVersion: middle.alauda.cn/v1alpha1 -kind: RedisShake -metadata: - name: sentinel-to-sentinel -spec: - image: - keyExists: rewrite - modelType: sync - replicas: 1 - source: - address: - - "mymaster@:" - passwordSecret: - type: sentinel - target: - address: - - "mymaster@:" - passwordSecret: - type: sentinel +:::warning +If the Sentinel nodes of an instance have their own password (`spec.sentinel.passwordSecret`), the `mymaster@` address does not work on either side: RedisShake cannot authenticate to Sentinel and the log shows `no sentinels available; last error: NOAUTH Authentication required.` Connect to the data nodes directly instead. Set `type: standalone`, use the Secret of the **data node** password, and use the external address of the read-write Service `rfr--read-write`. To read its NodePort: + +```bash +kubectl -n get svc rfr--read-write \ + -o jsonpath='{.spec.ports[0].nodePort}' ``` +::: -### 4. Standalone to Cluster +### 3. Create the RedisShake Resource ```yaml apiVersion: middle.alauda.cn/v1alpha1 kind: RedisShake metadata: - name: standalone-to-cluster + name: + namespace: spec: - image: - keyExists: rewrite - modelType: sync - replicas: 1 + modelType: sync # or rump, see "Choose a Migration Mode" + keyExists: rewrite # overwrite keys that already exist on the target source: + type: sentinel # standalone, sentinel or cluster address: - - ":" + - "mymaster@:" passwordSecret: - type: standalone target: + type: cluster # standalone, sentinel or cluster address: - ":" passwordSecret: - type: cluster + resources: # optional + requests: + cpu: "2" + memory: 2Gi + limits: + cpu: "4" + memory: 4Gi ``` -:::note -For a cluster target, the address can be **any one** of the cluster access endpoints. Find them in **Data Services > Instance Detail > Access Method > In-cluster Access / External Access**. -::: +- Change `source` and `target` for your scenario, using the address formats from step 2. +- Leave out `image`: redis-operator 5.1 and later use the RedisShake image delivered with the operator. +- `keyExists` also accepts `ignore`, which keeps the target's value, and `none`, which stops the migration at the first key that already exists on the target. `none` is the default. +- Without `resources`, the migration pod gets 500m CPU and 500Mi memory, with limits of 1 CPU and 1Gi. For large datasets, allocate 2–8 CPU; 4 is a good starting point. -### 5. Sentinel to Cluster +Apply the file with `kubectl apply -f`, or create the resource in the platform UI. redis-operator then creates the workload `-shake`: a Deployment for `sync`, or a Job for `rump`. -```yaml -apiVersion: middle.alauda.cn/v1alpha1 -kind: RedisShake -metadata: - name: sentinel-to-cluster -spec: - image: - keyExists: rewrite - modelType: sync - replicas: 1 - resumeFromBreakPoint: false - source: - address: - - "mymaster@:" - passwordSecret: - type: sentinel - target: - address: - - ":" - passwordSecret: - type: cluster -``` +### 4. Monitor the Migration -### 6. Cluster to Cluster +Check the migration pod and its log: -```yaml -apiVersion: middle.alauda.cn/v1alpha1 -kind: RedisShake -metadata: - name: cluster-to-cluster -spec: - image: - keyExists: rewrite - modelType: sync - replicas: 1 - source: - address: - - "master@:" - passwordSecret: - type: cluster - target: - address: - - ":" - passwordSecret: - type: cluster +```bash +kubectl -n get pods | grep -shake +kubectl -n logs deploy/-shake -c redis-shake --tail=50 # sync +kubectl -n logs job/-shake -c redis-shake --tail=50 # rump ``` -:::warning -- When the **source is a cluster**, prefix the address with `master@`. The target cluster address does **not** need this prefix. -- For both source and target clusters, any one of the cluster endpoints is sufficient — RedisShake discovers the rest of the topology automatically. -::: +- **`sync`**: the log line `DbSyncer[0] sync rdb done` means the full copy has finished and incremental replication has started. With a cluster source there is one `DbSyncer` per source shard. +- **`rump`**: the Job completes, and the log ends with `all rumpers finish!`. -## Verifying Synchronization +A `sync` migration also exposes Prometheus metrics on port 9320: -RedisShake performs continuous replication. Even after the initial dataset is copied, replication continues until you delete the `RedisShake` resource. You must therefore verify completion manually. +```bash +kubectl get --raw "/api/v1/namespaces//pods/:9320/proxy/metrics" \ + | grep -E 'redisshake_(full_sync_process_percent|fake_slave_delay_offset)' +``` -### Platform Version Below 3.10 +| Metric | Meaning | +| --- | --- | +| `redisshake_full_sync_process_percent` | Progress of the full copy; `100` when it is done. One series per source shard, labelled `db_syncer`. | +| `redisshake_fake_slave_delay_offset` | How far the target lags behind the source, in bytes. Replication heartbeats keep it moving between `0` and about 150 even with no writes; the target has caught up when it keeps returning to `0` while writes are paused. | -There is no monitoring dashboard on these versions. Use a key-count comparison: +redis-operator also creates a ServiceMonitor named `redis-shake` in its own namespace, so the platform Prometheus can collect these metrics. No Grafana dashboard is shipped for them. -1. Run `DBSIZE` on the source. For cluster sources, sum `DBSIZE` across all primaries. -2. Run `DBSIZE` on the target. For cluster targets, sum across all primaries. -3. With writes paused on the source, the totals should match when synchronization is complete. +### 5. Verify the Data -For cluster mode, the following one-liner returns the total key count across all primaries: +With writes paused on the source, compare the key counts on both sides. Use the `REDISCLI_AUTH` environment variable so the password does not appear on the command line: ```bash -redis-cli -a --cluster call : dbsize --cluster-only-masters +# standalone or Sentinel: run against the primary +REDISCLI_AUTH= redis-cli -h -p dbsize + +# cluster: prints one line per primary; add them up +REDISCLI_AUTH= redis-cli --cluster call : dbsize --cluster-only-masters ``` -### Platform Version 3.10.2 and Later +Keys with an expiry time can make the counts differ slightly, because each side expires keys on its own schedule. + +### 6. Cut Over and Clean Up -The key-count method still works. In addition, a Grafana dashboard **"Redis Shake Dashboard"** is available under **Platform Management > Operations Center > Monitoring > Grafana > Dashboards**. +1. Pause writes on the source. +2. For `sync`, wait until `redisshake_fake_slave_delay_offset` settles at `0`, then compare the key counts again. +3. Point the applications at the target. +4. Delete the resource. This removes the Deployment or Job, its ConfigMaps and its Service: -#### Sentinel Source + ```bash + kubectl -n delete redisshake + ``` -- `SyncProcessPercent` — reaches `100` when the initial sync is complete. -- `SlaveDelayOffset` — periodic dips to `0` indicate that the target has caught up. Brief spikes from replication ping packets are expected and not data. +5. Delete the password Secrets if you no longer need them. -#### Cluster Source +## Changing a Running Migration -- `SyncProcessPercent` — one progress series per source shard. All shards reaching the completion mark indicates initial sync is done. -- `SlaveDelayOffset` — when the offset reaches `0`, replication is up to date. +Edits to a running `RedisShake` resource only partly take effect: -## Appendix: RedisShake Image by Platform Version +- **Applied**: fields of the migration pod, such as `resources`, `nodeSelector`, `tolerations` and `affinity`. redis-operator updates the Deployment and the pod restarts. +- **Ignored**: `source`, `target`, `filter`, `keyExists`, `parallel` and the other migration settings. They are written into a ConfigMap only when the resource is created. -Use the RedisShake image matching your platform version: +To change a migration setting, delete the resource and create it again. A `rump` Job is never updated; to run it again, delete and recreate the resource. + +## Troubleshooting + +| Symptom | Cause | Action | +| --- | --- | --- | +| `invalid RDB version number 11, 9` (or `10, 9`) in the `redis-shake` log, and the container restarts | The RedisShake image predates Redis 7.x support | Upgrade redis-operator to a release later than 5.1.1, or use `modelType: rump` | +| `no sentinels available; last error: NOAUTH Authentication required.` | The Sentinel nodes have their own password | Connect to the data nodes directly, as described in step 2 | +| `target key name is busy` and the container restarts | `keyExists` is `none`, and the key already exists on the target | Set `keyExists: rewrite` or `ignore`, or empty the target first | +| `spec.template.spec.containers[0].image: Required value` in the operator log, and no migration pod | redis-operator earlier than 5.1 does not fill in the image | Set `spec.image` to the RedisShake image of your release | +| The `shake-init` container log shows `Could not connect to Redis at mymaster@...: Name does not resolve` | The init script also tries the Sentinel-style address as a host name | None needed; this message is harmless | +| Keys from databases other than 0 are missing on a cluster target | Redis Cluster has only database 0 | Move the keys to database 0 on the source before migrating | + +## Important Considerations + +- **Replication never stops automatically.** A `sync` migration keeps running until you delete the resource. Always verify before cutover. +- **Consumer groups on a cluster target.** Consumer groups that exist on the source when the migration starts are copied with their streams. But `XGROUP` commands run on the source *during* a migration to a cluster target can be sent to the wrong shard, so a group created then may be missing on the target, with no error. Create consumer groups before you start, or recreate them on the target after cutover. +- **No Redis 7.4 or later.** Their RDB format is not supported, and the migration stops at once. +- **No migration to an older Redis.** A Redis 7.x source cannot be migrated to a 5.x or 6.x target. +- **Network stability.** Persistent network instability between the two sides causes replication restarts and growing lag. Make sure the connection is stable before you start. + +## Appendix: Platform 3.x + +On platform 3.8.x – 3.16.x, the procedure above applies with these differences. + +**Set the image.** Add `spec.image` to the resource, using the image for your platform version: | ACP Version | RedisShake Image | | --- | --- | @@ -242,11 +231,8 @@ Use the RedisShake image matching your platform version: | 3.16.1 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2-9bb65a7b` | | 3.16.2 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2-3777a73b` | -## Important Considerations +**Redis versions.** Only Redis 5.x and 6.x sources and targets are supported. + +**Monitoring.** From 3.10.2, the Grafana dashboard **"Redis Shake Dashboard"** is available under **Platform Management > Operations Center > Monitoring > Grafana > Dashboards**. Its panel `SyncProcessPercent` reaches `100` when the full copy is done, and `SlaveDelayOffset` returns to `0` when the target has caught up. Before 3.10.2 there is no dashboard; compare key counts instead. -- **No Redis 7.x support.** Use a different migration path (for example RDB export/import) for Redis 7.x source or target. -- **Replication never stops automatically.** Always validate completion before cutover and delete the `RedisShake` resource when migration is finished. -- **Pause writes during cutover.** A short freeze on the source allows the offset to reach zero before applications are repointed. -- **CR is immutable.** To change the configuration, delete and recreate the `RedisShake` resource. -- **Sentinel and cluster address format.** Sentinel addresses require the `mymaster@` prefix; cluster source addresses require the `master@` prefix; cluster target addresses do not. -- **Network stability.** Persistent network instability between source and target leads to replication restarts and increased lag. Ensure stable connectivity before kicking off migration. +**Known issue.** On 3.16.0 and 3.16.1, when the source is a Sentinel instance, the source instance status may be reported as `Processing` even though the instance is healthy. This is fixed in 3.16.2. From 301c78f4aaf8c25089e4532bfa4cea798d1defe1 Mon Sep 17 00:00:00 2001 From: hfxia Date: Mon, 9 Nov 2026 01:19:49 -0500 Subject: [PATCH 2/2] docs(redis): translate the rewritten migration guide into Chinese The pull request check compares each translation with the English page the branch changes, and the Chinese page was still the translation of the old guide. Translate the new guide, and set its sourceSHA to the English file so the translation run on main keeps it. Give the two headings that the page links to explicit ids, so those links also work on the Chinese page. --- .../How_to_Migrate_Redis_Across_Clusters.md | 4 +- .../How_to_Migrate_Redis_Across_Clusters.md | 330 +++++++++--------- 2 files changed, 159 insertions(+), 175 deletions(-) diff --git a/docs/en/solutions/ecosystem/redis/How_to_Migrate_Redis_Across_Clusters.md b/docs/en/solutions/ecosystem/redis/How_to_Migrate_Redis_Across_Clusters.md index 0eb8e4cea..d61381931 100644 --- a/docs/en/solutions/ecosystem/redis/How_to_Migrate_Redis_Across_Clusters.md +++ b/docs/en/solutions/ecosystem/redis/How_to_Migrate_Redis_Across_Clusters.md @@ -31,7 +31,7 @@ A `RedisShake` resource runs in one of two modes: - **`sync`**: copies the full dataset, then keeps replicating every write made on the source. It never stops on its own; you end it by deleting the resource. Use it when the source must stay in service until cutover. - **`rump`**: copies the keys once, by scanning the source, and then finishes. Writes made after a key was copied are not carried over. Use it when you can stop writes on the source for the whole copy. -## Choose a Migration Mode +## Choose a Migration Mode {#choose-a-migration-mode} | Source Redis | Target Redis | Mode | | --- | --- | --- | @@ -215,7 +215,7 @@ To change a migration setting, delete the resource and create it again. A `rump` - **No migration to an older Redis.** A Redis 7.x source cannot be migrated to a 5.x or 6.x target. - **Network stability.** Persistent network instability between the two sides causes replication restarts and growing lag. Make sure the connection is stable before you start. -## Appendix: Platform 3.x +## Appendix: Platform 3.x {#appendix-platform-3x} On platform 3.8.x – 3.16.x, the procedure above applies with these differences. diff --git a/docs/zh/solutions/ecosystem/redis/How_to_Migrate_Redis_Across_Clusters.md b/docs/zh/solutions/ecosystem/redis/How_to_Migrate_Redis_Across_Clusters.md index b01ac8452..119ed02ab 100644 --- a/docs/zh/solutions/ecosystem/redis/How_to_Migrate_Redis_Across_Clusters.md +++ b/docs/zh/solutions/ecosystem/redis/How_to_Migrate_Redis_Across_Clusters.md @@ -5,251 +5,235 @@ kind: - Solution ProductsVersion: - 3.x + - 4.x id: KB260500095 -sourceSHA: 17d794a44095f6bc32371f8fdcf2387b7c1fad87834cc0bd16707f61f6b62bfa +sourceSHA: 22946602fd053867ad56641a5c11009970009a4f00a29b22bb9d450286020e3e --- # 在 Kubernetes 集群之间迁移 Redis 数据 :::info 适用版本 - -- 平台: **3.8.x – 3.16.x**(下表中的镜像截至 3.16.2) - ::: +- 平台 **4.x**,**redis-operator 5.1** 或更高版本。源端为 Redis 7.x 时,需要晚于 **5.1.1** 的 redis-operator 版本,参见[选择迁移模式](#choose-a-migration-mode)。 +- 平台 **3.8.x – 3.16.x**:操作步骤相同,但需要自行指定镜像,并且只支持 Redis 5.x 和 6.x。参见[附录:平台 3.x](#appendix-platform-3x)。 +::: ## 介绍 -本指南解释了如何使用平台内置的 **RedisShake** 迁移功能在不同 Kubernetes 集群中运行的 Redis 实例之间迁移数据。该操作步骤支持以下源/目标架构组合: +本文介绍如何使用平台内置的、基于 **RedisShake** 的迁移功能,把数据从一个 Redis 实例复制到另一个 Redis 实例,两个实例可以位于不同的 Kubernetes 集群中。一次迁移用一个 `RedisShake` 自定义资源描述,由 redis-operator 执行。源端和目标端都可以是以下任意一种架构: -| 架构 | 描述 | -| ----------- | -------------------------------------- | -| `standalone` | 单节点 Redis | -| `sentinel` | 基于 Sentinel 的主/从实例 | -| `cluster` | 分片 Redis 集群 | +| 架构 | 说明 | +| --- | --- | +| `standalone` | 单节点 Redis | +| `sentinel` | 基于 Sentinel 的主从实例 | +| `cluster` | 分片的 Redis Cluster | -RedisShake 执行持续复制:只要源实例继续接受写入,这些写入就会传播到目标实例。同步不会自动停止。 +`RedisShake` 资源有两种运行模式: -## 术语表 +- **`sync`**:先复制全量数据,再持续同步源端的每一次写入。它不会自行结束,删除资源后才会停止。适用于切换之前源端必须持续提供服务的场景。 +- **`rump`**:扫描源端,把所有 key 复制一次后结束。某个 key 复制完成后,源端对它的写入不会同步到目标端。适用于整个复制期间都可以停止源端写入的场景。 -| 术语 | 释义 | -| ------ | -------------------------------------------------- | -| Source | 正在迁移数据的 Redis 实例。 | -| Target | 接收迁移数据的 Redis 实例。 | +## 选择迁移模式 {#choose-a-migration-mode} -## 先决条件 +| 源端 Redis | 目标端 Redis | 模式 | +| --- | --- | --- | +| 5.x 或 6.x | 6.x 或 7.2 | `sync` | +| 7.0 或 7.2 | 7.2 | redis-operator 晚于 5.1.1 时使用 `sync`;更早的版本使用 `rump` | +| 7.x | 5.x 或 6.x | 不支持 | +| 7.4 或更高版本 | 任意 | 不支持 | -1. **容量检查。** 确保目标实例有足够的内存来容纳源数据集: - - 目标为 Sentinel:目标内存应至少为 **源数据集大小的 5/4**。 - - 目标为集群,源为单节点:目标内存应至少为 **源数据集大小的 5/4**。 - - 目标为集群,源为集群:目标内存应至少为 **最大源分片的数据集大小的 5/4**。 - - 为了获得可接受的性能,为每个 RedisShake pod 分配 **2-8 vCPU**(推荐 4 vCPU)。 -2. **减少或暂停** 源实例上的写入,以降低复制压力。 -3. **选择正确的 RedisShake 镜像** 以匹配您的平台版本(请参见 [附录](#appendix-redisshake-image-by-platform-version))。 -4. **网络连接。** RedisShake pod 必须能够通过稳定的 IPv4 网络访问源和目标实例。 -5. **Redis 版本。** 当前的 RedisShake 镜像 **不支持 Redis 7.x**。仅在 Redis 5.x 或 6.x 源/目标实例上使用此方法。 +redis-operator 5.1.1 之后的版本中,RedisShake 才支持 Redis 7.0 和 7.2 的 RDB 格式。起决定作用的是创建 `RedisShake` 资源的那个集群中的 redis-operator,因为迁移 pod 使用的是该 operator 自带的 RedisShake 镜像。如果那里的 redis-operator 版本更早,从 Redis 7.x 源端发起的 `sync` 迁移会立即失败,目标端保持为空,`redis-shake` 容器日志中会出现: -### 已知问题 +```text +[PANIC] parse rdb header error +[error]: verify version, invalid RDB version number 11, 9 +``` -- 在平台版本 **3.16.0** 和 **3.16.1** 中,当源为 Sentinel 实例时,源实例状态可能会错误地报告为 `Processing`,即使实例是健康的。状态报告在 **3.16.2** 中已修复。 +遇到这种情况,可以升级 redis-operator,也可以删除该资源,再以 `modelType: rump` 重新创建。 -## 迁移操作步骤 +## 前提条件 -对于下面的每种场景,您需要创建一个单独的 `RedisShake` 自定义资源。更新镜像、源/目标地址和密码 Secrets 以匹配您的环境,然后使用 `kubectl` 或平台 UI 应用该资源。要在部署后更改配置,请删除现有资源并创建一个新资源——不支持就地编辑。 +1. **容量。** 目标端的内存必须能容纳源端的数据集: + - 目标端为 Sentinel:至少为**源端数据集大小的 5/4**。 + - 目标端为集群、源端为单节点或 Sentinel:至少为**源端数据集大小的 5/4**。 + - 目标端为集群、源端也为集群:至少为**源端最大分片数据集大小的 5/4**。 +2. **迁移的运行位置。** 在安装了 redis-operator 的集群中创建 `RedisShake` 资源,通常是目标端所在的集群。迁移 pod 必须能通过稳定的 IPv4 网络访问两个实例。 +3. **外部访问。** 如果实例与 `RedisShake` 资源不在同一个集群中,该实例必须开启 NodePort 或 LoadBalancer 访问: + - 集群实例:RedisShake 按每个分片对外宣告的地址连接各个分片,因此集群内地址在其他集群中无法使用。 + - Sentinel 实例:数据节点(`spec.expose.type`)和 Sentinel 节点(`spec.sentinel.expose.type`)都要开启外部访问。RedisShake 先向 Sentinel 查询主节点的地址,再连接该主节点。 +4. **数据库。** Redis Cluster 只有 0 号数据库。目标端为集群时,源端其他数据库中的 key 会被**跳过,且不报任何错误**。迁移到集群之前,请在源端执行 `INFO keyspace`,确认所有 key 都在 `db0` 中。 +5. **Redis 函数。** 函数库(Redis 7 的 `FUNCTION LOAD`)不会被迁移。请在目标端单独加载,例如在源端执行 `FUNCTION DUMP`,再在目标端执行 `FUNCTION RESTORE`。 +6. **写入。** 减少源端的写入,以降低复制压力。使用 `rump` 时,整个复制期间都要停止写入。 -### 1. 创建密码 Secrets +## 迁移步骤 -如果任一方受密码保护,请为每一方创建一个 Kubernetes Secret。Secret 必须使用数据键 `password`。 +### 1. 创建密码 Secret + +如果某一端设置了密码,请在创建 `RedisShake` 资源的命名空间中为它创建一个 Secret,数据键为 `password`: ```bash kubectl -n create secret generic \ --from-literal=password= ``` -### 2. 从单节点到 Sentinel +### 2. 查找地址 -```yaml -apiVersion: middle.alauda.cn/v1alpha1 -kind: RedisShake -metadata: - name: standalone-to-sentinel -spec: - image: # 请参见附录以获取适合您版本的正确镜像 - keyExists: rewrite - modelType: sync - replicas: 1 - source: - address: - - ":" - passwordSecret: - type: standalone - target: - address: - - "mymaster@:" - passwordSecret: - type: sentinel -``` +在 **数据服务 > 实例详情 > 访问方式 > 集群内访问 / 外部访问** 中找到访问端点,再按架构和角色把每个地址写成对应的格式: -:::warning -当 **源或目标是 Sentinel 实例** 时,地址必须以 `mymaster@` 为前缀,值应为 Sentinel 访问端点。在 **数据服务 > 实例详情 > 访问方式 > 集群内访问 / 外部访问** 中找到它。 -::: +| 架构 | 作为源端 | 作为目标端 | +| --- | --- | --- | +| `standalone` | `:` | `:` | +| `sentinel` | `mymaster@:` | `mymaster@:` | +| `cluster` | `master@:` | `:` | -### 3. 从 Sentinel 到 Sentinel +- Sentinel 实例使用 **Sentinel** 端点。`mymaster` 是平台为每个 Sentinel 实例设置的主节点名称。 +- 集群实例只需填写任意一个节点的端点,RedisShake 会自动发现其他分片。只有作为**源端**的集群才需要 `master@` 前缀。 -```yaml -apiVersion: middle.alauda.cn/v1alpha1 -kind: RedisShake -metadata: - name: sentinel-to-sentinel -spec: - image: - keyExists: rewrite - modelType: sync - replicas: 1 - source: - address: - - "mymaster@:" - passwordSecret: - type: sentinel - target: - address: - - "mymaster@:" - passwordSecret: - type: sentinel +:::warning +如果实例的 Sentinel 节点设置了独立的密码(`spec.sentinel.passwordSecret`),无论用作源端还是目标端,`mymaster@` 地址都无法使用:RedisShake 无法通过 Sentinel 的认证,日志中会出现 `no sentinels available; last error: NOAUTH Authentication required.`。此时请直接连接数据节点:设置 `type: standalone`,使用**数据节点**密码的 Secret,并使用读写 Service `rfr--read-write` 的外部地址。查看它的 NodePort: + +```bash +kubectl -n get svc rfr--read-write \ + -o jsonpath='{.spec.ports[0].nodePort}' ``` +::: -### 4. 从单节点到集群 +### 3. 创建 RedisShake 资源 ```yaml apiVersion: middle.alauda.cn/v1alpha1 kind: RedisShake metadata: - name: standalone-to-cluster + name: + namespace: spec: - image: - keyExists: rewrite - modelType: sync - replicas: 1 + modelType: sync # 或 rump,参见“选择迁移模式” + keyExists: rewrite # 覆盖目标端已存在的 key source: + type: sentinel # standalone、sentinel 或 cluster address: - - ":" + - "mymaster@:" passwordSecret: - type: standalone target: + type: cluster # standalone、sentinel 或 cluster address: - ":" passwordSecret: - type: cluster + resources: # 可选 + requests: + cpu: "2" + memory: 2Gi + limits: + cpu: "4" + memory: 4Gi ``` -:::note -对于集群目标,地址可以是 **集群访问端点中的任意一个**。在 **数据服务 > 实例详情 > 访问方式 > 集群内访问 / 外部访问** 中找到它们。 -::: +- 按实际场景修改 `source` 和 `target`,地址格式见第 2 步。 +- 不要设置 `image`:redis-operator 5.1 及更高版本会使用该 operator 自带的 RedisShake 镜像。 +- `keyExists` 还可以取 `ignore`,即保留目标端的值;或取 `none`,即遇到第一个在目标端已存在的 key 时停止迁移。默认值为 `none`。 +- 不设置 `resources` 时,迁移 pod 的请求为 500m CPU 和 500Mi 内存,上限为 1 CPU 和 1Gi。数据集较大时,请分配 2–8 个 CPU,可以从 4 个开始。 -### 5. 从 Sentinel 到集群 +使用 `kubectl apply -f` 应用该文件,或在平台界面中创建该资源。随后 redis-operator 会创建工作负载 `-shake`:`sync` 模式为 Deployment,`rump` 模式为 Job。 -```yaml -apiVersion: middle.alauda.cn/v1alpha1 -kind: RedisShake -metadata: - name: sentinel-to-cluster -spec: - image: - keyExists: rewrite - modelType: sync - replicas: 1 - resumeFromBreakPoint: false - source: - address: - - "mymaster@:" - passwordSecret: - type: sentinel - target: - address: - - ":" - passwordSecret: - type: cluster -``` +### 4. 监控迁移 -### 6. 从集群到集群 +查看迁移 pod 及其日志: -```yaml -apiVersion: middle.alauda.cn/v1alpha1 -kind: RedisShake -metadata: - name: cluster-to-cluster -spec: - image: - keyExists: rewrite - modelType: sync - replicas: 1 - source: - address: - - "master@:" - passwordSecret: - type: cluster - target: - address: - - ":" - passwordSecret: - type: cluster +```bash +kubectl -n get pods | grep -shake +kubectl -n logs deploy/-shake -c redis-shake --tail=50 # sync +kubectl -n logs job/-shake -c redis-shake --tail=50 # rump ``` -:::warning +- **`sync`**:日志中出现 `DbSyncer[0] sync rdb done`,表示全量复制已完成,增量同步已开始。源端为集群时,每个源端分片对应一个 `DbSyncer`。 +- **`rump`**:Job 完成,日志以 `all rumpers finish!` 结尾。 -- 当 **源是集群** 时,地址前缀为 `master@`。目标集群地址 **不需要** 此前缀。 -- 对于源和目标集群,任意一个集群端点即可——RedisShake 会自动发现其余拓扑。 - ::: +`sync` 迁移还会在 9320 端口提供 Prometheus 指标: -## 验证同步 - -RedisShake 执行持续复制。即使初始数据集已复制,复制仍会继续,直到您删除 `RedisShake` 资源。因此,您必须手动验证完成情况。 +```bash +kubectl get --raw "/api/v1/namespaces//pods/:9320/proxy/metrics" \ + | grep -E 'redisshake_(full_sync_process_percent|fake_slave_delay_offset)' +``` -### 平台版本低于 3.10 +| 指标 | 含义 | +| --- | --- | +| `redisshake_full_sync_process_percent` | 全量复制的进度,完成时为 `100`。每个源端分片一条时间序列,标签为 `db_syncer`。 | +| `redisshake_fake_slave_delay_offset` | 目标端落后于源端的字节数。即使没有写入,复制心跳也会让它在 `0` 到 150 左右之间变化;暂停写入后,如果它反复回到 `0`,说明目标端已经追上源端。 | -这些版本没有监控面板。使用键计数比较: +redis-operator 还会在它自己所在的命名空间中创建名为 `redis-shake` 的 ServiceMonitor,供平台的 Prometheus 采集这些指标。平台没有为这些指标提供 Grafana 仪表板。 -1. 在源上运行 `DBSIZE`。对于集群源,计算所有主节点的 `DBSIZE` 之和。 -2. 在目标上运行 `DBSIZE`。对于集群目标,计算所有主节点的总和。 -3. 在源上暂停写入时,当同步完成时,总数应匹配。 +### 5. 校验数据 -对于集群模式,以下一行命令返回所有主节点的总键计数: +暂停源端写入后,比较两端的 key 数量。请通过 `REDISCLI_AUTH` 环境变量传入密码,以免密码出现在命令行中: ```bash -redis-cli -a --cluster call : dbsize --cluster-only-masters +# 单节点或 Sentinel:在主节点上执行 +REDISCLI_AUTH= redis-cli -h -p dbsize + +# 集群:每个主节点输出一行,把它们相加 +REDISCLI_AUTH= redis-cli --cluster call : dbsize --cluster-only-masters ``` -### 平台版本 3.10.2 及以上 +设置了过期时间的 key 可能使两端的数量略有差异,因为两端各自按自己的节奏让 key 过期。 -键计数方法仍然有效。此外,在 **平台管理 > 操作中心 > 监控 > Grafana > 仪表板** 下提供了 **"Redis Shake Dashboard"**。 +### 6. 切换与清理 -#### Sentinel 源 +1. 暂停源端写入。 +2. 对于 `sync`,等待 `redisshake_fake_slave_delay_offset` 稳定在 `0`,再次比较 key 数量。 +3. 把应用切换到目标端。 +4. 删除该资源。这会同时删除 Deployment 或 Job,以及它的 ConfigMap 和 Service: -- `SyncProcessPercent` — 当初始同步完成时达到 `100`。 -- `SlaveDelayOffset` — 定期下降到 `0` 表示目标已赶上。来自复制 ping 数据包的短暂峰值是预期的,而不是数据。 + ```bash + kubectl -n delete redisshake + ``` -#### 集群源 +5. 如果不再需要密码 Secret,将它们删除。 -- `SyncProcessPercent` — 每个源分片一个进度系列。所有分片达到完成标记表示初始同步完成。 -- `SlaveDelayOffset` — 当偏移量达到 `0` 时,复制已更新。 +## 修改运行中的迁移 -## 附录:按平台版本的 RedisShake 镜像 +对运行中的 `RedisShake` 资源所做的修改,只有一部分会生效: -使用与您的平台版本匹配的 RedisShake 镜像: +- **会生效**:迁移 pod 的字段,例如 `resources`、`nodeSelector`、`tolerations` 和 `affinity`。redis-operator 会更新 Deployment,pod 随之重启。 +- **不会生效**:`source`、`target`、`filter`、`keyExists`、`parallel` 以及其他迁移设置。这些设置只在创建资源时写入 ConfigMap。 -| ACP 版本 | RedisShake 镜像 | -| ---------------- | ------------------------------------------------------------- | -| 3.8.1 - 3.8.3 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2` | -| 3.10.1 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2` | -| 3.10.2, 3.10.3 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.3` | -| 3.12.1 - 3.12.3 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2-5ad6d091` | -| 3.14.1, 3.14.2 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2-063e3b5d` | -| 3.16.1 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2-9bb65a7b` | -| 3.16.2 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2-3777a73b` | +要修改迁移设置,请删除该资源后重新创建。`rump` 的 Job 从不更新;要再次执行,请删除并重新创建该资源。 + +## 故障排查 + +| 现象 | 原因 | 处理 | +| --- | --- | --- | +| `redis-shake` 日志中出现 `invalid RDB version number 11, 9`(或 `10, 9`),容器随之重启 | RedisShake 镜像早于支持 Redis 7.x 的版本 | 把 redis-operator 升级到晚于 5.1.1 的版本,或使用 `modelType: rump` | +| `no sentinels available; last error: NOAUTH Authentication required.` | Sentinel 节点设置了独立的密码 | 按第 2 步的说明,直接连接数据节点 | +| 出现 `target key name is busy`,容器随之重启 | `keyExists` 为 `none`,且该 key 已存在于目标端 | 设置 `keyExists: rewrite` 或 `ignore`,或先清空目标端 | +| operator 日志中出现 `spec.template.spec.containers[0].image: Required value`,且没有迁移 pod | 早于 5.1 的 redis-operator 不会自动填写镜像 | 把 `spec.image` 设置为对应版本的 RedisShake 镜像 | +| `shake-init` 容器日志显示 `Could not connect to Redis at mymaster@...: Name does not resolve` | 初始化脚本也会把 Sentinel 格式的地址当作主机名尝试连接 | 无需处理,该信息无害 | +| 集群目标端缺少源端 0 号以外数据库中的 key | Redis Cluster 只有 0 号数据库 | 迁移前在源端把这些 key 移到 0 号数据库 | ## 重要注意事项 -- **不支持 Redis 7.x。** 对于 Redis 7.x 源或目标,请使用其他迁移路径(例如 RDB 导出/导入)。 -- **复制不会自动停止。** 在切换之前始终验证完成情况,并在迁移完成后删除 `RedisShake` 资源。 -- **在切换期间暂停写入。** 源上的短暂冻结允许偏移量在应用程序重新指向之前达到零。 -- **CR 是不可变的。** 要更改配置,请删除并重新创建 `RedisShake` 资源。 -- **Sentinel 和集群地址格式。** Sentinel 地址需要 `mymaster@` 前缀;集群源地址需要 `master@` 前缀;集群目标地址不需要。 -- **网络稳定性。** 源和目标之间的持续网络不稳定会导致复制重启和堆积量增加。在开始迁移之前确保连接稳定。 +- **复制不会自动停止。** `sync` 迁移会一直运行,直到删除该资源。切换之前务必先校验数据。 +- **集群目标端上的消费组。** 迁移开始时源端已有的消费组会随 stream 一起复制。但迁移到集群目标端*期间*,在源端执行的 `XGROUP` 命令可能被发往错误的分片,因此这期间创建的消费组可能在目标端缺失,且没有任何报错。请在开始迁移之前创建消费组,或在切换后到目标端重新创建。 +- **不支持 Redis 7.4 及更高版本。** 它们的 RDB 格式不受支持,迁移会立即停止。 +- **不支持迁移到更低版本的 Redis。** Redis 7.x 源端不能迁移到 5.x 或 6.x 目标端。 +- **网络稳定性。** 两端之间持续的网络不稳定会导致复制反复重启、延迟不断增大。开始迁移之前,请确保连接稳定。 + +## 附录:平台 3.x {#appendix-platform-3x} + +在平台 3.8.x – 3.16.x 上,上述步骤同样适用,区别如下。 + +**设置镜像。** 在资源中添加 `spec.image`,使用与平台版本对应的镜像: + +| ACP 版本 | RedisShake 镜像 | +| --- | --- | +| 3.8.1 - 3.8.3 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2` | +| 3.10.1 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2` | +| 3.10.2, 3.10.3 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.3` | +| 3.12.1 - 3.12.3 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2-5ad6d091` | +| 3.14.1, 3.14.2 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2-063e3b5d` | +| 3.16.1 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2-9bb65a7b` | +| 3.16.2 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2-3777a73b` | + +**Redis 版本。** 源端和目标端都只支持 Redis 5.x 和 6.x。 + +**监控。** 从 3.10.2 起,可以在 **平台管理 > 操作中心 > 监控 > Grafana > 仪表板** 下使用 Grafana 仪表板 **"Redis Shake Dashboard"**。全量复制完成时,其面板 `SyncProcessPercent` 达到 `100`;目标端追上源端时,`SlaveDelayOffset` 回到 `0`。3.10.2 之前没有该仪表板,请改为比较 key 数量。 + +**已知问题。** 在 3.16.0 和 3.16.1 上,源端为 Sentinel 实例时,即使实例运行正常,源实例的状态也可能显示为 `Processing`。该问题已在 3.16.2 中修复。