diff --git a/docs/en/solutions/ecosystem/redis/How_to_Migrate_Redis_Across_Clusters.md b/docs/en/solutions/ecosystem/redis/How_to_Migrate_Redis_Across_Clusters.md index 28999e1d1..d61381931 100644 --- a/docs/en/solutions/ecosystem/redis/How_to_Migrate_Redis_Across_Clusters.md +++ b/docs/en/solutions/ecosystem/redis/How_to_Migrate_Redis_Across_Clusters.md @@ -5,18 +5,20 @@ kind: - Solution ProductsVersion: - 3.x + - 4.x id: KB260500095 --- # Migrate Redis Data Across Kubernetes Clusters :::info Applicable Versions -- Platform: **3.8.x – 3.16.x** (image table below stops at 3.16.2) +- Platform **4.x** with **redis-operator 5.1** or later. Redis 7.x sources need a redis-operator release later than **5.1.1**; see [Choose a Migration Mode](#choose-a-migration-mode). +- Platform **3.8.x – 3.16.x**: the procedure is the same, but you must set the image yourself and only Redis 5.x and 6.x are supported. See [Appendix: Platform 3.x](#appendix-platform-3x). ::: ## Introduction -This guide explains how to migrate data between Redis instances running in different Kubernetes clusters using the platform's built-in **RedisShake**-based migration feature. The procedure supports the following source/target architecture combinations: +This guide explains how to copy data from one Redis instance to another, including instances in different Kubernetes clusters, with the platform's built-in migration based on **RedisShake**. You describe a migration with one `RedisShake` custom resource, and redis-operator runs it. The source and the target can each be any of these architectures: | Architecture | Description | | --- | --- | @@ -24,213 +26,200 @@ This guide explains how to migrate data between Redis instances running in diffe | `sentinel` | Sentinel-based primary/replica instance | | `cluster` | Sharded Redis Cluster | -RedisShake performs continuous replication: as long as the source keeps accepting writes, those writes are propagated to the target. Synchronization does not stop on its own. +A `RedisShake` resource runs in one of two modes: -## Glossary +- **`sync`**: copies the full dataset, then keeps replicating every write made on the source. It never stops on its own; you end it by deleting the resource. Use it when the source must stay in service until cutover. +- **`rump`**: copies the keys once, by scanning the source, and then finishes. Writes made after a key was copied are not carried over. Use it when you can stop writes on the source for the whole copy. -| Term | Meaning | -| --- | --- | -| Source | The Redis instance whose data is being migrated. | -| Target | The Redis instance that receives the migrated data. | +## Choose a Migration Mode {#choose-a-migration-mode} -## Prerequisites +| Source Redis | Target Redis | Mode | +| --- | --- | --- | +| 5.x or 6.x | 6.x or 7.2 | `sync` | +| 7.0 or 7.2 | 7.2 | `sync` on redis-operator later than 5.1.1; `rump` on earlier releases | +| 7.x | 5.x or 6.x | Not supported | +| 7.4 or later | Any | Not supported | -1. **Capacity check.** Ensure the target has enough memory to hold the source dataset: - - Target is sentinel: target memory should be at least **5/4 of source dataset size**. - - Target is cluster, source is standalone: target memory should be at least **5/4 of source dataset size**. - - Target is cluster, source is cluster: target memory should be at least **5/4 of the largest source shard's dataset size**. - - For acceptable performance, allocate **2-8 vCPU** per RedisShake pod (4 vCPU recommended). -2. **Reduce or pause writes** on the source to lower replication pressure. -3. **Pick the correct RedisShake image** for your platform version (see the [appendix](#appendix-redisshake-image-by-platform-version)). -4. **Network connectivity.** The source and target instances must be reachable from the RedisShake pod over a stable IPv4 network. -5. **Redis version.** The current RedisShake image **does not support Redis 7.x**. Use this method only with Redis 5.x or 6.x source/target instances. +RedisShake added support for the RDB format of Redis 7.0 and 7.2 after redis-operator 5.1.1. What matters is the redis-operator in the cluster where you create the `RedisShake` resource, because the migration pod uses the RedisShake image delivered with that operator. With an earlier release there, a `sync` migration from a Redis 7.x source fails at once, the target stays empty, and the `redis-shake` container log shows: -### Known Issues +```text +[PANIC] parse rdb header error +[error]: verify version, invalid RDB version number 11, 9 +``` -- On platform versions **3.16.0** and **3.16.1**, when the source is a Sentinel instance, the source instance status may be incorrectly reported as `Processing` even though the instance is healthy. The status reporting is fixed in **3.16.2**. +If you see this, either upgrade redis-operator or delete the resource and recreate it with `modelType: rump`. -## Migration Procedure +## Prerequisites -For every scenario below, you create a single `RedisShake` custom resource. Update the image, source/target addresses, and password Secrets to match your environment, then apply the resource using `kubectl` or the platform UI. To change a configuration after deployment, delete the existing resource and create a new one — in-place edits are not supported. +1. **Capacity.** The target must have enough memory for the source dataset: + - Sentinel target: at least **5/4 of the source dataset size**. + - Cluster target with a standalone or Sentinel source: at least **5/4 of the source dataset size**. + - Cluster target with a cluster source: at least **5/4 of the dataset size of the largest source shard**. +2. **Where the migration runs.** Create the `RedisShake` resource in a cluster where redis-operator is installed, usually the target cluster. The migration pod must reach both instances over a stable IPv4 network. +3. **External access.** An instance in a different cluster from the `RedisShake` resource must have NodePort or LoadBalancer access enabled: + - Cluster instance: RedisShake connects to every shard at the address the shard announces, so in-cluster addresses do not work from another cluster. + - Sentinel instance: enable external access for both the data nodes (`spec.expose.type`) and the Sentinel nodes (`spec.sentinel.expose.type`). RedisShake asks Sentinel for the primary's address and then connects to that primary. +4. **Databases.** A Redis Cluster has only database 0. When the target is a cluster, keys in other databases of the source are **skipped without any error**. Run `INFO keyspace` on the source and make sure every key is in `db0` before you migrate to a cluster. +5. **Redis functions.** Function libraries (Redis 7 `FUNCTION LOAD`) are not migrated. Load them on the target separately, for example with `FUNCTION DUMP` on the source and `FUNCTION RESTORE` on the target. +6. **Writes.** Reduce writes on the source to lower replication pressure. With `rump`, stop writes for the whole copy. -### 1. Create Password Secrets +## Migration Procedure + +### 1. Create the Password Secrets -If either side is password-protected, create a Kubernetes Secret per side. The Secret must use the data key `password`. +If a side is password-protected, create one Secret for it, with the data key `password`, in the namespace where you create the `RedisShake` resource: ```bash kubectl -n create secret generic \ --from-literal=password= ``` -### 2. Standalone to Sentinel +### 2. Find the Addresses -```yaml -apiVersion: middle.alauda.cn/v1alpha1 -kind: RedisShake -metadata: - name: standalone-to-sentinel -spec: - image: # See appendix for the right image for your version - keyExists: rewrite - modelType: sync - replicas: 1 - source: - address: - - ":" - passwordSecret: - type: standalone - target: - address: - - "mymaster@:" - passwordSecret: - type: sentinel -``` +Find the access endpoints in **Data Services > Instance Detail > Access Method > In-cluster Access / External Access**, then write each address in the format for its architecture and role: -:::warning -When the **source or target is a Sentinel instance**, the address must be prefixed with `mymaster@`, and the value should be the Sentinel access endpoint. Find it in **Data Services > Instance Detail > Access Method > In-cluster Access / External Access**. -::: +| Architecture | As source | As target | +| --- | --- | --- | +| `standalone` | `:` | `:` | +| `sentinel` | `mymaster@:` | `mymaster@:` | +| `cluster` | `master@:` | `:` | -### 3. Sentinel to Sentinel +- For a Sentinel instance, use the **Sentinel** endpoint. `mymaster` is the primary name the platform gives every Sentinel instance. +- For a cluster instance, any one node endpoint is enough: RedisShake discovers the other shards. Only a cluster **source** takes the `master@` prefix. -```yaml -apiVersion: middle.alauda.cn/v1alpha1 -kind: RedisShake -metadata: - name: sentinel-to-sentinel -spec: - image: - keyExists: rewrite - modelType: sync - replicas: 1 - source: - address: - - "mymaster@:" - passwordSecret: - type: sentinel - target: - address: - - "mymaster@:" - passwordSecret: - type: sentinel +:::warning +If the Sentinel nodes of an instance have their own password (`spec.sentinel.passwordSecret`), the `mymaster@` address does not work on either side: RedisShake cannot authenticate to Sentinel and the log shows `no sentinels available; last error: NOAUTH Authentication required.` Connect to the data nodes directly instead. Set `type: standalone`, use the Secret of the **data node** password, and use the external address of the read-write Service `rfr--read-write`. To read its NodePort: + +```bash +kubectl -n get svc rfr--read-write \ + -o jsonpath='{.spec.ports[0].nodePort}' ``` +::: -### 4. Standalone to Cluster +### 3. Create the RedisShake Resource ```yaml apiVersion: middle.alauda.cn/v1alpha1 kind: RedisShake metadata: - name: standalone-to-cluster + name: + namespace: spec: - image: - keyExists: rewrite - modelType: sync - replicas: 1 + modelType: sync # or rump, see "Choose a Migration Mode" + keyExists: rewrite # overwrite keys that already exist on the target source: + type: sentinel # standalone, sentinel or cluster address: - - ":" + - "mymaster@:" passwordSecret: - type: standalone target: + type: cluster # standalone, sentinel or cluster address: - ":" passwordSecret: - type: cluster + resources: # optional + requests: + cpu: "2" + memory: 2Gi + limits: + cpu: "4" + memory: 4Gi ``` -:::note -For a cluster target, the address can be **any one** of the cluster access endpoints. Find them in **Data Services > Instance Detail > Access Method > In-cluster Access / External Access**. -::: +- Change `source` and `target` for your scenario, using the address formats from step 2. +- Leave out `image`: redis-operator 5.1 and later use the RedisShake image delivered with the operator. +- `keyExists` also accepts `ignore`, which keeps the target's value, and `none`, which stops the migration at the first key that already exists on the target. `none` is the default. +- Without `resources`, the migration pod gets 500m CPU and 500Mi memory, with limits of 1 CPU and 1Gi. For large datasets, allocate 2–8 CPU; 4 is a good starting point. -### 5. Sentinel to Cluster +Apply the file with `kubectl apply -f`, or create the resource in the platform UI. redis-operator then creates the workload `-shake`: a Deployment for `sync`, or a Job for `rump`. -```yaml -apiVersion: middle.alauda.cn/v1alpha1 -kind: RedisShake -metadata: - name: sentinel-to-cluster -spec: - image: - keyExists: rewrite - modelType: sync - replicas: 1 - resumeFromBreakPoint: false - source: - address: - - "mymaster@:" - passwordSecret: - type: sentinel - target: - address: - - ":" - passwordSecret: - type: cluster -``` +### 4. Monitor the Migration -### 6. Cluster to Cluster +Check the migration pod and its log: -```yaml -apiVersion: middle.alauda.cn/v1alpha1 -kind: RedisShake -metadata: - name: cluster-to-cluster -spec: - image: - keyExists: rewrite - modelType: sync - replicas: 1 - source: - address: - - "master@:" - passwordSecret: - type: cluster - target: - address: - - ":" - passwordSecret: - type: cluster +```bash +kubectl -n get pods | grep -shake +kubectl -n logs deploy/-shake -c redis-shake --tail=50 # sync +kubectl -n logs job/-shake -c redis-shake --tail=50 # rump ``` -:::warning -- When the **source is a cluster**, prefix the address with `master@`. The target cluster address does **not** need this prefix. -- For both source and target clusters, any one of the cluster endpoints is sufficient — RedisShake discovers the rest of the topology automatically. -::: +- **`sync`**: the log line `DbSyncer[0] sync rdb done` means the full copy has finished and incremental replication has started. With a cluster source there is one `DbSyncer` per source shard. +- **`rump`**: the Job completes, and the log ends with `all rumpers finish!`. -## Verifying Synchronization +A `sync` migration also exposes Prometheus metrics on port 9320: -RedisShake performs continuous replication. Even after the initial dataset is copied, replication continues until you delete the `RedisShake` resource. You must therefore verify completion manually. +```bash +kubectl get --raw "/api/v1/namespaces//pods/:9320/proxy/metrics" \ + | grep -E 'redisshake_(full_sync_process_percent|fake_slave_delay_offset)' +``` -### Platform Version Below 3.10 +| Metric | Meaning | +| --- | --- | +| `redisshake_full_sync_process_percent` | Progress of the full copy; `100` when it is done. One series per source shard, labelled `db_syncer`. | +| `redisshake_fake_slave_delay_offset` | How far the target lags behind the source, in bytes. Replication heartbeats keep it moving between `0` and about 150 even with no writes; the target has caught up when it keeps returning to `0` while writes are paused. | -There is no monitoring dashboard on these versions. Use a key-count comparison: +redis-operator also creates a ServiceMonitor named `redis-shake` in its own namespace, so the platform Prometheus can collect these metrics. No Grafana dashboard is shipped for them. -1. Run `DBSIZE` on the source. For cluster sources, sum `DBSIZE` across all primaries. -2. Run `DBSIZE` on the target. For cluster targets, sum across all primaries. -3. With writes paused on the source, the totals should match when synchronization is complete. +### 5. Verify the Data -For cluster mode, the following one-liner returns the total key count across all primaries: +With writes paused on the source, compare the key counts on both sides. Use the `REDISCLI_AUTH` environment variable so the password does not appear on the command line: ```bash -redis-cli -a --cluster call : dbsize --cluster-only-masters +# standalone or Sentinel: run against the primary +REDISCLI_AUTH= redis-cli -h -p dbsize + +# cluster: prints one line per primary; add them up +REDISCLI_AUTH= redis-cli --cluster call : dbsize --cluster-only-masters ``` -### Platform Version 3.10.2 and Later +Keys with an expiry time can make the counts differ slightly, because each side expires keys on its own schedule. + +### 6. Cut Over and Clean Up -The key-count method still works. In addition, a Grafana dashboard **"Redis Shake Dashboard"** is available under **Platform Management > Operations Center > Monitoring > Grafana > Dashboards**. +1. Pause writes on the source. +2. For `sync`, wait until `redisshake_fake_slave_delay_offset` settles at `0`, then compare the key counts again. +3. Point the applications at the target. +4. Delete the resource. This removes the Deployment or Job, its ConfigMaps and its Service: -#### Sentinel Source + ```bash + kubectl -n delete redisshake + ``` -- `SyncProcessPercent` — reaches `100` when the initial sync is complete. -- `SlaveDelayOffset` — periodic dips to `0` indicate that the target has caught up. Brief spikes from replication ping packets are expected and not data. +5. Delete the password Secrets if you no longer need them. -#### Cluster Source +## Changing a Running Migration -- `SyncProcessPercent` — one progress series per source shard. All shards reaching the completion mark indicates initial sync is done. -- `SlaveDelayOffset` — when the offset reaches `0`, replication is up to date. +Edits to a running `RedisShake` resource only partly take effect: -## Appendix: RedisShake Image by Platform Version +- **Applied**: fields of the migration pod, such as `resources`, `nodeSelector`, `tolerations` and `affinity`. redis-operator updates the Deployment and the pod restarts. +- **Ignored**: `source`, `target`, `filter`, `keyExists`, `parallel` and the other migration settings. They are written into a ConfigMap only when the resource is created. -Use the RedisShake image matching your platform version: +To change a migration setting, delete the resource and create it again. A `rump` Job is never updated; to run it again, delete and recreate the resource. + +## Troubleshooting + +| Symptom | Cause | Action | +| --- | --- | --- | +| `invalid RDB version number 11, 9` (or `10, 9`) in the `redis-shake` log, and the container restarts | The RedisShake image predates Redis 7.x support | Upgrade redis-operator to a release later than 5.1.1, or use `modelType: rump` | +| `no sentinels available; last error: NOAUTH Authentication required.` | The Sentinel nodes have their own password | Connect to the data nodes directly, as described in step 2 | +| `target key name is busy` and the container restarts | `keyExists` is `none`, and the key already exists on the target | Set `keyExists: rewrite` or `ignore`, or empty the target first | +| `spec.template.spec.containers[0].image: Required value` in the operator log, and no migration pod | redis-operator earlier than 5.1 does not fill in the image | Set `spec.image` to the RedisShake image of your release | +| The `shake-init` container log shows `Could not connect to Redis at mymaster@...: Name does not resolve` | The init script also tries the Sentinel-style address as a host name | None needed; this message is harmless | +| Keys from databases other than 0 are missing on a cluster target | Redis Cluster has only database 0 | Move the keys to database 0 on the source before migrating | + +## Important Considerations + +- **Replication never stops automatically.** A `sync` migration keeps running until you delete the resource. Always verify before cutover. +- **Consumer groups on a cluster target.** Consumer groups that exist on the source when the migration starts are copied with their streams. But `XGROUP` commands run on the source *during* a migration to a cluster target can be sent to the wrong shard, so a group created then may be missing on the target, with no error. Create consumer groups before you start, or recreate them on the target after cutover. +- **No Redis 7.4 or later.** Their RDB format is not supported, and the migration stops at once. +- **No migration to an older Redis.** A Redis 7.x source cannot be migrated to a 5.x or 6.x target. +- **Network stability.** Persistent network instability between the two sides causes replication restarts and growing lag. Make sure the connection is stable before you start. + +## Appendix: Platform 3.x {#appendix-platform-3x} + +On platform 3.8.x – 3.16.x, the procedure above applies with these differences. + +**Set the image.** Add `spec.image` to the resource, using the image for your platform version: | ACP Version | RedisShake Image | | --- | --- | @@ -242,11 +231,8 @@ Use the RedisShake image matching your platform version: | 3.16.1 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2-9bb65a7b` | | 3.16.2 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2-3777a73b` | -## Important Considerations +**Redis versions.** Only Redis 5.x and 6.x sources and targets are supported. + +**Monitoring.** From 3.10.2, the Grafana dashboard **"Redis Shake Dashboard"** is available under **Platform Management > Operations Center > Monitoring > Grafana > Dashboards**. Its panel `SyncProcessPercent` reaches `100` when the full copy is done, and `SlaveDelayOffset` returns to `0` when the target has caught up. Before 3.10.2 there is no dashboard; compare key counts instead. -- **No Redis 7.x support.** Use a different migration path (for example RDB export/import) for Redis 7.x source or target. -- **Replication never stops automatically.** Always validate completion before cutover and delete the `RedisShake` resource when migration is finished. -- **Pause writes during cutover.** A short freeze on the source allows the offset to reach zero before applications are repointed. -- **CR is immutable.** To change the configuration, delete and recreate the `RedisShake` resource. -- **Sentinel and cluster address format.** Sentinel addresses require the `mymaster@` prefix; cluster source addresses require the `master@` prefix; cluster target addresses do not. -- **Network stability.** Persistent network instability between source and target leads to replication restarts and increased lag. Ensure stable connectivity before kicking off migration. +**Known issue.** On 3.16.0 and 3.16.1, when the source is a Sentinel instance, the source instance status may be reported as `Processing` even though the instance is healthy. This is fixed in 3.16.2. diff --git a/docs/zh/solutions/ecosystem/redis/How_to_Migrate_Redis_Across_Clusters.md b/docs/zh/solutions/ecosystem/redis/How_to_Migrate_Redis_Across_Clusters.md index b01ac8452..119ed02ab 100644 --- a/docs/zh/solutions/ecosystem/redis/How_to_Migrate_Redis_Across_Clusters.md +++ b/docs/zh/solutions/ecosystem/redis/How_to_Migrate_Redis_Across_Clusters.md @@ -5,251 +5,235 @@ kind: - Solution ProductsVersion: - 3.x + - 4.x id: KB260500095 -sourceSHA: 17d794a44095f6bc32371f8fdcf2387b7c1fad87834cc0bd16707f61f6b62bfa +sourceSHA: 22946602fd053867ad56641a5c11009970009a4f00a29b22bb9d450286020e3e --- # 在 Kubernetes 集群之间迁移 Redis 数据 :::info 适用版本 - -- 平台: **3.8.x – 3.16.x**(下表中的镜像截至 3.16.2) - ::: +- 平台 **4.x**,**redis-operator 5.1** 或更高版本。源端为 Redis 7.x 时,需要晚于 **5.1.1** 的 redis-operator 版本,参见[选择迁移模式](#choose-a-migration-mode)。 +- 平台 **3.8.x – 3.16.x**:操作步骤相同,但需要自行指定镜像,并且只支持 Redis 5.x 和 6.x。参见[附录:平台 3.x](#appendix-platform-3x)。 +::: ## 介绍 -本指南解释了如何使用平台内置的 **RedisShake** 迁移功能在不同 Kubernetes 集群中运行的 Redis 实例之间迁移数据。该操作步骤支持以下源/目标架构组合: +本文介绍如何使用平台内置的、基于 **RedisShake** 的迁移功能,把数据从一个 Redis 实例复制到另一个 Redis 实例,两个实例可以位于不同的 Kubernetes 集群中。一次迁移用一个 `RedisShake` 自定义资源描述,由 redis-operator 执行。源端和目标端都可以是以下任意一种架构: -| 架构 | 描述 | -| ----------- | -------------------------------------- | -| `standalone` | 单节点 Redis | -| `sentinel` | 基于 Sentinel 的主/从实例 | -| `cluster` | 分片 Redis 集群 | +| 架构 | 说明 | +| --- | --- | +| `standalone` | 单节点 Redis | +| `sentinel` | 基于 Sentinel 的主从实例 | +| `cluster` | 分片的 Redis Cluster | -RedisShake 执行持续复制:只要源实例继续接受写入,这些写入就会传播到目标实例。同步不会自动停止。 +`RedisShake` 资源有两种运行模式: -## 术语表 +- **`sync`**:先复制全量数据,再持续同步源端的每一次写入。它不会自行结束,删除资源后才会停止。适用于切换之前源端必须持续提供服务的场景。 +- **`rump`**:扫描源端,把所有 key 复制一次后结束。某个 key 复制完成后,源端对它的写入不会同步到目标端。适用于整个复制期间都可以停止源端写入的场景。 -| 术语 | 释义 | -| ------ | -------------------------------------------------- | -| Source | 正在迁移数据的 Redis 实例。 | -| Target | 接收迁移数据的 Redis 实例。 | +## 选择迁移模式 {#choose-a-migration-mode} -## 先决条件 +| 源端 Redis | 目标端 Redis | 模式 | +| --- | --- | --- | +| 5.x 或 6.x | 6.x 或 7.2 | `sync` | +| 7.0 或 7.2 | 7.2 | redis-operator 晚于 5.1.1 时使用 `sync`;更早的版本使用 `rump` | +| 7.x | 5.x 或 6.x | 不支持 | +| 7.4 或更高版本 | 任意 | 不支持 | -1. **容量检查。** 确保目标实例有足够的内存来容纳源数据集: - - 目标为 Sentinel:目标内存应至少为 **源数据集大小的 5/4**。 - - 目标为集群,源为单节点:目标内存应至少为 **源数据集大小的 5/4**。 - - 目标为集群,源为集群:目标内存应至少为 **最大源分片的数据集大小的 5/4**。 - - 为了获得可接受的性能,为每个 RedisShake pod 分配 **2-8 vCPU**(推荐 4 vCPU)。 -2. **减少或暂停** 源实例上的写入,以降低复制压力。 -3. **选择正确的 RedisShake 镜像** 以匹配您的平台版本(请参见 [附录](#appendix-redisshake-image-by-platform-version))。 -4. **网络连接。** RedisShake pod 必须能够通过稳定的 IPv4 网络访问源和目标实例。 -5. **Redis 版本。** 当前的 RedisShake 镜像 **不支持 Redis 7.x**。仅在 Redis 5.x 或 6.x 源/目标实例上使用此方法。 +redis-operator 5.1.1 之后的版本中,RedisShake 才支持 Redis 7.0 和 7.2 的 RDB 格式。起决定作用的是创建 `RedisShake` 资源的那个集群中的 redis-operator,因为迁移 pod 使用的是该 operator 自带的 RedisShake 镜像。如果那里的 redis-operator 版本更早,从 Redis 7.x 源端发起的 `sync` 迁移会立即失败,目标端保持为空,`redis-shake` 容器日志中会出现: -### 已知问题 +```text +[PANIC] parse rdb header error +[error]: verify version, invalid RDB version number 11, 9 +``` -- 在平台版本 **3.16.0** 和 **3.16.1** 中,当源为 Sentinel 实例时,源实例状态可能会错误地报告为 `Processing`,即使实例是健康的。状态报告在 **3.16.2** 中已修复。 +遇到这种情况,可以升级 redis-operator,也可以删除该资源,再以 `modelType: rump` 重新创建。 -## 迁移操作步骤 +## 前提条件 -对于下面的每种场景,您需要创建一个单独的 `RedisShake` 自定义资源。更新镜像、源/目标地址和密码 Secrets 以匹配您的环境,然后使用 `kubectl` 或平台 UI 应用该资源。要在部署后更改配置,请删除现有资源并创建一个新资源——不支持就地编辑。 +1. **容量。** 目标端的内存必须能容纳源端的数据集: + - 目标端为 Sentinel:至少为**源端数据集大小的 5/4**。 + - 目标端为集群、源端为单节点或 Sentinel:至少为**源端数据集大小的 5/4**。 + - 目标端为集群、源端也为集群:至少为**源端最大分片数据集大小的 5/4**。 +2. **迁移的运行位置。** 在安装了 redis-operator 的集群中创建 `RedisShake` 资源,通常是目标端所在的集群。迁移 pod 必须能通过稳定的 IPv4 网络访问两个实例。 +3. **外部访问。** 如果实例与 `RedisShake` 资源不在同一个集群中,该实例必须开启 NodePort 或 LoadBalancer 访问: + - 集群实例:RedisShake 按每个分片对外宣告的地址连接各个分片,因此集群内地址在其他集群中无法使用。 + - Sentinel 实例:数据节点(`spec.expose.type`)和 Sentinel 节点(`spec.sentinel.expose.type`)都要开启外部访问。RedisShake 先向 Sentinel 查询主节点的地址,再连接该主节点。 +4. **数据库。** Redis Cluster 只有 0 号数据库。目标端为集群时,源端其他数据库中的 key 会被**跳过,且不报任何错误**。迁移到集群之前,请在源端执行 `INFO keyspace`,确认所有 key 都在 `db0` 中。 +5. **Redis 函数。** 函数库(Redis 7 的 `FUNCTION LOAD`)不会被迁移。请在目标端单独加载,例如在源端执行 `FUNCTION DUMP`,再在目标端执行 `FUNCTION RESTORE`。 +6. **写入。** 减少源端的写入,以降低复制压力。使用 `rump` 时,整个复制期间都要停止写入。 -### 1. 创建密码 Secrets +## 迁移步骤 -如果任一方受密码保护,请为每一方创建一个 Kubernetes Secret。Secret 必须使用数据键 `password`。 +### 1. 创建密码 Secret + +如果某一端设置了密码,请在创建 `RedisShake` 资源的命名空间中为它创建一个 Secret,数据键为 `password`: ```bash kubectl -n create secret generic \ --from-literal=password= ``` -### 2. 从单节点到 Sentinel +### 2. 查找地址 -```yaml -apiVersion: middle.alauda.cn/v1alpha1 -kind: RedisShake -metadata: - name: standalone-to-sentinel -spec: - image: # 请参见附录以获取适合您版本的正确镜像 - keyExists: rewrite - modelType: sync - replicas: 1 - source: - address: - - ":" - passwordSecret: - type: standalone - target: - address: - - "mymaster@:" - passwordSecret: - type: sentinel -``` +在 **数据服务 > 实例详情 > 访问方式 > 集群内访问 / 外部访问** 中找到访问端点,再按架构和角色把每个地址写成对应的格式: -:::warning -当 **源或目标是 Sentinel 实例** 时,地址必须以 `mymaster@` 为前缀,值应为 Sentinel 访问端点。在 **数据服务 > 实例详情 > 访问方式 > 集群内访问 / 外部访问** 中找到它。 -::: +| 架构 | 作为源端 | 作为目标端 | +| --- | --- | --- | +| `standalone` | `:` | `:` | +| `sentinel` | `mymaster@:` | `mymaster@:` | +| `cluster` | `master@:` | `:` | -### 3. 从 Sentinel 到 Sentinel +- Sentinel 实例使用 **Sentinel** 端点。`mymaster` 是平台为每个 Sentinel 实例设置的主节点名称。 +- 集群实例只需填写任意一个节点的端点,RedisShake 会自动发现其他分片。只有作为**源端**的集群才需要 `master@` 前缀。 -```yaml -apiVersion: middle.alauda.cn/v1alpha1 -kind: RedisShake -metadata: - name: sentinel-to-sentinel -spec: - image: - keyExists: rewrite - modelType: sync - replicas: 1 - source: - address: - - "mymaster@:" - passwordSecret: - type: sentinel - target: - address: - - "mymaster@:" - passwordSecret: - type: sentinel +:::warning +如果实例的 Sentinel 节点设置了独立的密码(`spec.sentinel.passwordSecret`),无论用作源端还是目标端,`mymaster@` 地址都无法使用:RedisShake 无法通过 Sentinel 的认证,日志中会出现 `no sentinels available; last error: NOAUTH Authentication required.`。此时请直接连接数据节点:设置 `type: standalone`,使用**数据节点**密码的 Secret,并使用读写 Service `rfr--read-write` 的外部地址。查看它的 NodePort: + +```bash +kubectl -n get svc rfr--read-write \ + -o jsonpath='{.spec.ports[0].nodePort}' ``` +::: -### 4. 从单节点到集群 +### 3. 创建 RedisShake 资源 ```yaml apiVersion: middle.alauda.cn/v1alpha1 kind: RedisShake metadata: - name: standalone-to-cluster + name: + namespace: spec: - image: - keyExists: rewrite - modelType: sync - replicas: 1 + modelType: sync # 或 rump,参见“选择迁移模式” + keyExists: rewrite # 覆盖目标端已存在的 key source: + type: sentinel # standalone、sentinel 或 cluster address: - - ":" + - "mymaster@:" passwordSecret: - type: standalone target: + type: cluster # standalone、sentinel 或 cluster address: - ":" passwordSecret: - type: cluster + resources: # 可选 + requests: + cpu: "2" + memory: 2Gi + limits: + cpu: "4" + memory: 4Gi ``` -:::note -对于集群目标,地址可以是 **集群访问端点中的任意一个**。在 **数据服务 > 实例详情 > 访问方式 > 集群内访问 / 外部访问** 中找到它们。 -::: +- 按实际场景修改 `source` 和 `target`,地址格式见第 2 步。 +- 不要设置 `image`:redis-operator 5.1 及更高版本会使用该 operator 自带的 RedisShake 镜像。 +- `keyExists` 还可以取 `ignore`,即保留目标端的值;或取 `none`,即遇到第一个在目标端已存在的 key 时停止迁移。默认值为 `none`。 +- 不设置 `resources` 时,迁移 pod 的请求为 500m CPU 和 500Mi 内存,上限为 1 CPU 和 1Gi。数据集较大时,请分配 2–8 个 CPU,可以从 4 个开始。 -### 5. 从 Sentinel 到集群 +使用 `kubectl apply -f` 应用该文件,或在平台界面中创建该资源。随后 redis-operator 会创建工作负载 `-shake`:`sync` 模式为 Deployment,`rump` 模式为 Job。 -```yaml -apiVersion: middle.alauda.cn/v1alpha1 -kind: RedisShake -metadata: - name: sentinel-to-cluster -spec: - image: - keyExists: rewrite - modelType: sync - replicas: 1 - resumeFromBreakPoint: false - source: - address: - - "mymaster@:" - passwordSecret: - type: sentinel - target: - address: - - ":" - passwordSecret: - type: cluster -``` +### 4. 监控迁移 -### 6. 从集群到集群 +查看迁移 pod 及其日志: -```yaml -apiVersion: middle.alauda.cn/v1alpha1 -kind: RedisShake -metadata: - name: cluster-to-cluster -spec: - image: - keyExists: rewrite - modelType: sync - replicas: 1 - source: - address: - - "master@:" - passwordSecret: - type: cluster - target: - address: - - ":" - passwordSecret: - type: cluster +```bash +kubectl -n get pods | grep -shake +kubectl -n logs deploy/-shake -c redis-shake --tail=50 # sync +kubectl -n logs job/-shake -c redis-shake --tail=50 # rump ``` -:::warning +- **`sync`**:日志中出现 `DbSyncer[0] sync rdb done`,表示全量复制已完成,增量同步已开始。源端为集群时,每个源端分片对应一个 `DbSyncer`。 +- **`rump`**:Job 完成,日志以 `all rumpers finish!` 结尾。 -- 当 **源是集群** 时,地址前缀为 `master@`。目标集群地址 **不需要** 此前缀。 -- 对于源和目标集群,任意一个集群端点即可——RedisShake 会自动发现其余拓扑。 - ::: +`sync` 迁移还会在 9320 端口提供 Prometheus 指标: -## 验证同步 - -RedisShake 执行持续复制。即使初始数据集已复制,复制仍会继续,直到您删除 `RedisShake` 资源。因此,您必须手动验证完成情况。 +```bash +kubectl get --raw "/api/v1/namespaces//pods/:9320/proxy/metrics" \ + | grep -E 'redisshake_(full_sync_process_percent|fake_slave_delay_offset)' +``` -### 平台版本低于 3.10 +| 指标 | 含义 | +| --- | --- | +| `redisshake_full_sync_process_percent` | 全量复制的进度,完成时为 `100`。每个源端分片一条时间序列,标签为 `db_syncer`。 | +| `redisshake_fake_slave_delay_offset` | 目标端落后于源端的字节数。即使没有写入,复制心跳也会让它在 `0` 到 150 左右之间变化;暂停写入后,如果它反复回到 `0`,说明目标端已经追上源端。 | -这些版本没有监控面板。使用键计数比较: +redis-operator 还会在它自己所在的命名空间中创建名为 `redis-shake` 的 ServiceMonitor,供平台的 Prometheus 采集这些指标。平台没有为这些指标提供 Grafana 仪表板。 -1. 在源上运行 `DBSIZE`。对于集群源,计算所有主节点的 `DBSIZE` 之和。 -2. 在目标上运行 `DBSIZE`。对于集群目标,计算所有主节点的总和。 -3. 在源上暂停写入时,当同步完成时,总数应匹配。 +### 5. 校验数据 -对于集群模式,以下一行命令返回所有主节点的总键计数: +暂停源端写入后,比较两端的 key 数量。请通过 `REDISCLI_AUTH` 环境变量传入密码,以免密码出现在命令行中: ```bash -redis-cli -a --cluster call : dbsize --cluster-only-masters +# 单节点或 Sentinel:在主节点上执行 +REDISCLI_AUTH= redis-cli -h -p dbsize + +# 集群:每个主节点输出一行,把它们相加 +REDISCLI_AUTH= redis-cli --cluster call : dbsize --cluster-only-masters ``` -### 平台版本 3.10.2 及以上 +设置了过期时间的 key 可能使两端的数量略有差异,因为两端各自按自己的节奏让 key 过期。 -键计数方法仍然有效。此外,在 **平台管理 > 操作中心 > 监控 > Grafana > 仪表板** 下提供了 **"Redis Shake Dashboard"**。 +### 6. 切换与清理 -#### Sentinel 源 +1. 暂停源端写入。 +2. 对于 `sync`,等待 `redisshake_fake_slave_delay_offset` 稳定在 `0`,再次比较 key 数量。 +3. 把应用切换到目标端。 +4. 删除该资源。这会同时删除 Deployment 或 Job,以及它的 ConfigMap 和 Service: -- `SyncProcessPercent` — 当初始同步完成时达到 `100`。 -- `SlaveDelayOffset` — 定期下降到 `0` 表示目标已赶上。来自复制 ping 数据包的短暂峰值是预期的,而不是数据。 + ```bash + kubectl -n delete redisshake + ``` -#### 集群源 +5. 如果不再需要密码 Secret,将它们删除。 -- `SyncProcessPercent` — 每个源分片一个进度系列。所有分片达到完成标记表示初始同步完成。 -- `SlaveDelayOffset` — 当偏移量达到 `0` 时,复制已更新。 +## 修改运行中的迁移 -## 附录:按平台版本的 RedisShake 镜像 +对运行中的 `RedisShake` 资源所做的修改,只有一部分会生效: -使用与您的平台版本匹配的 RedisShake 镜像: +- **会生效**:迁移 pod 的字段,例如 `resources`、`nodeSelector`、`tolerations` 和 `affinity`。redis-operator 会更新 Deployment,pod 随之重启。 +- **不会生效**:`source`、`target`、`filter`、`keyExists`、`parallel` 以及其他迁移设置。这些设置只在创建资源时写入 ConfigMap。 -| ACP 版本 | RedisShake 镜像 | -| ---------------- | ------------------------------------------------------------- | -| 3.8.1 - 3.8.3 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2` | -| 3.10.1 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2` | -| 3.10.2, 3.10.3 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.3` | -| 3.12.1 - 3.12.3 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2-5ad6d091` | -| 3.14.1, 3.14.2 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2-063e3b5d` | -| 3.16.1 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2-9bb65a7b` | -| 3.16.2 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2-3777a73b` | +要修改迁移设置,请删除该资源后重新创建。`rump` 的 Job 从不更新;要再次执行,请删除并重新创建该资源。 + +## 故障排查 + +| 现象 | 原因 | 处理 | +| --- | --- | --- | +| `redis-shake` 日志中出现 `invalid RDB version number 11, 9`(或 `10, 9`),容器随之重启 | RedisShake 镜像早于支持 Redis 7.x 的版本 | 把 redis-operator 升级到晚于 5.1.1 的版本,或使用 `modelType: rump` | +| `no sentinels available; last error: NOAUTH Authentication required.` | Sentinel 节点设置了独立的密码 | 按第 2 步的说明,直接连接数据节点 | +| 出现 `target key name is busy`,容器随之重启 | `keyExists` 为 `none`,且该 key 已存在于目标端 | 设置 `keyExists: rewrite` 或 `ignore`,或先清空目标端 | +| operator 日志中出现 `spec.template.spec.containers[0].image: Required value`,且没有迁移 pod | 早于 5.1 的 redis-operator 不会自动填写镜像 | 把 `spec.image` 设置为对应版本的 RedisShake 镜像 | +| `shake-init` 容器日志显示 `Could not connect to Redis at mymaster@...: Name does not resolve` | 初始化脚本也会把 Sentinel 格式的地址当作主机名尝试连接 | 无需处理,该信息无害 | +| 集群目标端缺少源端 0 号以外数据库中的 key | Redis Cluster 只有 0 号数据库 | 迁移前在源端把这些 key 移到 0 号数据库 | ## 重要注意事项 -- **不支持 Redis 7.x。** 对于 Redis 7.x 源或目标,请使用其他迁移路径(例如 RDB 导出/导入)。 -- **复制不会自动停止。** 在切换之前始终验证完成情况,并在迁移完成后删除 `RedisShake` 资源。 -- **在切换期间暂停写入。** 源上的短暂冻结允许偏移量在应用程序重新指向之前达到零。 -- **CR 是不可变的。** 要更改配置,请删除并重新创建 `RedisShake` 资源。 -- **Sentinel 和集群地址格式。** Sentinel 地址需要 `mymaster@` 前缀;集群源地址需要 `master@` 前缀;集群目标地址不需要。 -- **网络稳定性。** 源和目标之间的持续网络不稳定会导致复制重启和堆积量增加。在开始迁移之前确保连接稳定。 +- **复制不会自动停止。** `sync` 迁移会一直运行,直到删除该资源。切换之前务必先校验数据。 +- **集群目标端上的消费组。** 迁移开始时源端已有的消费组会随 stream 一起复制。但迁移到集群目标端*期间*,在源端执行的 `XGROUP` 命令可能被发往错误的分片,因此这期间创建的消费组可能在目标端缺失,且没有任何报错。请在开始迁移之前创建消费组,或在切换后到目标端重新创建。 +- **不支持 Redis 7.4 及更高版本。** 它们的 RDB 格式不受支持,迁移会立即停止。 +- **不支持迁移到更低版本的 Redis。** Redis 7.x 源端不能迁移到 5.x 或 6.x 目标端。 +- **网络稳定性。** 两端之间持续的网络不稳定会导致复制反复重启、延迟不断增大。开始迁移之前,请确保连接稳定。 + +## 附录:平台 3.x {#appendix-platform-3x} + +在平台 3.8.x – 3.16.x 上,上述步骤同样适用,区别如下。 + +**设置镜像。** 在资源中添加 `spec.image`,使用与平台版本对应的镜像: + +| ACP 版本 | RedisShake 镜像 | +| --- | --- | +| 3.8.1 - 3.8.3 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2` | +| 3.10.1 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2` | +| 3.10.2, 3.10.3 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.3` | +| 3.12.1 - 3.12.3 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2-5ad6d091` | +| 3.14.1, 3.14.2 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2-063e3b5d` | +| 3.16.1 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2-9bb65a7b` | +| 3.16.2 | `build-harbor.alauda.cn/middleware/redis-shake:v3.8.2-3777a73b` | + +**Redis 版本。** 源端和目标端都只支持 Redis 5.x 和 6.x。 + +**监控。** 从 3.10.2 起,可以在 **平台管理 > 操作中心 > 监控 > Grafana > 仪表板** 下使用 Grafana 仪表板 **"Redis Shake Dashboard"**。全量复制完成时,其面板 `SyncProcessPercent` 达到 `100`;目标端追上源端时,`SlaveDelayOffset` 回到 `0`。3.10.2 之前没有该仪表板,请改为比较 key 数量。 + +**已知问题。** 在 3.16.0 和 3.16.1 上,源端为 Sentinel 实例时,即使实例运行正常,源实例的状态也可能显示为 `Processing`。该问题已在 3.16.2 中修复。