Add redis-proxy: dual-write Redis proxy with pub/sub forwarding#351
Add redis-proxy: dual-write Redis proxy with pub/sub forwarding#351
Conversation
Implement a Redis-protocol proxy that supports dual-write to primary (Redis) and secondary (ElasticKV) backends. Includes full pub/sub message forwarding via redcon Detach + go-redis PubSub. Verified all 581 Misskey unit tests pass through the proxy.
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request delivers a robust Redis-protocol proxy designed to facilitate the migration of Redis data to an ElasticKV store. It provides a flexible architecture for routing commands, allowing for various stages of migration from simple passthrough to full dual-write with shadow reads. The proxy ensures operational stability through bounded asynchronous operations, comprehensive metrics, and proactive error reporting, making it a critical component for managing data consistency during transitions. Highlights
Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive Redis proxy with dual-write capabilities, pub/sub forwarding, metrics, and error reporting. The overall architecture is well-structured, with clear separation of concerns into different packages for configuration, backends, command handling, and proxy logic. The use of a semaphore for bounding asynchronous operations and the detailed command classification are excellent. However, I've identified a few critical issues related to binary data handling that could lead to data corruption, as well as some opportunities for improving graceful shutdown and simplifying transaction handling logic.
There was a problem hiding this comment.
Pull request overview
Adds a new redis-proxy component to the codebase: a Redis-protocol proxy that routes commands to a primary backend and optionally dual-writes / shadow-reads against a secondary backend, with Prometheus metrics and optional Sentry reporting.
Changes:
- Introduce
proxypackage implementing command classification/routing, dual-write + shadow-read comparison, and Redis protocol response translation. - Add Pub/Sub forwarding support using
redcondetached connections bridged togo-redisPubSub. - Add unit + integration tests plus a
cmd/redis-proxyexecutable with metrics + Sentry wiring.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| cmd/redis-proxy/main.go | CLI entrypoint that wires config, backends, metrics endpoint, and proxy server lifecycle. |
| proxy/backend.go | Backend abstraction and go-redis implementation (including optional PubSub factory). |
| proxy/command.go | Command classification table used to decide routing (read/write/admin/blocking/txn/pubsub/script). |
| proxy/compare.go | Shadow-read comparison logic and divergence classification + reporting. |
| proxy/config.go | Proxy configuration and mode parsing/defaults. |
| proxy/dualwrite.go | Dual-write/shadow-read execution logic, async bounding, metrics, and Sentry reporting hooks. |
| proxy/integration_test.go | End-to-end integration tests against real Redis instances (skipped if unavailable). |
| proxy/metrics.go | Prometheus metrics definitions and registration. |
| proxy/proxy.go | Redcon server implementation: command dispatch, txn handling, pubsub session startup, response encoding. |
| proxy/proxy_test.go | Unit tests covering command classification, divergence logic, dual-write behavior, and Sentry cooldown logic. |
| proxy/pubsub.go | Detached pub/sub session implementation bridging client <-> upstream PubSub. |
| proxy/sentry.go | Sentry reporter wrapper with fingerprint-based cooldown and bounded history tracking. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
- Extract dispatchPubSubCommand to reduce readClientCommands complexity (CC=11→8) - Use metricsSrv.Shutdown for graceful metrics server stop - Guard against nil logger in NewSentryReporter - Cap ShouldReport map after eviction to prevent unbounded growth - Normalize command names with strings.ToUpper in DualWriter methods - Fix goAsync doc comment to match actual behavior - Simplify execTxn by merging duplicate result-handling branches
There was a problem hiding this comment.
Pull request overview
This PR introduces a new Redis-protocol proxy (cmd/redis-proxy) that can route reads/writes across a primary Redis backend and a secondary ElasticKV backend, including optional shadow-read divergence detection, Prometheus metrics, Sentry anomaly reporting, and Pub/Sub forwarding.
Changes:
- Added a redcon-based proxy server with command classification, transaction handling, and Pub/Sub session bridging.
- Implemented dual-write + optional shadow-read comparison logic with Prometheus metrics and Sentry reporting.
- Added unit and integration tests plus a runnable
redis-proxyCLI entrypoint.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| cmd/redis-proxy/main.go | CLI entrypoint wiring config, metrics server, Sentry, and proxy server lifecycle. |
| proxy/backend.go | Backend abstraction and go-redis based RedisBackend implementation (incl. pipelining + PubSub creation). |
| proxy/command.go | Redis command classification table for routing (read/write/admin/txn/pubsub/etc.). |
| proxy/compare.go | Shadow-read comparison + divergence classification and reporting. |
| proxy/config.go | Proxy configuration and mode parsing/defaults. |
| proxy/dualwrite.go | Dual-write/shadow-read execution, async bounding, and Sentry/metrics instrumentation. |
| proxy/integration_test.go | Integration tests that run against real Redis instances when available. |
| proxy/metrics.go | Prometheus metrics definitions and registration. |
| proxy/proxy.go | Main redcon proxy server: dispatch/routing, txn support, RESP writing, Pub/Sub session startup. |
| proxy/proxy_test.go | Unit tests covering classification, dual-writer behavior, shadow reader, sentry reporter, and backend defaults. |
| proxy/pubsub.go | Detached-connection Pub/Sub session that forwards messages and handles pubsub-mode commands. |
| proxy/sentry.go | Sentry reporter with cooldown-based de-duplication and flush support. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
- Pass []byte directly in bytesArgsToInterfaces instead of converting to string - Handle []byte type assertion in argsToBytes for correct Sentry reporting - Fix responseEqual panic when comparing mismatched interface types - Return false in ShouldReport when at capacity to prevent Sentry flooding
When all subscriptions are removed, the detached connection transitions to normal command mode instead of being stuck in pub/sub-only mode. Clients can then execute regular Redis commands (GET, SET, transactions, etc.) or re-enter pub/sub mode with a new SUBSCRIBE/PSUBSCRIBE. Key changes: - Add respWriter interface so writeResponse works with both Conn and DetachedConn - Refactor pubsubSession with commandLoop that handles both pub/sub and normal modes - Support transactions (MULTI/EXEC/DISCARD) in normal mode on detached connections - Track forwardMessages goroutine lifecycle for clean pub/sub mode transitions - Extract command name constants (MULTI, EXEC, DISCARD, PING, QUIT)
There was a problem hiding this comment.
Pull request overview
Adds a new redis-proxy component to the codebase that speaks the Redis protocol and routes commands to primary/secondary backends to support dual-write migration to ElasticKV, with observability (Prometheus + Sentry) and pub/sub forwarding.
Changes:
- Implement Redis-protocol proxy server (redcon) with command classification, transactions, and backend routing (dual-write + optional shadow reads).
- Add pub/sub bridging using
redcon.Detach+go-redisPubSub and support transitioning back to normal command mode. - Add Prometheus metrics, Sentry reporting/dedup, and accompanying unit + integration tests; introduce
cmd/redis-proxyentrypoint.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| cmd/redis-proxy/main.go | New CLI binary wiring config, metrics server, and proxy server lifecycle. |
| proxy/backend.go | Backend abstraction + go-redis implementation, including PubSub capability. |
| proxy/command.go | Command classification table for routing decisions. |
| proxy/compare.go | Shadow-read comparison + divergence classification/metrics/Sentry reporting. |
| proxy/config.go | Proxy configuration + supported operating modes. |
| proxy/dualwrite.go | Core dual-write/shadow-read execution, bounded async work, and metrics/Sentry hooks. |
| proxy/integration_test.go | Optional integration tests against real Redis instances. |
| proxy/metrics.go | Prometheus metric definitions/registration for proxy operations. |
| proxy/proxy.go | redcon server handlers, transaction execution, and RESP response writing helpers. |
| proxy/proxy_test.go | Unit tests covering classification, dual-writer behavior, metrics, and Sentry dedup. |
| proxy/pubsub.go | Pub/sub session implementation with message forwarding and mode switching. |
| proxy/sentry.go | Sentry initialization + cooldown-based dedup for anomaly reporting. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
- Fix startForwarding race: capture upstream under lock before goroutine start - Reject SUBSCRIBE/PSUBSCRIBE during MULTI transaction to prevent state corruption - Normalize non-redis.Error to "ERR ..." prefix in writeRedisError for valid RESP - Truncate divergence values in Sentry reports to prevent data leakage
- Add PrimaryDB, PrimaryPassword, SecondaryDB, SecondaryPassword to ProxyConfig - Add DB and Password fields to BackendOptions, passed to go-redis client - Add CLI flags: -primary-db, -primary-password, -secondary-db, -secondary-password - Silently accept SELECT and AUTH commands (return OK without forwarding) since DB/auth are configured at the connection-pool level
There was a problem hiding this comment.
Pull request overview
Adds a Redis-protocol proxy service (redis-proxy) that can route reads/writes across two Redis-protocol backends (Redis + ElasticKV), with optional shadow reads for divergence detection and support for pub/sub forwarding via detached redcon connections.
Changes:
- Introduces the proxy server core (command classification/routing, dual-write + shadow-read logic, RESP response writer, transaction handling).
- Adds pub/sub support using
redcon.Detach()+go-redisPubSubforwarding, including a “return to normal mode” when unsubscribed. - Adds observability: Prometheus metrics + optional Sentry reporting, plus unit/integration tests for non-pubsub behavior.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
cmd/redis-proxy/main.go |
CLI entrypoint wiring config, backends, metrics endpoint, and graceful shutdown. |
proxy/backend.go |
Backend abstraction and go-redis implementation (including a dedicated PubSub connection). |
proxy/command.go |
Redis command classification table for routing decisions. |
proxy/config.go |
Proxy config + modes (redis-only, dual-write, shadow, elastickv-primary/only). |
proxy/dualwrite.go |
Dual-writer/shadow-reader execution paths + bounded async work. |
proxy/compare.go |
Shadow-read comparison, divergence classification, and divergence reporting hook. |
proxy/metrics.go |
Prometheus metrics definitions/registration. |
proxy/sentry.go |
Optional Sentry reporter with cooldown-based de-duplication support. |
proxy/proxy.go |
redcon server, RESP writing helpers, command dispatch, and transaction support. |
proxy/pubsub.go |
Detached pub/sub session implementation and upstream message forwarding. |
proxy/proxy_test.go |
Unit tests for core proxy components (classification, dual-write, sentry gating, etc.). |
proxy/integration_test.go |
Integration tests against real Redis instances for basic command flows. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
proxy/sentry: fix exhaustive lint error in truncateValue
- exitPubSubMode and cleanup now close dconn after timeout to unblock forwardMessages stuck on Flush, preventing writeMu deadlock - Add nolint:exhaustive for reflect.Kind switch that intentionally uses default for all non-container types
There was a problem hiding this comment.
Pull request overview
This PR introduces a Redis-protocol “redis-proxy” that can dual-write to a primary (Redis) and secondary (ElasticKV) backend, with shadow-read divergence detection and full Pub/Sub forwarding using redcon detached connections.
Changes:
- Added a detached Pub/Sub session implementation that forwards upstream Pub/Sub messages and supports transitioning back to normal command mode without reconnecting.
- Implemented dual-write + optional shadow-read comparison with Prometheus metrics and Sentry reporting/deduplication.
- Added unit and integration tests for command classification, txn handling, Pub/Sub behavior, and dual-write flows.
Reviewed changes
Copilot reviewed 14 out of 15 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
proxy/backend.go |
Adds backend abstraction + Redis backend implementation (Do/Pipeline/PubSub). |
proxy/command.go |
Adds command classification table for routing (read/write/blocking/pubsub/admin/txn/script). |
proxy/compare.go |
Adds shadow-read comparison and divergence classification logic. |
proxy/config.go |
Adds proxy configuration + mode parsing/defaults. |
proxy/dualwrite.go |
Implements dual-write, shadow reads, bounded async execution, and metrics/Sentry hooks. |
proxy/metrics.go |
Adds Prometheus metrics definitions + registration. |
proxy/proxy.go |
Implements the redcon server, command dispatch, txn handling, and Pub/Sub session entry. |
proxy/pubsub.go |
Implements detached Pub/Sub session, message forwarding, and Pub/Sub-mode command loop. |
proxy/sentry.go |
Adds Sentry reporter with cooldown-based de-duplication and value truncation helpers. |
proxy/proxy_test.go |
Adds unit tests for classification, divergence logic, Sentry cooldown, dual-write behavior, etc. |
proxy/pubsub_test.go |
Adds unit tests for detached Pub/Sub session behavior and RESP writes. |
proxy/integration_test.go |
Adds integration tests against real Redis instances (skipped if unavailable). |
cmd/redis-proxy/main.go |
Adds a runnable redis-proxy binary with flags + metrics HTTP server. |
go.mod / go.sum |
Adds Sentry dependency and updates module sums. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
| func (b *RedisBackend) Pipeline(ctx context.Context, cmds [][]any) ([]*redis.Cmd, error) { | ||
| pipe := b.client.Pipeline() | ||
| results := make([]*redis.Cmd, len(cmds)) | ||
| for i, args := range cmds { | ||
| results[i] = pipe.Do(ctx, args...) | ||
| } | ||
| _, err := pipe.Exec(ctx) | ||
| if err != nil { | ||
| return results, fmt.Errorf("pipeline exec: %w", err) | ||
| } | ||
| return results, nil |
proxy/proxy.go
Outdated
| if err != nil { | ||
| // Pipeline-level error (connection/transport failure) takes precedence. | ||
| writeRedisError(conn, err) | ||
| } else if len(results) > 0 { | ||
| // Write the EXEC result (last command in the pipeline). | ||
| lastResult := results[len(results)-1] | ||
| resp, rErr := lastResult.Result() | ||
| writeResponse(conn, resp, rErr) |
| results, err := s.proxy.dual.Primary().Pipeline(ctx, cmds) | ||
|
|
||
| s.writeMu.Lock() | ||
| if err != nil { | ||
| // Pipeline-level error (connection/transport failure) takes precedence. | ||
| writeRedisError(s.dconn, err) | ||
| } else if len(results) > 0 { | ||
| lastResult := results[len(results)-1] | ||
| resp, rErr := lastResult.Result() | ||
| writeResponse(s.dconn, resp, rErr) | ||
| } |
proxy/pubsub.go
Outdated
| s.writeMu.Lock() | ||
| defer s.writeMu.Unlock() | ||
|
|
||
| if len(set) == 0 { | ||
| // No subscriptions: single reply with null channel (matching Redis). | ||
| s.dconn.WriteArray(pubsubArrayReply) | ||
| s.dconn.WriteBulkString(kind) | ||
| s.dconn.WriteNull() | ||
| s.dconn.WriteInt64(int64(s.subCount())) | ||
| _ = s.dconn.Flush() | ||
| return | ||
| } | ||
|
|
||
| // Collect names, then remove one-by-one to decrement count per reply. | ||
| names := make([]string, 0, len(set)) | ||
| for n := range set { | ||
| names = append(names, n) | ||
| } | ||
|
|
||
| for _, n := range names { | ||
| if isPattern { | ||
| delete(s.patternSet, n) | ||
| } else { | ||
| delete(s.channelSet, n) | ||
| } | ||
| s.dconn.WriteArray(pubsubArrayReply) | ||
| s.dconn.WriteBulkString(kind) | ||
| s.dconn.WriteBulkString(n) | ||
| s.dconn.WriteInt64(int64(s.subCount())) |
- Pipeline now returns results with nil error for redis.Error/redis.Nil, only propagating transport/context errors. This preserves Redis transaction semantics in execTxn callers. - handleUnsub and writeUnsubAll now mutate goroutine-confined state and pre-compute counts before acquiring writeMu, keeping writeMu strictly for dconn write serialization.
Implement shadow pub/sub that subscribes to the secondary backend for the same channels/patterns and compares received messages against the primary using a time-windowed matching strategy. - New shadowPubSub struct with sliding-window message comparison - Messages matched by (channel, payload) key within configurable window - Primary-only messages after window expiry reported as data_mismatch - Secondary-only messages with no primary match reported as extra_data - All shadow operations are fire-and-forget (never block primary path) - Active only in ModeDualWriteShadow / ModeElasticKVPrimary - New metrics: pubsub_shadow_divergences_total, pubsub_shadow_errors_total - New config: PubSubCompareWindow (default 2s) - Fix Pipeline to not wrap redis.Error (preserves EXEC result semantics)
There was a problem hiding this comment.
Pull request overview
This PR adds a Redis-protocol proxy (redis-proxy) that supports dual-write routing across primary (Redis) and secondary (ElasticKV) backends, including Pub/Sub support with optional “shadow” comparison against the secondary to detect divergence.
Changes:
- Introduces a detached Pub/Sub session handler that forwards upstream Pub/Sub messages to clients while allowing normal commands once subscriptions are cleared.
- Adds shadow Pub/Sub comparison logic and new Prometheus/Sentry reporting for Pub/Sub divergences.
- Adds extensive unit and integration tests covering routing, transactions, Pub/Sub session behavior, and shadow comparison.
Reviewed changes
Copilot reviewed 16 out of 17 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| proxy/pubsub.go | Implements detached Pub/Sub session handling, forwarding, and transition back to normal command mode |
| proxy/shadow_pubsub.go | Adds shadow Pub/Sub subscriber that matches primary vs secondary messages and reports divergences |
| proxy/sentry.go | Adds Sentry reporter with cooldown-based de-duplication and value truncation helpers |
| proxy/metrics.go | Introduces Prometheus metrics for proxy operations and Pub/Sub shadowing |
| proxy/proxy.go | Wires Pub/Sub sessions into the main redcon handler; creates shadow Pub/Sub based on mode |
| proxy/dualwrite.go | Implements dual-write + shadow-read routing with bounded async execution |
| proxy/*_test.go | Adds unit tests for Pub/Sub session logic, shadow Pub/Sub matching, and core proxy behaviors |
| proxy/integration_test.go | Adds integration tests that exercise the proxy against real Redis instances |
| cmd/redis-proxy/main.go | Adds CLI entrypoint and metrics server wiring |
| go.mod / go.sum | Updates dependencies (adds sentry-go; removes raft-boltdb related deps) |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
| // Record for shadow comparison (outside writeMu to avoid nested locking). | ||
| if s.shadow != nil { | ||
| s.shadow.RecordPrimary(msg) | ||
| } | ||
| } |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
- Track started flag so Close is safe before Start (prevents deadlock when subscribe fails during setup) - Guard s.shadow access in forwardMessages with mu to prevent data race with concurrent cleanup/exitPubSubMode - Move closeShadow after fwdDone wait so RecordPrimary completes first - Remove unused reportDivergenceLocked
There was a problem hiding this comment.
Pull request overview
Implements a Redis-protocol proxy that supports dual-write to primary/secondary backends, adds shadow-read/pubsub divergence detection, and exposes observability (Prometheus + Sentry) around mismatches and failures.
Changes:
- Added detached Pub/Sub session handling (SUBSCRIBE/PSUBSCRIBE, idempotent counts, txn handling in normal mode) and optional shadow pub/sub comparison.
- Added Sentry reporter with cooldown-based de-duplication and value truncation to reduce data leakage / event size.
- Added Prometheus metrics, extensive unit tests, and an integration test suite plus a
redis-proxyCLI.
Reviewed changes
Copilot reviewed 16 out of 17 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| proxy/shadow_pubsub.go | Implements shadow pub/sub comparator between primary and secondary streams. |
| proxy/shadow_pubsub_test.go | Unit tests for shadow pub/sub matching/sweeping/divergence metrics. |
| proxy/pubsub.go | Detached pub/sub session implementation bridging redcon ↔ go-redis, with shadow mirroring hooks. |
| proxy/pubsub_test.go | Unit tests for pub/sub session command dispatch, unsubscribe behavior, txn handling, and wire writes. |
| proxy/sentry.go | Sentry reporting with fingerprint cooldown + truncation helpers. |
| proxy/metrics.go | Prometheus metrics definitions/registration, including pub/sub shadow metrics. |
| proxy/dualwrite.go | Dual-write routing + bounded async executor + shadow read support. |
| proxy/proxy.go | Main redcon server: command routing, transaction support, pub/sub session bootstrap, shadow pub/sub creation. |
| proxy/compare.go | Shadow-read comparison logic, divergence classification, and divergence reporting. |
| proxy/command.go | Command classification table (read/write/blocking/pubsub/admin/txn/script). |
| proxy/backend.go | Backend abstraction + Redis backend implementation with pipelining and pub/sub support. |
| proxy/proxy_test.go | Broad unit test coverage for command classification, config, shadow reader, dual writer, sentry, etc. |
| proxy/integration_test.go | End-to-end integration tests against real Redis instances (skipped if unavailable). |
| cmd/redis-proxy/main.go | New CLI entrypoint: configuration flags, metrics server, and proxy startup/shutdown. |
| go.mod | Adds direct dependency on sentry-go; removes unused raft-boltdb dependency. |
| go.sum | Dependency checksum updates reflecting module graph changes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…, divergence reporting - Include Pattern in msgKey for correct pmessage matching - Release lock before reporting in matchSecondary to avoid blocking I/O under mu - Add sweepAll to flush all pending on channel close (not just expired) - Set Primary/Secondary in reportDivergence based on DivergenceKind - Guard s.shadow assignment in reenterPubSub with mu - Promote divergenceEvent to package-level type shared by sweep methods
There was a problem hiding this comment.
Pull request overview
This PR introduces a Redis-protocol proxy server that can route commands to a primary backend (Redis) while optionally dual-writing to a secondary backend (ElasticKV), including support for Pub/Sub with optional shadow (compare) subscriptions for divergence detection.
Changes:
- Added
ProxyServer+DualWritercommand routing (read/write/blocking/admin/script/txn) with Prometheus metrics and optional Sentry reporting. - Implemented detached Pub/Sub session handling (SUBSCRIBE/PSUBSCRIBE, unsubscribe semantics, ping/quit, txn behavior in normal mode) plus shadow Pub/Sub comparison.
- Added unit/integration tests for command classification, divergence detection, pub/sub session behavior, and shadow pub/sub matching.
Reviewed changes
Copilot reviewed 16 out of 17 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
proxy/proxy.go |
Core redcon server handler; detaches connections for pub/sub sessions; creates shadow pub/sub. |
proxy/pubsub.go |
Implements detached pub/sub session state machine, forwarding, and command handling. |
proxy/shadow_pubsub.go |
Shadow subscriber comparison logic + divergence reporting for pub/sub messages. |
proxy/sentry.go |
Sentry reporter with cooldown-based dedup + truncation helpers. |
proxy/metrics.go |
Prometheus metrics definitions/registration (including pub/sub shadow metrics). |
proxy/*.go tests |
Unit/integration tests for the new proxy, pub/sub handling, and shadow behavior. |
cmd/redis-proxy/main.go |
New executable wiring config/metrics/sentry/backends and starting the proxy. |
go.mod, go.sum |
Dependency updates (notably adding Sentry). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
proxy/metrics.go
Outdated
| Namespace: "proxy", | ||
| Name: "pubsub_shadow_divergences_total", | ||
| Help: "Total pub/sub message mismatches detected by shadow subscribe.", | ||
| }, []string{"channel", "kind"}), |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| } | ||
|
|
||
| // sweepExpired reports primary messages that were not matched within the window. | ||
| func (sp *shadowPubSub) sweepExpired() { |
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
calculated cyclomatic complexity for function sweepExpired is 12, max is 10 (cyclop)
Correctness: - Remove redis.Nil double-wrapping in DualWriter methods - Add empty transaction result handling in execTxn - Add WATCH/UNWATCH to command table - Intercept HELLO to prevent RESP3 upgrade - Reject SELECT for non-configured DB - Fix UNSUBSCRIBE to emit per-argument replies Concurrency/distributed failures: - Split asyncSem into writeSem + shadowSem to prevent starvation - Add DualWriter.Close() with WaitGroup for graceful shutdown drain - Add AsyncDrops metric for semaphore backpressure visibility Performance: - Uppercase command name once in handleCommand, pass through to DualWriter - Remove channel label from PubSubShadowDivergences to prevent cardinality explosion Test coverage: - Add writeRedisValue/writeRedisError/writeResponse tests - Add forwardMessages message/pmessage branch tests - Add reenterPubSub error path tests - Add truncateValue type coverage tests - Add ClassifyCommand tests for new commands - Add Pipeline transport error test
matchSecondary now buffers unmatched secondaries instead of reporting immediately. Adjust the test to wait for the comparison window and call sweepExpired to trigger divergence reporting.
Implement a Redis-protocol proxy that supports dual-write to primary (Redis) and secondary (ElasticKV) backends. Includes full pub/sub message forwarding via redcon Detach + go-redis PubSub.
Verified all 581 Misskey unit tests pass through the proxy.