From 03c2728c1c57aa95d7c690d22f1208c0e38647c2 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Mon, 21 Sep 2026 14:02:39 +1200 Subject: [PATCH 01/24] Gemini first draft. Signed-off-by: Sam Barker --- _drafts/routing-post1.md | 139 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 139 insertions(+) create mode 100644 _drafts/routing-post1.md diff --git a/_drafts/routing-post1.md b/_drafts/routing-post1.md new file mode 100644 index 00000000..35d8cd06 --- /dev/null +++ b/_drafts/routing-post1.md @@ -0,0 +1,139 @@ +--- +layout: post +title: "Kroxylicious release 0.24.0" +date: 2026-09-04 15:00:00 +1200 +author: "Sam Barker" +author_url: "https://github.com/sambarker" +# noinspection YAMLSchemaValidation +categories: blog kroxylicious-proxy +tags: [ "routing" ] +--- + +# Layer 7 Kafka Routing: A Pattern Taxonomy for the Consolidation Minefield + +If you've ever tried to migrate a live Kafka workload between clusters, consolidate regional deployments, or shift cloud providers, you already know the bottom line: Kafka clients are remarkably opinionated about network topology. They don't just talk to a virtual endpoint; they demand exact broker metadata, explicit partition assignments, and direct TCP connections to specific physical nodes. + +When you try to reshape that underlying physical infrastructure without breaking application teams, you usually end up picking which operational headache you dislike the least. + +## Pick your poison: Dual writes, MirrorMaker, or scheduled downtime + +Before looking at proxy-level patterns, it helps to review the standard tools people use when trying to move or consolidate Kafka traffic—and why they so often result in late-night incident reviews. + +* **Application-level dual writes:** You ask application teams to update their producer code to write to both Cluster A and Cluster B simultaneously. In architectural diagrams, this looks clean. In production, network blips cause asymmetric failures, message ordering drifts instantly, and handling duplicate delivery becomes the application's problem. Worse, getting twenty product teams to deploy matching code changes on the same timeline is an exercise in cat-herding. +* **Replication pipelines (MirrorMaker 2, etc.):** Running an intermediary replication cluster works reasonably well for asynchronous backup, but relying on it for live consolidation adds latency, doubles your storage and network bills, and forces consumers to deal with offset translation. You're essentially running twice as much hardware to move bytes you already owned. +* **Hard cutovers and maintenance windows:** You schedule a Sunday 2:00 AM window, drain topic queues, update DNS records, and restart clients. This is conceptually simple right up until a legacy service ignores DNS TTLs, holds onto stale socket connections indefinitely, and drops messages silently when the old brokers finally go dark. + +Intercepting traffic at Layer 7—the Kafka wire protocol itself—offers an alternative. By placing a proxy like Kroxylicious between clients and brokers, we can manipulate metadata and route requests on the fly. + +To be clear: introducing an L7 proxy adds a hop, consumes CPU, and gives you another piece of infrastructure to manage. If a simple DNS CNAME flip actually solves your problem, do that instead. But when you need to decouple physical cluster topologies from what clients see, proxying gives you control back. + +--- + +## Where does the proxy run? Forward, Reverse, and Sidecars + +Choosing a routing pattern is only half the battle. You also have to decide where the proxy tier physically lives, who owns it, and whether it acts as a forward proxy for egress or a reverse proxy for ingress. + +``` + ┌─────────────────────────────────────────────────────────┐ + │ FORWARD PROXY │ + │ (Client-Side Egress) │ + │ ┌───────────────────────┐ ┌───────────────────────┐ │ + │ │ Client App + Pod │ │ Client Cluster │ │ + │ │ Sidecar Proxy │ │ Centralized Gateway │ │ + │ └───────────┬───────────┘ └───────────┬───────────┘ │ + └──────────────┼───────────────────────────┼──────────────┘ + │ │ + NETWORK BOUNDARY / TRANSIT / VPC PEERING │ + │ │ + ┌──────────────┼───────────────────────────┼──────────────┐ + │ ▼ ▼ │ + │ ┌───────────────────────┐ ┌───────────────────────┐ │ + │ │ Broker Cluster │ │ Broker Node + │ │ + │ │ Ingress Gateway │ │ Broker Sidecar Proxy │ │ + │ └───────────────────────┘ └───────────────────────┘ │ + │ REVERSE PROXY │ + │ (Broker-Side Ingress) │ + └└─────────────────────────────────────────────────────────┘ + +``` + +### The Forward Proxy Model (Client-Side Egress) + +Lives in the client's network boundary and is managed by application or client-platform teams. + +* **Client Cluster Gateway:** A shared proxy fleet inside the client Kubernetes cluster or VPC. Applications point to a local gateway service, and the proxy handles cross-cluster egress across network boundaries. +* **App Pod Sidecar:** Co-located inside the application pod as an egress proxy. Crucially, the proxy does *not* flatten or hide the Kafka cluster model—the client driver still receives metadata mapped to local endpoints (e.g., port ranges on `localhost`) and maintains individual TCP sockets per broker. You get isolated blast radius per pod, but running hundreds of Netty/JVM proxy containers across a microservice fleet levies a noticeable baseline memory tax. + +### The Reverse Proxy Model (Broker-Side Ingress) + +Lives in the Kafka cluster's network boundary and is managed by the central infrastructure/platform team. + +* **Broker Cluster Ingress Gateway:** A shared ingress fleet sitting in front of physical Kafka brokers. Provides a unified front door for incoming client connections and shields underlying cluster topology, though a gateway outage impacts all incoming traffic. +* **Broker Node Sidecar:** Co-located on the actual physical broker hardware (one proxy instance per broker node). While it eliminates an internal network hop on paper, almost nobody does this in production. You risk L7 proxy bugs or memory spikes starving the underlying broker JVM or page cache, and a broker-side proxy loses most of its cross-cluster routing superpowers anyway. + +--- + +## The four L7 traffic patterns + +With deployment boundaries established, here are the four architectural patterns we use to manipulate Kafka traffic at Layer 7. + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Kroxylicious L7 Proxy │ +├─────────────────┬─────────────────┬──────────────┬──────────────┤ +│ Cluster Aliasing│ Union Clusters │ Union Topics │ Topology- │ +│ │ │ │ Aware │ +│ [Virtual A] │ [Virtual Hub] │ [Logical T] │ [AZ-a Client]│ +│ │ │ ┌─────┴─────┐ │ ┌─────┴───┐ │ │ │ +│ ▼ │ ▼ ▼ │ ▼ ▼ │ ▼ │ +│ [Physical A/B] │ [Phys 1] [Phys 2]│ [P0-1] [P2-3]│ [Broker-a] │ +└─────────────────┴─────────────────┴──────────────┴──────────────┤ + │ + ▼ │ +┌─────────────────────────────────────────────────────────────────┐ +│ Physical Kafka Clusters │ +└─────────────────────────────────────────────────────────────────┘ + +``` + +### 1. Cluster Aliasing + +**What it is:** Mapping a static virtual cluster endpoint to actual physical backend clusters, allowing you to switch the target backend at the proxy layer without reconfiguring or restarting clients. + +**How it works:** Clients connect to `kafka-virtual.company.internal`. The proxy inspects incoming Kafka frames (`Produce`, `Fetch`, `Metadata`) and maps them to physical brokers in `cluster-blue`. When you want to migrate to `cluster-green`, you update the proxy's routing target. The proxy handles the connection handoff to the new brokers under the hood. + +**Real-world caveat:** Swapping backend targets at the proxy layer solves client connectivity, but it doesn't magically sync topic data or offset state between backends. If you flip the pointer without state replication, your consumers will hit offset mismatches. (We'll cover how we pair this with byte-level replication and KIP-1279 in Post 2). + +### 2. Union Clusters + +**What it is:** Exposing multiple distinct backend physical clusters through a single virtual cluster endpoint. + +**How it works:** The proxy intercepts `Metadata` requests and synthesizes a single, unified cluster layout for the client. To an incoming producer or consumer, it looks like one massive cluster. Behind the scenes, the proxy routes requests for `orders-*` topics to a high-throughput physical cluster, while `analytics-*` topics head to a cheaper, storage-optimized cluster. + +**Real-world caveat:** Namespace collisions will ruin your day. If `orders-v1` exists on two backend clusters, the proxy has to decide which physical cluster wins or enforce explicit topic-prefix rules. Also, we haven't benchmarked proxy metadata synthesis overhead at tens of thousands of topics across dozens of physical backends yet, so expect memory usage to scale with metadata volume. + +### 3. Union Topics + +**What it is:** Presenting a single logical Kafka topic to clients while sharding its underlying partitions across multiple physical clusters. + +**How it works:** A client asks for metadata for topic `events`, which appears to have 32 partitions. The proxy returns metadata where partitions 0–15 point to physical brokers in Cluster A, and partitions 16–31 point to physical brokers in Cluster B. When a client sends a `ProduceRequest` for partition 20, the proxy routes those specific frames to Cluster B. + +**Real-world caveat:** Transaction coordinator boundaries stop working cleanly here. If your applications rely on multi-topic transactions or read-committed isolation levels across partitions, splitting a topic across physical cluster boundaries breaks those guarantees today. Treat this pattern as a fit for simple, un-keyed, or independently partition-keyed workloads until proxy transaction handling matures. + +### 4. Topology-Aware Routing + +**What it is:** Directing client traffic dynamically based on client metadata, network topology, or locality attributes (like cloud availability zones). + +**How it works:** Kafka cross-AZ data transfer fees are a recurring budget surprise for infra teams. With topology-aware routing, the proxy reads client IP blocks or rack attributes and routes fetch requests to brokers or read-replicas inside the same availability zone, cutting down on inter-zone bandwidth costs. + +**Real-world caveat:** Local routing savings disappear if your proxy fleet is deployed inefficiently. If a client in `us-east-1a` sends frames to a proxy instance running in `us-east-1b`, which then forwards bytes to a broker in `us-east-1a`, you've just doubled your cross-AZ costs instead of eliminating them. Proxy placement must match client topology. + +--- + +## What's next + +This taxonomy gives us a common vocabulary for describing how L7 proxies reshape Kafka traffic and where they fit into physical deployment topologies. Over the coming weeks leading up to Current in San Francisco, we're going to dive deeper into the actual implementations. + +Next week in Post 2, we'll take a close look at **Active-Passive DR with KIP-1279 & Cluster Aliasing**, breaking down how Virtual Cluster Keys swap backend targets without dropping client sockets, and showing the code behind our live demo. + +If you're playing with Kafka traffic routing, building custom extensions, or just want to tell us where our architecture assumptions are wrong, drop by our [GitHub](https://github.com/kroxylicious/kroxylicious?utm_source=gemini), join us on [Slack](https://www.google.com/search?q=https://kroxylicious.slack.com&utm_source=gemini), or find us on [Bluesky](https://bsky.app/profile/kroxylicious.io?utm_source=gemini). From dcd31b3f23e8f1d2e93d18474c58f6412ed61073 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Mon, 21 Sep 2026 14:23:43 +1200 Subject: [PATCH 02/24] Bob's rewrite Signed-off-by: Sam Barker --- _drafts/routing-post1.md | 129 ++++++++++----------------------------- 1 file changed, 32 insertions(+), 97 deletions(-) diff --git a/_drafts/routing-post1.md b/_drafts/routing-post1.md index 35d8cd06..d01dea5b 100644 --- a/_drafts/routing-post1.md +++ b/_drafts/routing-post1.md @@ -1,6 +1,6 @@ --- layout: post -title: "Kroxylicious release 0.24.0" +title: "These are not the brokers you are connecting to" date: 2026-09-04 15:00:00 +1200 author: "Sam Barker" author_url: "https://github.com/sambarker" @@ -9,131 +9,66 @@ categories: blog kroxylicious-proxy tags: [ "routing" ] --- -# Layer 7 Kafka Routing: A Pattern Taxonomy for the Consolidation Minefield +Back in May, Tom wrote about [a proof of concept for routing]({% post_url 2026-05-21-topic-routing %}) — Kafka clients producing and consuming from topics scattered across multiple clusters, with no idea that's what they're doing. The proxy handles the fan-out, the response merging, the session state. The client just sees topics. -If you've ever tried to migrate a live Kafka workload between clusters, consolidate regional deployments, or shift cloud providers, you already know the bottom line: Kafka clients are remarkably opinionated about network topology. They don't just talk to a virtual endpoint; they demand exact broker metadata, explicit partition assignments, and direct TCP connections to specific physical nodes. +That POC has been quietly maturing. We're now in the process of turning it into something you can actually run in production, and in a few weeks we'll be talking about it at [Current in San Francisco](https://current.confluent.io/). The talk is called *"These are not the brokers you are connecting to"*, which felt like the honest description of what the proxy is doing. -When you try to reshape that underlying physical infrastructure without breaking application teams, you usually end up picking which operational headache you dislike the least. +This post is the first in a short series leading up to that talk. The goal here is to establish some vocabulary — I've been thinking about the routing design space and I'm convinced there are a handful of distinct patterns worth naming separately, because each one carries different tradeoffs and breaks down in different ways. The names are mine and I'll happily accept better ones, but I think having *something* to call them makes it easier to reason about the problems without conflating them. -## Pick your poison: Dual writes, MirrorMaker, or scheduled downtime +The deeper dives on each pattern will follow in subsequent posts. -Before looking at proxy-level patterns, it helps to review the standard tools people use when trying to move or consolidate Kafka traffic—and why they so often result in late-night incident reviews. +## Pick your poison -* **Application-level dual writes:** You ask application teams to update their producer code to write to both Cluster A and Cluster B simultaneously. In architectural diagrams, this looks clean. In production, network blips cause asymmetric failures, message ordering drifts instantly, and handling duplicate delivery becomes the application's problem. Worse, getting twenty product teams to deploy matching code changes on the same timeline is an exercise in cat-herding. -* **Replication pipelines (MirrorMaker 2, etc.):** Running an intermediary replication cluster works reasonably well for asynchronous backup, but relying on it for live consolidation adds latency, doubles your storage and network bills, and forces consumers to deal with offset translation. You're essentially running twice as much hardware to move bytes you already owned. -* **Hard cutovers and maintenance windows:** You schedule a Sunday 2:00 AM window, drain topic queues, update DNS records, and restart clients. This is conceptually simple right up until a legacy service ignores DNS TTLs, holds onto stale socket connections indefinitely, and drops messages silently when the old brokers finally go dark. +If you've tried to migrate a live Kafka workload between clusters, consolidate regional deployments, or shift cloud providers, you know the bottom line: Kafka clients are remarkably opinionated about where their brokers live. They don't just talk to a virtual endpoint — they demand exact broker metadata, explicit partition assignments, and direct TCP connections to specific physical nodes. You can't just update a load balancer and go home. -Intercepting traffic at Layer 7—the Kafka wire protocol itself—offers an alternative. By placing a proxy like Kroxylicious between clients and brokers, we can manipulate metadata and route requests on the fly. +So when you need to reshape the underlying infrastructure without breaking application teams, you usually end up picking which operational headache you dislike the least. -To be clear: introducing an L7 proxy adds a hop, consumes CPU, and gives you another piece of infrastructure to manage. If a simple DNS CNAME flip actually solves your problem, do that instead. But when you need to decouple physical cluster topologies from what clients see, proxying gives you control back. +**Application-level dual writes** look clean in architecture diagrams. In production, network blips cause asymmetric failures, message ordering drifts, and duplicate delivery becomes every application team's problem. Getting twenty teams to deploy matching code changes on the same timeline is an exercise in cat-herding that usually ends with someone's Friday afternoon becoming someone else's Saturday morning. ---- - -## Where does the proxy run? Forward, Reverse, and Sidecars - -Choosing a routing pattern is only half the battle. You also have to decide where the proxy tier physically lives, who owns it, and whether it acts as a forward proxy for egress or a reverse proxy for ingress. - -``` - ┌─────────────────────────────────────────────────────────┐ - │ FORWARD PROXY │ - │ (Client-Side Egress) │ - │ ┌───────────────────────┐ ┌───────────────────────┐ │ - │ │ Client App + Pod │ │ Client Cluster │ │ - │ │ Sidecar Proxy │ │ Centralized Gateway │ │ - │ └───────────┬───────────┘ └───────────┬───────────┘ │ - └──────────────┼───────────────────────────┼──────────────┘ - │ │ - NETWORK BOUNDARY / TRANSIT / VPC PEERING │ - │ │ - ┌──────────────┼───────────────────────────┼──────────────┐ - │ ▼ ▼ │ - │ ┌───────────────────────┐ ┌───────────────────────┐ │ - │ │ Broker Cluster │ │ Broker Node + │ │ - │ │ Ingress Gateway │ │ Broker Sidecar Proxy │ │ - │ └───────────────────────┘ └───────────────────────┘ │ - │ REVERSE PROXY │ - │ (Broker-Side Ingress) │ - └└─────────────────────────────────────────────────────────┘ +**Replication pipelines** (MirrorMaker 2, et al.) work fine for asynchronous backup, but for live consolidation you're adding latency, doubling storage and network costs, and asking consumers to deal with offset translation. You're running twice as much hardware to move bytes you already owned. -``` +**Maintenance windows** are conceptually simple right up until a legacy service ignores DNS TTLs, holds onto stale sockets, and drops messages silently when the old brokers finally go dark. Sunday 2am is a fine time for this to happen. -### The Forward Proxy Model (Client-Side Egress) +## What a Layer 7 proxy buys you -Lives in the client's network boundary and is managed by application or client-platform teams. +Intercepting at Layer 7 — the Kafka wire protocol itself — is the alternative. The proxy sits between clients and brokers, inspects and rewrites Kafka frames, and presents whatever cluster topology it wants to clients regardless of what's actually behind it. Clients connect to a virtual cluster. What they get told about that cluster is up to us. -* **Client Cluster Gateway:** A shared proxy fleet inside the client Kubernetes cluster or VPC. Applications point to a local gateway service, and the proxy handles cross-cluster egress across network boundaries. -* **App Pod Sidecar:** Co-located inside the application pod as an egress proxy. Crucially, the proxy does *not* flatten or hide the Kafka cluster model—the client driver still receives metadata mapped to local endpoints (e.g., port ranges on `localhost`) and maintains individual TCP sockets per broker. You get isolated blast radius per pod, but running hundreds of Netty/JVM proxy containers across a microservice fleet levies a noticeable baseline memory tax. +To be clear about the tradeoffs: an L7 proxy adds a network hop, consumes CPU for frame inspection and rewriting, and gives you another piece of infrastructure to keep alive. If a DNS CNAME swap actually solves your problem, do that instead. But when you need to decouple what clients see from how your physical infrastructure is actually laid out, this is the lever. -### The Reverse Proxy Model (Broker-Side Ingress) - -Lives in the Kafka cluster's network boundary and is managed by the central infrastructure/platform team. - -* **Broker Cluster Ingress Gateway:** A shared ingress fleet sitting in front of physical Kafka brokers. Provides a unified front door for incoming client connections and shields underlying cluster topology, though a gateway outage impacts all incoming traffic. -* **Broker Node Sidecar:** Co-located on the actual physical broker hardware (one proxy instance per broker node). While it eliminates an internal network hop on paper, almost nobody does this in production. You risk L7 proxy bugs or memory spikes starving the underlying broker JVM or page cache, and a broker-side proxy loses most of its cross-cluster routing superpowers anyway. - ---- +## Four patterns (working names) -## The four L7 traffic patterns +After reviewing the [routing design proposal](https://github.com/kroxylicious/design/pull/70) and thinking through the problem space, I think there are four distinct patterns worth naming. They're not mutually exclusive and they compose, but each one has its own failure modes, so it helps to be able to talk about them separately. -With deployment boundaries established, here are the four architectural patterns we use to manipulate Kafka traffic at Layer 7. +### Cluster aliasing -``` -┌─────────────────────────────────────────────────────────────────┐ -│ Kroxylicious L7 Proxy │ -├─────────────────┬─────────────────┬──────────────┬──────────────┤ -│ Cluster Aliasing│ Union Clusters │ Union Topics │ Topology- │ -│ │ │ │ Aware │ -│ [Virtual A] │ [Virtual Hub] │ [Logical T] │ [AZ-a Client]│ -│ │ │ ┌─────┴─────┐ │ ┌─────┴───┐ │ │ │ -│ ▼ │ ▼ ▼ │ ▼ ▼ │ ▼ │ -│ [Physical A/B] │ [Phys 1] [Phys 2]│ [P0-1] [P2-3]│ [Broker-a] │ -└─────────────────┴─────────────────┴──────────────┴──────────────┤ - │ - ▼ │ -┌─────────────────────────────────────────────────────────────────┐ -│ Physical Kafka Clusters │ -└─────────────────────────────────────────────────────────────────┘ +A static virtual cluster endpoint maps to one physical backend, but the mapping can be changed at the proxy layer without touching clients. -``` +Clients connect to `kafka-virtual.company.internal`. The proxy maps their connections to `cluster-blue`. When you want to migrate to `cluster-green`, you update the proxy config. Clients never reconnect, never need new bootstrap addresses, never need to know a migration happened. -### 1. Cluster Aliasing +The catch is that this only solves connectivity. Flipping the pointer without replicating state means consumers hit offset mismatches on the new cluster. The routing layer isn't magic — it's just decoupling one specific piece of the problem. In Post 2 we'll look at pairing this with KIP-1279 to handle the state problem, which makes this pattern actually useful for DR rather than just theoretically interesting. -**What it is:** Mapping a static virtual cluster endpoint to actual physical backend clusters, allowing you to switch the target backend at the proxy layer without reconfiguring or restarting clients. +### Union clusters -**How it works:** Clients connect to `kafka-virtual.company.internal`. The proxy inspects incoming Kafka frames (`Produce`, `Fetch`, `Metadata`) and maps them to physical brokers in `cluster-blue`. When you want to migrate to `cluster-green`, you update the proxy's routing target. The proxy handles the connection handoff to the new brokers under the hood. +Multiple distinct physical clusters exposed through a single virtual cluster endpoint. The proxy synthesises a unified metadata view — to a producer or consumer it looks like one cluster. Behind the scenes, `orders-*` topics route to a high-throughput cluster, `analytics-*` topics go to a cheaper storage-optimised one. -**Real-world caveat:** Swapping backend targets at the proxy layer solves client connectivity, but it doesn't magically sync topic data or offset state between backends. If you flip the pointer without state replication, your consumers will hit offset mismatches. (We'll cover how we pair this with byte-level replication and KIP-1279 in Post 2). +The obvious failure mode is namespace collisions. If `orders-v1` exists on two backends, the proxy has to pick a winner or enforce prefix rules. We haven't pushed the metadata synthesis hard enough at scale yet to characterise the memory behaviour at tens of thousands of topics across many backends — worth flagging if that's your situation. -### 2. Union Clusters +### Union topics -**What it is:** Exposing multiple distinct backend physical clusters through a single virtual cluster endpoint. +A single logical topic whose partitions are sharded across multiple physical clusters. A client asks for metadata for `events` with 32 partitions; partitions 0–15 live on Cluster A, 16–31 on Cluster B. A `ProduceRequest` for partition 20 gets routed to Cluster B transparently. -**How it works:** The proxy intercepts `Metadata` requests and synthesizes a single, unified cluster layout for the client. To an incoming producer or consumer, it looks like one massive cluster. Behind the scenes, the proxy routes requests for `orders-*` topics to a high-throughput physical cluster, while `analytics-*` topics head to a cheaper, storage-optimized cluster. +The hard limit today: transaction coordinator boundaries break across physical clusters. If you're relying on multi-topic transactions or read-committed isolation, splitting a topic across cluster boundaries removes those guarantees. This pattern is a good fit for simple, independently partition-keyed workloads. For anything transactional, wait. -**Real-world caveat:** Namespace collisions will ruin your day. If `orders-v1` exists on two backend clusters, the proxy has to decide which physical cluster wins or enforce explicit topic-prefix rules. Also, we haven't benchmarked proxy metadata synthesis overhead at tens of thousands of topics across dozens of physical backends yet, so expect memory usage to scale with metadata volume. +### Topology-aware routing -### 3. Union Topics +Traffic directed dynamically based on client metadata or locality — the main use case being cross-AZ bandwidth costs, which have a habit of appearing as a surprise line item in cloud bills. The proxy reads rack attributes or client IP blocks and routes fetch requests to brokers inside the same availability zone. -**What it is:** Presenting a single logical Kafka topic to clients while sharding its underlying partitions across multiple physical clusters. - -**How it works:** A client asks for metadata for topic `events`, which appears to have 32 partitions. The proxy returns metadata where partitions 0–15 point to physical brokers in Cluster A, and partitions 16–31 point to physical brokers in Cluster B. When a client sends a `ProduceRequest` for partition 20, the proxy routes those specific frames to Cluster B. - -**Real-world caveat:** Transaction coordinator boundaries stop working cleanly here. If your applications rely on multi-topic transactions or read-committed isolation levels across partitions, splitting a topic across physical cluster boundaries breaks those guarantees today. Treat this pattern as a fit for simple, un-keyed, or independently partition-keyed workloads until proxy transaction handling matures. - -### 4. Topology-Aware Routing - -**What it is:** Directing client traffic dynamically based on client metadata, network topology, or locality attributes (like cloud availability zones). - -**How it works:** Kafka cross-AZ data transfer fees are a recurring budget surprise for infra teams. With topology-aware routing, the proxy reads client IP blocks or rack attributes and routes fetch requests to brokers or read-replicas inside the same availability zone, cutting down on inter-zone bandwidth costs. - -**Real-world caveat:** Local routing savings disappear if your proxy fleet is deployed inefficiently. If a client in `us-east-1a` sends frames to a proxy instance running in `us-east-1b`, which then forwards bytes to a broker in `us-east-1a`, you've just doubled your cross-AZ costs instead of eliminating them. Proxy placement must match client topology. +The failure mode here is proxy placement. A client in `us-east-1a` hitting a proxy in `us-east-1b` that forwards to a broker in `us-east-1a` doubles your cross-AZ cost instead of eliminating it. The proxy fleet has to be co-located with clients for the locality savings to materialise. --- -## What's next - -This taxonomy gives us a common vocabulary for describing how L7 proxies reshape Kafka traffic and where they fit into physical deployment topologies. Over the coming weeks leading up to Current in San Francisco, we're going to dive deeper into the actual implementations. +These four patterns give us a working vocabulary for the rest of the series. They're not all at the same maturity level — cluster aliasing and union topics are closest to production-ready, topology-aware routing is further out — and the posts that follow will be honest about where the code is versus where the design points. -Next week in Post 2, we'll take a close look at **Active-Passive DR with KIP-1279 & Cluster Aliasing**, breaking down how Virtual Cluster Keys swap backend targets without dropping client sockets, and showing the code behind our live demo. +Next up: **Active-Passive DR with cluster aliasing and KIP-1279**. Fair warning: this one is going to live as a POC on a branch rather than shipped code, but it's what we're demoing at Current and the mechanics are worth understanding on their own terms. -If you're playing with Kafka traffic routing, building custom extensions, or just want to tell us where our architecture assumptions are wrong, drop by our [GitHub](https://github.com/kroxylicious/kroxylicious?utm_source=gemini), join us on [Slack](https://www.google.com/search?q=https://kroxylicious.slack.com&utm_source=gemini), or find us on [Bluesky](https://bsky.app/profile/kroxylicious.io?utm_source=gemini). +If you're thinking about Kafka topology problems, have a use case that doesn't fit cleanly into any of these patterns, or want to tell us where the vocabulary breaks down — find us on [GitHub](https://github.com/kroxylicious/kroxylicious), [Slack](https://kroxylicious.slack.com), or [Bluesky](https://bsky.app/profile/kroxylicious.io). From fc2212c5c1eacb543442b7f94fbc15d36d3996bb Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Thu, 24 Sep 2026 11:52:50 +1200 Subject: [PATCH 03/24] Rewrite for voice and detail Signed-off-by: Sam Barker --- _drafts/routing-post1.md | 59 ++++++++++++++++++++++++---------------- 1 file changed, 35 insertions(+), 24 deletions(-) diff --git a/_drafts/routing-post1.md b/_drafts/routing-post1.md index d01dea5b..2cf51758 100644 --- a/_drafts/routing-post1.md +++ b/_drafts/routing-post1.md @@ -11,64 +11,75 @@ tags: [ "routing" ] Back in May, Tom wrote about [a proof of concept for routing]({% post_url 2026-05-21-topic-routing %}) — Kafka clients producing and consuming from topics scattered across multiple clusters, with no idea that's what they're doing. The proxy handles the fan-out, the response merging, the session state. The client just sees topics. -That POC has been quietly maturing. We're now in the process of turning it into something you can actually run in production, and in a few weeks we'll be talking about it at [Current in San Francisco](https://current.confluent.io/). The talk is called *"These are not the brokers you are connecting to"*, which felt like the honest description of what the proxy is doing. +That POC has been quietly maturing. We're now in the process of turning it into something you can actually run in production, and in a few weeks we'll be talking about it at [Current in San Francisco](https://current.confluent.io/san-francisco/sessions#session-SESS-165). In a session called *"These are not the brokers you are connecting to"* — my employer does not condone lying, but the proxy will quite happily tell your clients whatever you need it to. -This post is the first in a short series leading up to that talk. The goal here is to establish some vocabulary — I've been thinking about the routing design space and I'm convinced there are a handful of distinct patterns worth naming separately, because each one carries different tradeoffs and breaks down in different ways. The names are mine and I'll happily accept better ones, but I think having *something* to call them makes it easier to reason about the problems without conflating them. +This post is the first in a short series leading up to that talk. The goal here is to establish some vocabulary and get into the gory details I won't have time for onstage. I've been thinking about the design space routing enables and I'm convinced there are a handful of distinct patterns worth naming separately. There are no hard boundaries between the patterns, they all leverage the same underlying infrastructure, but they are distinct because they trade off different aspects of the problem space. The names are mine and while I'll happily accept better ones, we need something to start a conversation (plus I'm right :wink: ). The deeper dives on each pattern will follow in subsequent posts. ## Pick your poison -If you've tried to migrate a live Kafka workload between clusters, consolidate regional deployments, or shift cloud providers, you know the bottom line: Kafka clients are remarkably opinionated about where their brokers live. They don't just talk to a virtual endpoint — they demand exact broker metadata, explicit partition assignments, and direct TCP connections to specific physical nodes. You can't just update a load balancer and go home. +If you've tried to migrate a live Kafka workload between clusters, consolidate regional deployments, or shift cloud providers, you will have run into the same core problem: Kafka is not HTTP-based (stop rolling your eyes at the back). Kafka clients are genuinely indifferent to geography — they'll talk to a broker in the same pod just as happily as one in Timbuktu. What they care about, deeply and lastingly, is *which* broker they're talking to. Broker 1 is Broker 1. You can't quietly swap it out, move it between availability zones, or retire it without a lot of fuss. The load balancer doesn't get a say. Most proxies can only manage Kafka as a layer 4 protocol — and layer 4 is BOOOOORRRRING (I will not listen to arguments from the network engineer in the corner). So when you need to reshape the underlying infrastructure without breaking application teams, you usually end up picking which operational headache you dislike the least. -**Application-level dual writes** look clean in architecture diagrams. In production, network blips cause asymmetric failures, message ordering drifts, and duplicate delivery becomes every application team's problem. Getting twenty teams to deploy matching code changes on the same timeline is an exercise in cat-herding that usually ends with someone's Friday afternoon becoming someone else's Saturday morning. +**Application-level dual writes** look clean in architecture diagrams. In production, network blips cause asymmetric failures, message ordering drifts, and duplicate delivery becomes every application team's problem. Getting twenty teams to deploy matching code changes on the same timeline is an exercise in cat-herding that usually ends with someone's Friday afternoon becoming someone else's out of hours page. -**Replication pipelines** (MirrorMaker 2, et al.) work fine for asynchronous backup, but for live consolidation you're adding latency, doubling storage and network costs, and asking consumers to deal with offset translation. You're running twice as much hardware to move bytes you already owned. +**Replication pipelines** (MirrorMaker 2, et al.) work great for asynchronous backup, but for live consolidation knowing you are in sync is very difficult question, not to mention doubling storage and network costs, and asking consumers to deal with offset translation. You're running twice as much hardware to move bytes you already owned, fine if the migration ever actually ends. -**Maintenance windows** are conceptually simple right up until a legacy service ignores DNS TTLs, holds onto stale sockets, and drops messages silently when the old brokers finally go dark. Sunday 2am is a fine time for this to happen. +**Maintenance windows** are conceptually simple right up until a legacy service ignores DNS TTLs, holds onto stale sockets, and drops messages silently when the old brokers finally go dark. Sunday 2am is when the core reporting pipeline splutters to a halt. ## What a Layer 7 proxy buys you -Intercepting at Layer 7 — the Kafka wire protocol itself — is the alternative. The proxy sits between clients and brokers, inspects and rewrites Kafka frames, and presents whatever cluster topology it wants to clients regardless of what's actually behind it. Clients connect to a virtual cluster. What they get told about that cluster is up to us. +Jumping a few layers up the OSI stack to Layer 7 — the Kafka wire protocol itself — changes what's possible. Rather than blindly forwarding bytes, the proxy can inspect, rewrite, and route individual Kafka frames. It knows a `Metadata` response from a `Produce` request. It can answer the client's question about where the brokers are with whatever answer it likes. -To be clear about the tradeoffs: an L7 proxy adds a network hop, consumes CPU for frame inspection and rewriting, and gives you another piece of infrastructure to keep alive. If a DNS CNAME swap actually solves your problem, do that instead. But when you need to decouple what clients see from how your physical infrastructure is actually laid out, this is the lever. +Kroxylicious has used the concept of a Virtual Kafka Cluster (VKC) from the start — it's the endpoint clients connect to. What building routing support gave us was a breakthrough — the VKC is more than a networking construct. It's a stable, client-facing identity that we fully control. The brokers behind it can change. The topology behind it can change. The client doesn't need to know. -## Four patterns (working names) +That shift — from "the VKC is where you point your bootstrap servers" to "the VKC is the contract between the client and whatever we've decided is behind it" — is what opens up the design space this series is exploring. The posts that follow get into the mechanics of each pattern and are honest about where the implementation is today versus where the design points. The talk covers the highlights; the blog is where the details live. + +We get it, adding a proxy is operational overhead and we wish as much as you do swapping a CNAME from site A to site B was enough — but as far as Kafka clients are concerned, that's a divorce and why we are all here. + +## Name them we must + +After reviewing the [routing design proposal](https://github.com/kroxylicious/design/pull/70) and thinking through the problem space, I think there are multiple distinct deployment patterns worth naming. As with all patterns the boundaries are fuzzy and often more than one applies at once. The thing that convinces me they're real is that they have descriptive power — each one has its own tradeoffs, failure modes, and a distinct place on the spectrum from the clear(ish) waters of cluster aliasing to the mangrove swamp of virtual topics. Like all good abstractions, they're useful. + +If you're reaching for the [Enterprise Integration Patterns](https://www.enterpriseintegrationpatterns.com/patterns/messaging/MessageRoutingIntro.html) book right now — well done for being as old as I am, the rest of you just Googled it. It describes what happens to individual messages, which is interesting, but I'm talking about whole streams. Those patterns are what makes all this possible under the hood; what I'm describing sits a layer or two higher. + +The patterns do build on each other in terms of what the proxy has to understand about the Kafka protocol — each one introduces a new class of work the router stage has to own. But the complexity doesn't compound when you compose them, because the routing pipeline is a DAG: each stage is narrowly scoped to its own job and doesn't need to know what the stages around it are doing. A topology-aware routing stage doesn't care whether the topic it's directing traffic to is a virtual topic or a plain one. A virtual topic stage doesn't care whether the cluster it's dispatching to is an alias or a physical cluster. You pay the cost of each stage once, and only if you need it. -After reviewing the [routing design proposal](https://github.com/kroxylicious/design/pull/70) and thinking through the problem space, I think there are four distinct patterns worth naming. They're not mutually exclusive and they compose, but each one has its own failure modes, so it helps to be able to talk about them separately. ### Cluster aliasing -A static virtual cluster endpoint maps to one physical backend, but the mapping can be changed at the proxy layer without touching clients. +Until recently a VKC provided a 1:1 mapping between the thing the client addressed and the thing the proxy dialled. The routing API breaks that constraint — one VKC can now connect to one or more physical backends, and the target isn't fixed. The VKC is stable; what's behind it doesn't have to be. + +*Boom.* A semi truck just drove through the wall of your DC and into your primary cage. What now? Kroxylicious already sits in the client cage — you tell it `VKC_ALPHA` now targets `blue-cluster` instead of `green-cluster`. Done. -Clients connect to `kafka-virtual.company.internal`. The proxy maps their connections to `cluster-blue`. When you want to migrate to `cluster-green`, you update the proxy config. Clients never reconnect, never need new bootstrap addresses, never need to know a migration happened. +This is the Kafka DR dream. So what's new? You could always update a config file and restart. The difference is the failover can now happen in flight — a router that listens to cluster health checks, or one that responds to being paged, can flip the switch without dropping client connections. That was always the easy part though. The hard part is knowing it's safe to do so. *KIP-1279: Cluster Mirroring enters stage left.* By mirroring `green` to `blue` we can trust that when we flip the switch, offset state is intact and it's safe to do so. -The catch is that this only solves connectivity. Flipping the pointer without replicating state means consumers hit offset mismatches on the new cluster. The routing layer isn't magic — it's just decoupling one specific piece of the problem. In Post 2 we'll look at pairing this with KIP-1279 to handle the state problem, which makes this pattern actually useful for DR rather than just theoretically interesting. +Aliasing is more powerful than a dead man's switch though. Consider: you're chasing an issue in the reporting pipeline that only shows up in the sixth hour of the run and nobody can figure out which record trips it up. You've been there, right? What you really want is access to the live data with a debugger. Alas, this job is stuffed full of Personally Identifiable Information — so you're out of luck. You've been there and got *that* t-shirt. What now? You could build a duplication router that shadows production traffic to a staging cluster, piping it through a redaction filter on the way so the PII becomes gibberish. The client never knows. Post 2 gets into the detail. + +Before routing, Kroxylicious was a pipe with opinions — it could inspect and rewrite the stream, but it was inherently connection-oriented. One client socket, one backend, straight through. Cluster aliasing breaks the static part of that: the backend can now change. But it's still 1:1. ### Union clusters -Multiple distinct physical clusters exposed through a single virtual cluster endpoint. The proxy synthesises a unified metadata view — to a producer or consumer it looks like one cluster. Behind the scenes, `orders-*` topics route to a high-throughput cluster, `analytics-*` topics go to a cheaper storage-optimised one. +Kroxylicious has always been like a cycle courier with opinions — it inspects, rewrites, and has strong views about what is appropriate to carry, but the destination was fixed when the client dropped off the parcel. Routing turns it into a sorting office with standing orders. You write *George Street* on the parcel and drop it at the counter. The sorting office decides whether you mean George Street, Edinburgh or George Street, Dunedin — a city Scottish settlers named after Edinburgh, gave the same street names, and promptly scrambled the layout. (Guess why I know.) The sender doesn't need to know both exist. The brokers behind a union cluster work the same way: they still have to exist somewhere, we're not making them up, but the client only ever sees the one address it was given. Because the proxy controls what gets returned in a metadata response, you decide which brokers are visible, under what names, and what maps where. And we don't need a distributed consensus layer to do it — the Kafka community just spent considerable effort going from two consensus systems down to one with KRaft. Nobody wants a third. -The obvious failure mode is namespace collisions. If `orders-v1` exists on two backends, the proxy has to pick a winner or enforce prefix rules. We haven't pushed the metadata synthesis hard enough at scale yet to characterise the memory behaviour at tens of thousands of topics across many backends — worth flagging if that's your situation. +You look after a gaggle of Kafka clusters. You know how they got there. You're not proud of all of them. You can't herd them — geese are worse than cats, they honk back — but you don't have to admit to anyone else how many geese there actually are. -### Union topics +Point your application teams at one VKC and the proxy stitches the backing clusters together into a single cluster view. The client asks for metadata and gets back a broker list. It has no idea that list was assembled from three physical clusters, one of which is on-prem and nobody's quite sure what breed it is. You implement the routing logic — the proxy gives you the framework to dispatch requests to the right place. -A single logical topic whose partitions are sharded across multiple physical clusters. A client asks for metadata for `events` with 32 partitions; partitions 0–15 live on Cluster A, 16–31 on Cluster B. A `ProduceRequest` for partition 20 gets routed to Cluster B transparently. +Two things to keep in mind. Consumer group co-ordination stays pinned to physical clusters — that's almost always fine, because a group's offset state is tied to the topic-partitions it's consuming and those live on one backend, but it's worth knowing the seam is there. And transactions don't cross cluster boundaries: a transaction that writes to topics on two different backing clusters is two independent transactions whether your code believes that or not. If cross-topic atomicity matters, those topics need to share a physical cluster. -The hard limit today: transaction coordinator boundaries break across physical clusters. If you're relying on multi-topic transactions or read-committed isolation, splitting a topic across cluster boundaries removes those guarantees. This pattern is a good fit for simple, independently partition-keyed workloads. For anything transactional, wait. +### Virtual topics -### Topology-aware routing +A single logical topic whose partitions are composed from physical topics on multiple clusters — which don't even need to share a name. A client asks for metadata for `topic-x` and gets back 32 partitions; partitions 0–15 are sourced from `topic-x` on `us-east`, 16–31 from `topic-eu` on `eu-west`. A `ProduceRequest` for partition 20 gets routed to `eu-west` transparently, with partition numbers rewritten to match the physical layout. The client sees one topic. It has no idea. -Traffic directed dynamically based on client metadata or locality — the main use case being cross-AZ bandwidth costs, which have a habit of appearing as a surprise line item in cloud bills. The proxy reads rack attributes or client IP blocks and routes fetch requests to brokers inside the same availability zone. +The motivating example here is one union clusters can't solve. Your reporting pipeline in `us-east` is hardcoded to read from `topic-x`. It has always read from `topic-x`. It will continue to read from `topic-x`. The problem is on the producer side: your business is growing in Europe, and sending every EU event across the Atlantic to land on the `us-east` cluster is expensive, slow, and fragile. The EU team provisions `topic-eu` on a cluster local to them, sized for their producer load. The router presents both physical topics as one logical `topic-x` to every client. EU producers write locally. The reporting pipeline reads the full partition space without a config change. Nobody crosses the Atlantic unnecessarily. -The failure mode here is proxy placement. A client in `us-east-1a` hitting a proxy in `us-east-1b` that forwards to a broker in `us-east-1a` doubles your cross-AZ cost instead of eliminating it. The proxy fleet has to be co-located with clients for the locality savings to materialise. +What the proxy has to do here is more demanding than in any of the previous patterns: partition numbers must be rewritten consistently across every Kafka RPC that mentions them — `Metadata`, `Produce`, `Fetch`, `OffsetCommit`, `OffsetFetch`, `ListOffsets`, group coordinator lookups. That's the essential complexity of the pattern, not incidental overhead. It's a lot of surfaces to get right. The operational contract also shifts: both physical topics have to honour the mapping, and the router config is now the source of truth for what `topic-x` means. Transactions don't cross cluster boundaries here either — same caveat as union clusters, but worth repeating because with virtual topics it's easier to forget the seam is there. --- -These four patterns give us a working vocabulary for the rest of the series. They're not all at the same maturity level — cluster aliasing and union topics are closest to production-ready, topology-aware routing is further out — and the posts that follow will be honest about where the code is versus where the design points. - -Next up: **Active-Passive DR with cluster aliasing and KIP-1279**. Fair warning: this one is going to live as a POC on a branch rather than shipped code, but it's what we're demoing at Current and the mechanics are worth understanding on their own terms. +These three patterns give us a working vocabulary for the rest of the series. The posts that follow take each one in turn — what it can do, what it can't, and why. Some of the limits are ours and will close over time. Some are the Kafka protocol's: transactions have no external coordinator, and no amount of clever routing changes that. And some would require the proxy to grow a consensus layer of its own — which is a whole different animal, and not one we're planning to domesticate any time soon. If you're thinking about Kafka topology problems, have a use case that doesn't fit cleanly into any of these patterns, or want to tell us where the vocabulary breaks down — find us on [GitHub](https://github.com/kroxylicious/kroxylicious), [Slack](https://kroxylicious.slack.com), or [Bluesky](https://bsky.app/profile/kroxylicious.io). From 199d23249cd5b86d0b3c04c9c0480a420d1cb253 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Thu, 24 Sep 2026 12:31:17 +1200 Subject: [PATCH 04/24] clarify a few points from gemini. Signed-off-by: Sam Barker --- _drafts/routing-post1.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/_drafts/routing-post1.md b/_drafts/routing-post1.md index 2cf51758..da8a5bfe 100644 --- a/_drafts/routing-post1.md +++ b/_drafts/routing-post1.md @@ -37,7 +37,9 @@ Kroxylicious has used the concept of a Virtual Kafka Cluster (VKC) from the star That shift — from "the VKC is where you point your bootstrap servers" to "the VKC is the contract between the client and whatever we've decided is behind it" — is what opens up the design space this series is exploring. The posts that follow get into the mechanics of each pattern and are honest about where the implementation is today versus where the design points. The talk covers the highlights; the blog is where the details live. -We get it, adding a proxy is operational overhead and we wish as much as you do swapping a CNAME from site A to site B was enough — but as far as Kafka clients are concerned, that's a divorce and why we are all here. +Kroxylicious swims happily on either side of the network boundary — helping traffic egress from the client domain (forward proxy territory) or ingress into the broker domain (reverse proxy territory). We prefer egress and ingress: they describe traffic flows from the user's perspective. As much as I pour my time and energy into Kroxylicious, it's always the supporting cast — Kafka is the leading light here, and the vocabulary should reflect that. Some of the patterns that follow land more naturally on one side than the other, and I'll call that out as we go. + +We get it, adding a proxy is operational overhead and we wish as much as you do swapping a CNAME from site A to site B was enough — but as far as Kafka clients are concerned, a CNAME flip is an unexpected divorce. That's why we're all here. ## Name them we must @@ -54,7 +56,7 @@ Until recently a VKC provided a 1:1 mapping between the thing the client address *Boom.* A semi truck just drove through the wall of your DC and into your primary cage. What now? Kroxylicious already sits in the client cage — you tell it `VKC_ALPHA` now targets `blue-cluster` instead of `green-cluster`. Done. -This is the Kafka DR dream. So what's new? You could always update a config file and restart. The difference is the failover can now happen in flight — a router that listens to cluster health checks, or one that responds to being paged, can flip the switch without dropping client connections. That was always the easy part though. The hard part is knowing it's safe to do so. *KIP-1279: Cluster Mirroring enters stage left.* By mirroring `green` to `blue` we can trust that when we flip the switch, offset state is intact and it's safe to do so. +This is the Kafka DR dream. So what's new? You could always update a config file and restart. The difference is the target swap can happen without restarting the proxy or touching a single application config — a router that listens to cluster health checks, or one that responds to being paged, can flip the switch while traffic is flowing. Today that means the client connection resets and the client driver does what it was always going to do: reconnect and retry against the new target. No app code changes, no pod restarts. Getting to a genuinely seamless in-flight handover — no reset, no retry — is somewhere we would love to go, but we have other roads to travel first. That was always the easy part though. The hard part is knowing it's safe to do so. *KIP-1279: Cluster Mirroring enters stage left.* By mirroring `green` to `blue` we can trust that when we flip the switch, offset state is intact and it's safe to do so. Aliasing is more powerful than a dead man's switch though. Consider: you're chasing an issue in the reporting pipeline that only shows up in the sixth hour of the run and nobody can figure out which record trips it up. You've been there, right? What you really want is access to the live data with a debugger. Alas, this job is stuffed full of Personally Identifiable Information — so you're out of luck. You've been there and got *that* t-shirt. What now? You could build a duplication router that shadows production traffic to a staging cluster, piping it through a redaction filter on the way so the PII becomes gibberish. The client never knows. Post 2 gets into the detail. From c140cbef514ae0b269814a488bba0446080a1981 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Thu, 1 Oct 2026 17:33:48 +1300 Subject: [PATCH 05/24] Update post 1 draft with routing metaphor and tenant dispatch framing Signed-off-by: Sam Barker --- _drafts/routing-post1.md | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/_drafts/routing-post1.md b/_drafts/routing-post1.md index da8a5bfe..a0d46dc9 100644 --- a/_drafts/routing-post1.md +++ b/_drafts/routing-post1.md @@ -31,20 +31,26 @@ So when you need to reshape the underlying infrastructure without breaking appli ## What a Layer 7 proxy buys you -Jumping a few layers up the OSI stack to Layer 7 — the Kafka wire protocol itself — changes what's possible. Rather than blindly forwarding bytes, the proxy can inspect, rewrite, and route individual Kafka frames. It knows a `Metadata` response from a `Produce` request. It can answer the client's question about where the brokers are with whatever answer it likes. +Jumping up to Layer 7 changes the game. Rather than blindly punting TCP bytes around, the proxy speaks Kafka. It knows a `Metadata` response from a `Produce` request, and it can answer a client's question about where the brokers live with whatever answer you need. -Kroxylicious has used the concept of a Virtual Kafka Cluster (VKC) from the start — it's the endpoint clients connect to. What building routing support gave us was a breakthrough — the VKC is more than a networking construct. It's a stable, client-facing identity that we fully control. The brokers behind it can change. The topology behind it can change. The client doesn't need to know. +From day one, Kroxylicious gave you a Virtual Kafka Cluster (VKC) — a stable endpoint for your clients. But until recently, it was an opinionated pipe. That was a compliment: it did real work. It could encrypt records, enforce auth, rewrite headers — all without the client noticing. But it was still a fixed pipe: one client socket in, one backend socket out, wired to whatever cluster you pointed it at on boot. The filters were stateless with respect to the broker topology: an encryption filter might talk out-of-band to a KMS for keys, but it didn't care which physical cluster lived behind the socket. -That shift — from "the VKC is where you point your bootstrap servers" to "the VKC is the contract between the client and whatever we've decided is behind it" — is what opens up the design space this series is exploring. The posts that follow get into the mechanics of each pattern and are honest about where the implementation is today versus where the design points. The talk covers the highlights; the blog is where the details live. +Think of it like the horse in the late 19th century. A horse gets you from A to B across familiar ground without needing a national highway project. You feed it, you mount it, and you go. Horses were great. -Kroxylicious swims happily on either side of the network boundary — helping traffic egress from the client domain (forward proxy territory) or ingress into the broker domain (reverse proxy territory). We prefer egress and ingress: they describe traffic flows from the user's perspective. As much as I pour my time and energy into Kroxylicious, it's always the supporting cast — Kafka is the leading light here, and the vocabulary should reflect that. Some of the patterns that follow land more naturally on one side than the other, and I'll call that out as we go. +[Proposal 70](https://github.com/kroxylicious/design/blob/main/proposals/070-routing-api.md), the Routing API, is our combustion engine. The filters haven't gone anywhere — encryption, auth enforcement, and header rewriting all ride along inside the car. But the platform underneath can now make routing decisions on the fly. As a project, we provide the API so *any* car is possible, but we only supply the Model T — apocryphally in any colour you like, so long as it's black. If you need a bespoke Bentley or Bugatti router, you can build it. -We get it, adding a proxy is operational overhead and we wish as much as you do swapping a CNAME from site A to site B was enough — but as far as Kafka clients are concerned, a CNAME flip is an unexpected divorce. That's why we're all here. +The trade-off is the same one history made. A horse rider and a car driver both pick their own destination. But while the horse is happy on grass, the car only delivers that speed and range if there are roads, fuel stops, and mechanics. + +Routing is the exact same trade: more range, more power, but you depend on the world outside the proxy. Kroxylicious can't and shouldn't make your business policy decisions — it doesn't know which tenant belongs on which tier or when a cluster swap is safe. Those decisions belong to you. What the proxy gives you is the steering wheel, the throttle, and a much bigger map. + +We get it: running a proxy is operational overhead. We wish swapping a CNAME was enough as much as you do — but to a Kafka client, a CNAME flip is an unexpected divorce. That's why we're all here. ## Name them we must After reviewing the [routing design proposal](https://github.com/kroxylicious/design/pull/70) and thinking through the problem space, I think there are multiple distinct deployment patterns worth naming. As with all patterns the boundaries are fuzzy and often more than one applies at once. The thing that convinces me they're real is that they have descriptive power — each one has its own tradeoffs, failure modes, and a distinct place on the spectrum from the clear(ish) waters of cluster aliasing to the mangrove swamp of virtual topics. Like all good abstractions, they're useful. +You can deploy the proxy on either side of the network boundary: helping traffic egress from the client domain (forward proxy territory), or managing ingress into the broker domain (reverse proxy territory). We prefer *egress* and *ingress* because they describe traffic flows from your perspective rather than abstract networking theory. Some patterns land far more naturally on one side than the other, and we'll call that out as we go, but the underlying mechanics are the same. + If you're reaching for the [Enterprise Integration Patterns](https://www.enterpriseintegrationpatterns.com/patterns/messaging/MessageRoutingIntro.html) book right now — well done for being as old as I am, the rest of you just Googled it. It describes what happens to individual messages, which is interesting, but I'm talking about whole streams. Those patterns are what makes all this possible under the hood; what I'm describing sits a layer or two higher. The patterns do build on each other in terms of what the proxy has to understand about the Kafka protocol — each one introduces a new class of work the router stage has to own. But the complexity doesn't compound when you compose them, because the routing pipeline is a DAG: each stage is narrowly scoped to its own job and doesn't need to know what the stages around it are doing. A topology-aware routing stage doesn't care whether the topic it's directing traffic to is a virtual topic or a plain one. A virtual topic stage doesn't care whether the cluster it's dispatching to is an alias or a physical cluster. You pay the cost of each stage once, and only if you need it. @@ -54,11 +60,13 @@ The patterns do build on each other in terms of what the proxy has to understand Until recently a VKC provided a 1:1 mapping between the thing the client addressed and the thing the proxy dialled. The routing API breaks that constraint — one VKC can now connect to one or more physical backends, and the target isn't fixed. The VKC is stable; what's behind it doesn't have to be. -*Boom.* A semi truck just drove through the wall of your DC and into your primary cage. What now? Kroxylicious already sits in the client cage — you tell it `VKC_ALPHA` now targets `blue-cluster` instead of `green-cluster`. Done. +It's 2am on the last day of the month. The sales team has just launched an unannounced, wildly successful viral promotion. At the exact same moment, the scheduled end-of-month financial analytics job spins up and starts hammering the shared cluster with gigantic batch reads. Brokers are sweating, disk queues are spiking, and p99 produce latencies are heading for the stratosphere. + +In the world before routing, rescuing that shared cluster means waking up engineers across six different departments, getting them to edit `bootstrap.servers` in approximately one million config files, flushing DNS, restarting pods, and hoping nobody typos a port number in the panic. -This is the Kafka DR dream. So what's new? You could always update a config file and restart. The difference is the target swap can happen without restarting the proxy or touching a single application config — a router that listens to cluster health checks, or one that responds to being paged, can flip the switch while traffic is flowing. Today that means the client connection resets and the client driver does what it was always going to do: reconnect and retry against the new target. No app code changes, no pod restarts. Getting to a genuinely seamless in-flight handover — no reset, no retry — is somewhere we would love to go, but we have other roads to travel first. That was always the easy part though. The hard part is knowing it's safe to do so. *KIP-1279: Cluster Mirroring enters stage left.* By mirroring `green` to `blue` we can trust that when we flip the switch, offset state is intact and it's safe to do so. +With cluster aliasing, everyone still points at the exact same VKC. The proxy inspects who's knocking on the door when the connection establishes and dispatches them to the right physical cluster behind the scenes. The viral promotion gets its own dedicated, high-IOPS cluster; the analytics job lands quietly on bulk storage; everybody else keeps humming along on the shared tier. When you need to split or isolate workloads, the platform team updates a route. Zero application deployments. Zero out-of-hours pages to change a connection string. -Aliasing is more powerful than a dead man's switch though. Consider: you're chasing an issue in the reporting pipeline that only shows up in the sixth hour of the run and nobody can figure out which record trips it up. You've been there, right? What you really want is access to the live data with a debugger. Alas, this job is stuffed full of Personally Identifiable Information — so you're out of luck. You've been there and got *that* t-shirt. What now? You could build a duplication router that shadows production traffic to a staging cluster, piping it through a redaction filter on the way so the PII becomes gibberish. The client never knows. Post 2 gets into the detail. +Aliasing is more than workload dispatch, though. Consider: you're chasing an issue in the reporting pipeline that only shows up in the sixth hour of the run and nobody can figure out which record trips it up. You've been there, right? What you really want is access to the live data with a debugger. Alas, this job is stuffed full of Personally Identifiable Information — so you're out of luck. You've been there and got *that* t-shirt. What now? You could build a duplication router that shadows production traffic to a staging cluster, piping it through a redaction filter on the way so the PII becomes gibberish before it touches staging brokers. The client never knows. Post 2 gets into the detail. Before routing, Kroxylicious was a pipe with opinions — it could inspect and rewrite the stream, but it was inherently connection-oriented. One client socket, one backend, straight through. Cluster aliasing breaks the static part of that: the backend can now change. But it's still 1:1. From 68ec5a28dc5b1da46a0380ececf0e533b82886cc Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Mon, 5 Oct 2026 11:16:04 +1300 Subject: [PATCH 06/24] Polish 'How the proxy makes it work' section of routing post 2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Clarify MetadataResponse structure (broker list + partition→nodeId mapping) - Tighten node ID collision explanation: proxy cannot have two entries for node ID 2 - Reframe bijection as per-DAG-node with downstream/upstream ID spaces - Add N upstream ID spaces explanation — the reason a bijection is necessary - Replace LaTeX formula (doesn't render) with plain English arithmetic - Make diagram bidirectional and show fan-out to multiple clusters - Topology discovery: add 'no independent view' framing and per-router shared cache explanation with additive semantics safety argument - PID section: lead with the multi-cluster collision problem, add honest note that translation table is in-flight (topic partition routing work) Signed-off-by: Sam Barker --- _drafts/routing-post2.md | 167 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 167 insertions(+) create mode 100644 _drafts/routing-post2.md diff --git a/_drafts/routing-post2.md b/_drafts/routing-post2.md new file mode 100644 index 00000000..ecd766d5 --- /dev/null +++ b/_drafts/routing-post2.md @@ -0,0 +1,167 @@ +--- +layout: post +title: "Cluster Aliasing: Identity without commitment" +date: 2026-09-11 15:00:00 +1200 +author: "Sam Barker" +author_url: "https://github.com/sambarker" +# noinspection YAMLSchemaValidation +categories: blog kroxylicious-proxy +tags: [ "routing" ] +--- + +[Post 1]({link) introduced the Routing API and the fundamental shift it brings: a Virtual Kafka Cluster (VKC) is no longer a fixed pipe wired to a single physical cluster on boot. It’s a stable identity. What actually lives behind it is now the proxy's problem, not the client's. + +We called the first pattern **Cluster Aliasing**. + +At its core, aliasing is still a 1:1 mapping between a client and a physical cluster. The proxy isn't doing fan-out, merging responses, or stitching disparate topics together (we’ll get to those later in the series). But breaking the static link at boot means *which* cluster a client talks to can be decided dynamically when the connection is established—or redirected when the platform team needs to reshape the backends. + +It sounds simple. But when you decouple client configuration from physical infrastructure, the dream of flexible Kafka topologies starts to look feasible. + +## What you can do with it (and the dream it sets up) + +### The single front door: Tenant & workload dispatch + +Remember that 2am nightmare from post 1? The sales team launches an unannounced viral promotion right as end-of-month financial batch analytics fire up on the same shared cluster. + +In the old world, separating those workloads meant a multi-week coordination exercise: provision new hardware, issue new bootstrap URLs, cajole twenty application teams into updating their config repositories, and schedule rolling restarts. + +With cluster aliasing, everyone points at `kafka.corp.internal:9092`. When a client connects, the router inspects the authentication context established during handshake—who is knocking on the door?—and dispatches the connection to the appropriate physical cluster behind the scenes: + +- High-throughput production services get routed to dedicated, high-IOPS NVMe brokers. +- Heavy batch analytics jobs get dispatched to high-capacity bulk storage clusters. +- General application traffic stays on the shared tier. + +Now, let's be honest about the boundaries: migrating a producer pumping ephemeral clickstream data to a new cluster is trivial with aliasing—the platform team updates the route, and the producer begins writing to the new cluster with zero config changes and zero downtime. But what about the downstream analytics consumers that need to process that data? If they need continuous historical context, or if data was split across clusters mid-stream, the downstream consumer story is much more complicated. The proxy solves the connection and routing problem, but physical log replication and consumer offset synchronisation remain out-of-band challenges (more on that reality later). Still, for producers emitting fresh streams or teams operating in clean, discrete domains, cluster aliasing eliminates the connection coordination nightmare entirely. + +### Client-change-free cluster migrations and DR failover + +Let’s not pretend this is "seamless"—switching a distributed state machine like Kafka between physical backends always carries operational friction. But traditionally, both planned migrations (e.g. major version bumps, moving to KRaft, cloud region shifts) and unplanned Disaster Recovery (DR) failovers share an expensive logistical bottleneck: changing client endpoints. + +In a conventional setup, redirecting traffic to a secondary cluster or a freshly provisioned "Green" environment means: +- Coordinating with dozens of independent application teams. +- Updating DNS records (with all the painful quirks as DNS propagates, seemingly at random[^1]) or editing hundreds of configuration maps. +- Bouncing client fleets across an agreed maintenance window—or watching an incident drag out while waiting for client restarts. + +Cluster aliasing doesn't solve the hard data replication or consumer offset reconciliation problem, but it **completely eliminates the client-side migration cost**. The bootstrap configuration remains stable (`kafka.corp.internal:9092`). + +When executing a planned migration or a DR failover: + +1. **The data plane is handled out-of-band:** Topics and offsets are synced beforehand via active replication (MirrorMaker 2, KIP-1279 cluster mirroring), or recovered from storage volume snapshots. +2. **The connection cutover is a control-plane flip:** The proxy updates its route target to the secondary cluster—either globally or tenant-by-tenant. +3. **Rollback is instant:** If the secondary cluster misbehaves under live load, you can revert the routing target in seconds rather than triggering another round of client config rollbacks. + +Whether you're doing a planned blue/green transition or invoking DR in an emergency, cluster aliasing transforms a massive cross-team coordination exercise into an internal platform switch. + +[^1]: There is always some bit of infra that fails to honour the TTL properly or doesn't understand a CNAME or PTR record. + +### Production shadowing without the compliance conversation + +You're chasing an elusive bug that only manifests in the sixth hour of a high-volume run. Nobody can figure out which specific record payload triggers it. What you really want is to attach a debugger to the live stream in staging—except this production topic is stuffed full of Personally Identifiable Information (PII), and your compliance team has very reasonable objections. + +Because routing stages compose cleanly into a pipeline with standard filters, a duplicating router can shadow live traffic to a staging cluster alias while passing it through a redaction filter on the way. The PII is transformed into gibberish before the bytes ever touch staging disks. + +The client never knows. You get the real stream volume, the real cadence, the real record shapes—just not the real customer names. + +### The cloud bill from hell: Replica routing + +If you run Kafka in the cloud across multiple Availability Zones, you've probably had this exact conversation with your finance team. You opened the monthly AWS bill, choked on the 5-figure line item for cross-AZ data transfer, and dutifully turned on Kafka's built-in follower fetching (`client.rack`). + +And it worked! The bill plummeted. You closed the Jira ticket, posted a triumphant graph in `#infra-cost-savings`, and moved on. + +Then, six months later, the finance team knocks on your door again. The cross-AZ tax has silently crept right back up. + +Why? Because relying on every client in the company to configure rack awareness is an unwinnable game of whack-a-mole: +- Someone copy-pastes a Helm chart that hardcodes `client.rack=eu-west-1a` into a pod deployed in `eu-west-1b`. +- A data science team spins up a fleet of Python consumers using a library where nobody realised follower fetching requires a specific flag. +- A new microservice goes live with the defaults, happily pulling multi-gigabyte streams across the AZ boundary from whatever partition leader happens to be awake. + +Kafka's native rack awareness is great, but client compliance is leaky. Centralized problems need centralized controls. + +And this isn't just about inspecting frames at Layer 7. If all you have is a filter pipeline on a static 1:1 pipe, understanding that a `Fetch` belongs in `eu-west-1a` doesn't help you—the pipe is already nailed to broker 1 in `eu-west-1c`. The breakthrough comes from the Routing API making the backend connection dynamic. + +When a proxy instance deployed in `eu-west-1a` intercepts a `Fetch` request arriving on a virtual connection to the leader over in `eu-west-1c`, it inspects the partition's in-sync replica (ISR) set. If broker 3 is sitting right next door in `eu-west-1a` with a caught-up replica, the proxy doesn't rewrite the frame—it dynamically dispatches that individual request down a separate backend pipe to broker 3: + +``` +[ Consumer (AZ-a) ] ──Fetch──► [ Kroxylicious (AZ-a) ] ────► [ Broker 3 (AZ-a, Replica) ] + │ + (Skips AZ tax) + ▼ + [ Broker 1 (AZ-c, Leader) ] +``` + +The application didn't configure `client.rack`. The request payload is completely untouched. It simply travelled down a different pipe. Broker 1 never sees the read, and the cross-AZ request never happened so never reaches the invoice. + +Rack-local fetching is just leveraging the existing Kafka protocol. In theory, dynamic per-request dispatch opens up other intriguing possibilities to explore in your custom Bugatti router: +- **Replica lag awareness:** Steering reads strictly to the most caught-up followers to avoid trailing-edge staleness. +- **Broker load shedding:** Diverting fetch traffic away from CPU- or IO-bound leaders to give them headroom for critical writes. +- **Tiered storage offloading:** Routing historical backfills to brokers backed by cheap object storage while keeping hot tail-reads on fast NVMe. + +Whether those speculative policies prove practical in production remains to be seen—they require telemetry the proxy might not easily have—but proximity routing alone solves a very real, very expensive problem right now. + +There is, of course, a hard constraint: **proxy placement**. If a client in `eu-west-1a` talks to a proxy sitting in `eu-west-1b`, which then forwards to a broker in `eu-west-1a`, you’ve just invented a two-hop cross-AZ detour. For replica routing to pay off, your proxy fleet must be co-located with the clients it fronts. + +--- + +## How the proxy makes it work + +Making one physical cluster look like a different one at Layer 7 requires more than TCP forwarding. There are three places where the proxy has to do actual work. + +### Broker IDs are integers, not names + +Kafka wire frames identify brokers by integer `NodeId` — not hostnames. A `MetadataResponse` carries a broker list (node ID, host, port, rack) and maps each partition to its leader and replicas by node ID. The client resolves those node IDs to connections itself. That's fine when there's one physical cluster. When you have multiple physical clusters behind a VKC, you immediately run into a collision problem: broker 2 on cluster A and broker 2 on cluster B are completely different machines, but the proxy can only send one broker list to the client — it cannot have two different entries for node ID 2. + +The proxy solves this with a deterministic, stateless mapping applied at every node in the routing DAG. Each router speaks two consistent ID languages: a downstream ID space shared with whatever is below it, and an upstream ID space shared with whatever is above it. Requests flow upstream with IDs translated from downstream to upstream; responses flow downstream with IDs translated from upstream to downstream. The router reads and writes both spaces fluently in both directions. A node can have multiple upstream ID spaces — one per route — all of which must be collapsed into a single, unambiguous downstream space. The mapping is strictly one-to-one in both directions — a bijection, if you want the precise term — so every downstream ID resolves to exactly one upstream ID and route, with no collisions and no ambiguity. No database lookup, no distributed state, just arithmetic: the downstream node ID is the upstream node ID multiplied by the number of routes, plus the route index. For a leaf router, the upstream ID space happens to be the physical IDs coming off the wire from a real broker — but the mapping rule is the same regardless. In theory the upstream could itself be another Kroxylicious instance — the mapping rule is indifferent to what sits above the leaf. The only realistic reason you'd pay that latency tax is Conway's Law: separate teams owning separate proxy tiers with their own trust boundaries and operational responsibilities. The proxy runtime handles the translation at each hop transparently; routers work with opaque `VirtualNode` handles scoped to their own ID space and never need to care about the underlying integer arithmetic. + +``` + ◄──(cluster A IDs)──► [ broker1.cluster-a, broker2.cluster-a, ... ] +[ Client ] ◄──(downstream IDs)──► [ Router ] + ◄──(cluster B IDs)──► [ broker1.cluster-b, broker2.cluster-b, ... ] +``` + +### Topology discovery happens in-band + +The proxy has no independent view of the Kafka topology. It doesn't poll brokers in the background, run health checks, or hold idle connections open just to keep its metadata warm. Clients and routers drive connections; the proxy learns what it needs to know from the traffic that flows through it. + +Topology state is cached per-router and shared across all connections using that router — so a second connection to the same virtual cluster benefits from what the first one already discovered. The cache uses additive semantics: entries are added as they are learned, never removed. That matters for safety. A low-privilege connection might get a filtered `MetadataResponse` covering only the topics it can see; an additive cache means it can only *contribute* knowledge, never silently evict entries that a higher-privilege connection established. And cache poisoning doesn't translate into an access control bypass anyway — the cache only influences routing decisions. Brokers enforce ACLs on every request regardless; a misdirected frame gets an error, not someone else's data. + +When a client sends a request to a route whose topology the proxy doesn't know yet — on first connection, or after a stale entry — it discovers it on demand. The proxy synthesises a `MetadataRequest` on the active client channel, uses the response to populate its topology cache, and then forwards the original request as normal. Crucially, discovery inherits the client's authenticated context, so there's no privileged out-of-band management channel to secure or misconfigure. + +### Idempotent producers and the PID problem + +Idempotent producers get a Producer ID (`PID`) from the broker and use it to deduplicate retried writes. The moment you have multiple upstream clusters, you have a problem: each broker issues PIDs independently, with no coordination between clusters. Two brokers will cheerfully hand out the same PID to different producers, and the idempotency guarantee falls apart. + +The solution — currently being developed as part of the topic partition routing work — is a session-scoped translation table. When a client sends `InitProducerIdRequest` to a VKC, the proxy intercepts it and forwards the initialisation to the target routes. Each upstream broker issues its own PID. The proxy assigns the client a single virtual PID and maintains the mapping to the physical PIDs behind it. On every subsequent `ProduceRequest`, it rewrites the virtual PID to the correct physical one before forwarding. From the client's perspective it's talking to one cluster with one PID. From the broker's perspective every request arrives with a PID it recognises. The idempotency guarantee holds end-to-end. + +--- + +## Proxies aren't magic + +Layer 7 protocol inspection gives you a lot of leverage. It doesn't give you the ability to reach inside a Kafka broker and synchronise its internal state with another Kafka broker. That distinction matters, and it's worth being honest about it. + +### State lives on the physical cluster, not in the proxy + +Consumer group offsets, fetch sessions, and transaction coordination are all managed by coordinator brokers on a specific physical cluster. The proxy routes frames. It does not replicate or merge coordinator state between backends. + +If the routing target changes while a client is active: + +- **Transactional producers:** A transaction is coordinated by the Transaction Coordinator on whichever physical cluster the `InitProducerIdRequest` landed on. That transaction cannot be committed against a different cluster. If a failover happens mid-transaction, the subsequent `EndTxnRequest` will arrive at the new target and get back `INVALID_TXN_STATE` or `UNKNOWN_PRODUCER_ID`. Clients using transactions need to handle that abort cleanly and restart the transactional loop — ideally they already do, because the same failure happens on a plain broker restart. + +- **Consumer groups:** Offset state lives in `__consumer_offsets` on the physical cluster. The secondary cluster's Group Coordinator has no knowledge of what the primary's consumers have committed. Failing over without a replication pipeline that includes offsets means consumers either re-read data they've already processed, or skip ahead and lose it. Neither is silent. + +### Cutovers are an explicit decision, not a proxy heuristic + +Because the proxy can't silently catch up coordinator state, automated failover *inside the proxy* is a deliberate non-goal. + +A brief network hiccup and a genuine primary failure look identical from inside the proxy. The cost of getting that call wrong — dropping in-flight transactions, replaying messages, committing offsets against the wrong cluster — isn't the proxy's to bear. That judgement belongs to a human operator or an orchestration playbook that can actually assess cluster health and replication lag. + +Kroxylicious gives you the mechanism to flip the routing target without restarting anything. When to pull that trigger is your call. + +### Offset continuity requires out-of-band replication + +The proxy routes frames; it doesn't replicate log data. If consumers need to pick up where they left off on the secondary cluster, offsets have to line up — which means your replication pipeline ([KIP-1279](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1279%3A+Cluster+Mirroring) or equivalent) needs to be in place and caught up *before* you flip the switch, not after. Flipping the switch is not a substitute for having done that work. + +--- + +Cluster aliasing is the simplest pattern in this series, and it's already doing something genuinely useful: platform teams can change the physical backend without the application teams ever knowing it happened. + +The next post moves from one-to-one aliasing to stitching multiple physical clusters into a single logical view: **Union Clusters**. From b7f52c1fc0462953c7758c5fcb3f7f1c3d360b05 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Mon, 5 Oct 2026 11:50:18 +1300 Subject: [PATCH 07/24] Refine PID translation section - Ground readers with an explanation of idempotent delivery and PIDs before introducing the multi-cluster problem - Note idempotent delivery is default (and non-optional in Kafka 4.0) - Clarify the collision is between brokers on different clusters - Explain why a translation table rather than a bijection: PIDs are opaque broker-assigned integers with no exploitable arithmetic structure, unlike node IDs - Per-router scoping: mirrors the node ID bijection structure with downstream virtual PIDs mapping to upstream broker-allocated PIDs - Replace 'physical PID' with 'broker-allocated' throughout - Explain why session-scoped is sufficient: brokers own transactional PID continuity, proxy rebuilds downstream-to-upstream mapping on each new session Signed-off-by: Sam Barker --- _drafts/routing-post2.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/_drafts/routing-post2.md b/_drafts/routing-post2.md index ecd766d5..2389698b 100644 --- a/_drafts/routing-post2.md +++ b/_drafts/routing-post2.md @@ -128,9 +128,13 @@ When a client sends a request to a route whose topology the proxy doesn't know y ### Idempotent producers and the PID problem -Idempotent producers get a Producer ID (`PID`) from the broker and use it to deduplicate retried writes. The moment you have multiple upstream clusters, you have a problem: each broker issues PIDs independently, with no coordination between clusters. Two brokers will cheerfully hand out the same PID to different producers, and the idempotency guarantee falls apart. +Kafka producers use idempotent delivery by default. The broker deduplicates retried writes so that a message is committed exactly once even if the network drops the acknowledgement and the producer retries. The mechanism relies on a Producer ID (`PID`) that the broker assigns on initialisation, combined with a per-partition sequence number on every batch. If the broker sees a batch it's already committed — same PID, same sequence — it silently discards the duplicate. The client never needs to know. -The solution — currently being developed as part of the topic partition routing work — is a session-scoped translation table. When a client sends `InitProducerIdRequest` to a VKC, the proxy intercepts it and forwards the initialisation to the target routes. Each upstream broker issues its own PID. The proxy assigns the client a single virtual PID and maintains the mapping to the physical PIDs behind it. On every subsequent `ProduceRequest`, it rewrites the virtual PID to the correct physical one before forwarding. From the client's perspective it's talking to one cluster with one PID. From the broker's perspective every request arrives with a PID it recognises. The idempotency guarantee holds end-to-end. +That works cleanly with a single cluster. The moment you have multiple upstream clusters behind a VKC, you have a problem: each cluster's brokers issue PIDs independently, with no coordination between clusters. A broker on cluster A and a broker on cluster B will independently issue their own PIDs to the same producer — and the proxy needs to present a single coherent PID to the client while keeping track of both broker-allocated ones behind it. + +The solution — currently being developed — is a per-router, session-scoped translation table. The structure mirrors the node ID bijection — each router maintains its own mapping between the virtual PIDs on its downstream face and the broker-allocated PIDs on its upstream face — but the mechanism is different. Node IDs can be remapped arithmetically because the proxy controls the virtual ID space and can assign values deterministically. PIDs are opaque integers assigned by the broker; the proxy has no say in what values it receives and no arithmetic relationship to exploit between them. So instead of computing the translation, it remembers it. When a client sends `InitProducerIdRequest` to a VKC, each router intercepts it and fans the initialisation out to its upstream routes. Each upstream issues its own PID. The router assigns a single virtual PID on its downstream face and maintains the mapping to the broker-allocated PIDs behind it. On every subsequent `ProduceRequest`, it rewrites the downstream virtual PID to the correct broker-allocated one before forwarding. From the client's perspective it's talking to one cluster with one PID. From each broker's perspective every request arrives with a PID it recognises. The idempotency guarantee holds end-to-end. + +The translation table is session-scoped, with no durable store and no sharing across sessions — and that's fine. For transactional producers, PID continuity (the same broker-allocated PID handed back with a bumped epoch for the same `transactional.id`) is state the backing brokers already own and persist themselves. The proxy doesn't need to shadow that durability. On every new session — whether the client is brand new or reconnecting with an existing `transactional.id` — the proxy simply re-runs the fan-out and rebuilds the downstream-to-upstream mapping from scratch. The `transactional.id` itself passes through to every route unchanged; it's not something the proxy translates. --- From 7c79206fc9e370b9c7f7aabc51665922fa8855cb Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Tue, 6 Oct 2026 13:17:59 +1300 Subject: [PATCH 08/24] Rework 'Proxies aren't magic' section with clearer voice and structure Signed-off-by: Sam Barker --- _drafts/routing-post2.md | 38 ++++++++++++++++++++++++++------------ 1 file changed, 26 insertions(+), 12 deletions(-) diff --git a/_drafts/routing-post2.md b/_drafts/routing-post2.md index 2389698b..27706a15 100644 --- a/_drafts/routing-post2.md +++ b/_drafts/routing-post2.md @@ -13,9 +13,7 @@ tags: [ "routing" ] We called the first pattern **Cluster Aliasing**. -At its core, aliasing is still a 1:1 mapping between a client and a physical cluster. The proxy isn't doing fan-out, merging responses, or stitching disparate topics together (we’ll get to those later in the series). But breaking the static link at boot means *which* cluster a client talks to can be decided dynamically when the connection is established—or redirected when the platform team needs to reshape the backends. - -It sounds simple. But when you decouple client configuration from physical infrastructure, the dream of flexible Kafka topologies starts to look feasible. +Cluster aliasing makes the smallest possible change to Kafka protocol semantics. A connection to a VKC is still a connection to exactly one physical cluster — no fan-out, no merging, no stitching. The broker on the other end behaves like any Kafka broker, because it is one. What the proxy adds is a single degree of freedom: *which* cluster. Think of it in DNS terms: we transform a hardwired A record into a CNAME. The client has a stable name; what it resolves to is now the proxy's problem, not the client's. ## What you can do with it (and the dream it sets up) @@ -130,28 +128,44 @@ When a client sends a request to a route whose topology the proxy doesn't know y Kafka producers use idempotent delivery by default. The broker deduplicates retried writes so that a message is committed exactly once even if the network drops the acknowledgement and the producer retries. The mechanism relies on a Producer ID (`PID`) that the broker assigns on initialisation, combined with a per-partition sequence number on every batch. If the broker sees a batch it's already committed — same PID, same sequence — it silently discards the duplicate. The client never needs to know. -That works cleanly with a single cluster. The moment you have multiple upstream clusters behind a VKC, you have a problem: each cluster's brokers issue PIDs independently, with no coordination between clusters. A broker on cluster A and a broker on cluster B will independently issue their own PIDs to the same producer — and the proxy needs to present a single coherent PID to the client while keeping track of both broker-allocated ones behind it. +When you first look at PID management through the proxy, it looks like a problem we already solved. The previous section introduced a bijection for node IDs — a deterministic, stateless mapping between downstream IDs and upstream IDs at each router. PID translation looks like the same pattern, and the surface similarities are convincing: both are integers, both are cluster-scoped, both are assigned by the cluster rather than chosen by the client, and both are opaque values the client simply carries and passes through. So the proxy can just maintain a downstream-to-upstream mapping for both, right? This is the obvious answer. The seams show up quickly, but each one looks stitchable — until you're looking at a patchwork quilt and wondering why it keeps unravelling. The reason is that despite identical protocol syntax, node IDs and producer IDs have completely different semantics within the protocol. A node ID is **spatial** — a label, a coordinate. A PID is **temporal** — not just a label but a claim of active identity. In other words: we can mathematically fake a Node ID because it's just an address. We have to statefully track a Producer ID because it's a weaponised identity claim. + +**Node ID translation is transport-scoped.** A `MetadataResponse` tells you where brokers are. The proxy can tell as many lies as it likes here — the broker never sees them, let alone tries to call them out. And the proxy can be forgetful — stateless, even: if it discards its mapping and rebuilds it from the next `MetadataResponse`, it just tells the same lies again. Nothing notices, nothing breaks. + +**PID allocation is negotiation-scoped.** Since Kafka 3.0, producers run in idempotent mode by default — you don't have to opt in, it just happens. To support that, the producer calls `InitProducerIdRequest` to ask the broker to allocate it an identity: a (PID, epoch) pair. The broker tracks that identity and uses it to deduplicate retried batches within that session. It will also forcibly evict any connection that presents the same PID with a lower epoch — that is the fencing mechanism, preventing two connections from simultaneously claiming the same identity. If you need that identity to survive across sessions — so a restarted producer can pick up where it left off rather than starting fresh — that is what `transactional.id` is for. A second `InitProducerIdRequest` carrying the same `transactional.id` is treated as a hostile takeover, deliberately: fencing is how Kafka ensures only one producer instance is ever active for a given transactional identity. + +Both node IDs and PIDs are cluster-scoped — issued by a specific cluster, honoured within it, meaningless outside it. But the node ID relationship is stateless: if the proxy loses its mapping, it recovers via discovery and rederivation. A PID is unrecoverable. What was negotiated is gone. + +All of that is a long way of saying: the proxy cannot manage PIDs globally, and the Routing API makes no attempt to. There is no built-in abstraction that handles this for you — it is left to the team implementing the router to understand what their routing topology implies for producer identity. That is not as bleak as it sounds. There are shapes that work cleanly. If your problem fits one of them, go hard or go home. + +**Single physical cluster.** When all routes lead to the same physical cluster, there is nothing to solve. This is just standard Kafka protocol semantics — the proxy is a transparent conduit for a PID relationship that exists entirely between the client and the cluster. No mapping, no invention, no state to lose. Two things to keep in mind: routing decisions must be deterministic at the cluster level (a given producer must always land on the same physical cluster), and if the cluster changes — whether through a failover or a routing reconfiguration — the PID means nothing to the new cluster. Not fenced — a brand new client. -The solution — currently being developed — is a per-router, session-scoped translation table. The structure mirrors the node ID bijection — each router maintains its own mapping between the virtual PIDs on its downstream face and the broker-allocated PIDs on its upstream face — but the mechanism is different. Node IDs can be remapped arithmetically because the proxy controls the virtual ID space and can assign values deterministically. PIDs are opaque integers assigned by the broker; the proxy has no say in what values it receives and no arithmetic relationship to exploit between them. So instead of computing the translation, it remembers it. When a client sends `InitProducerIdRequest` to a VKC, each router intercepts it and fans the initialisation out to its upstream routes. Each upstream issues its own PID. The router assigns a single virtual PID on its downstream face and maintains the mapping to the broker-allocated PIDs behind it. On every subsequent `ProduceRequest`, it rewrites the downstream virtual PID to the correct broker-allocated one before forwarding. From the client's perspective it's talking to one cluster with one PID. From each broker's perspective every request arrives with a PID it recognises. The idempotency guarantee holds end-to-end. +**Duplication routing (traffic shadowing).** The shadow write to the secondary cluster is entirely router-invented — the client does not know Cluster B exists, and nothing downstream will ever attempt to resume or share that identity. The router negotiates its own PID with the shadow cluster and owns it from start to finish. What makes this safe is precisely the client's ignorance: because the shadow identity is invisible to the client, nothing can collide with it. There is one edge case worth naming: the shadow cluster has no knowledge of the primary's deduplication history. Writes the primary would have fenced as duplicates may be accepted on the shadow side, particularly after a proxy restart, when the negotiated PID is lost and a fresh one is issued to the shadow cluster. This is a dual-write pattern — do not use it where total accuracy is required. Shadow data is for observability. -The translation table is session-scoped, with no durable store and no sharing across sessions — and that's fine. For transactional producers, PID continuity (the same broker-allocated PID handed back with a bumped epoch for the same `transactional.id`) is state the backing brokers already own and persist themselves. The proxy doesn't need to shadow that durability. On every new session — whether the client is brand new or reconnecting with an existing `transactional.id` — the proxy simply re-runs the fan-out and rebuilds the downstream-to-upstream mapping from scratch. The `transactional.id` itself passes through to every route unchanged; it's not something the proxy translates. +**Multiple active clusters.** This is where the whole quilt unravels — we will pick it up in the union clusters post. --- ## Proxies aren't magic -Layer 7 protocol inspection gives you a lot of leverage. It doesn't give you the ability to reach inside a Kafka broker and synchronise its internal state with another Kafka broker. That distinction matters, and it's worth being honest about it. +Layer 7 protocol inspection gives you a lot of leverage. What it doesn't give you is the ability to reach inside a Kafka broker and move its internal state somewhere else. The proxy can route frames to a different cluster. It cannot conjure the state those frames depend on. -### State lives on the physical cluster, not in the proxy - -Consumer group offsets, fetch sessions, and transaction coordination are all managed by coordinator brokers on a specific physical cluster. The proxy routes frames. It does not replicate or merge coordinator state between backends. +### When you flip the target, the state doesn't follow If the routing target changes while a client is active: -- **Transactional producers:** A transaction is coordinated by the Transaction Coordinator on whichever physical cluster the `InitProducerIdRequest` landed on. That transaction cannot be committed against a different cluster. If a failover happens mid-transaction, the subsequent `EndTxnRequest` will arrive at the new target and get back `INVALID_TXN_STATE` or `UNKNOWN_PRODUCER_ID`. Clients using transactions need to handle that abort cleanly and restart the transactional loop — ideally they already do, because the same failure happens on a plain broker restart. +- **Transactional producers:** Kafka treats the Transaction Coordinator as an internal role within the cluster. Transactions are therefore scoped to a single cluster by definition — there is no extension point or API through which an external coordinator could be plugged in. A transaction that starts on cluster A cannot be committed on cluster B; if a routing cutover happens mid-transaction, the subsequent `EndTxnRequest` arrives at the new target and gets back `INVALID_TXN_STATE` or `UNKNOWN_PRODUCER_ID`. That abort needs to be handled cleanly and the transactional loop restarted — the same failure mode as a plain broker restart, which well-written transactional producers already handle. A proxy restart itself is safe: the proxy never invents a PID in the single-cluster case, so the KIP-360 fencing fields the client sends on reconnect are the broker's own real values. - **Consumer groups:** Offset state lives in `__consumer_offsets` on the physical cluster. The secondary cluster's Group Coordinator has no knowledge of what the primary's consumers have committed. Failing over without a replication pipeline that includes offsets means consumers either re-read data they've already processed, or skip ahead and lose it. Neither is silent. +### Layer 7 can only do so much + +Aliasing works because there is a single physical cluster backing it. The proxy isn't maintaining an illusion — it's directing traffic to something real, and real Kafka semantics apply throughout. The moment a router exposes brokers from multiple physical clusters as a unified address space, it has to ensure those semantics stay valid. For some operations that's fine — routing a `Fetch` to a local replica is cheap and safe. But for anything that touches producer identity or coordination — PIDs, epochs, transaction coordinators, consumer group coordinators — the router has taken on the obligation of upholding guarantees the protocol no longer provides on its behalf. + +The most immediate consequence is session affinity. The router must maintain a session-scoped mapping between the client's virtual PID and the real (PID, epoch) on each backend cluster. That mapping lives in the proxy instance that negotiated it. If the load balancer routes a reconnecting client to a different proxy replica, the mapping is gone — the router re-negotiates, the epoch bumps, and the backend fences the producer. The proxy can't prevent this. Only sticky L4 routing can. + +Friends don't let friends cross that line without knowing exactly what they've signed up for. + ### Cutovers are an explicit decision, not a proxy heuristic Because the proxy can't silently catch up coordinator state, automated failover *inside the proxy* is a deliberate non-goal. @@ -166,6 +180,6 @@ The proxy routes frames; it doesn't replicate log data. If consumers need to pic --- -Cluster aliasing is the simplest pattern in this series, and it's already doing something genuinely useful: platform teams can change the physical backend without the application teams ever knowing it happened. +Cluster aliasing keeps the operational promise simple: platform teams can change the physical backend without the application teams ever knowing it happened. That's not a small thing. The next post moves from one-to-one aliasing to stitching multiple physical clusters into a single logical view: **Union Clusters**. From 77c8443ee4d14523cd5e3346842e307be4cae5d4 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Tue, 6 Oct 2026 14:18:42 +1300 Subject: [PATCH 09/24] Reorder PID cases, footnote to bottom, tighten aliasing boundary framing Signed-off-by: Sam Barker --- _drafts/routing-post2.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/_drafts/routing-post2.md b/_drafts/routing-post2.md index 27706a15..6cc894b8 100644 --- a/_drafts/routing-post2.md +++ b/_drafts/routing-post2.md @@ -50,8 +50,6 @@ When executing a planned migration or a DR failover: Whether you're doing a planned blue/green transition or invoking DR in an emergency, cluster aliasing transforms a massive cross-team coordination exercise into an internal platform switch. -[^1]: There is always some bit of infra that fails to honour the TTL properly or doesn't understand a CNAME or PTR record. - ### Production shadowing without the compliance conversation You're chasing an elusive bug that only manifests in the sixth hour of a high-volume run. Nobody can figure out which specific record payload triggers it. What you really want is to attach a debugger to the live stream in staging—except this production topic is stuffed full of Personally Identifiable Information (PII), and your compliance team has very reasonable objections. @@ -140,9 +138,9 @@ All of that is a long way of saying: the proxy cannot manage PIDs globally, and **Single physical cluster.** When all routes lead to the same physical cluster, there is nothing to solve. This is just standard Kafka protocol semantics — the proxy is a transparent conduit for a PID relationship that exists entirely between the client and the cluster. No mapping, no invention, no state to lose. Two things to keep in mind: routing decisions must be deterministic at the cluster level (a given producer must always land on the same physical cluster), and if the cluster changes — whether through a failover or a routing reconfiguration — the PID means nothing to the new cluster. Not fenced — a brand new client. -**Duplication routing (traffic shadowing).** The shadow write to the secondary cluster is entirely router-invented — the client does not know Cluster B exists, and nothing downstream will ever attempt to resume or share that identity. The router negotiates its own PID with the shadow cluster and owns it from start to finish. What makes this safe is precisely the client's ignorance: because the shadow identity is invisible to the client, nothing can collide with it. There is one edge case worth naming: the shadow cluster has no knowledge of the primary's deduplication history. Writes the primary would have fenced as duplicates may be accepted on the shadow side, particularly after a proxy restart, when the negotiated PID is lost and a fresh one is issued to the shadow cluster. This is a dual-write pattern — do not use it where total accuracy is required. Shadow data is for observability. +**Multiple active clusters.** This is where aliasing ends. The full treatment — what can be made to work, what can't, and at what cost — is in the union clusters post. But the boundary is worth drawing clearly here before we get there: see *Layer 7 can only do so much* below. -**Multiple active clusters.** This is where the whole quilt unravels — we will pick it up in the union clusters post. +**Duplication routing (traffic shadowing).** One pattern that stays cleanly within the aliasing boundary: the shadow write to the secondary cluster is entirely router-invented — the client does not know Cluster B exists, and nothing downstream will ever attempt to resume or share that identity. The router negotiates its own PID with the shadow cluster and owns it from start to finish. What makes this safe is precisely the client's ignorance: because the shadow identity is invisible to the client, nothing can collide with it. There is one edge case worth naming: the shadow cluster has no knowledge of the primary's deduplication history. Writes the primary would have fenced as duplicates may be accepted on the shadow side, particularly after a proxy restart, when the negotiated PID is lost and a fresh one is issued to the shadow cluster. This is a dual-write pattern — do not use it where total accuracy is required. Shadow data is for observability. There will be other safe patterns depending on your topology — the constraint is client visibility, not the number of physical clusters involved. --- @@ -183,3 +181,5 @@ The proxy routes frames; it doesn't replicate log data. If consumers need to pic Cluster aliasing keeps the operational promise simple: platform teams can change the physical backend without the application teams ever knowing it happened. That's not a small thing. The next post moves from one-to-one aliasing to stitching multiple physical clusters into a single logical view: **Union Clusters**. + +[^1]: There is always some bit of infra that fails to honour the TTL properly or doesn't understand a CNAME or PTR record. From 0fd22a3139fe7120c1da30ed63c88a2cb4d5e800 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Tue, 6 Oct 2026 14:21:23 +1300 Subject: [PATCH 10/24] Promote routing post 1 to dated post, publish date 2026-10-10 Signed-off-by: Sam Barker --- _drafts/stash-replica-routing.md | 44 +++++++++++++++++++ ...e-not-the-brokers-you-are-connecting-to.md | 2 +- 2 files changed, 45 insertions(+), 1 deletion(-) create mode 100644 _drafts/stash-replica-routing.md rename _drafts/routing-post1.md => _posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md (99%) diff --git a/_drafts/stash-replica-routing.md b/_drafts/stash-replica-routing.md new file mode 100644 index 00000000..a65fa585 --- /dev/null +++ b/_drafts/stash-replica-routing.md @@ -0,0 +1,44 @@ +# Stash: replica routing pattern + +Removed from `routing-post1.md` — to be picked up in post 2 (cluster aliasing) as an intra-cluster optimisation. + +## Why it's not a peer pattern + +The other three patterns (cluster aliasing, union clusters, virtual topics) all answer "which cluster/topic does this request go to". Replica routing answers a different question: "given we've already decided which cluster, which broker within it serves this fetch". It's a different level of abstraction, and it only meaningfully applies to `Fetch` requests — produce requires the leader, and most admin operations that are partition-aware could use it but it's not the primary motivation. + +## The pattern + +- Client is connected to a VKC backed by, say, the DE cluster +- Client sends a `Fetch` for a partition whose leader is broker 1 in `eu-west-1c` +- The proxy is sitting in `eu-west-1a`, where broker 3 holds a valid in-sync replica of that partition +- Rather than forwarding down the existing pipe to broker 1, the proxy routes the request down a separate connection to broker 3 +- Broker 1 never sees the request. The client never knows there was a choice. + +The key implementation detail: the proxy doesn't rewrite the request and send it to a different address on the same connection. It has a separate backend connection to broker 3 and routes the `Fetch` down that pipe instead. The request itself is unchanged — what changes is which pipe it travels down. + +## Why this needs Layer 7 + +A Layer 4 proxy sees "connection to broker 1" and can only forward or drop. A Layer 7 proxy reads the `Fetch` request, understands which partition is being requested, knows the replica set and where each replica sits, and can choose which backend connection to use. The routing decision happens at the Kafka protocol level, not the TCP level. + +## Relationship to Kafka's built-in rack awareness + +This is a complement to rack awareness, not a replacement. `client.rack` + follower fetching works — when clients set it, and set it correctly. Replica routing extends the same idea: the proxy knows its own position in the topology and can apply locality policy centrally, covering clients that haven't configured rack awareness without requiring them to change anything. + +## Naming considered + +- **Topology-aware routing** — too tied to the implementation word "topology", was the original name +- **Locality routing** — too narrow, implies proximity is the only policy +- **Broker aliasing** — implies a static mapping, but the choice is dynamic +- **Replica routing** — landed here: names the thing being chosen between (replicas), leaves the policy open (proximity, lag, load, cost), intra-cluster by definition + +## Other policies beyond proximity + +- Replica lag — prefer the most caught-up replica +- Load — prefer the least busy broker +- Cost — prefer replicas on cheaper storage tiers + +None of these are proximity, which is why the name shouldn't bake in "locality". + +## Failure mode + +Proxy placement. A client in `us-east-1a` hitting a proxy in `us-east-1b` that then routes to a broker in `us-east-1a` adds a cross-AZ hop instead of eliminating one. The proxy fleet has to be co-located with clients for locality savings to materialise. diff --git a/_drafts/routing-post1.md b/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md similarity index 99% rename from _drafts/routing-post1.md rename to _posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md index a0d46dc9..2022a7ed 100644 --- a/_drafts/routing-post1.md +++ b/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md @@ -1,7 +1,7 @@ --- layout: post title: "These are not the brokers you are connecting to" -date: 2026-09-04 15:00:00 +1200 +date: 2026-10-10 15:00:00 +0100 author: "Sam Barker" author_url: "https://github.com/sambarker" # noinspection YAMLSchemaValidation From 1735dd8ca09c745ec89c3d54fad37c2e225d6388 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Tue, 6 Oct 2026 16:42:39 +1300 Subject: [PATCH 11/24] fix emoji usage Co-authored-by: Robert Young Signed-off-by: Sam Barker --- ...026-10-10-these-are-not-the-brokers-you-are-connecting-to.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md b/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md index 2022a7ed..f8fdf850 100644 --- a/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md +++ b/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md @@ -13,7 +13,7 @@ Back in May, Tom wrote about [a proof of concept for routing]({% post_url 2026-0 That POC has been quietly maturing. We're now in the process of turning it into something you can actually run in production, and in a few weeks we'll be talking about it at [Current in San Francisco](https://current.confluent.io/san-francisco/sessions#session-SESS-165). In a session called *"These are not the brokers you are connecting to"* — my employer does not condone lying, but the proxy will quite happily tell your clients whatever you need it to. -This post is the first in a short series leading up to that talk. The goal here is to establish some vocabulary and get into the gory details I won't have time for onstage. I've been thinking about the design space routing enables and I'm convinced there are a handful of distinct patterns worth naming separately. There are no hard boundaries between the patterns, they all leverage the same underlying infrastructure, but they are distinct because they trade off different aspects of the problem space. The names are mine and while I'll happily accept better ones, we need something to start a conversation (plus I'm right :wink: ). +This post is the first in a short series leading up to that talk. The goal here is to establish some vocabulary and get into the gory details I won't have time for onstage. I've been thinking about the design space routing enables and I'm convinced there are a handful of distinct patterns worth naming separately. There are no hard boundaries between the patterns, they all leverage the same underlying infrastructure, but they are distinct because they trade off different aspects of the problem space. The names are mine and while I'll happily accept better ones, we need something to start a conversation (plus I'm right 😉). The deeper dives on each pattern will follow in subsequent posts. From 9ea596e491b6ecb63df0597ff857e8f2d0873dda Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Tue, 6 Oct 2026 16:48:03 +1300 Subject: [PATCH 12/24] Grammar fix Co-authored-by: Robert Young Signed-off-by: Sam Barker --- ...026-10-10-these-are-not-the-brokers-you-are-connecting-to.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md b/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md index f8fdf850..b9072496 100644 --- a/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md +++ b/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md @@ -25,7 +25,7 @@ So when you need to reshape the underlying infrastructure without breaking appli **Application-level dual writes** look clean in architecture diagrams. In production, network blips cause asymmetric failures, message ordering drifts, and duplicate delivery becomes every application team's problem. Getting twenty teams to deploy matching code changes on the same timeline is an exercise in cat-herding that usually ends with someone's Friday afternoon becoming someone else's out of hours page. -**Replication pipelines** (MirrorMaker 2, et al.) work great for asynchronous backup, but for live consolidation knowing you are in sync is very difficult question, not to mention doubling storage and network costs, and asking consumers to deal with offset translation. You're running twice as much hardware to move bytes you already owned, fine if the migration ever actually ends. +**Replication pipelines** (MirrorMaker 2, et al.) work great for asynchronous backup, but for live consolidation knowing whether you're in sync is a very difficult question, not to mention doubling storage and network costs, and asking consumers to deal with offset translation. You're running twice as much hardware to move bytes you already owned, fine if the migration ever actually ends. **Maintenance windows** are conceptually simple right up until a legacy service ignores DNS TTLs, holds onto stale sockets, and drops messages silently when the old brokers finally go dark. Sunday 2am is when the core reporting pipeline splutters to a halt. From 2ee8c2b61a8c5383d7dc6e4091c6f641c66a0915 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Wed, 7 Oct 2026 15:04:38 +1300 Subject: [PATCH 13/24] Revise routing taxonomy: rename patterns, add stream branching section, restructure topic weaving - Rename 'cluster aliasing' -> 'connection switching' in post 1 and post 2 - Rename 'union clusters' -> 'topic weaving' in post 1 - Rename 'virtual topics' -> 'partition weaving' in post 1 - Add stream branching as a new pattern section in post 1 - Fix connection switching closing paragraph (T3 resolution) - Restructure topic weaving: Keith's NY/london/tokyo example leads, George Street becomes the collision case (T4 resolution) - Remove forward post number commitments; use pattern names instead - Update post 2 title and all body references to connection switching Signed-off-by: Sam Barker --- _drafts/routing-post2.md | 24 +++++------ ...e-not-the-brokers-you-are-connecting-to.md | 40 +++++++++++-------- 2 files changed, 36 insertions(+), 28 deletions(-) diff --git a/_drafts/routing-post2.md b/_drafts/routing-post2.md index 6cc894b8..886661f4 100644 --- a/_drafts/routing-post2.md +++ b/_drafts/routing-post2.md @@ -1,6 +1,6 @@ --- layout: post -title: "Cluster Aliasing: Identity without commitment" +title: "Connection Switching: Identity without commitment" date: 2026-09-11 15:00:00 +1200 author: "Sam Barker" author_url: "https://github.com/sambarker" @@ -11,9 +11,9 @@ tags: [ "routing" ] [Post 1]({link) introduced the Routing API and the fundamental shift it brings: a Virtual Kafka Cluster (VKC) is no longer a fixed pipe wired to a single physical cluster on boot. It’s a stable identity. What actually lives behind it is now the proxy's problem, not the client's. -We called the first pattern **Cluster Aliasing**. +We called the first pattern **Connection Switching**. -Cluster aliasing makes the smallest possible change to Kafka protocol semantics. A connection to a VKC is still a connection to exactly one physical cluster — no fan-out, no merging, no stitching. The broker on the other end behaves like any Kafka broker, because it is one. What the proxy adds is a single degree of freedom: *which* cluster. Think of it in DNS terms: we transform a hardwired A record into a CNAME. The client has a stable name; what it resolves to is now the proxy's problem, not the client's. +Connection switching makes the smallest possible change to Kafka protocol semantics. A connection to a VKC is still a connection to exactly one physical cluster — no fan-out, no merging, no stitching. The broker on the other end behaves like any Kafka broker, because it is one. What the proxy adds is a single degree of freedom: *which* cluster. Think of it in DNS terms: we transform a hardwired A record into a CNAME. The client has a stable name; what it resolves to is now the proxy's problem, not the client's. ## What you can do with it (and the dream it sets up) @@ -23,13 +23,13 @@ Remember that 2am nightmare from post 1? The sales team launches an unannounced In the old world, separating those workloads meant a multi-week coordination exercise: provision new hardware, issue new bootstrap URLs, cajole twenty application teams into updating their config repositories, and schedule rolling restarts. -With cluster aliasing, everyone points at `kafka.corp.internal:9092`. When a client connects, the router inspects the authentication context established during handshake—who is knocking on the door?—and dispatches the connection to the appropriate physical cluster behind the scenes: +With connection switching, everyone points at `kafka.corp.internal:9092`. When a client connects, the router inspects the authentication context established during handshake—who is knocking on the door?—and dispatches the connection to the appropriate physical cluster behind the scenes: - High-throughput production services get routed to dedicated, high-IOPS NVMe brokers. - Heavy batch analytics jobs get dispatched to high-capacity bulk storage clusters. - General application traffic stays on the shared tier. -Now, let's be honest about the boundaries: migrating a producer pumping ephemeral clickstream data to a new cluster is trivial with aliasing—the platform team updates the route, and the producer begins writing to the new cluster with zero config changes and zero downtime. But what about the downstream analytics consumers that need to process that data? If they need continuous historical context, or if data was split across clusters mid-stream, the downstream consumer story is much more complicated. The proxy solves the connection and routing problem, but physical log replication and consumer offset synchronisation remain out-of-band challenges (more on that reality later). Still, for producers emitting fresh streams or teams operating in clean, discrete domains, cluster aliasing eliminates the connection coordination nightmare entirely. +Now, let's be honest about the boundaries: migrating a producer pumping ephemeral clickstream data to a new cluster is trivial with connection switching—the platform team updates the route, and the producer begins writing to the new cluster with zero config changes and zero downtime. But what about the downstream analytics consumers that need to process that data? If they need continuous historical context, or if data was split across clusters mid-stream, the downstream consumer story is much more complicated. The proxy solves the connection and routing problem, but physical log replication and consumer offset synchronisation remain out-of-band challenges (more on that reality later). Still, for producers emitting fresh streams or teams operating in clean, discrete domains, connection switching eliminates the connection coordination nightmare entirely. ### Client-change-free cluster migrations and DR failover @@ -40,7 +40,7 @@ In a conventional setup, redirecting traffic to a secondary cluster or a freshly - Updating DNS records (with all the painful quirks as DNS propagates, seemingly at random[^1]) or editing hundreds of configuration maps. - Bouncing client fleets across an agreed maintenance window—or watching an incident drag out while waiting for client restarts. -Cluster aliasing doesn't solve the hard data replication or consumer offset reconciliation problem, but it **completely eliminates the client-side migration cost**. The bootstrap configuration remains stable (`kafka.corp.internal:9092`). +Connection switching doesn't solve the hard data replication or consumer offset reconciliation problem, but it **completely eliminates the client-side migration cost**. The bootstrap configuration remains stable (`kafka.corp.internal:9092`). When executing a planned migration or a DR failover: @@ -48,7 +48,7 @@ When executing a planned migration or a DR failover: 2. **The connection cutover is a control-plane flip:** The proxy updates its route target to the secondary cluster—either globally or tenant-by-tenant. 3. **Rollback is instant:** If the secondary cluster misbehaves under live load, you can revert the routing target in seconds rather than triggering another round of client config rollbacks. -Whether you're doing a planned blue/green transition or invoking DR in an emergency, cluster aliasing transforms a massive cross-team coordination exercise into an internal platform switch. +Whether you're doing a planned blue/green transition or invoking DR in an emergency, connection switching transforms a massive cross-team coordination exercise into an internal platform switch. ### Production shadowing without the compliance conversation @@ -138,9 +138,9 @@ All of that is a long way of saying: the proxy cannot manage PIDs globally, and **Single physical cluster.** When all routes lead to the same physical cluster, there is nothing to solve. This is just standard Kafka protocol semantics — the proxy is a transparent conduit for a PID relationship that exists entirely between the client and the cluster. No mapping, no invention, no state to lose. Two things to keep in mind: routing decisions must be deterministic at the cluster level (a given producer must always land on the same physical cluster), and if the cluster changes — whether through a failover or a routing reconfiguration — the PID means nothing to the new cluster. Not fenced — a brand new client. -**Multiple active clusters.** This is where aliasing ends. The full treatment — what can be made to work, what can't, and at what cost — is in the union clusters post. But the boundary is worth drawing clearly here before we get there: see *Layer 7 can only do so much* below. +**Multiple active clusters.** This is where connection switching ends. The full treatment — what can be made to work, what can't, and at what cost — is in the post on topic weaving. But the boundary is worth drawing clearly here before we get there: see *Layer 7 can only do so much* below. -**Duplication routing (traffic shadowing).** One pattern that stays cleanly within the aliasing boundary: the shadow write to the secondary cluster is entirely router-invented — the client does not know Cluster B exists, and nothing downstream will ever attempt to resume or share that identity. The router negotiates its own PID with the shadow cluster and owns it from start to finish. What makes this safe is precisely the client's ignorance: because the shadow identity is invisible to the client, nothing can collide with it. There is one edge case worth naming: the shadow cluster has no knowledge of the primary's deduplication history. Writes the primary would have fenced as duplicates may be accepted on the shadow side, particularly after a proxy restart, when the negotiated PID is lost and a fresh one is issued to the shadow cluster. This is a dual-write pattern — do not use it where total accuracy is required. Shadow data is for observability. There will be other safe patterns depending on your topology — the constraint is client visibility, not the number of physical clusters involved. +**Duplication routing (traffic shadowing).** One pattern that stays cleanly within the connection switching boundary: the shadow write to the secondary cluster is entirely router-invented — the client does not know Cluster B exists, and nothing downstream will ever attempt to resume or share that identity. The router negotiates its own PID with the shadow cluster and owns it from start to finish. What makes this safe is precisely the client's ignorance: because the shadow identity is invisible to the client, nothing can collide with it. There is one edge case worth naming: the shadow cluster has no knowledge of the primary's deduplication history. Writes the primary would have fenced as duplicates may be accepted on the shadow side, particularly after a proxy restart, when the negotiated PID is lost and a fresh one is issued to the shadow cluster. This is a dual-write pattern — do not use it where total accuracy is required. Shadow data is for observability. There will be other safe patterns depending on your topology — the constraint is client visibility, not the number of physical clusters involved. --- @@ -158,7 +158,7 @@ If the routing target changes while a client is active: ### Layer 7 can only do so much -Aliasing works because there is a single physical cluster backing it. The proxy isn't maintaining an illusion — it's directing traffic to something real, and real Kafka semantics apply throughout. The moment a router exposes brokers from multiple physical clusters as a unified address space, it has to ensure those semantics stay valid. For some operations that's fine — routing a `Fetch` to a local replica is cheap and safe. But for anything that touches producer identity or coordination — PIDs, epochs, transaction coordinators, consumer group coordinators — the router has taken on the obligation of upholding guarantees the protocol no longer provides on its behalf. +Connection switching works because there is a single physical cluster backing it. The proxy isn't maintaining an illusion — it's directing traffic to something real, and real Kafka semantics apply throughout. The moment a router exposes brokers from multiple physical clusters as a unified address space, it has to ensure those semantics stay valid. For some operations that's fine — routing a `Fetch` to a local replica is cheap and safe. But for anything that touches producer identity or coordination — PIDs, epochs, transaction coordinators, consumer group coordinators — the router has taken on the obligation of upholding guarantees the protocol no longer provides on its behalf. The most immediate consequence is session affinity. The router must maintain a session-scoped mapping between the client's virtual PID and the real (PID, epoch) on each backend cluster. That mapping lives in the proxy instance that negotiated it. If the load balancer routes a reconnecting client to a different proxy replica, the mapping is gone — the router re-negotiates, the epoch bumps, and the backend fences the producer. The proxy can't prevent this. Only sticky L4 routing can. @@ -178,8 +178,8 @@ The proxy routes frames; it doesn't replicate log data. If consumers need to pic --- -Cluster aliasing keeps the operational promise simple: platform teams can change the physical backend without the application teams ever knowing it happened. That's not a small thing. +Connection switching keeps the operational promise simple: platform teams can change the physical backend without the application teams ever knowing it happened. That's not a small thing. -The next post moves from one-to-one aliasing to stitching multiple physical clusters into a single logical view: **Union Clusters**. +The next post moves from one-to-one connection switching to stitching multiple physical clusters into a single logical view: **Topic Weaving**. [^1]: There is always some bit of infra that fails to honour the TTL properly or doesn't understand a CNAME or PTR record. diff --git a/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md b/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md index b9072496..cac3413a 100644 --- a/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md +++ b/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md @@ -47,7 +47,7 @@ We get it: running a proxy is operational overhead. We wish swapping a CNAME was ## Name them we must -After reviewing the [routing design proposal](https://github.com/kroxylicious/design/pull/70) and thinking through the problem space, I think there are multiple distinct deployment patterns worth naming. As with all patterns the boundaries are fuzzy and often more than one applies at once. The thing that convinces me they're real is that they have descriptive power — each one has its own tradeoffs, failure modes, and a distinct place on the spectrum from the clear(ish) waters of cluster aliasing to the mangrove swamp of virtual topics. Like all good abstractions, they're useful. +After reviewing the [routing design proposal](https://github.com/kroxylicious/design/pull/70) and thinking through the problem space, I think there are multiple distinct deployment patterns worth naming. As with all patterns the boundaries are fuzzy and often more than one applies at once. The thing that convinces me they're real is that they have descriptive power — each one has its own tradeoffs, failure modes, and a distinct place on the spectrum from the clear(ish) waters of connection switching to the mangrove swamp of partition weaving. Like all good abstractions, they're useful. You can deploy the proxy on either side of the network boundary: helping traffic egress from the client domain (forward proxy territory), or managing ingress into the broker domain (reverse proxy territory). We prefer *egress* and *ingress* because they describe traffic flows from your perspective rather than abstract networking theory. Some patterns land far more naturally on one side than the other, and we'll call that out as we go, but the underlying mechanics are the same. @@ -56,40 +56,48 @@ If you're reaching for the [Enterprise Integration Patterns](https://www.enterpr The patterns do build on each other in terms of what the proxy has to understand about the Kafka protocol — each one introduces a new class of work the router stage has to own. But the complexity doesn't compound when you compose them, because the routing pipeline is a DAG: each stage is narrowly scoped to its own job and doesn't need to know what the stages around it are doing. A topology-aware routing stage doesn't care whether the topic it's directing traffic to is a virtual topic or a plain one. A virtual topic stage doesn't care whether the cluster it's dispatching to is an alias or a physical cluster. You pay the cost of each stage once, and only if you need it. -### Cluster aliasing +### Connection switching Until recently a VKC provided a 1:1 mapping between the thing the client addressed and the thing the proxy dialled. The routing API breaks that constraint — one VKC can now connect to one or more physical backends, and the target isn't fixed. The VKC is stable; what's behind it doesn't have to be. -It's 2am on the last day of the month. The sales team has just launched an unannounced, wildly successful viral promotion. At the exact same moment, the scheduled end-of-month financial analytics job spins up and starts hammering the shared cluster with gigantic batch reads. Brokers are sweating, disk queues are spiking, and p99 produce latencies are heading for the stratosphere. +Every organisation has a cage it can't close. The CFO wants it gone, but somewhere upstream is "the Thing" nobody dares touch that still talks to the Kafka cluster inside it. The cluster is full. The cage is full. You need more capacity, and you can't get it here. -In the world before routing, rescuing that shared cluster means waking up engineers across six different departments, getting them to edit `bootstrap.servers` in approximately one million config files, flushing DNS, restarting pods, and hoping nobody typos a port number in the panic. +Before routing, that means updating `bootstrap.servers` in half a million places — and crucially, touching the Thing. -With cluster aliasing, everyone still points at the exact same VKC. The proxy inspects who's knocking on the door when the connection establishes and dispatches them to the right physical cluster behind the scenes. The viral promotion gets its own dedicated, high-IOPS cluster; the analytics job lands quietly on bulk storage; everybody else keeps humming along on the shared tier. When you need to split or isolate workloads, the platform team updates a route. Zero application deployments. Zero out-of-hours pages to change a connection string. +With connection switching, you leave a Kroxylicious instance in the cage and point it at a new cluster wherever you actually have headroom. Every client still connects to the same address. The proxy dials the new cluster behind the scenes. When you're ready, update the route. When the cage is empty, tell the CFO. Nobody had to touch the Thing. -Aliasing is more than workload dispatch, though. Consider: you're chasing an issue in the reporting pipeline that only shows up in the sixth hour of the run and nobody can figure out which record trips it up. You've been there, right? What you really want is access to the live data with a debugger. Alas, this job is stuffed full of Personally Identifiable Information — so you're out of luck. You've been there and got *that* t-shirt. What now? You could build a duplication router that shadows production traffic to a staging cluster, piping it through a redaction filter on the way so the PII becomes gibberish before it touches staging brokers. The client never knows. Post 2 gets into the detail. +Before routing, Kroxylicious was a pipe with opinions — it could inspect and rewrite the stream, but it was inherently connection-oriented. One client socket, one backend, straight through. Connection switching breaks the static part of that: the backend can now change. The routing decision is made once at connection time — the points are thrown before the train departs, and after that it's a straight run to the destination. It's still 1:1; the interesting thing is that the 1 on the right side is now chosen, not fixed. -Before routing, Kroxylicious was a pipe with opinions — it could inspect and rewrite the stream, but it was inherently connection-oriented. One client socket, one backend, straight through. Cluster aliasing breaks the static part of that: the backend can now change. But it's still 1:1. +### Stream branching -### Union clusters +You're chasing an issue in the reporting pipeline that only shows up in the sixth hour of the run and nobody can figure out which record trips it up. You've been there, right? What you really want is access to the live data with a debugger. Alas, this job is stuffed full of Personally Identifiable Information — so you're out of luck. You've been there and got *that* t-shirt. What now? You build a router that shadows production traffic to a staging cluster, piping it through a redaction filter on the way so the PII becomes gibberish before it touches staging brokers. The client is still producing to one VKC. But the proxy is now opening *two* backend connections — one to the primary cluster, one to staging — and writing every record to both. The acknowledgement the client gets back is from the primary; the shadow write happens on the side. One stream in, two streams out. If you've got your EIP book off the shelf already, yes — it's a Wire Tap with a Message Translator on the diverted stream. -Kroxylicious has always been like a cycle courier with opinions — it inspects, rewrites, and has strong views about what is appropriate to carry, but the destination was fixed when the client dropped off the parcel. Routing turns it into a sorting office with standing orders. You write *George Street* on the parcel and drop it at the counter. The sorting office decides whether you mean George Street, Edinburgh or George Street, Dunedin — a city Scottish settlers named after Edinburgh, gave the same street names, and promptly scrambled the layout. (Guess why I know.) The sender doesn't need to know both exist. The brokers behind a union cluster work the same way: they still have to exist somewhere, we're not making them up, but the client only ever sees the one address it was given. Because the proxy controls what gets returned in a metadata response, you decide which brokers are visible, under what names, and what maps where. And we don't need a distributed consensus layer to do it — the Kafka community just spent considerable effort going from two consensus systems down to one with KRaft. Nobody wants a third. +That's one flavour of stream branching — the secondary stream is a copy of the primary, diverted somewhere and the client never realises. But the diverted branch doesn't have to be a copy. Every record passes through the routing DAG before it hits the broker. Something in that DAG can inspect the payload, compute something from it — consumer lag, record counts by key, an audit trail your compliance team needs — and emit that derived data to a topic anywhere the routing config points — a different topic on the same cluster, or a cluster the client has never heard of. The primary stream is untouched. The additional streams just appear, derived from the original client's activity. The clients don't change. Which means anything that previously required every application to cooperate — the audit trail, the telemetry, the thing all but one of your clients never supported — you can now just invent that stream yourself. -You look after a gaggle of Kafka clusters. You know how they got there. You're not proud of all of them. You can't herd them — geese are worse than cats, they honk back — but you don't have to admit to anyone else how many geese there actually are. +What the routing layer has to own here is the fan-out. It has to manage the additional backend connections, generate the rewritten requests for each branch, and decide what to do if a secondary destination is slow or unavailable. Stream branching is in the category of dual writes, and dual writes have sharp edges. How sharply they cut depends entirely on your use case — the post on stream branching gets into it. -Point your application teams at one VKC and the proxy stitches the backing clusters together into a single cluster view. The client asks for metadata and gets back a broker list. It has no idea that list was assembled from three physical clusters, one of which is on-prem and nobody's quite sure what breed it is. You implement the routing logic — the proxy gives you the framework to dispatch requests to the right place. +### Topic weaving -Two things to keep in mind. Consumer group co-ordination stays pinned to physical clusters — that's almost always fine, because a group's offset state is tied to the topic-partitions it's consuming and those live on one backend, but it's worth knowing the seam is there. And transactions don't cross cluster boundaries: a transaction that writes to topics on two different backing clusters is two independent transactions whether your code believes that or not. If cross-topic atomicity matters, those topics need to share a physical cluster. +Your New York application team wants to consume `london_orders`, `tokyo_orders`, and `newyork_orders`. Those topics live on three different clusters — one in each region, provisioned by three different teams at three different points in time. Without topic weaving, the NY team needs bootstrap addresses for all three clusters, separate client configurations for all three, and the operational joy of managing credentials and failover for all three. With topic weaving, you point them at one VKC. The routing layer stitches the backing clusters together into a single logical view. The client asks for metadata and gets back a broker list that includes all brokers from all regions as part of the same cluster. As far as the client belives they all live in its local cluster. -### Virtual topics +That's the common case: different topics, different clusters, one coherent view for the client. Your router is what holds the map — which topic lives where, which broker list to return, what gets dispatched where. Because the proxy controls what gets returned in a metadata response, you decide which brokers are visible, under what names, and what maps where. + +There's a subtler case. What if two of your clusters both host a topic called `orders`? Think of it like a sorting office: you write *George Street* on the parcel and drop it at the counter. The sorting office decides whether you mean George Street, Edinburgh or George Street, Dunedin — a city in New Zealand that Scottish settlers named after Edinburgh, gave the same street names, and promptly scrambled the layout. (Guess why I know.) The sender doesn't need to know both exist. Neither does your client. Because the proxy controls what gets returned in a metadata response, your router decides which topics are visible and under what names — which means in principle it can arbitrate which names reach clients, or whether the duplicate surfaces at all. + +You look after a gaggle of Kafka clusters. You know how they got there. You're not proud of all of them. You can't herd them — geese are worse than cats, they honk back — but you don't have to admit to anyone else how many geese there actually are, or what they're called. + +Two things to keep in mind. Consumer group co-ordination stays pinned to physical clusters — that's almost always fine, because a group's offset state is tied to the topic-partitions it's consuming and those live on one backend, but it's worth knowing the seam is there. And transactions don't cross cluster boundaries: a transaction that writes to topics on two different backing clusters is two independent transactions, and there is no way to make it otherwise — the Kafka protocol has no cross-cluster transaction coordinator and routing can't invent one. If cross-topic atomicity matters, those topics need to share a physical cluster. + +### Partition weaving A single logical topic whose partitions are composed from physical topics on multiple clusters — which don't even need to share a name. A client asks for metadata for `topic-x` and gets back 32 partitions; partitions 0–15 are sourced from `topic-x` on `us-east`, 16–31 from `topic-eu` on `eu-west`. A `ProduceRequest` for partition 20 gets routed to `eu-west` transparently, with partition numbers rewritten to match the physical layout. The client sees one topic. It has no idea. -The motivating example here is one union clusters can't solve. Your reporting pipeline in `us-east` is hardcoded to read from `topic-x`. It has always read from `topic-x`. It will continue to read from `topic-x`. The problem is on the producer side: your business is growing in Europe, and sending every EU event across the Atlantic to land on the `us-east` cluster is expensive, slow, and fragile. The EU team provisions `topic-eu` on a cluster local to them, sized for their producer load. The router presents both physical topics as one logical `topic-x` to every client. EU producers write locally. The reporting pipeline reads the full partition space without a config change. Nobody crosses the Atlantic unnecessarily. +The motivating example here is one topic weaving can't solve. Your reporting pipeline in `us-east` is hardcoded to read from `topic-x`. It has always read from `topic-x`. It will continue to read from `topic-x`. The problem is on the producer side: your business is growing in Europe, and sending every EU event across the Atlantic to land on the `us-east` cluster is expensive, slow, and fragile. The EU team provisions `topic-eu` on a cluster local to them, sized for their producer load. The router presents both physical topics as one logical `topic-x` to every client. EU producers write locally. The reporting pipeline reads the full partition space without a config change. Nobody crosses the Atlantic unnecessarily. -What the proxy has to do here is more demanding than in any of the previous patterns: partition numbers must be rewritten consistently across every Kafka RPC that mentions them — `Metadata`, `Produce`, `Fetch`, `OffsetCommit`, `OffsetFetch`, `ListOffsets`, group coordinator lookups. That's the essential complexity of the pattern, not incidental overhead. It's a lot of surfaces to get right. The operational contract also shifts: both physical topics have to honour the mapping, and the router config is now the source of truth for what `topic-x` means. Transactions don't cross cluster boundaries here either — same caveat as union clusters, but worth repeating because with virtual topics it's easier to forget the seam is there. +What the proxy has to do here is more demanding than in any of the previous patterns: partition numbers must be rewritten consistently across every Kafka RPC that mentions them — `Metadata`, `Produce`, `Fetch`, `OffsetCommit`, `OffsetFetch`, `ListOffsets`, group coordinator lookups. That's the essential complexity of the pattern, not incidental overhead. It's a lot of surfaces to get right. The operational contract also shifts: both physical topics have to honour the mapping, and the router config is now the source of truth for what `topic-x` means. Transactions don't cross cluster boundaries here either — same caveat as topic weaving, but worth repeating because with partition weaving it's easier to forget the seam is there. --- -These three patterns give us a working vocabulary for the rest of the series. The posts that follow take each one in turn — what it can do, what it can't, and why. Some of the limits are ours and will close over time. Some are the Kafka protocol's: transactions have no external coordinator, and no amount of clever routing changes that. And some would require the proxy to grow a consensus layer of its own — which is a whole different animal, and not one we're planning to domesticate any time soon. +These four patterns give us a working vocabulary for the rest of the series. The posts that follow take each one in turn — what it can do, what it can't, and why. Some of the limits are ours and will close over time. Some are the Kafka protocol's: transactions have no external coordinator, and no amount of clever routing changes that. And some would require the proxy to grow a consensus layer of its own — which is a whole different animal, and not one we're planning to domesticate any time soon. If you're thinking about Kafka topology problems, have a use case that doesn't fit cleanly into any of these patterns, or want to tell us where the vocabulary breaks down — find us on [GitHub](https://github.com/kroxylicious/kroxylicious), [Slack](https://kroxylicious.slack.com), or [Bluesky](https://bsky.app/profile/kroxylicious.io). From 98b7c299fd26adf54da941eb323779f9cae5a96f Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Wed, 7 Oct 2026 16:00:35 +1300 Subject: [PATCH 14/24] Polish post 1: intro taxonomy, single-axis framing, topic weaving restructure - Introduce all four pattern names upfront in the intro paragraph - Replace DAG composability paragraph with single-axis protocol depth framing - Fix stale 'virtual topic' and 'alias' references in DAG paragraph - Restructure topic weaving: Keith's NY/london/tokyo example leads, George Street becomes the collision case - Tighten transaction boundary warning: 'no way to make it otherwise' - Replace exhaustive RPC list with 'they get everywhere, as you might expect' - Fix 'believes' typo; remove redundant broker-control sentence Signed-off-by: Sam Barker --- ...hese-are-not-the-brokers-you-are-connecting-to.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md b/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md index cac3413a..2f0d22bf 100644 --- a/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md +++ b/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md @@ -47,13 +47,13 @@ We get it: running a proxy is operational overhead. We wish swapping a CNAME was ## Name them we must -After reviewing the [routing design proposal](https://github.com/kroxylicious/design/pull/70) and thinking through the problem space, I think there are multiple distinct deployment patterns worth naming. As with all patterns the boundaries are fuzzy and often more than one applies at once. The thing that convinces me they're real is that they have descriptive power — each one has its own tradeoffs, failure modes, and a distinct place on the spectrum from the clear(ish) waters of connection switching to the mangrove swamp of partition weaving. Like all good abstractions, they're useful. +After reviewing the [routing design proposal](https://github.com/kroxylicious/design/pull/70) and thinking through the problem space, I think there are four distinct deployment patterns worth naming: **connection switching**, **stream branching**, **topic weaving**, and **partition weaving**. As with all patterns the boundaries are fuzzy and often more than one applies at once. The thing that convinces me they're real is that they have descriptive power — each one has its own tradeoffs, failure modes, and a distinct place on the spectrum from the clear(ish) waters of connection switching to the mangrove swamp of partition weaving. Like all good abstractions, they're useful. -You can deploy the proxy on either side of the network boundary: helping traffic egress from the client domain (forward proxy territory), or managing ingress into the broker domain (reverse proxy territory). We prefer *egress* and *ingress* because they describe traffic flows from your perspective rather than abstract networking theory. Some patterns land far more naturally on one side than the other, and we'll call that out as we go, but the underlying mechanics are the same. +You can deploy the proxy on either side of the network boundary: helping traffic egress from the client domain (forward proxy territory), or managing ingress into the broker domain (reverse proxy territory). I prefer *egress* and *ingress* because they describe traffic flows from your perspective rather than abstract networking theory. Some patterns land far more naturally on one side than the other, and we'll call that out as we go, but the underlying mechanics are the same. If you're reaching for the [Enterprise Integration Patterns](https://www.enterpriseintegrationpatterns.com/patterns/messaging/MessageRoutingIntro.html) book right now — well done for being as old as I am, the rest of you just Googled it. It describes what happens to individual messages, which is interesting, but I'm talking about whole streams. Those patterns are what makes all this possible under the hood; what I'm describing sits a layer or two higher. -The patterns do build on each other in terms of what the proxy has to understand about the Kafka protocol — each one introduces a new class of work the router stage has to own. But the complexity doesn't compound when you compose them, because the routing pipeline is a DAG: each stage is narrowly scoped to its own job and doesn't need to know what the stages around it are doing. A topology-aware routing stage doesn't care whether the topic it's directing traffic to is a virtual topic or a plain one. A virtual topic stage doesn't care whether the cluster it's dispatching to is an alias or a physical cluster. You pay the cost of each stage once, and only if you need it. +The patterns do build on each other along a single axis: how deep into the Kafka protocol the router has to reach. Connection switching operates at the connection level — which broker, which forms part of which cluster, is this connection actually being sent to? Stream branching works at the message level — where does a copy of this record go? Topic weaving reaches into the topic catalog — which topics are visible and where do they live? Partition weaving goes deepest — how do physical partition spaces map onto a logical one? Each step requires the router(s) to understand more of the protocol than the last. But the complexity doesn't compound as routing is managed through a DAG which allows them to be composed rather than monolithic. A router handling topic visibility doesn't care whether the partitions behind it are woven from multiple clusters. A router rewriting partition numbers doesn't care whether the topic catalog above it spans two physical clusters or twenty. ### Connection switching @@ -78,9 +78,9 @@ What the routing layer has to own here is the fan-out. It has to manage the addi ### Topic weaving -Your New York application team wants to consume `london_orders`, `tokyo_orders`, and `newyork_orders`. Those topics live on three different clusters — one in each region, provisioned by three different teams at three different points in time. Without topic weaving, the NY team needs bootstrap addresses for all three clusters, separate client configurations for all three, and the operational joy of managing credentials and failover for all three. With topic weaving, you point them at one VKC. The routing layer stitches the backing clusters together into a single logical view. The client asks for metadata and gets back a broker list that includes all brokers from all regions as part of the same cluster. As far as the client belives they all live in its local cluster. +Your New York application team wants to consume `london_orders`, `tokyo_orders`, and `newyork_orders`. Those topics live on three different clusters — one in each region, provisioned by three different teams at three different points in time. Without topic weaving, the NY team needs bootstrap addresses for all three clusters, separate client configurations for all three, and the operational joy of managing credentials and failover for all three. With topic weaving, you point them at one VKC. The routing layer stitches the backing clusters together into a single logical view. The client asks for metadata and gets back a broker list that includes all brokers from all regions as part of the same cluster. As far as the client believes, they all live in its local cluster. -That's the common case: different topics, different clusters, one coherent view for the client. Your router is what holds the map — which topic lives where, which broker list to return, what gets dispatched where. Because the proxy controls what gets returned in a metadata response, you decide which brokers are visible, under what names, and what maps where. +That's the common case: different topics, different clusters, one coherent view for the client. Your router is what holds the map — which topic lives where, which broker list to return, what gets dispatched where. There's a subtler case. What if two of your clusters both host a topic called `orders`? Think of it like a sorting office: you write *George Street* on the parcel and drop it at the counter. The sorting office decides whether you mean George Street, Edinburgh or George Street, Dunedin — a city in New Zealand that Scottish settlers named after Edinburgh, gave the same street names, and promptly scrambled the layout. (Guess why I know.) The sender doesn't need to know both exist. Neither does your client. Because the proxy controls what gets returned in a metadata response, your router decides which topics are visible and under what names — which means in principle it can arbitrate which names reach clients, or whether the duplicate surfaces at all. @@ -94,7 +94,7 @@ A single logical topic whose partitions are composed from physical topics on mul The motivating example here is one topic weaving can't solve. Your reporting pipeline in `us-east` is hardcoded to read from `topic-x`. It has always read from `topic-x`. It will continue to read from `topic-x`. The problem is on the producer side: your business is growing in Europe, and sending every EU event across the Atlantic to land on the `us-east` cluster is expensive, slow, and fragile. The EU team provisions `topic-eu` on a cluster local to them, sized for their producer load. The router presents both physical topics as one logical `topic-x` to every client. EU producers write locally. The reporting pipeline reads the full partition space without a config change. Nobody crosses the Atlantic unnecessarily. -What the proxy has to do here is more demanding than in any of the previous patterns: partition numbers must be rewritten consistently across every Kafka RPC that mentions them — `Metadata`, `Produce`, `Fetch`, `OffsetCommit`, `OffsetFetch`, `ListOffsets`, group coordinator lookups. That's the essential complexity of the pattern, not incidental overhead. It's a lot of surfaces to get right. The operational contract also shifts: both physical topics have to honour the mapping, and the router config is now the source of truth for what `topic-x` means. Transactions don't cross cluster boundaries here either — same caveat as topic weaving, but worth repeating because with partition weaving it's easier to forget the seam is there. +What the proxy has to do here is more demanding than in any of the previous patterns: partition numbers must be rewritten consistently across every Kafka RPC that mentions them — and they get everywhere, as you might expect. That's the essential complexity of the pattern, not incidental overhead. It's a lot of surfaces to get right. The operational contract also shifts: both physical topics have to honour the mapping, and your router is now the source of truth for what `topic-x` means. Transactions don't cross cluster boundaries here either — same caveat as topic weaving, but worth repeating because with partition weaving it's easier to forget the seam is there. --- From 4468ac13998424982bd27e645078324c09bc6e70 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Wed, 7 Oct 2026 21:07:11 +1300 Subject: [PATCH 15/24] Revise post 2: taxonomy alignment, remove production shadowing, tighten copy - Fix broken post_url tag in intro - Replace '2am nightmare' post 1 callback with self-contained opener - Remove production shadowing section (stream branching pattern, not connection switching) - Replace 'no stitching' with 'no weaving' to use established taxonomy - Stash PID section to stash-replica-routing.md; add concise PID notes under 'Proxies aren't magic' - New 'When is 2 != 2?' heading for node ID collision section - Various copy fixes: spinning rust, vendor-neutral replication tool list, Bugatti callback, overclaiming on cutover promise Signed-off-by: Sam Barker --- _drafts/routing-post2.md | 66 ++++++++++---------------------- _drafts/stash-replica-routing.md | 26 +++++++++++++ 2 files changed, 47 insertions(+), 45 deletions(-) diff --git a/_drafts/routing-post2.md b/_drafts/routing-post2.md index 886661f4..3bc4ddc1 100644 --- a/_drafts/routing-post2.md +++ b/_drafts/routing-post2.md @@ -9,27 +9,25 @@ categories: blog kroxylicious-proxy tags: [ "routing" ] --- -[Post 1]({link) introduced the Routing API and the fundamental shift it brings: a Virtual Kafka Cluster (VKC) is no longer a fixed pipe wired to a single physical cluster on boot. It’s a stable identity. What actually lives behind it is now the proxy's problem, not the client's. +[Post 1]({% post_url 2026-10-10-these-are-not-the-brokers-you-are-connecting-to %}) introduced the Routing API and the fundamental shift it brings: a Virtual Kafka Cluster (VKC) is no longer a fixed pipe wired to a single physical cluster on boot. It's a stable identity. What actually lives behind it is now the proxy's problem, not the client's. We called the first pattern **Connection Switching**. -Connection switching makes the smallest possible change to Kafka protocol semantics. A connection to a VKC is still a connection to exactly one physical cluster — no fan-out, no merging, no stitching. The broker on the other end behaves like any Kafka broker, because it is one. What the proxy adds is a single degree of freedom: *which* cluster. Think of it in DNS terms: we transform a hardwired A record into a CNAME. The client has a stable name; what it resolves to is now the proxy's problem, not the client's. +Connection switching makes the smallest possible change to Kafka protocol semantics. A connection to a VKC is still a connection to exactly one physical cluster — no fan-out, no merging, no weaving. The broker on the other end behaves like any Kafka broker, because it is one. What the proxy adds is a single degree of freedom: *which* cluster. Think of it in DNS terms: we transform a hardwired A record into a CNAME. The client has a stable name; what it resolves to is now the proxy's problem, not the client's. ## What you can do with it (and the dream it sets up) ### The single front door: Tenant & workload dispatch -Remember that 2am nightmare from post 1? The sales team launches an unannounced viral promotion right as end-of-month financial batch analytics fire up on the same shared cluster. +Workload collision is one of Kafka's classic growing pains. When operations eventually wants to separate workloads onto dedicated hardware — high-IOPS brokers for latency-sensitive services, bulk storage for batch analytics, a shared tier for everything else — the old world makes it a multi-week coordination exercise: provision new hardware, issue new bootstrap URLs, cajole twenty application teams into updating their config repositories, and schedule rolling restarts. -In the old world, separating those workloads meant a multi-week coordination exercise: provision new hardware, issue new bootstrap URLs, cajole twenty application teams into updating their config repositories, and schedule rolling restarts. - -With connection switching, everyone points at `kafka.corp.internal:9092`. When a client connects, the router inspects the authentication context established during handshake—who is knocking on the door?—and dispatches the connection to the appropriate physical cluster behind the scenes: +With connection switching, everyone bootstraps at `kafka.corp.internal:9092`. When a client connects, the router inspects the authentication context established during handshake—who is knocking on the door? — and dispatches the connection to the appropriate physical cluster behind the scenes: - High-throughput production services get routed to dedicated, high-IOPS NVMe brokers. -- Heavy batch analytics jobs get dispatched to high-capacity bulk storage clusters. +- Heavy batch analytics jobs get dispatched to the spinning rust cluster. - General application traffic stays on the shared tier. -Now, let's be honest about the boundaries: migrating a producer pumping ephemeral clickstream data to a new cluster is trivial with connection switching—the platform team updates the route, and the producer begins writing to the new cluster with zero config changes and zero downtime. But what about the downstream analytics consumers that need to process that data? If they need continuous historical context, or if data was split across clusters mid-stream, the downstream consumer story is much more complicated. The proxy solves the connection and routing problem, but physical log replication and consumer offset synchronisation remain out-of-band challenges (more on that reality later). Still, for producers emitting fresh streams or teams operating in clean, discrete domains, connection switching eliminates the connection coordination nightmare entirely. +Now, let's be honest about the boundaries: migrating a producer pumping ephemeral clickstream data to a new cluster is trivial with connection switching—the platform team updates the route, and the producer begins writing to the new cluster with zero config changes and zero downtime. But what about the downstream analytics consumers that need to process that data? If they need continuous historical context, or if data was split across clusters mid-stream, the downstream consumer story is much more complicated. The proxy solves the connection and routing problem, but physical log replication and consumer offset synchronisation remain out-of-band challenges (the migrations section below gets into what that actually requires). Still, for producers emitting fresh streams or teams operating in clean, discrete domains, connection switching eliminates the connection coordination nightmare entirely. ### Client-change-free cluster migrations and DR failover @@ -44,19 +42,11 @@ Connection switching doesn't solve the hard data replication or consumer offset When executing a planned migration or a DR failover: -1. **The data plane is handled out-of-band:** Topics and offsets are synced beforehand via active replication (MirrorMaker 2, KIP-1279 cluster mirroring), or recovered from storage volume snapshots. +1. **The data plane is handled out-of-band:** Topics and offsets are synced beforehand via active replication (MirrorMaker 2, KIP-1279 cluster mirroring, or your vendor's equivalent), or recovered from storage volume snapshots. 2. **The connection cutover is a control-plane flip:** The proxy updates its route target to the secondary cluster—either globally or tenant-by-tenant. 3. **Rollback is instant:** If the secondary cluster misbehaves under live load, you can revert the routing target in seconds rather than triggering another round of client config rollbacks. -Whether you're doing a planned blue/green transition or invoking DR in an emergency, connection switching transforms a massive cross-team coordination exercise into an internal platform switch. - -### Production shadowing without the compliance conversation - -You're chasing an elusive bug that only manifests in the sixth hour of a high-volume run. Nobody can figure out which specific record payload triggers it. What you really want is to attach a debugger to the live stream in staging—except this production topic is stuffed full of Personally Identifiable Information (PII), and your compliance team has very reasonable objections. - -Because routing stages compose cleanly into a pipeline with standard filters, a duplicating router can shadow live traffic to a staging cluster alias while passing it through a redaction filter on the way. The PII is transformed into gibberish before the bytes ever touch staging disks. - -The client never knows. You get the real stream volume, the real cadence, the real record shapes—just not the real customer names. +Whether you're doing a planned blue/green transition or invoking DR in an emergency, connection switching takes client coordination off the critical path. The data still has to be in sync — that's on you — but at least the cutover itself isn't a multi-team event. ### The cloud bill from hell: Replica routing @@ -75,7 +65,7 @@ Kafka's native rack awareness is great, but client compliance is leaky. Centrali And this isn't just about inspecting frames at Layer 7. If all you have is a filter pipeline on a static 1:1 pipe, understanding that a `Fetch` belongs in `eu-west-1a` doesn't help you—the pipe is already nailed to broker 1 in `eu-west-1c`. The breakthrough comes from the Routing API making the backend connection dynamic. -When a proxy instance deployed in `eu-west-1a` intercepts a `Fetch` request arriving on a virtual connection to the leader over in `eu-west-1c`, it inspects the partition's in-sync replica (ISR) set. If broker 3 is sitting right next door in `eu-west-1a` with a caught-up replica, the proxy doesn't rewrite the frame—it dynamically dispatches that individual request down a separate backend pipe to broker 3: +When a `Fetch` request arrives at a Kroxylicious instance in `eu-west-1a` — on the pipe for the leader over in `eu-west-1c` — the router inspects the partition's in-sync replica (ISR) set. If broker 3 is sitting right next door in `eu-west-1a` with a caught-up replica, the router doesn't rewrite the frame—it dynamically dispatches that individual request down a separate backend pipe to broker 3: ``` [ Consumer (AZ-a) ] ──Fetch──► [ Kroxylicious (AZ-a) ] ────► [ Broker 3 (AZ-a, Replica) ] @@ -87,12 +77,12 @@ When a proxy instance deployed in `eu-west-1a` intercepts a `Fetch` request arri The application didn't configure `client.rack`. The request payload is completely untouched. It simply travelled down a different pipe. Broker 1 never sees the read, and the cross-AZ request never happened so never reaches the invoice. -Rack-local fetching is just leveraging the existing Kafka protocol. In theory, dynamic per-request dispatch opens up other intriguing possibilities to explore in your custom Bugatti router: +Rack-local fetching is just leveraging the existing Kafka protocol. In theory, dynamic per-request dispatch opens up other intriguing possibilities to explore in your custom Bugatti router from post 1: - **Replica lag awareness:** Steering reads strictly to the most caught-up followers to avoid trailing-edge staleness. - **Broker load shedding:** Diverting fetch traffic away from CPU- or IO-bound leaders to give them headroom for critical writes. - **Tiered storage offloading:** Routing historical backfills to brokers backed by cheap object storage while keeping hot tail-reads on fast NVMe. -Whether those speculative policies prove practical in production remains to be seen—they require telemetry the proxy might not easily have—but proximity routing alone solves a very real, very expensive problem right now. +Whether those speculative policies prove practical in production remains to be seen—they require telemetry the router might not easily have—but proximity routing alone solves a very real, very expensive problem right now. There is, of course, a hard constraint: **proxy placement**. If a client in `eu-west-1a` talks to a proxy sitting in `eu-west-1b`, which then forwards to a broker in `eu-west-1a`, you’ve just invented a two-hop cross-AZ detour. For replica routing to pay off, your proxy fleet must be co-located with the clients it fronts. @@ -102,9 +92,11 @@ There is, of course, a hard constraint: **proxy placement**. If a client in `eu- Making one physical cluster look like a different one at Layer 7 requires more than TCP forwarding. There are three places where the proxy has to do actual work. -### Broker IDs are integers, not names +### When is 2 != 2? -Kafka wire frames identify brokers by integer `NodeId` — not hostnames. A `MetadataResponse` carries a broker list (node ID, host, port, rack) and maps each partition to its leader and replicas by node ID. The client resolves those node IDs to connections itself. That's fine when there's one physical cluster. When you have multiple physical clusters behind a VKC, you immediately run into a collision problem: broker 2 on cluster A and broker 2 on cluster B are completely different machines, but the proxy can only send one broker list to the client — it cannot have two different entries for node ID 2. +When broker 2 on cluster A and broker 2 on cluster B are both behind the same VKC. + +Kafka identifies brokers by an integer called `NodeId` — not hostnames. A `MetadataResponse` carries a broker list (node ID, host, port, rack) and maps each partition to its leader and replicas by node ID. The client resolves those node IDs to connections itself. That's fine when there's one physical cluster. When you have multiple physical clusters behind a VKC, you immediately run into a collision problem: broker 2 on cluster A and broker 2 on cluster B are completely different machines, but the proxy can only send one broker list to the client — it cannot have two different entries for node ID 2. The proxy solves this with a deterministic, stateless mapping applied at every node in the routing DAG. Each router speaks two consistent ID languages: a downstream ID space shared with whatever is below it, and an upstream ID space shared with whatever is above it. Requests flow upstream with IDs translated from downstream to upstream; responses flow downstream with IDs translated from upstream to downstream. The router reads and writes both spaces fluently in both directions. A node can have multiple upstream ID spaces — one per route — all of which must be collapsed into a single, unambiguous downstream space. The mapping is strictly one-to-one in both directions — a bijection, if you want the precise term — so every downstream ID resolves to exactly one upstream ID and route, with no collisions and no ambiguity. No database lookup, no distributed state, just arithmetic: the downstream node ID is the upstream node ID multiplied by the number of routes, plus the route index. For a leaf router, the upstream ID space happens to be the physical IDs coming off the wire from a real broker — but the mapping rule is the same regardless. In theory the upstream could itself be another Kroxylicious instance — the mapping rule is indifferent to what sits above the leaf. The only realistic reason you'd pay that latency tax is Conway's Law: separate teams owning separate proxy tiers with their own trust boundaries and operational responsibilities. The proxy runtime handles the translation at each hop transparently; routers work with opaque `VirtualNode` handles scoped to their own ID space and never need to care about the underlying integer arithmetic. @@ -114,34 +106,16 @@ The proxy solves this with a deterministic, stateless mapping applied at every n ◄──(cluster B IDs)──► [ broker1.cluster-b, broker2.cluster-b, ... ] ``` +A router could in principle choose which brokers it exposes to the client — but the moment you start hiding brokers, you own the consequences for partition leadership, replica placement, and fetch routing. That's a deep rabbit hole, and the current API doesn't give you a way in: the bijection and topology discovery are handled by the runtime, not the router. + ### Topology discovery happens in-band The proxy has no independent view of the Kafka topology. It doesn't poll brokers in the background, run health checks, or hold idle connections open just to keep its metadata warm. Clients and routers drive connections; the proxy learns what it needs to know from the traffic that flows through it. -Topology state is cached per-router and shared across all connections using that router — so a second connection to the same virtual cluster benefits from what the first one already discovered. The cache uses additive semantics: entries are added as they are learned, never removed. That matters for safety. A low-privilege connection might get a filtered `MetadataResponse` covering only the topics it can see; an additive cache means it can only *contribute* knowledge, never silently evict entries that a higher-privilege connection established. And cache poisoning doesn't translate into an access control bypass anyway — the cache only influences routing decisions. Brokers enforce ACLs on every request regardless; a misdirected frame gets an error, not someone else's data. +Topology state is cached per-router and shared across all connections using that router — so a second connection to the same virtual cluster benefits from what the first one already discovered. The cache uses additive semantics: entries are added as they are learned, never removed. That matters for safety. A low-privilege connection might get a filtered `MetadataResponse` covering only the topics it can see; an additive cache means it can only *contribute* knowledge, never silently evict entries that a higher-privilege connection established. And cache poisoning doesn't translate into an access control bypass either — the cache only influences routing decisions. Brokers enforce ACLs on every request regardless; a misdirected frame gets an error, not someone else's data. When a client sends a request to a route whose topology the proxy doesn't know yet — on first connection, or after a stale entry — it discovers it on demand. The proxy synthesises a `MetadataRequest` on the active client channel, uses the response to populate its topology cache, and then forwards the original request as normal. Crucially, discovery inherits the client's authenticated context, so there's no privileged out-of-band management channel to secure or misconfigure. -### Idempotent producers and the PID problem - -Kafka producers use idempotent delivery by default. The broker deduplicates retried writes so that a message is committed exactly once even if the network drops the acknowledgement and the producer retries. The mechanism relies on a Producer ID (`PID`) that the broker assigns on initialisation, combined with a per-partition sequence number on every batch. If the broker sees a batch it's already committed — same PID, same sequence — it silently discards the duplicate. The client never needs to know. - -When you first look at PID management through the proxy, it looks like a problem we already solved. The previous section introduced a bijection for node IDs — a deterministic, stateless mapping between downstream IDs and upstream IDs at each router. PID translation looks like the same pattern, and the surface similarities are convincing: both are integers, both are cluster-scoped, both are assigned by the cluster rather than chosen by the client, and both are opaque values the client simply carries and passes through. So the proxy can just maintain a downstream-to-upstream mapping for both, right? This is the obvious answer. The seams show up quickly, but each one looks stitchable — until you're looking at a patchwork quilt and wondering why it keeps unravelling. The reason is that despite identical protocol syntax, node IDs and producer IDs have completely different semantics within the protocol. A node ID is **spatial** — a label, a coordinate. A PID is **temporal** — not just a label but a claim of active identity. In other words: we can mathematically fake a Node ID because it's just an address. We have to statefully track a Producer ID because it's a weaponised identity claim. - -**Node ID translation is transport-scoped.** A `MetadataResponse` tells you where brokers are. The proxy can tell as many lies as it likes here — the broker never sees them, let alone tries to call them out. And the proxy can be forgetful — stateless, even: if it discards its mapping and rebuilds it from the next `MetadataResponse`, it just tells the same lies again. Nothing notices, nothing breaks. - -**PID allocation is negotiation-scoped.** Since Kafka 3.0, producers run in idempotent mode by default — you don't have to opt in, it just happens. To support that, the producer calls `InitProducerIdRequest` to ask the broker to allocate it an identity: a (PID, epoch) pair. The broker tracks that identity and uses it to deduplicate retried batches within that session. It will also forcibly evict any connection that presents the same PID with a lower epoch — that is the fencing mechanism, preventing two connections from simultaneously claiming the same identity. If you need that identity to survive across sessions — so a restarted producer can pick up where it left off rather than starting fresh — that is what `transactional.id` is for. A second `InitProducerIdRequest` carrying the same `transactional.id` is treated as a hostile takeover, deliberately: fencing is how Kafka ensures only one producer instance is ever active for a given transactional identity. - -Both node IDs and PIDs are cluster-scoped — issued by a specific cluster, honoured within it, meaningless outside it. But the node ID relationship is stateless: if the proxy loses its mapping, it recovers via discovery and rederivation. A PID is unrecoverable. What was negotiated is gone. - -All of that is a long way of saying: the proxy cannot manage PIDs globally, and the Routing API makes no attempt to. There is no built-in abstraction that handles this for you — it is left to the team implementing the router to understand what their routing topology implies for producer identity. That is not as bleak as it sounds. There are shapes that work cleanly. If your problem fits one of them, go hard or go home. - -**Single physical cluster.** When all routes lead to the same physical cluster, there is nothing to solve. This is just standard Kafka protocol semantics — the proxy is a transparent conduit for a PID relationship that exists entirely between the client and the cluster. No mapping, no invention, no state to lose. Two things to keep in mind: routing decisions must be deterministic at the cluster level (a given producer must always land on the same physical cluster), and if the cluster changes — whether through a failover or a routing reconfiguration — the PID means nothing to the new cluster. Not fenced — a brand new client. - -**Multiple active clusters.** This is where connection switching ends. The full treatment — what can be made to work, what can't, and at what cost — is in the post on topic weaving. But the boundary is worth drawing clearly here before we get there: see *Layer 7 can only do so much* below. - -**Duplication routing (traffic shadowing).** One pattern that stays cleanly within the connection switching boundary: the shadow write to the secondary cluster is entirely router-invented — the client does not know Cluster B exists, and nothing downstream will ever attempt to resume or share that identity. The router negotiates its own PID with the shadow cluster and owns it from start to finish. What makes this safe is precisely the client's ignorance: because the shadow identity is invisible to the client, nothing can collide with it. There is one edge case worth naming: the shadow cluster has no knowledge of the primary's deduplication history. Writes the primary would have fenced as duplicates may be accepted on the shadow side, particularly after a proxy restart, when the negotiated PID is lost and a fresh one is issued to the shadow cluster. This is a dual-write pattern — do not use it where total accuracy is required. Shadow data is for observability. There will be other safe patterns depending on your topology — the constraint is client visibility, not the number of physical clusters involved. - --- ## Proxies aren't magic @@ -152,10 +126,12 @@ Layer 7 protocol inspection gives you a lot of leverage. What it doesn't give yo If the routing target changes while a client is active: -- **Transactional producers:** Kafka treats the Transaction Coordinator as an internal role within the cluster. Transactions are therefore scoped to a single cluster by definition — there is no extension point or API through which an external coordinator could be plugged in. A transaction that starts on cluster A cannot be committed on cluster B; if a routing cutover happens mid-transaction, the subsequent `EndTxnRequest` arrives at the new target and gets back `INVALID_TXN_STATE` or `UNKNOWN_PRODUCER_ID`. That abort needs to be handled cleanly and the transactional loop restarted — the same failure mode as a plain broker restart, which well-written transactional producers already handle. A proxy restart itself is safe: the proxy never invents a PID in the single-cluster case, so the KIP-360 fencing fields the client sends on reconnect are the broker's own real values. +- **Transactional producers:** Kafka treats the Transaction Coordinator as an internal role within the cluster. Transactions are therefore scoped to a single cluster by definition — there is no extension point or API through which an external coordinator could be plugged in. A transaction that starts on cluster A cannot be committed on cluster B; if a routing cutover happens mid-transaction, the subsequent `EndTxnRequest` arrives at the new target and gets back `INVALID_TXN_STATE` or `UNKNOWN_PRODUCER_ID`. That abort needs to be handled cleanly and the transactional loop restarted — the same failure mode as a plain broker restart, which well-written transactional producers already handle. A proxy restart itself is safe: the proxy never invents a PID in the single-cluster case, so the KIP-360 fencing fields the client sends on reconnect are the broker's own real values. But if the cluster itself changes — through a failover or routing reconfiguration — the PID the client holds is meaningless to the new cluster. Not fenced; simply unknown. The producer reconnects as a stranger. - **Consumer groups:** Offset state lives in `__consumer_offsets` on the physical cluster. The secondary cluster's Group Coordinator has no knowledge of what the primary's consumers have committed. Failing over without a replication pipeline that includes offsets means consumers either re-read data they've already processed, or skip ahead and lose it. Neither is silent. +- **Idempotent producers:** Producer IDs are issued by a specific cluster and are meaningless to any other. Any time a route targets a new cluster — planned migration, DR failover, or routing reconfiguration — every active producer starts fresh. The new cluster has no deduplication history; retried batches that the old cluster would have silently discarded may be accepted and committed. For producers that care about exactly-once semantics, this is a consideration to plan for, not a surprise to discover in production. + ### Layer 7 can only do so much Connection switching works because there is a single physical cluster backing it. The proxy isn't maintaining an illusion — it's directing traffic to something real, and real Kafka semantics apply throughout. The moment a router exposes brokers from multiple physical clusters as a unified address space, it has to ensure those semantics stay valid. For some operations that's fine — routing a `Fetch` to a local replica is cheap and safe. But for anything that touches producer identity or coordination — PIDs, epochs, transaction coordinators, consumer group coordinators — the router has taken on the obligation of upholding guarantees the protocol no longer provides on its behalf. diff --git a/_drafts/stash-replica-routing.md b/_drafts/stash-replica-routing.md index a65fa585..cc14fd2c 100644 --- a/_drafts/stash-replica-routing.md +++ b/_drafts/stash-replica-routing.md @@ -42,3 +42,29 @@ None of these are proximity, which is why the name shouldn't bake in "locality". ## Failure mode Proxy placement. A client in `us-east-1a` hitting a proxy in `us-east-1b` that then routes to a broker in `us-east-1a` adds a cross-AZ hop instead of eliminating one. The proxy fleet has to be co-located with clients for locality savings to materialise. + +--- + +# Stash: PID section from post 2 + +Removed from `routing-post2.md` — the full treatment belongs in the stream branching or topic weaving post where PIDs across multiple clusters actually bite. Connection switching stays in the single-cluster safe zone so the detail isn't earned here. + +### Idempotent producers and the PID problem + +Kafka producers use idempotent delivery by default. The broker deduplicates retried writes so that a message is committed exactly once even if the network drops the acknowledgement and the producer retries. The mechanism relies on a Producer ID (`PID`) that the broker assigns on initialisation, combined with a per-partition sequence number on every batch. If the broker sees a batch it's already committed — same PID, same sequence — it silently discards the duplicate. The client never needs to know. + +When you first look at PID management through the proxy, it looks like a problem we already solved. The previous section introduced a bijection for node IDs — a deterministic, stateless mapping between downstream IDs and upstream IDs at each router. PID translation looks like the same pattern, and the surface similarities are convincing: both are integers, both are cluster-scoped, both are assigned by the cluster rather than chosen by the client, and both are opaque values the client simply carries and passes through. So the proxy can just maintain a downstream-to-upstream mapping for both, right? This is the obvious answer. The seams show up quickly, but each one looks stitchable — until you're looking at a patchwork quilt and wondering why it keeps unravelling. The reason is that despite identical protocol syntax, node IDs and producer IDs have completely different semantics within the protocol. A node ID is **spatial** — a label, a coordinate. A PID is **temporal** — not just a label but a claim of active identity. In other words: we can mathematically fake a Node ID because it's just an address. We have to statefully track a Producer ID because it's a weaponised identity claim. + +**Node ID translation is transport-scoped.** A `MetadataResponse` tells you where brokers are. The proxy can tell as many lies as it likes here — the broker never sees them, let alone tries to call them out. And the proxy can be forgetful — stateless, even: if it discards its mapping and rebuilds it from the next `MetadataResponse`, it just tells the same lies again. Nothing notices, nothing breaks. + +**PID allocation is negotiation-scoped.** Since Kafka 3.0, producers run in idempotent mode by default — you don't have to opt in, it just happens. To support that, the producer calls `InitProducerIdRequest` to ask the broker to allocate it an identity: a (PID, epoch) pair. The broker tracks that identity and uses it to deduplicate retried batches within that session. It will also forcibly evict any connection that presents the same PID with a lower epoch — that is the fencing mechanism, preventing two connections from simultaneously claiming the same identity. If you need that identity to survive across sessions — so a restarted producer can pick up where it left off rather than starting fresh — that is what `transactional.id` is for. A second `InitProducerIdRequest` carrying the same `transactional.id` is treated as a hostile takeover, deliberately: fencing is how Kafka ensures only one producer instance is ever active for a given transactional identity. + +Both node IDs and PIDs are cluster-scoped — issued by a specific cluster, honoured within it, meaningless outside it. But the node ID relationship is stateless: if the proxy loses its mapping, it recovers via discovery and rederivation. A PID is unrecoverable. What was negotiated is gone. + +All of that is a long way of saying: the proxy cannot manage PIDs globally, and the Routing API makes no attempt to. There is no built-in abstraction that handles this for you — it is left to the team implementing the router to understand what their routing topology implies for producer identity. That is not as bleak as it sounds. There are shapes that work cleanly. If your problem fits one of them, go hard or go home. + +**Single physical cluster.** When all routes lead to the same physical cluster, there is nothing to solve. This is just standard Kafka protocol semantics — the proxy is a transparent conduit for a PID relationship that exists entirely between the client and the cluster. No mapping, no invention, no state to lose. Two things to keep in mind: routing decisions must be deterministic at the cluster level (a given producer must always land on the same physical cluster), and if the cluster changes — whether through a failover or a routing reconfiguration — the PID means nothing to the new cluster. Not fenced — a brand new client. + +**Multiple active clusters.** This is where connection switching ends. The full treatment — what can be made to work, what can't, and at what cost — is in the post on topic weaving. But the boundary is worth drawing clearly here before we get there: see *Layer 7 can only do so much* below. + +**Duplication routing (traffic shadowing).** One pattern that stays cleanly within the connection switching boundary: the shadow write to the secondary cluster is entirely router-invented — the client does not know Cluster B exists, and nothing downstream will ever attempt to resume or share that identity. The router negotiates its own PID with the shadow cluster and owns it from start to finish. What makes this safe is precisely the client's ignorance: because the shadow identity is invisible to the client, nothing can collide with it. There is one edge case worth naming: the shadow cluster has no knowledge of the primary's deduplication history. Writes the primary would have fenced as duplicates may be accepted on the shadow side, particularly after a proxy restart, when the negotiated PID is lost and a fresh one is issued to the shadow cluster. This is a dual-write pattern — do not use it where total accuracy is required. Shadow data is for observability. There will be other safe patterns depending on your topology — the constraint is client visibility, not the number of physical clusters involved. From e3d5ed1c6e039eef67da73f0f3e87f589cb4e2b6 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Thu, 8 Oct 2026 13:57:31 +1300 Subject: [PATCH 16/24] Address Keith's review comments on post 1 Signed-off-by: Sam Barker --- ...e-not-the-brokers-you-are-connecting-to.md | 12 ++++---- proposals/289-review-comments.json | 30 +++++++++++++++++++ 2 files changed, 37 insertions(+), 5 deletions(-) create mode 100644 proposals/289-review-comments.json diff --git a/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md b/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md index 2f0d22bf..50e958f4 100644 --- a/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md +++ b/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md @@ -33,7 +33,7 @@ So when you need to reshape the underlying infrastructure without breaking appli Jumping up to Layer 7 changes the game. Rather than blindly punting TCP bytes around, the proxy speaks Kafka. It knows a `Metadata` response from a `Produce` request, and it can answer a client's question about where the brokers live with whatever answer you need. -From day one, Kroxylicious gave you a Virtual Kafka Cluster (VKC) — a stable endpoint for your clients. But until recently, it was an opinionated pipe. That was a compliment: it did real work. It could encrypt records, enforce auth, rewrite headers — all without the client noticing. But it was still a fixed pipe: one client socket in, one backend socket out, wired to whatever cluster you pointed it at on boot. The filters were stateless with respect to the broker topology: an encryption filter might talk out-of-band to a KMS for keys, but it didn't care which physical cluster lived behind the socket. +From day one, Kroxylicious gave you a Virtual Kafka Cluster (VKC) — a stable endpoint for your clients. But until recently, it was an opinionated pipe. That was a compliment: it did real work. It could encrypt records, enforce auth, rewrite headers — all without the client noticing. But it was still a fixed pipe: one client socket in, one backend socket out, wired to a broker on whatever cluster you pointed it at on boot. The filters were stateless with respect to the broker topology: an encryption filter might talk out-of-band to a KMS for keys, but it didn't care which physical cluster lived behind the socket. Think of it like the horse in the late 19th century. A horse gets you from A to B across familiar ground without needing a national highway project. You feed it, you mount it, and you go. Horses were great. @@ -53,7 +53,7 @@ You can deploy the proxy on either side of the network boundary: helping traffic If you're reaching for the [Enterprise Integration Patterns](https://www.enterpriseintegrationpatterns.com/patterns/messaging/MessageRoutingIntro.html) book right now — well done for being as old as I am, the rest of you just Googled it. It describes what happens to individual messages, which is interesting, but I'm talking about whole streams. Those patterns are what makes all this possible under the hood; what I'm describing sits a layer or two higher. -The patterns do build on each other along a single axis: how deep into the Kafka protocol the router has to reach. Connection switching operates at the connection level — which broker, which forms part of which cluster, is this connection actually being sent to? Stream branching works at the message level — where does a copy of this record go? Topic weaving reaches into the topic catalog — which topics are visible and where do they live? Partition weaving goes deepest — how do physical partition spaces map onto a logical one? Each step requires the router(s) to understand more of the protocol than the last. But the complexity doesn't compound as routing is managed through a DAG which allows them to be composed rather than monolithic. A router handling topic visibility doesn't care whether the partitions behind it are woven from multiple clusters. A router rewriting partition numbers doesn't care whether the topic catalog above it spans two physical clusters or twenty. +The patterns do build on each other along a single axis: how deep into the Kafka protocol the router has to reach. Connection switching operates at the connection level — which broker, which forms part of which cluster, is this connection actually being sent to? Stream branching works at the stream level — where does this request go? Topic weaving reaches into the topic catalog — which topics are visible and where do they live? Partition weaving goes deepest — how do physical partition spaces map onto a logical one? Each step requires the router(s) to understand more of the protocol than the last. But the complexity doesn't compound as routing is managed through a DAG which allows them to be composed rather than monolithic. A router handling topic visibility doesn't care whether the partitions behind it are woven from multiple clusters. A router rewriting partition numbers doesn't care whether the topic catalog above it spans two physical clusters or twenty. ### Connection switching @@ -72,9 +72,11 @@ Before routing, Kroxylicious was a pipe with opinions — it could inspect and r You're chasing an issue in the reporting pipeline that only shows up in the sixth hour of the run and nobody can figure out which record trips it up. You've been there, right? What you really want is access to the live data with a debugger. Alas, this job is stuffed full of Personally Identifiable Information — so you're out of luck. You've been there and got *that* t-shirt. What now? You build a router that shadows production traffic to a staging cluster, piping it through a redaction filter on the way so the PII becomes gibberish before it touches staging brokers. The client is still producing to one VKC. But the proxy is now opening *two* backend connections — one to the primary cluster, one to staging — and writing every record to both. The acknowledgement the client gets back is from the primary; the shadow write happens on the side. One stream in, two streams out. If you've got your EIP book off the shelf already, yes — it's a Wire Tap with a Message Translator on the diverted stream. -That's one flavour of stream branching — the secondary stream is a copy of the primary, diverted somewhere and the client never realises. But the diverted branch doesn't have to be a copy. Every record passes through the routing DAG before it hits the broker. Something in that DAG can inspect the payload, compute something from it — consumer lag, record counts by key, an audit trail your compliance team needs — and emit that derived data to a topic anywhere the routing config points — a different topic on the same cluster, or a cluster the client has never heard of. The primary stream is untouched. The additional streams just appear, derived from the original client's activity. The clients don't change. Which means anything that previously required every application to cooperate — the audit trail, the telemetry, the thing all but one of your clients never supported — you can now just invent that stream yourself. +That's one flavour of stream branching — the secondary stream is a copy of the primary, diverted somewhere and the client never realises. But the diverted branch doesn't have to be a copy. Every record passes through the routing DAG before it hits the broker. Something in that DAG can inspect the payload, compute something from it — consumer lag, record counts by key, an audit trail your compliance team needs — and emit that derived data to a topic anywhere the routing config points — a different topic on the same cluster, or a cluster the client has never heard of. The primary stream is untouched. The additional streams just appear, derived from the original client's activity. The clients don't change. Which means anything that previously required every application to cooperate — the audit trail, the telemetry, the thing all but one of your clients never supported — you can now just invent that stream yourself. This all works for stateless transformations performed inline. If you need state, joins, or windowing, Kafka Streams or Kafka Connect have your back. -What the routing layer has to own here is the fan-out. It has to manage the additional backend connections, generate the rewritten requests for each branch, and decide what to do if a secondary destination is slow or unavailable. Stream branching is in the category of dual writes, and dual writes have sharp edges. How sharply they cut depends entirely on your use case — the post on stream branching gets into it. +I think of stream branching more like a railway network. The client talking to the broker is the mainline. In Britain the mainline has two tracks: the town line and the country line. Requests are like trains running on the town line. Responses are on the country line. The routing DAG is the station network. At each station the stationmaster is responsible for dispatching the train on down the main line. However, as we are working with bytes and not hundreds of tonnes of steel, we can assemble a new train as it passes through the station. That stationmaster is free to assemble his new train however he wants, moving passengers (better known as records) between carriages (aka batches), and dispatching it down whichever branch line he wants. He's free to do this as it has no effect on the express trains on the mainline. + +Stream branching is a dual-write pattern, and dual writes have sharp edges. How sharply they cut depends on your use case — the post on stream branching gets into it. ### Topic weaving @@ -92,7 +94,7 @@ Two things to keep in mind. Consumer group co-ordination stays pinned to physica A single logical topic whose partitions are composed from physical topics on multiple clusters — which don't even need to share a name. A client asks for metadata for `topic-x` and gets back 32 partitions; partitions 0–15 are sourced from `topic-x` on `us-east`, 16–31 from `topic-eu` on `eu-west`. A `ProduceRequest` for partition 20 gets routed to `eu-west` transparently, with partition numbers rewritten to match the physical layout. The client sees one topic. It has no idea. -The motivating example here is one topic weaving can't solve. Your reporting pipeline in `us-east` is hardcoded to read from `topic-x`. It has always read from `topic-x`. It will continue to read from `topic-x`. The problem is on the producer side: your business is growing in Europe, and sending every EU event across the Atlantic to land on the `us-east` cluster is expensive, slow, and fragile. The EU team provisions `topic-eu` on a cluster local to them, sized for their producer load. The router presents both physical topics as one logical `topic-x` to every client. EU producers write locally. The reporting pipeline reads the full partition space without a config change. Nobody crosses the Atlantic unnecessarily. +The motivating example here is one topic weaving can't solve. Your reporting pipeline in `us-east` is hardcoded to read from `topic-x`. It has always read from `topic-x`. It will continue to read from `topic-x`. The problem is on the producer side: your business is growing in Europe, and sending every EU event across the Atlantic to land on the `us-east` cluster is expensive, slow, and fragile. The EU team provisions `topic-eu` on a cluster local to them, sized for their producer load. EU producers write to `topic-eu` on that cluster, proxies optional. The router presents both physical topics as one logical `topic-x` to the reporting pipeline. It reads the full partition space without a config change. Nobody crosses the Atlantic unnecessarily. What the proxy has to do here is more demanding than in any of the previous patterns: partition numbers must be rewritten consistently across every Kafka RPC that mentions them — and they get everywhere, as you might expect. That's the essential complexity of the pattern, not incidental overhead. It's a lot of surfaces to get right. The operational contract also shifts: both physical topics have to honour the mapping, and your router is now the source of truth for what `topic-x` means. Transactions don't cross cluster boundaries here either — same caveat as topic weaving, but worth repeating because with partition weaving it's easier to forget the seam is there. diff --git a/proposals/289-review-comments.json b/proposals/289-review-comments.json new file mode 100644 index 00000000..69aef675 --- /dev/null +++ b/proposals/289-review-comments.json @@ -0,0 +1,30 @@ +[ + { + "type": "review_thread_reply", + "thread": "T5", + "replyToId": 4210755206, + "body": "Good point — you're right that we don't want to focus on record-level routing here. The design space does cover it (a router reconstituting a branch request can go as fine-grained as it likes), but that's not where current implementations sit. I've tweaked the language in the axis paragraph to avoid the ambiguity, and the stream branching section now covers the routing granularity point in more detail.", + "status": "pending" + }, + { + "type": "review_thread_reply", + "thread": "T6", + "replyToId": 4211918741, + "body": "Horses for courses — Kafka Streams is absolutely the right answer for some of this. Where the proxy approach has the edge: no duplicate storage cost (you're deriving data in flight rather than materialising a new topic), and no framework overhead for what are often stateless transforms. Kafka Streams and Connect are powerful but they're not lightweight — if all you need is \"emit a record count to a metrics topic as this batch passes through\", standing up a Streams application is a lot of machinery for a stateless operation. I've added a sentence to the section to make this clearer.", + "status": "pending" + }, + { + "type": "review_thread_reply", + "thread": "T9", + "replyToId": 4212088090, + "body": "Oops that wasn't quite the example I was trying to make. I've tweaked the text to make it clearer the woven topic is only on the consumer side.", + "status": "pending" + }, + { + "type": "review_thread_reply", + "thread": "T4", + "replyToId": 4210723236, + "body": "Glad the others landed. *Stream branching* was the best I could come up with — open to better suggestions. *Branching streams* is another candidate; the difference is whether you're naming the act (branching a stream) or the result (streams that branch). I went with the former but I'm not attached to it.", + "status": "pending" + } +] From be3f9522c056d8175bd94890f484982686f2a353 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Fri, 9 Oct 2026 11:43:56 +1300 Subject: [PATCH 17/24] Refine partition weaving section and clarify consumer group coordination - Introduce partition weaving as extending weaving to Kafka's unit of parallelism - Switch illustrative example from topic-x to orders with third-party risk engine - Clarify consumer group coordinator mechanics and offset tracking across clusters Signed-off-by: Sam Barker --- ...se-are-not-the-brokers-you-are-connecting-to.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md b/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md index 50e958f4..9bd25ba3 100644 --- a/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md +++ b/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md @@ -92,11 +92,19 @@ Two things to keep in mind. Consumer group co-ordination stays pinned to physica ### Partition weaving -A single logical topic whose partitions are composed from physical topics on multiple clusters — which don't even need to share a name. A client asks for metadata for `topic-x` and gets back 32 partitions; partitions 0–15 are sourced from `topic-x` on `us-east`, 16–31 from `topic-eu` on `eu-west`. A `ProduceRequest` for partition 20 gets routed to `eu-west` transparently, with partition numbers rewritten to match the physical layout. The client sees one topic. It has no idea. +Topic weaving weaves (obviously 😉) whole topics together to present a unified catalog. Which invites the natural question: why stop at the topic catalog? Kafka's real unit of parallelism and storage is the partition. Can we weave at that layer instead? -The motivating example here is one topic weaving can't solve. Your reporting pipeline in `us-east` is hardcoded to read from `topic-x`. It has always read from `topic-x`. It will continue to read from `topic-x`. The problem is on the producer side: your business is growing in Europe, and sending every EU event across the Atlantic to land on the `us-east` cluster is expensive, slow, and fragile. The EU team provisions `topic-eu` on a cluster local to them, sized for their producer load. EU producers write to `topic-eu` on that cluster, proxies optional. The router presents both physical topics as one logical `topic-x` to the reporting pipeline. It reads the full partition space without a config change. Nobody crosses the Atlantic unnecessarily. +Yes — but now we have to be *really* careful with the protocol. -What the proxy has to do here is more demanding than in any of the previous patterns: partition numbers must be rewritten consistently across every Kafka RPC that mentions them — and they get everywhere, as you might expect. That's the essential complexity of the pattern, not incidental overhead. It's a lot of surfaces to get right. The operational contract also shifts: both physical topics have to honour the mapping, and your router is now the source of truth for what `topic-x` means. Transactions don't cross cluster boundaries here either — same caveat as topic weaving, but worth repeating because with partition weaving it's easier to forget the seam is there. +With partition weaving, a single logical topic has its partitions composed from physical topics across multiple clusters — which don't even need to share a name. + +Take an illustrative scenario: your third-party risk engine in `us-east` is hardcoded to read from a topic called `orders`. It has always read from `orders`. It will continue to read from `orders` (because vendor software rarely accommodates your topology reorganisations). Meanwhile, your business is growing in Europe, and sending every EU event across the Atlantic to land on `us-east` is expensive, slow, and fragile. The EU team provisions `orders-eu` on a local cluster sized for their producer load, and EU producers write there directly. + +Rather than waiting on a vendor feature request or building intermediate replication pipelines, the router presents both physical topics as one logical `orders` topic to the risk engine. When the engine requests metadata for `orders`, it gets back 32 partitions: partitions 0–15 are backed by `orders` on `us-east`, and 16–31 by `orders-eu` on `eu-west`. When a consumer fetches from partition 20, the proxy transparently routes the `FetchRequest` to `eu-west` (rewriting it for partition 4) and translates the response back. The risk engine reads the full partition space without a single configuration change, and EU producers write locally without crossing the Atlantic. + +Perceptive readers, all of you, will be asking how do consumer groups work with this fiction? Luckily, partition IDs are like node IDs: they are semantically address markers. The group coordinator is a trusting beast and lets you have any address you like. So we pick a cluster and send all consumer group management to that one. The group coordinator happily records committed offsets for partition 20 in `__consumer_offsets` alongside 0–15 without ever checking the router's white lies — it doesn't verify whether partition 20 physically exists on its local brokers. Only the data plane RPCs need to get routed to the other cluster. + +What the proxy has to do here is far more demanding than in any of the previous patterns: partition numbers must be rewritten consistently across every Kafka RPC that mentions them — and they get everywhere, as you might expect. That's the essential complexity of the pattern, not incidental overhead. It's a lot of surfaces to get right. The operational contract also shifts: both physical topics have to honour the mapping, and your router is now the source of truth for what `orders` means. Transactions don't cross cluster boundaries here either — same caveat as topic weaving, but worth repeating because with partition weaving it's easier to forget the seam is there. --- From 0630c16e3d324b00a0dd3996a1745113f3b32a64 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Fri, 9 Oct 2026 12:11:47 +1300 Subject: [PATCH 18/24] Refine stream branching: add sampling for staging budgets and clarify railway metaphor Signed-off-by: Sam Barker --- ...10-10-these-are-not-the-brokers-you-are-connecting-to.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md b/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md index 9bd25ba3..92ba4613 100644 --- a/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md +++ b/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md @@ -72,9 +72,11 @@ Before routing, Kroxylicious was a pipe with opinions — it could inspect and r You're chasing an issue in the reporting pipeline that only shows up in the sixth hour of the run and nobody can figure out which record trips it up. You've been there, right? What you really want is access to the live data with a debugger. Alas, this job is stuffed full of Personally Identifiable Information — so you're out of luck. You've been there and got *that* t-shirt. What now? You build a router that shadows production traffic to a staging cluster, piping it through a redaction filter on the way so the PII becomes gibberish before it touches staging brokers. The client is still producing to one VKC. But the proxy is now opening *two* backend connections — one to the primary cluster, one to staging — and writing every record to both. The acknowledgement the client gets back is from the primary; the shadow write happens on the side. One stream in, two streams out. If you've got your EIP book off the shelf already, yes — it's a Wire Tap with a Message Translator on the diverted stream. -That's one flavour of stream branching — the secondary stream is a copy of the primary, diverted somewhere and the client never realises. But the diverted branch doesn't have to be a copy. Every record passes through the routing DAG before it hits the broker. Something in that DAG can inspect the payload, compute something from it — consumer lag, record counts by key, an audit trail your compliance team needs — and emit that derived data to a topic anywhere the routing config points — a different topic on the same cluster, or a cluster the client has never heard of. The primary stream is untouched. The additional streams just appear, derived from the original client's activity. The clients don't change. Which means anything that previously required every application to cooperate — the audit trail, the telemetry, the thing all but one of your clients never supported — you can now just invent that stream yourself. This all works for stateless transformations performed inline. If you need state, joins, or windowing, Kafka Streams or Kafka Connect have your back. +That's one flavour of stream branching — the secondary stream is a copy of the primary, diverted somewhere and the client never realises. But the diverted branch doesn't have to be a copy. All staging clusters are capable of handling production load, aren't they? No? Ohh right, you have budgets. We still want to stage some real traffic, but we can't handle the full volume. A filter on the staging route drops 90% of the produce requests it sees and forwards the rest — we can do that as nothing cares about the acks. -I think of stream branching more like a railway network. The client talking to the broker is the mainline. In Britain the mainline has two tracks: the town line and the country line. Requests are like trains running on the town line. Responses are on the country line. The routing DAG is the station network. At each station the stationmaster is responsible for dispatching the train on down the main line. However, as we are working with bytes and not hundreds of tonnes of steel, we can assemble a new train as it passes through the station. That stationmaster is free to assemble his new train however he wants, moving passengers (better known as records) between carriages (aka batches), and dispatching it down whichever branch line he wants. He's free to do this as it has no effect on the express trains on the mainline. +Every record passes through the routing DAG before it hits the broker. Something in that DAG can inspect the payload, derive something from it — an audit log your compliance team needs, record telemetry for downstream aggregation, or an anonymised event feed — and emit that derived data to a topic anywhere the routing config points — a different topic on the same cluster, or a cluster the client has never heard of. The primary stream is untouched. The additional streams just appear, derived from the original client's activity. The clients don't change. Which means anything that previously required every application to cooperate — the audit trail, the telemetry, the thing all but one of your clients never supported — you can now just invent that stream yourself. This all works for stateless transformations performed inline. If you need state, joins, or windowing, Kafka Streams or Kafka Connect have your back. + +I think of stream branching more like a railway network. The client talking to the broker is the mainline. In Britain the mainline has two tracks: the town line (inbound) and the country line (outbound). Requests are like trains running on the town line; responses are heading back out on the country line. The routing DAG is the station network. At each station the stationmaster is responsible for dispatching the train on down the main line. However, as we are working with bytes and not hundreds of tonnes of steel, we can assemble a new train as it passes through the station. That stationmaster is free to assemble his new train however he wants, moving passengers (better known as records) between carriages (aka batches), and dispatching it down whichever branch line he wants. He's free to do this as it has no effect on the express trains on the mainline. Stream branching is a dual-write pattern, and dual writes have sharp edges. How sharply they cut depends on your use case — the post on stream branching gets into it. From adb613618c817eca0f4d29e8532866da0fdcf7ea Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Fri, 9 Oct 2026 13:04:55 +1300 Subject: [PATCH 19/24] Update review comments status to posted for PR 289 Signed-off-by: Sam Barker --- proposals/289-review-comments.json | 35 ++++++++++++++++++++++++------ 1 file changed, 28 insertions(+), 7 deletions(-) diff --git a/proposals/289-review-comments.json b/proposals/289-review-comments.json index 69aef675..551912b0 100644 --- a/proposals/289-review-comments.json +++ b/proposals/289-review-comments.json @@ -3,28 +3,49 @@ "type": "review_thread_reply", "thread": "T5", "replyToId": 4210755206, - "body": "Good point — you're right that we don't want to focus on record-level routing here. The design space does cover it (a router reconstituting a branch request can go as fine-grained as it likes), but that's not where current implementations sit. I've tweaked the language in the axis paragraph to avoid the ambiguity, and the stream branching section now covers the routing granularity point in more detail.", - "status": "pending" + "body": "Good point \u2014 you're right that we don't want to focus on record-level routing here. The design space does cover it (a router reconstituting a branch request can go as fine-grained as it likes), but that's not where current implementations sit. I've tweaked the language in the axis paragraph to avoid the ambiguity, and the stream branching section now covers the routing granularity point in more detail.", + "status": "posted" }, { "type": "review_thread_reply", "thread": "T6", "replyToId": 4211918741, - "body": "Horses for courses — Kafka Streams is absolutely the right answer for some of this. Where the proxy approach has the edge: no duplicate storage cost (you're deriving data in flight rather than materialising a new topic), and no framework overhead for what are often stateless transforms. Kafka Streams and Connect are powerful but they're not lightweight — if all you need is \"emit a record count to a metrics topic as this batch passes through\", standing up a Streams application is a lot of machinery for a stateless operation. I've added a sentence to the section to make this clearer.", - "status": "pending" + "body": "Horses for courses \u2014 Kafka Streams is absolutely the right answer for some of this. Where the proxy approach has the edge: no duplicate storage cost (you're deriving data in flight rather than materialising a new topic), and no framework overhead for what are often stateless transforms. Kafka Streams and Connect are powerful but they're not lightweight \u2014 if all you need is \"emit a record count to a metrics topic as this batch passes through\", standing up a Streams application is a lot of machinery for a stateless operation. I've added a sentence to the section to make this clearer.", + "status": "posted" }, { "type": "review_thread_reply", "thread": "T9", "replyToId": 4212088090, "body": "Oops that wasn't quite the example I was trying to make. I've tweaked the text to make it clearer the woven topic is only on the consumer side.", - "status": "pending" + "status": "posted" }, { "type": "review_thread_reply", "thread": "T4", "replyToId": 4210723236, - "body": "Glad the others landed. *Stream branching* was the best I could come up with — open to better suggestions. *Branching streams* is another candidate; the difference is whether you're naming the act (branching a stream) or the result (streams that branch). I went with the former but I'm not attached to it.", - "status": "pending" + "body": "Glad the others landed. *Stream branching* was the best I could come up with \u2014 open to better suggestions. *Branching streams* is another candidate; the difference is whether you're naming the act (branching a stream) or the result (streams that branch). I went with the former but I'm not attached to it.", + "status": "posted" + }, + { + "type": "review_thread_reply", + "thread": "T1", + "replyToId": 4210638469, + "body": "Fixed in df2f68b \u2014 updated to broker rather than cluster.", + "status": "posted" + }, + { + "type": "review_thread_reply", + "thread": "T8", + "replyToId": 4223067354, + "body": "Fair point on consumer lag \u2014 you're right that cluster-side observation is the proper place for complete lag metrics. On counts/telemetry, the proxy isn't doing the final aggregation \u2014 it's just emitting inline record telemetry for downstream systems to aggregate. I've updated the examples in this section to focus on clean, stateless inline operations (a redaction filter for PII, a sampling filter to thin traffic for smaller staging environments, and derivation filters for audit logs/telemetry).", + "status": "posted" + }, + { + "type": "review_thread_reply", + "thread": "T9", + "replyToId": 4223173007, + "body": "Yep, exactly \u2014 if you control the consumer and don't mind it managing multiple topic subscriptions/assignments, topic weaving gets you part of the way there. Partition weaving is what lets you keep that complexity entirely in the infrastructure so downstream consumers don't even need to know the second topic exists.\n\nI've reworked this section to frame it around extending weaving down to Kafka's fundamental unit of parallelism (partitions), swapped the example to a third-party risk engine consuming an `orders` topic, and explained how consumer group coordination remains pinned to a single cluster without the coordinator needing to know about the seam.", + "status": "posted" } ] From 23487a760793121b7b9c8e6b6666960274310bac Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Fri, 9 Oct 2026 13:59:00 +1300 Subject: [PATCH 20/24] Polish taxonomy post: flow vs topology framing, DAG station network, and consumer group RPC demuxing Signed-off-by: Sam Barker --- ...se-are-not-the-brokers-you-are-connecting-to.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md b/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md index 92ba4613..1d7a0ab7 100644 --- a/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md +++ b/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md @@ -53,7 +53,9 @@ You can deploy the proxy on either side of the network boundary: helping traffic If you're reaching for the [Enterprise Integration Patterns](https://www.enterpriseintegrationpatterns.com/patterns/messaging/MessageRoutingIntro.html) book right now — well done for being as old as I am, the rest of you just Googled it. It describes what happens to individual messages, which is interesting, but I'm talking about whole streams. Those patterns are what makes all this possible under the hood; what I'm describing sits a layer or two higher. -The patterns do build on each other along a single axis: how deep into the Kafka protocol the router has to reach. Connection switching operates at the connection level — which broker, which forms part of which cluster, is this connection actually being sent to? Stream branching works at the stream level — where does this request go? Topic weaving reaches into the topic catalog — which topics are visible and where do they live? Partition weaving goes deepest — how do physical partition spaces map onto a logical one? Each step requires the router(s) to understand more of the protocol than the last. But the complexity doesn't compound as routing is managed through a DAG which allows them to be composed rather than monolithic. A router handling topic visibility doesn't care whether the partitions behind it are woven from multiple clusters. A router rewriting partition numbers doesn't care whether the topic catalog above it spans two physical clusters or twenty. +The patterns build on each other along a single axis: how deep into the Kafka protocol the router has to reach. The first two — **connection switching** and **stream branching** — are flow patterns. Think of them like a railway network: you're directing moving trains, either throwing points at the junction before departure or dispatching carriages down branch lines as the train passes through. The second two — **topic weaving** and **partition weaving** — are topology patterns, where you step off the track to stitch discrete streams into a single logical map. + +Connection switching operates at the connection level — which broker, which forms part of which cluster, is this connection actually being sent to? Stream branching works at the stream level — where does this request go? Topic weaving reaches into the topic catalog — which topics are visible and where do they live? Partition weaving goes deepest — how do physical partition spaces map onto a logical one? Each step requires the router(s) to understand more of the protocol than the last. But the complexity doesn't compound as routing is managed through a DAG which allows them to be composed rather than monolithic. A router handling topic visibility doesn't care whether the partitions behind it are woven from multiple clusters. A router rewriting partition numbers doesn't care whether the topic catalog above it spans two physical clusters or twenty. ### Connection switching @@ -76,9 +78,11 @@ That's one flavour of stream branching — the secondary stream is a copy of the Every record passes through the routing DAG before it hits the broker. Something in that DAG can inspect the payload, derive something from it — an audit log your compliance team needs, record telemetry for downstream aggregation, or an anonymised event feed — and emit that derived data to a topic anywhere the routing config points — a different topic on the same cluster, or a cluster the client has never heard of. The primary stream is untouched. The additional streams just appear, derived from the original client's activity. The clients don't change. Which means anything that previously required every application to cooperate — the audit trail, the telemetry, the thing all but one of your clients never supported — you can now just invent that stream yourself. This all works for stateless transformations performed inline. If you need state, joins, or windowing, Kafka Streams or Kafka Connect have your back. -I think of stream branching more like a railway network. The client talking to the broker is the mainline. In Britain the mainline has two tracks: the town line (inbound) and the country line (outbound). Requests are like trains running on the town line; responses are heading back out on the country line. The routing DAG is the station network. At each station the stationmaster is responsible for dispatching the train on down the main line. However, as we are working with bytes and not hundreds of tonnes of steel, we can assemble a new train as it passes through the station. That stationmaster is free to assemble his new train however he wants, moving passengers (better known as records) between carriages (aka batches), and dispatching it down whichever branch line he wants. He's free to do this as it has no effect on the express trains on the mainline. +I think of stream branching as a railway network, where the routing DAG forms the station network itself. The client's primary stream runs straight down the express mainline. At each station along the line (a router node in the DAG), the stationmaster can inspect passing trains, reassemble carriages (record batches), or dispatch brand new trains down branch lines toward audit or staging clusters — all without delaying express traffic on the mainline. The power of the network is that complexity stays completely self-contained: one station handles PII redaction, the next branches an audit feed, and neither needs to know what happens up or down the line. + +Stream branching is fundamentally a dual-write pattern, and dual writes have sharp edges. Kafka logs have no undo button: if a write succeeds on the primary cluster but fails on the secondary, you cannot roll back the primary. Returning an error to the client triggers a retry, creating duplicate batches on the primary; returning success leaves the secondary silently behind. -Stream branching is a dual-write pattern, and dual writes have sharp edges. How sharply they cut depends on your use case — the post on stream branching gets into it. +That is why stream branching is best suited for auxiliary streams — shadowing PII-redacted traffic to staging, sampling load, or emitting derived telemetry. If your architecture strictly requires atomic, lossless replication across two physical clusters, an L7 proxy isn't the right tool; you want dedicated replication pipelines like MirrorMaker 2 (or KIP-1279 Cluster Mirroring). The dedicated post on stream branching gets into the gory details. ### Topic weaving @@ -104,7 +108,9 @@ Take an illustrative scenario: your third-party risk engine in `us-east` is hard Rather than waiting on a vendor feature request or building intermediate replication pipelines, the router presents both physical topics as one logical `orders` topic to the risk engine. When the engine requests metadata for `orders`, it gets back 32 partitions: partitions 0–15 are backed by `orders` on `us-east`, and 16–31 by `orders-eu` on `eu-west`. When a consumer fetches from partition 20, the proxy transparently routes the `FetchRequest` to `eu-west` (rewriting it for partition 4) and translates the response back. The risk engine reads the full partition space without a single configuration change, and EU producers write locally without crossing the Atlantic. -Perceptive readers, all of you, will be asking how do consumer groups work with this fiction? Luckily, partition IDs are like node IDs: they are semantically address markers. The group coordinator is a trusting beast and lets you have any address you like. So we pick a cluster and send all consumer group management to that one. The group coordinator happily records committed offsets for partition 20 in `__consumer_offsets` alongside 0–15 without ever checking the router's white lies — it doesn't verify whether partition 20 physically exists on its local brokers. Only the data plane RPCs need to get routed to the other cluster. +Perceptive readers, all of you, will be asking how consumer groups survive this fiction. You can’t simply commit an offset for partition 20 to a coordinator whose cluster only knows 0–15 — Kafka brokers validate committed topic-partitions against their local metadata and reject unmapped ones with `UNKNOWN_TOPIC_OR_PARTITION`. + +Instead, the proxy intercepts `OffsetCommit` and `OffsetFetch` requests, demuxes the partition tuples across the backing clusters, and translates virtual partition 20 to physical partition 4 on `eu-west`. Each coordinator only ever receives offset commits for partitions that genuinely exist on its cluster, and the proxy stitches the responses back together into a single seamless reply for the consumer. What the proxy has to do here is far more demanding than in any of the previous patterns: partition numbers must be rewritten consistently across every Kafka RPC that mentions them — and they get everywhere, as you might expect. That's the essential complexity of the pattern, not incidental overhead. It's a lot of surfaces to get right. The operational contract also shifts: both physical topics have to honour the mapping, and your router is now the source of truth for what `orders` means. Transactions don't cross cluster boundaries here either — same caveat as topic weaving, but worth repeating because with partition weaving it's easier to forget the seam is there. From 49d2f295b980de2b0b84ab9dd91dad4492c54059 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Fri, 9 Oct 2026 14:36:37 +1300 Subject: [PATCH 21/24] Update post date Signed-off-by: Sam Barker --- ...26-10-09-these-are-not-the-brokers-you-are-connecting-to.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename _posts/{2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md => 2026-10-09-these-are-not-the-brokers-you-are-connecting-to.md} (99%) diff --git a/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md b/_posts/2026-10-09-these-are-not-the-brokers-you-are-connecting-to.md similarity index 99% rename from _posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md rename to _posts/2026-10-09-these-are-not-the-brokers-you-are-connecting-to.md index 1d7a0ab7..b7e84a1f 100644 --- a/_posts/2026-10-10-these-are-not-the-brokers-you-are-connecting-to.md +++ b/_posts/2026-10-09-these-are-not-the-brokers-you-are-connecting-to.md @@ -1,7 +1,7 @@ --- layout: post title: "These are not the brokers you are connecting to" -date: 2026-10-10 15:00:00 +0100 +date: 2026-10-09 15:00:00 +1200 author: "Sam Barker" author_url: "https://github.com/sambarker" # noinspection YAMLSchemaValidation From 2dbae9ad66362c064b4959bb0000aa48217f176a Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Fri, 9 Oct 2026 14:51:47 +1300 Subject: [PATCH 22/24] Reschedule post 1 publication to Monday 12 October Signed-off-by: Sam Barker --- _drafts/routing-post2.md | 4 ++-- ...-10-12-these-are-not-the-brokers-you-are-connecting-to.md} | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) rename _posts/{2026-10-09-these-are-not-the-brokers-you-are-connecting-to.md => 2026-10-12-these-are-not-the-brokers-you-are-connecting-to.md} (99%) diff --git a/_drafts/routing-post2.md b/_drafts/routing-post2.md index 3bc4ddc1..62dde28b 100644 --- a/_drafts/routing-post2.md +++ b/_drafts/routing-post2.md @@ -9,7 +9,7 @@ categories: blog kroxylicious-proxy tags: [ "routing" ] --- -[Post 1]({% post_url 2026-10-10-these-are-not-the-brokers-you-are-connecting-to %}) introduced the Routing API and the fundamental shift it brings: a Virtual Kafka Cluster (VKC) is no longer a fixed pipe wired to a single physical cluster on boot. It's a stable identity. What actually lives behind it is now the proxy's problem, not the client's. +[Post 1]({% post_url 2026-10-12-these-are-not-the-brokers-you-are-connecting-to %}) introduced the Routing API and the fundamental shift it brings: a Virtual Kafka Cluster (VKC) is no longer a fixed pipe wired to a single physical cluster on boot. It's a stable identity. What actually lives behind it is now the proxy's problem, not the client's. We called the first pattern **Connection Switching**. @@ -156,6 +156,6 @@ The proxy routes frames; it doesn't replicate log data. If consumers need to pic Connection switching keeps the operational promise simple: platform teams can change the physical backend without the application teams ever knowing it happened. That's not a small thing. -The next post moves from one-to-one connection switching to stitching multiple physical clusters into a single logical view: **Topic Weaving**. +The next post moves from one-to-one connection switching to fanning out traffic and assembling secondary streams on the fly: **Stream Branching**. [^1]: There is always some bit of infra that fails to honour the TTL properly or doesn't understand a CNAME or PTR record. diff --git a/_posts/2026-10-09-these-are-not-the-brokers-you-are-connecting-to.md b/_posts/2026-10-12-these-are-not-the-brokers-you-are-connecting-to.md similarity index 99% rename from _posts/2026-10-09-these-are-not-the-brokers-you-are-connecting-to.md rename to _posts/2026-10-12-these-are-not-the-brokers-you-are-connecting-to.md index b7e84a1f..7c80c559 100644 --- a/_posts/2026-10-09-these-are-not-the-brokers-you-are-connecting-to.md +++ b/_posts/2026-10-12-these-are-not-the-brokers-you-are-connecting-to.md @@ -1,7 +1,7 @@ --- layout: post title: "These are not the brokers you are connecting to" -date: 2026-10-09 15:00:00 +1200 +date: 2026-10-12 15:00:00 +1200 author: "Sam Barker" author_url: "https://github.com/sambarker" # noinspection YAMLSchemaValidation From 73e7fc62324b80806566ebb2a9867325b1b5cf7f Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Mon, 12 Oct 2026 13:52:11 +1300 Subject: [PATCH 23/24] Add egress/ingress discussion to connection switching post Signed-off-by: Sam Barker --- _drafts/routing-post2.md | 56 +++++++++++----------------------------- 1 file changed, 15 insertions(+), 41 deletions(-) diff --git a/_drafts/routing-post2.md b/_drafts/routing-post2.md index 62dde28b..60cd0582 100644 --- a/_drafts/routing-post2.md +++ b/_drafts/routing-post2.md @@ -48,46 +48,6 @@ When executing a planned migration or a DR failover: Whether you're doing a planned blue/green transition or invoking DR in an emergency, connection switching takes client coordination off the critical path. The data still has to be in sync — that's on you — but at least the cutover itself isn't a multi-team event. -### The cloud bill from hell: Replica routing - -If you run Kafka in the cloud across multiple Availability Zones, you've probably had this exact conversation with your finance team. You opened the monthly AWS bill, choked on the 5-figure line item for cross-AZ data transfer, and dutifully turned on Kafka's built-in follower fetching (`client.rack`). - -And it worked! The bill plummeted. You closed the Jira ticket, posted a triumphant graph in `#infra-cost-savings`, and moved on. - -Then, six months later, the finance team knocks on your door again. The cross-AZ tax has silently crept right back up. - -Why? Because relying on every client in the company to configure rack awareness is an unwinnable game of whack-a-mole: -- Someone copy-pastes a Helm chart that hardcodes `client.rack=eu-west-1a` into a pod deployed in `eu-west-1b`. -- A data science team spins up a fleet of Python consumers using a library where nobody realised follower fetching requires a specific flag. -- A new microservice goes live with the defaults, happily pulling multi-gigabyte streams across the AZ boundary from whatever partition leader happens to be awake. - -Kafka's native rack awareness is great, but client compliance is leaky. Centralized problems need centralized controls. - -And this isn't just about inspecting frames at Layer 7. If all you have is a filter pipeline on a static 1:1 pipe, understanding that a `Fetch` belongs in `eu-west-1a` doesn't help you—the pipe is already nailed to broker 1 in `eu-west-1c`. The breakthrough comes from the Routing API making the backend connection dynamic. - -When a `Fetch` request arrives at a Kroxylicious instance in `eu-west-1a` — on the pipe for the leader over in `eu-west-1c` — the router inspects the partition's in-sync replica (ISR) set. If broker 3 is sitting right next door in `eu-west-1a` with a caught-up replica, the router doesn't rewrite the frame—it dynamically dispatches that individual request down a separate backend pipe to broker 3: - -``` -[ Consumer (AZ-a) ] ──Fetch──► [ Kroxylicious (AZ-a) ] ────► [ Broker 3 (AZ-a, Replica) ] - │ - (Skips AZ tax) - ▼ - [ Broker 1 (AZ-c, Leader) ] -``` - -The application didn't configure `client.rack`. The request payload is completely untouched. It simply travelled down a different pipe. Broker 1 never sees the read, and the cross-AZ request never happened so never reaches the invoice. - -Rack-local fetching is just leveraging the existing Kafka protocol. In theory, dynamic per-request dispatch opens up other intriguing possibilities to explore in your custom Bugatti router from post 1: -- **Replica lag awareness:** Steering reads strictly to the most caught-up followers to avoid trailing-edge staleness. -- **Broker load shedding:** Diverting fetch traffic away from CPU- or IO-bound leaders to give them headroom for critical writes. -- **Tiered storage offloading:** Routing historical backfills to brokers backed by cheap object storage while keeping hot tail-reads on fast NVMe. - -Whether those speculative policies prove practical in production remains to be seen—they require telemetry the router might not easily have—but proximity routing alone solves a very real, very expensive problem right now. - -There is, of course, a hard constraint: **proxy placement**. If a client in `eu-west-1a` talks to a proxy sitting in `eu-west-1b`, which then forwards to a broker in `eu-west-1a`, you’ve just invented a two-hop cross-AZ detour. For replica routing to pay off, your proxy fleet must be co-located with the clients it fronts. - ---- - ## How the proxy makes it work Making one physical cluster look like a different one at Layer 7 requires more than TCP forwarding. There are three places where the proxy has to do actual work. @@ -136,7 +96,7 @@ If the routing target changes while a client is active: Connection switching works because there is a single physical cluster backing it. The proxy isn't maintaining an illusion — it's directing traffic to something real, and real Kafka semantics apply throughout. The moment a router exposes brokers from multiple physical clusters as a unified address space, it has to ensure those semantics stay valid. For some operations that's fine — routing a `Fetch` to a local replica is cheap and safe. But for anything that touches producer identity or coordination — PIDs, epochs, transaction coordinators, consumer group coordinators — the router has taken on the obligation of upholding guarantees the protocol no longer provides on its behalf. -The most immediate consequence is session affinity. The router must maintain a session-scoped mapping between the client's virtual PID and the real (PID, epoch) on each backend cluster. That mapping lives in the proxy instance that negotiated it. If the load balancer routes a reconnecting client to a different proxy replica, the mapping is gone — the router re-negotiates, the epoch bumps, and the backend fences the producer. The proxy can't prevent this. Only sticky L4 routing can. +The most immediate consequence is that producer identity becomes connection-scoped. The router must maintain a mapping between the client's virtual PID and the real (PID, epoch) on each backend cluster, and that mapping lives with the client connection that negotiated it. Kafka connections are long-lived, but they aren't forever: when one drops, the proxy tears down everything attached to it, mapping included. Reconnecting doesn't help — not even to the same proxy instance — the router re-negotiates, and whatever sequence or epoch continuity the client was relying on is gone. Nor is there only one mapping to lose. A client holds a connection per leader broker, possibly spread across proxy instances that know nothing of each other, so a single client PID can end up fronting several unrelated backend identities. Sticky L4 routing doesn't fix that; the state isn't anywhere a load balancer can point at. Friends don't let friends cross that line without knowing exactly what they've signed up for. @@ -154,6 +114,20 @@ The proxy routes frames; it doesn't replicate log data. If consumers need to pic --- +## Egress, ingress, or does it matter? + +Post 1 promised we'd call out where each pattern lands on the egress/ingress axis. From a protocol standpoint connection switching is neutral — the mechanics work either side of the wire. In practice, your use case has a strong opinion about placement. + +When talking about DR failover it's obvious once you say it out loud. If the proxy lives in the broker domain — in front of the brokers, a reverse proxy — and the primary site goes down, so does the proxy. Your switch just went boom along with the primary site... The secondary site's ingress proxy was never in the traffic path to begin with; you can't flip a routing target on a proxy your clients weren't talking to. You'd still need to redirect clients to a different address, which is exactly the coordination problem connection switching was supposed to eliminate. + +Thus for DR, the proxy needs to sit on the client side of the link as an egress deployment. The proxy isn't collocated with what failed. It's still in the traffic path. You can flip the routing target, and clients never notice. + +Connection switching also sits quite happily on the ingress side of the fence. If your goal is to give every team in the organisation a single `kafka.corp.internal:9092` and route their connections to the right tenant cluster behind the scenes, the proxy will most likely sit in front of the brokers — the platform team controls that domain and the stable endpoint is theirs to publish. You could run tenant dispatch egress-side, but now you're managing a fleet of client-side proxies, one per team or workload, and keeping them all pointed at the same routing config. That's a lot of instances to babysit for a problem the platform team already owns. + +So what happens when you need to perform a failover of a multi-tenant system? You put a proxy at both ends of the link — egress on the client side, ingress on the broker side. What makes this work cleanly is that each proxy is completely indifferent to what sits behind it: it dials what it's told, speaks Kafka, and has no idea whether the thing on the other end is a broker or another Kroxylicious instance. There's no shared state between them, no coordination protocol, no handshake. That's the pay-off from being a native layer 7 proxy. + +--- + Connection switching keeps the operational promise simple: platform teams can change the physical backend without the application teams ever knowing it happened. That's not a small thing. The next post moves from one-to-one connection switching to fanning out traffic and assembling secondary streams on the fly: **Stream Branching**. From 8c77b7b0b753058451511bf56a0f512e18da6ada Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Mon, 12 Oct 2026 13:53:42 +1300 Subject: [PATCH 24/24] Add post 3 draft (stream branching); update stash and post 1 Signed-off-by: Sam Barker --- _drafts/routing-post3.md | 175 ++++++++++++++++++ _drafts/stash-replica-routing.md | 24 +-- ...e-not-the-brokers-you-are-connecting-to.md | 4 +- 3 files changed, 184 insertions(+), 19 deletions(-) create mode 100644 _drafts/routing-post3.md diff --git a/_drafts/routing-post3.md b/_drafts/routing-post3.md new file mode 100644 index 00000000..81dfb85b --- /dev/null +++ b/_drafts/routing-post3.md @@ -0,0 +1,175 @@ +--- +layout: post +title: "Stream Branching: Mind the gap" +date: 2026-10-26 15:00:00 +1200 +author: "Sam Barker" +author_url: "https://github.com/sambarker" +# noinspection YAMLSchemaValidation +categories: blog kroxylicious-proxy +tags: [ "routing" ] +--- + +[Post 2](/TODO-post2-url) established the station network: the routing DAG is a network of stations, the client's stream is the express running down the mainline, and at each station the stationmaster can inspect passing trains. Stream branching is what the stationmaster does next. There are two flavours. In the *substitution* case, the train is quietly re-routed to a different platform — the original destination never sees it. In the *additive* case, the express continues to its primary destination while new trains depart down branch lines. The first is simple. The second is where the sharp edges are, and most of this post is about those edges. + +## The simplest branch: Replica routing + +If you run Kafka across multiple Availability Zones, you've probably already had this conversation with your finance team. You opened the monthly bill, choked on the cross-AZ data transfer line item, and dutifully turned on Kafka's built-in follower fetching (`client.rack`). The bill plummeted. You closed the Jira ticket, posted a triumphant graph in `#infra-cost-savings`, and moved on. + +Then six months later finance knocks on your door again. The cross-AZ tax has silently crept right back up. + +Why? Because relying on every client in the company to configure rack awareness is an unwinnable game of whack-a-mole: +- Someone copy-pastes a Helm chart that hardcodes `client.rack=eu-west-1a` into a pod deployed in `eu-west-1b`. +- A data science team spins up a fleet of Python consumers using a library where nobody realised follower fetching requires a specific flag. +- A new microservice goes live with the defaults, happily pulling multi-gigabyte streams across the AZ boundary from whatever partition leader happens to be awake. + +Centralised problems need centralised controls. But there's a subtler obstacle. Even if a filter could see that a particular `FetchRequest` belongs in `eu-west-1a`, knowing that isn't enough — the pipe is already nailed to broker 1 in `eu-west-1c`. The filter has nowhere to send it. The Routing API allows you to fix this: the backend connection is no longer fixed at connection time, it can be chosen per request. + +A `FetchRequest` payload just says "partition 3, from offset N". The broker ID is encoded by which connection the client sends the request on — that's the seam a custom router can exploit. When the `FetchRequest` arrives on the virtual broker 1 connection, the router inspects the partition's replica set, sees that broker 3 is in the same AZ and fully caught up, and tells the runtime to fulfil the request down a separate backend socket to broker 3 instead: + +``` +[ Consumer (AZ-a) ] ──Fetch──► [ Kroxylicious (AZ-a) ] ────► [ Broker 3 (AZ-a, Replica) ] + │ + (Skips AZ tax) + ▼ + [ Broker 1 (AZ-c, Leader) ] ← never sees the request +``` + +The application didn't configure `client.rack`. The request payload is completely untouched. Broker 1 never receives the order, and the cross-AZ charge never reaches the invoice. What made that possible: the proxy is holding two backend sockets, one to the leader and one to the local replica, and choosing between them per request. To the client, that's still one broker on one cluster. + +The same per-request dispatch mechanism opens up other possibilities beyond proximity — your custom Bugatti router from post 1 could consider: +- **Replica lag awareness:** Steering reads to the most caught-up follower to avoid trailing-edge staleness. +- **Broker load shedding:** Diverting fetch traffic away from CPU- or IO-bound leaders to give them headroom for critical writes. +- **Tiered storage offloading:** Routing historical backfills to brokers backed by cheap object storage while keeping hot tail-reads on fast NVMe. + +Whether those prove practical in production remains to be seen — they require telemetry the router might not easily have — but proximity routing alone solves a very real, very expensive problem right now. + +One constraint the dispatcher can't escape: **proxy placement**. If the proxy is in `eu-west-1b` and the consumer is in `eu-west-1a`, pulling from a broker in `eu-west-1a` still costs a cross-AZ hop to reach the dispatcher first. For replica routing to pay off, the proxy fleet must be co-located with the clients it fronts. + +--- + +## The express doesn't wait + +Stream branching has one rule, and the rest of this post follows from it: + +> **The mainline is the source of truth. Branch lines are best effort.** + +The express never waits at a junction for a branch-line train. The client produces a batch. The router sends it to the primary cluster and, at the same time, sends zero or more batches down branch lines to other topics, other clusters, or both. Those can be copies, redacted copies, samples, or new records derived from the payload. The client gets back the primary's `ProduceResponse` and nothing else. Branch writes never add latency to that acknowledgement, never turn a success into an error, and never turn an error into a success. + +``` + ┌──(mainline)──► [ Primary cluster ] ──ack──┐ +[ Producer ] ──Produce──► [ Router ] │ + ▲ ├──(branch)────► [ Staging cluster ] (redacted, sampled) + │ └──(branch)────► [ audit-events topic ] (derived) + └──────────────── ProduceResponse (primary's verdict only) ◄───────────────┘ +``` + +The branch writes belong to the router, not the client. The client didn't send them and never sees their responses. The router opens its own connections to the branch targets, sends its own `ProduceRequest`s, and decides what to do when they fail. + +This rule is a design choice, not a missing feature. The obvious alternative is to hold the acknowledgement until every branch has landed. That doesn't make the writes atomic. It just ties production to your least reliable cluster, as [Mind the gap](#mind-the-gap) explains. + +## What it costs to stop at the station + +The cheapest branch is a faithful copy: same records, same batches, sent on to a different place. The router can pass the record batch bytes through with only the header changed (more on which fields later). It never has to decompress anything. + +Once the stationmaster needs to look *inside* the carriages, the cost goes up a lot. Redacting PII, deriving an audit record from the payload, or filtering individual records all mean decompressing the batch, deserialising records, transforming them, then re-serialising and recompressing. On a busy topic with `zstd` batches that's real CPU, and it happens inline on the proxy for every batch that takes the branch. Size your proxy fleet for the most expensive station on the line, not for the mainline. + +Sampling sits somewhere in the middle. Dropping whole batches is close to free. Dropping individual records means rewriting batches. Which one you want depends on whether "10% of the traffic" means 10% of requests or 10% of records. For load testing it's usually the first, which is lucky. + +Keep it stateless. Every station handles each batch on its own. Joins, windows, and aggregations over time need state that survives restarts, rebalances, and late data, and Kafka Streams and Kafka Connect exist to manage exactly that. A router that starts accumulating state across batches has become a badly built stream processor. + +## Mind the gap + +Here's what you're signing up for. + +### There is no undo + +A Kafka log only ever appends. When a batch lands on one cluster and fails on the other, nothing can take the successful write back. That leaves four outcomes for any batch: + +| Primary | Branch | What the client sees | What the branch ends up with | +|---------|---------|----------------------|------------------------------| +| ✅ | ✅ | Success | A copy. The happy path. | +| ✅ | ❌ | Success | A hole | +| ❌ | ✅ | Error, so it retries | A copy now, and another one after the retry | +| ❌ | ❌ | Error, so it retries | Whatever the retry produces | + +The second row is the one people expect: the branch falls behind and nobody tells the client. The third row is the one that catches people out. The primary returned an error (or the ack timed out), but the branch write already went through. The client retries. The primary deduplicates the retry if the first attempt actually landed, because that's what idempotent producers are for. The branch, though, receives the retried batch as a fresh write from the router. Unless the router has done the extra work described below, the branch now has a duplicate the primary doesn't. + +So in normal running the branch both loses data and duplicates it, independently of the primary. If that sentence makes you uncomfortable, stream branching is the wrong tool for that particular job. That isn't a criticism. It's the reason post 1 sent atomic, lossless replication off to MirrorMaker 2 or [KIP-1279](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1279%3A+Cluster+Mirroring) cluster mirroring. + +### Waiting for the branch makes the mainline only as reliable as the branch + +The obvious way to close the gap is to hold the client's acknowledgement until both writes succeed. It doesn't work. + +If the router waits for the branch, the client's produce latency becomes the slower of the two clusters, and the staging cluster you sized on a budget sets the latency of production. If the router *fails* the client whenever the branch fails, an outage on staging becomes an outage on production. You've wired production's reliability to your least reliable cluster. Even if you accept both of those, you still haven't got atomicity: a branch write can succeed while the primary times out, and you're back in row three of the table. + +A branch must never back up onto the mainline. Which leads to the next problem. + +### Backpressure: drop or block? + +The primary is healthy and the staging cluster is struggling. Branch writes start queueing in the proxy. That queue has to have a limit, because proxy memory isn't infinite and running out of it takes the mainline down as well. When the queue is full, the router has two options: + +- **Drop.** Throw away branch writes until the branch catches up. The mainline is unaffected and the branch gets a hole. This is the right default for nearly every branching use case, and it's what "best effort" means in practice. +- **Block.** Stop reading from the client until the branch drains. The branch stays complete, and the mainline slows down to match the branch. Now you're coupled again, just more slowly. + +Whichever one you choose, make the drops visible. A branch that loses data silently is one you'll eventually trust when you shouldn't. Count what was dropped, per branch, and alert on it, so that when someone asks whether staging saw the batch that triggered the bug, you can actually answer. + +### Partitions don't have to line up + +The client's partitioner picked a partition based on the *primary* topic's partition count. The branch topic might have a different number of partitions, or might not have partitions that mean the same thing. If the branch is a faithful shadow with the same partition count, partition *n* maps to partition *n* and per-key ordering survives. If not, the router is repartitioning. It has to choose a branch partition itself, which means either re-hashing the key or making some other decision, and the ordering guarantees on the branch are whatever that decision gives you. Derived streams, such as an audit feed keyed by user rather than by order, almost always fall into the second case. That's fine as long as you meant it. + +### Duplicates on the branch + +Kafka producers use idempotent delivery by default. The broker deduplicates retried writes, so a message is committed exactly once even if the network drops the acknowledgement and the producer retries. This depends on a Producer ID (`PID`) that the broker assigns at initialisation, plus a per-partition sequence number on every batch. If the broker sees a batch it has already committed (same PID, same sequence), it quietly discards the duplicate. The client never needs to know. + +**The mainline PID belongs to the client, and it is never translated.** The client negotiated its PID with the primary cluster, and the router passes it through unchanged, exactly as in connection switching. Every write that carries the client's identity goes to the cluster that issued it. Nothing to map, nothing to invent, nothing to lose. Not needing to translate it is a large part of what separates branching from weaving. Once one client identity has to be valid on several clusters at once, the proxy is translating something far less forgiving than a node ID. That's a problem for the weaving posts. + +**Branch PIDs belong to the router.** The branch cluster's identity is something the router created. The client doesn't know the branch cluster exists, and nothing downstream will ever try to resume or share that identity. The router sends its own `InitProducerIdRequest` to the branch cluster, rewrites the producer ID and epoch in each record batch header before sending it down the branch, and owns that identity from start to finish. This is safe *because* the client doesn't know about it: an identity the client can't see is one the client can't collide with. (Those header fields are covered by the batch CRC, so rewriting them means recomputing the checksum. It's cheap compared with decompressing anything, but it isn't free.) + +That ownership settles who is talking to the branch. It doesn't settle what the branch deduplicates, which brings back row three of the table. There are two ways to handle sequence numbers: + +- **Pass the client's sequence numbers through** under the router's PID. If the branch receives exactly the same batches as the primary, in the same order, from the start of the session, then a client retry carries the same sequence number to both clusters and *both* deduplicate it. That closes row three. It only works as long as the branch is a faithful copy. Sample a batch away, drop one under backpressure, or lose one to a branch error, and the branch sees a gap in the sequence and rejects everything after it with `OUT_OF_ORDER_SEQUENCE_NUMBER`. The router then has to notice, bump its epoch on the branch, and start again, and the batches in flight during that recovery are gone. +- **Assign the router's own sequence numbers.** The router numbers branch batches contiguously however it likes, so sampling, dropping, and derived streams all just work. But the branch can no longer recognise the client's retries, because each one is a new batch from the router. Row three now produces duplicates on the branch every time. + +Branches that only sample or derive have to use the second approach. Faithful shadows can try the first, as long as you accept that every gap forces a reset, and that it only helps while each partition's traffic stays on one client connection (more on that next). In both cases the branch's deduplication history is separate from the primary's, and writes the primary would have discarded as duplicates may be accepted on the branch. + +**The branch identity belongs to a connection, not to the client.** The router's branch PID only lasts as long as the client connection it was negotiated for. When that connection closes, the proxy tears down everything attached to it, branch identity included. And the client has one PID but several connections: a producer opens one to each broker that leads a partition it writes to, and in a horizontally scaled proxy fleet those connections can land on different proxy instances. The routers serving those connections have no way to coordinate, so each one negotiates its own PID with the branch cluster. One client identity on the primary becomes several unrelated identities on the branch. + +That's harmless while each partition's traffic stays on one connection. It stops being harmless when the traffic moves, and in Kafka it moves routinely: + +- **Leadership changes.** A partition's leader moves from broker 1 to broker 2, so the client starts sending that partition's batches down a different connection, possibly through a different proxy instance. It carries on with the same PID and the next sequence number. On the primary, that's an ordinary continuation. On the branch, the batches arrive under a PID that has never written to that partition. Whatever sequence numbers the router uses, the branch has no history for that PID, so it can't recognise any batch the client retries across the move. +- **Reconnects.** A network blip, an idle timeout, a broker restart, a proxy restart: anything that drops the client's connection drops the branch identity with it. The client reconnects and resends its in-flight batches, some of which had already reached the branch under the old PID. They arrive under a new PID, and the branch accepts them all as new writes. Reconnecting to the same proxy instance doesn't help, because the old connection's state is already gone. + +Both are the same failure. The client's retries for a partition move from one branch identity to another, and the branch can only deduplicate within a single identity. + +The mainline is unaffected. Its PID belongs to the client, which holds it across every connection, every proxy instance, and every reconnect, so the primary recognises the retries and discards the ones it has already committed. That's the asymmetry working in your favour: the identity that matters is held by the one party that sees all of the client's traffic. But leadership changes, reconnects, and rolling upgrades are routine, so duplicates on the branch are a normal part of running stream branching, not an edge case. + +Use shadow data for observability, not for accounting. If someone wants to reconcile the branch against the primary to the last record, the answer is replication, not branching. + +### Transactions don't branch + +If the client is a transactional producer, the batches it produces are marked transactional, and they're only meaningful together with the `AddPartitionsToTxn` and `EndTxn` calls the client sends to the *primary's* transaction coordinator. The commit and abort markers that tell consumers which records count aren't produced by the client at all. The primary's coordinator writes them. The router never sees them in the produce stream, so it can't copy them. + +That leaves the router with two bad options for a transactional branch: + +- **Strip the transactional flag** and write the branch batches as plain idempotent writes. They're visible on the branch straight away, *including records from transactions the primary later aborts*. A `read_committed` consumer on the branch will read data that, as far as the primary is concerned, never existed. +- **Run a shadow transaction** on the branch: the router starts its own transaction against the branch cluster's coordinator, adds partitions as the client does, and commits or aborts when the client's `EndTxn` succeeds or fails on the primary. That's a lot of protocol to mirror, and the two commits are still separate. The primary can commit while the branch aborts, and nothing coordinates the two. Post 1 put it bluntly and it still holds: the Kafka protocol has no cross-cluster transaction coordinator, and routing can't create one. + +For shadowing and sampling, stripping the flag and documenting that the branch may contain aborted data is usually the honest choice. For derived streams that feed something financial, don't branch from a transactional producer at all. + +## Know when to get off the train + +Stream branching is a great tool for a specific set of problems, and it's easy to use it for the wrong ones. A rough guide: + +- **You want production-shaped traffic somewhere safe, and losing some is fine.** Branch it. Redact it on the way, sample it if staging is small, and watch the drop counters. +- **You want a stream nobody produces yet, derived statelessly from one that already exists.** Branch it. That's the case that makes the proxy worth running: no application team has to deploy anything. +- **You need the derived stream to be stateful**, such as joins, windows, sessionisation, or deduplication across batches. Use Kafka Streams or a Connect pipeline that reads from the primary. They manage state. A proxy shouldn't. +- **You need a complete, ordered, offset-consistent copy of the primary on another cluster.** Use replication: MirrorMaker 2, KIP-1279 cluster mirroring, or your vendor's equivalent. They read the primary's committed log after the fact, so they can't have the asymmetric failures described above. +- **You need writes to two clusters to be atomic.** Nothing at layer 7 can give you that. Redesign so the writes share a cluster. + +The pattern works because it's honest about having two tiers: one stream that carries the guarantee and some that don't. Keep that distinction clear in your config, your dashboards, and your conversations with whoever consumes the branch, and stream branching lets you build streams nobody had to code for. Blur it, and sooner or later someone will be surprised by a gap you could have warned them about. + +--- + +Both flow patterns are now covered. Connection switching picks the track. Stream branching adds branch lines. Next we leave the railway for the sorting office: **topic weaving**, where the router no longer just directs streams but decides which topics exist, where they live, and what they're called. Producer identity comes up again there, and this time the client *does* know about every cluster involved. + +If you're branching streams already, or want to and have hit a gap this post didn't cover, find us on [GitHub](https://github.com/kroxylicious/kroxylicious), [Slack](https://kroxylicious.slack.com), or [Bluesky](https://bsky.app/profile/kroxylicious.io). diff --git a/_drafts/stash-replica-routing.md b/_drafts/stash-replica-routing.md index cc14fd2c..ff7ae6b0 100644 --- a/_drafts/stash-replica-routing.md +++ b/_drafts/stash-replica-routing.md @@ -45,26 +45,16 @@ Proxy placement. A client in `us-east-1a` hitting a proxy in `us-east-1b` that t --- -# Stash: PID section from post 2 +# Stash: why PIDs aren't node IDs (for the topic weaving post) -Removed from `routing-post2.md` — the full treatment belongs in the stream branching or topic weaving post where PIDs across multiple clusters actually bite. Connection switching stays in the single-cluster safe zone so the detail isn't earned here. +Cut from the stream branching post. Branching never translates the client's PID: the main line passes it through and branch PIDs are router-owned, so this argument isn't needed there. It belongs in weaving, where one client identity has to be valid on several clusters at once. -### Idempotent producers and the PID problem +When you first look at PID management through the proxy, it looks like a problem we've already solved. [Post 2](/TODO-post2-url#when-is-2--2) introduced a bijection for node IDs: a deterministic, stateless mapping between downstream and upstream IDs at each router. PID translation looks like the same problem, and the similarities are convincing. Both are integers. Both are cluster-scoped. Both are assigned by the cluster, not chosen by the client. Both are opaque values the client just carries around and hands back. So the proxy can keep a downstream-to-upstream mapping for both, right? That's the obvious answer, and each problem with it looks fixable on its own until you've fixed several and the whole thing still doesn't hold together. The reason is that although node IDs and producer IDs look identical on the wire, they mean completely different things. A node ID is **spatial**: a label, a coordinate. A PID is **temporal**: not just a label but a claim to be a particular live producer. We can fake a node ID with arithmetic because it's only an address. We have to track a producer ID with state because the broker enforces it as an identity, and fences anyone who presents a stale one. -Kafka producers use idempotent delivery by default. The broker deduplicates retried writes so that a message is committed exactly once even if the network drops the acknowledgement and the producer retries. The mechanism relies on a Producer ID (`PID`) that the broker assigns on initialisation, combined with a per-partition sequence number on every batch. If the broker sees a batch it's already committed — same PID, same sequence — it silently discards the duplicate. The client never needs to know. +**Node ID translation is transport-scoped.** A `MetadataResponse` tells the client where the brokers are. The proxy can lie as much as it likes here, because the broker never sees the lies, let alone challenges them. The proxy can also forget: if it throws away its mapping and rebuilds it from the next `MetadataResponse`, it tells the same lies again. Nothing notices and nothing breaks. -When you first look at PID management through the proxy, it looks like a problem we already solved. The previous section introduced a bijection for node IDs — a deterministic, stateless mapping between downstream IDs and upstream IDs at each router. PID translation looks like the same pattern, and the surface similarities are convincing: both are integers, both are cluster-scoped, both are assigned by the cluster rather than chosen by the client, and both are opaque values the client simply carries and passes through. So the proxy can just maintain a downstream-to-upstream mapping for both, right? This is the obvious answer. The seams show up quickly, but each one looks stitchable — until you're looking at a patchwork quilt and wondering why it keeps unravelling. The reason is that despite identical protocol syntax, node IDs and producer IDs have completely different semantics within the protocol. A node ID is **spatial** — a label, a coordinate. A PID is **temporal** — not just a label but a claim of active identity. In other words: we can mathematically fake a Node ID because it's just an address. We have to statefully track a Producer ID because it's a weaponised identity claim. +**PID allocation is negotiation-scoped.** Since Kafka 3.0, producers run in idempotent mode by default. To support that, the producer sends `InitProducerIdRequest` to ask the broker for an identity: a (PID, epoch) pair. The broker tracks that identity and uses it to deduplicate retried batches during that session. It will also evict any connection that presents the same PID with a lower epoch. That's the fencing mechanism, and it stops two connections claiming the same identity at once. If you need the identity to survive across sessions, so that a restarted producer picks up where it left off, that's what `transactional.id` is for. A second `InitProducerIdRequest` with the same `transactional.id` is deliberately treated as a hostile takeover: fencing is how Kafka makes sure only one producer instance is ever active for a given transactional identity. -**Node ID translation is transport-scoped.** A `MetadataResponse` tells you where brokers are. The proxy can tell as many lies as it likes here — the broker never sees them, let alone tries to call them out. And the proxy can be forgetful — stateless, even: if it discards its mapping and rebuilds it from the next `MetadataResponse`, it just tells the same lies again. Nothing notices, nothing breaks. +Node IDs and PIDs are both cluster-scoped: issued by one cluster, honoured within it, meaningless outside it. But the node ID mapping is stateless. If the proxy loses it, the proxy rediscovers and recomputes it. A PID can't be recovered that way. Once the negotiation is lost, so is the identity. -**PID allocation is negotiation-scoped.** Since Kafka 3.0, producers run in idempotent mode by default — you don't have to opt in, it just happens. To support that, the producer calls `InitProducerIdRequest` to ask the broker to allocate it an identity: a (PID, epoch) pair. The broker tracks that identity and uses it to deduplicate retried batches within that session. It will also forcibly evict any connection that presents the same PID with a lower epoch — that is the fencing mechanism, preventing two connections from simultaneously claiming the same identity. If you need that identity to survive across sessions — so a restarted producer can pick up where it left off rather than starting fresh — that is what `transactional.id` is for. A second `InitProducerIdRequest` carrying the same `transactional.id` is treated as a hostile takeover, deliberately: fencing is how Kafka ensures only one producer instance is ever active for a given transactional identity. - -Both node IDs and PIDs are cluster-scoped — issued by a specific cluster, honoured within it, meaningless outside it. But the node ID relationship is stateless: if the proxy loses its mapping, it recovers via discovery and rederivation. A PID is unrecoverable. What was negotiated is gone. - -All of that is a long way of saying: the proxy cannot manage PIDs globally, and the Routing API makes no attempt to. There is no built-in abstraction that handles this for you — it is left to the team implementing the router to understand what their routing topology implies for producer identity. That is not as bleak as it sounds. There are shapes that work cleanly. If your problem fits one of them, go hard or go home. - -**Single physical cluster.** When all routes lead to the same physical cluster, there is nothing to solve. This is just standard Kafka protocol semantics — the proxy is a transparent conduit for a PID relationship that exists entirely between the client and the cluster. No mapping, no invention, no state to lose. Two things to keep in mind: routing decisions must be deterministic at the cluster level (a given producer must always land on the same physical cluster), and if the cluster changes — whether through a failover or a routing reconfiguration — the PID means nothing to the new cluster. Not fenced — a brand new client. - -**Multiple active clusters.** This is where connection switching ends. The full treatment — what can be made to work, what can't, and at what cost — is in the post on topic weaving. But the boundary is worth drawing clearly here before we get there: see *Layer 7 can only do so much* below. - -**Duplication routing (traffic shadowing).** One pattern that stays cleanly within the connection switching boundary: the shadow write to the secondary cluster is entirely router-invented — the client does not know Cluster B exists, and nothing downstream will ever attempt to resume or share that identity. The router negotiates its own PID with the shadow cluster and owns it from start to finish. What makes this safe is precisely the client's ignorance: because the shadow identity is invisible to the client, nothing can collide with it. There is one edge case worth naming: the shadow cluster has no knowledge of the primary's deduplication history. Writes the primary would have fenced as duplicates may be accepted on the shadow side, particularly after a proxy restart, when the negotiated PID is lost and a fresh one is issued to the shadow cluster. This is a dual-write pattern — do not use it where total accuracy is required. Shadow data is for observability. There will be other safe patterns depending on your topology — the constraint is client visibility, not the number of physical clusters involved. +The upshot is that the proxy can't manage PIDs globally, and the Routing API doesn't try to. No built-in abstraction handles this for you. Whoever writes the router has to work out what their routing topology means for producer identity. diff --git a/_posts/2026-10-12-these-are-not-the-brokers-you-are-connecting-to.md b/_posts/2026-10-12-these-are-not-the-brokers-you-are-connecting-to.md index 7c80c559..222df16c 100644 --- a/_posts/2026-10-12-these-are-not-the-brokers-you-are-connecting-to.md +++ b/_posts/2026-10-12-these-are-not-the-brokers-you-are-connecting-to.md @@ -80,9 +80,9 @@ Every record passes through the routing DAG before it hits the broker. Something I think of stream branching as a railway network, where the routing DAG forms the station network itself. The client's primary stream runs straight down the express mainline. At each station along the line (a router node in the DAG), the stationmaster can inspect passing trains, reassemble carriages (record batches), or dispatch brand new trains down branch lines toward audit or staging clusters — all without delaying express traffic on the mainline. The power of the network is that complexity stays completely self-contained: one station handles PII redaction, the next branches an audit feed, and neither needs to know what happens up or down the line. -Stream branching is fundamentally a dual-write pattern, and dual writes have sharp edges. Kafka logs have no undo button: if a write succeeds on the primary cluster but fails on the secondary, you cannot roll back the primary. Returning an error to the client triggers a retry, creating duplicate batches on the primary; returning success leaves the secondary silently behind. +Stream branching is fundamentally a dual-write pattern, and dual writes have sharp edges. Kafka logs have no undo button: if a write succeeds on the primary cluster but fails on the secondary, you cannot roll back the primary. Returning an error to the client triggers a retry, sending duplicate batches to the primary; returning success leaves the secondary silently behind. -That is why stream branching is best suited for auxiliary streams — shadowing PII-redacted traffic to staging, sampling load, or emitting derived telemetry. If your architecture strictly requires atomic, lossless replication across two physical clusters, an L7 proxy isn't the right tool; you want dedicated replication pipelines like MirrorMaker 2 (or KIP-1279 Cluster Mirroring). The dedicated post on stream branching gets into the gory details. +That is why stream branching is best suited for auxiliary streams — PII-redacted shadows in staging, sampled load, derived telemetry — where losing or duplicating the odd record doesn't matter. If your architecture strictly requires atomic, lossless replication across two physical clusters, an L7 proxy isn't the right tool; you want dedicated replication pipelines like MirrorMaker 2 (or KIP-1279 Cluster Mirroring). The dedicated post on stream branching gets into the gory details. ### Topic weaving