diff --git a/_drafts/routing-post2.md b/_drafts/routing-post2.md new file mode 100644 index 00000000..60cd0582 --- /dev/null +++ b/_drafts/routing-post2.md @@ -0,0 +1,135 @@ +--- +layout: post +title: "Connection Switching: Identity without commitment" +date: 2026-09-11 15:00:00 +1200 +author: "Sam Barker" +author_url: "https://github.com/sambarker" +# noinspection YAMLSchemaValidation +categories: blog kroxylicious-proxy +tags: [ "routing" ] +--- + +[Post 1]({% post_url 2026-10-12-these-are-not-the-brokers-you-are-connecting-to %}) introduced the Routing API and the fundamental shift it brings: a Virtual Kafka Cluster (VKC) is no longer a fixed pipe wired to a single physical cluster on boot. It's a stable identity. What actually lives behind it is now the proxy's problem, not the client's. + +We called the first pattern **Connection Switching**. + +Connection switching makes the smallest possible change to Kafka protocol semantics. A connection to a VKC is still a connection to exactly one physical cluster — no fan-out, no merging, no weaving. The broker on the other end behaves like any Kafka broker, because it is one. What the proxy adds is a single degree of freedom: *which* cluster. Think of it in DNS terms: we transform a hardwired A record into a CNAME. The client has a stable name; what it resolves to is now the proxy's problem, not the client's. + +## What you can do with it (and the dream it sets up) + +### The single front door: Tenant & workload dispatch + +Workload collision is one of Kafka's classic growing pains. When operations eventually wants to separate workloads onto dedicated hardware — high-IOPS brokers for latency-sensitive services, bulk storage for batch analytics, a shared tier for everything else — the old world makes it a multi-week coordination exercise: provision new hardware, issue new bootstrap URLs, cajole twenty application teams into updating their config repositories, and schedule rolling restarts. + +With connection switching, everyone bootstraps at `kafka.corp.internal:9092`. When a client connects, the router inspects the authentication context established during handshake—who is knocking on the door? — and dispatches the connection to the appropriate physical cluster behind the scenes: + +- High-throughput production services get routed to dedicated, high-IOPS NVMe brokers. +- Heavy batch analytics jobs get dispatched to the spinning rust cluster. +- General application traffic stays on the shared tier. + +Now, let's be honest about the boundaries: migrating a producer pumping ephemeral clickstream data to a new cluster is trivial with connection switching—the platform team updates the route, and the producer begins writing to the new cluster with zero config changes and zero downtime. But what about the downstream analytics consumers that need to process that data? If they need continuous historical context, or if data was split across clusters mid-stream, the downstream consumer story is much more complicated. The proxy solves the connection and routing problem, but physical log replication and consumer offset synchronisation remain out-of-band challenges (the migrations section below gets into what that actually requires). Still, for producers emitting fresh streams or teams operating in clean, discrete domains, connection switching eliminates the connection coordination nightmare entirely. + +### Client-change-free cluster migrations and DR failover + +Let’s not pretend this is "seamless"—switching a distributed state machine like Kafka between physical backends always carries operational friction. But traditionally, both planned migrations (e.g. major version bumps, moving to KRaft, cloud region shifts) and unplanned Disaster Recovery (DR) failovers share an expensive logistical bottleneck: changing client endpoints. + +In a conventional setup, redirecting traffic to a secondary cluster or a freshly provisioned "Green" environment means: +- Coordinating with dozens of independent application teams. +- Updating DNS records (with all the painful quirks as DNS propagates, seemingly at random[^1]) or editing hundreds of configuration maps. +- Bouncing client fleets across an agreed maintenance window—or watching an incident drag out while waiting for client restarts. + +Connection switching doesn't solve the hard data replication or consumer offset reconciliation problem, but it **completely eliminates the client-side migration cost**. The bootstrap configuration remains stable (`kafka.corp.internal:9092`). + +When executing a planned migration or a DR failover: + +1. **The data plane is handled out-of-band:** Topics and offsets are synced beforehand via active replication (MirrorMaker 2, KIP-1279 cluster mirroring, or your vendor's equivalent), or recovered from storage volume snapshots. +2. **The connection cutover is a control-plane flip:** The proxy updates its route target to the secondary cluster—either globally or tenant-by-tenant. +3. **Rollback is instant:** If the secondary cluster misbehaves under live load, you can revert the routing target in seconds rather than triggering another round of client config rollbacks. + +Whether you're doing a planned blue/green transition or invoking DR in an emergency, connection switching takes client coordination off the critical path. The data still has to be in sync — that's on you — but at least the cutover itself isn't a multi-team event. + +## How the proxy makes it work + +Making one physical cluster look like a different one at Layer 7 requires more than TCP forwarding. There are three places where the proxy has to do actual work. + +### When is 2 != 2? + +When broker 2 on cluster A and broker 2 on cluster B are both behind the same VKC. + +Kafka identifies brokers by an integer called `NodeId` — not hostnames. A `MetadataResponse` carries a broker list (node ID, host, port, rack) and maps each partition to its leader and replicas by node ID. The client resolves those node IDs to connections itself. That's fine when there's one physical cluster. When you have multiple physical clusters behind a VKC, you immediately run into a collision problem: broker 2 on cluster A and broker 2 on cluster B are completely different machines, but the proxy can only send one broker list to the client — it cannot have two different entries for node ID 2. + +The proxy solves this with a deterministic, stateless mapping applied at every node in the routing DAG. Each router speaks two consistent ID languages: a downstream ID space shared with whatever is below it, and an upstream ID space shared with whatever is above it. Requests flow upstream with IDs translated from downstream to upstream; responses flow downstream with IDs translated from upstream to downstream. The router reads and writes both spaces fluently in both directions. A node can have multiple upstream ID spaces — one per route — all of which must be collapsed into a single, unambiguous downstream space. The mapping is strictly one-to-one in both directions — a bijection, if you want the precise term — so every downstream ID resolves to exactly one upstream ID and route, with no collisions and no ambiguity. No database lookup, no distributed state, just arithmetic: the downstream node ID is the upstream node ID multiplied by the number of routes, plus the route index. For a leaf router, the upstream ID space happens to be the physical IDs coming off the wire from a real broker — but the mapping rule is the same regardless. In theory the upstream could itself be another Kroxylicious instance — the mapping rule is indifferent to what sits above the leaf. The only realistic reason you'd pay that latency tax is Conway's Law: separate teams owning separate proxy tiers with their own trust boundaries and operational responsibilities. The proxy runtime handles the translation at each hop transparently; routers work with opaque `VirtualNode` handles scoped to their own ID space and never need to care about the underlying integer arithmetic. + +``` + ◄──(cluster A IDs)──► [ broker1.cluster-a, broker2.cluster-a, ... ] +[ Client ] ◄──(downstream IDs)──► [ Router ] + ◄──(cluster B IDs)──► [ broker1.cluster-b, broker2.cluster-b, ... ] +``` + +A router could in principle choose which brokers it exposes to the client — but the moment you start hiding brokers, you own the consequences for partition leadership, replica placement, and fetch routing. That's a deep rabbit hole, and the current API doesn't give you a way in: the bijection and topology discovery are handled by the runtime, not the router. + +### Topology discovery happens in-band + +The proxy has no independent view of the Kafka topology. It doesn't poll brokers in the background, run health checks, or hold idle connections open just to keep its metadata warm. Clients and routers drive connections; the proxy learns what it needs to know from the traffic that flows through it. + +Topology state is cached per-router and shared across all connections using that router — so a second connection to the same virtual cluster benefits from what the first one already discovered. The cache uses additive semantics: entries are added as they are learned, never removed. That matters for safety. A low-privilege connection might get a filtered `MetadataResponse` covering only the topics it can see; an additive cache means it can only *contribute* knowledge, never silently evict entries that a higher-privilege connection established. And cache poisoning doesn't translate into an access control bypass either — the cache only influences routing decisions. Brokers enforce ACLs on every request regardless; a misdirected frame gets an error, not someone else's data. + +When a client sends a request to a route whose topology the proxy doesn't know yet — on first connection, or after a stale entry — it discovers it on demand. The proxy synthesises a `MetadataRequest` on the active client channel, uses the response to populate its topology cache, and then forwards the original request as normal. Crucially, discovery inherits the client's authenticated context, so there's no privileged out-of-band management channel to secure or misconfigure. + +--- + +## Proxies aren't magic + +Layer 7 protocol inspection gives you a lot of leverage. What it doesn't give you is the ability to reach inside a Kafka broker and move its internal state somewhere else. The proxy can route frames to a different cluster. It cannot conjure the state those frames depend on. + +### When you flip the target, the state doesn't follow + +If the routing target changes while a client is active: + +- **Transactional producers:** Kafka treats the Transaction Coordinator as an internal role within the cluster. Transactions are therefore scoped to a single cluster by definition — there is no extension point or API through which an external coordinator could be plugged in. A transaction that starts on cluster A cannot be committed on cluster B; if a routing cutover happens mid-transaction, the subsequent `EndTxnRequest` arrives at the new target and gets back `INVALID_TXN_STATE` or `UNKNOWN_PRODUCER_ID`. That abort needs to be handled cleanly and the transactional loop restarted — the same failure mode as a plain broker restart, which well-written transactional producers already handle. A proxy restart itself is safe: the proxy never invents a PID in the single-cluster case, so the KIP-360 fencing fields the client sends on reconnect are the broker's own real values. But if the cluster itself changes — through a failover or routing reconfiguration — the PID the client holds is meaningless to the new cluster. Not fenced; simply unknown. The producer reconnects as a stranger. + +- **Consumer groups:** Offset state lives in `__consumer_offsets` on the physical cluster. The secondary cluster's Group Coordinator has no knowledge of what the primary's consumers have committed. Failing over without a replication pipeline that includes offsets means consumers either re-read data they've already processed, or skip ahead and lose it. Neither is silent. + +- **Idempotent producers:** Producer IDs are issued by a specific cluster and are meaningless to any other. Any time a route targets a new cluster — planned migration, DR failover, or routing reconfiguration — every active producer starts fresh. The new cluster has no deduplication history; retried batches that the old cluster would have silently discarded may be accepted and committed. For producers that care about exactly-once semantics, this is a consideration to plan for, not a surprise to discover in production. + +### Layer 7 can only do so much + +Connection switching works because there is a single physical cluster backing it. The proxy isn't maintaining an illusion — it's directing traffic to something real, and real Kafka semantics apply throughout. The moment a router exposes brokers from multiple physical clusters as a unified address space, it has to ensure those semantics stay valid. For some operations that's fine — routing a `Fetch` to a local replica is cheap and safe. But for anything that touches producer identity or coordination — PIDs, epochs, transaction coordinators, consumer group coordinators — the router has taken on the obligation of upholding guarantees the protocol no longer provides on its behalf. + +The most immediate consequence is that producer identity becomes connection-scoped. The router must maintain a mapping between the client's virtual PID and the real (PID, epoch) on each backend cluster, and that mapping lives with the client connection that negotiated it. Kafka connections are long-lived, but they aren't forever: when one drops, the proxy tears down everything attached to it, mapping included. Reconnecting doesn't help — not even to the same proxy instance — the router re-negotiates, and whatever sequence or epoch continuity the client was relying on is gone. Nor is there only one mapping to lose. A client holds a connection per leader broker, possibly spread across proxy instances that know nothing of each other, so a single client PID can end up fronting several unrelated backend identities. Sticky L4 routing doesn't fix that; the state isn't anywhere a load balancer can point at. + +Friends don't let friends cross that line without knowing exactly what they've signed up for. + +### Cutovers are an explicit decision, not a proxy heuristic + +Because the proxy can't silently catch up coordinator state, automated failover *inside the proxy* is a deliberate non-goal. + +A brief network hiccup and a genuine primary failure look identical from inside the proxy. The cost of getting that call wrong — dropping in-flight transactions, replaying messages, committing offsets against the wrong cluster — isn't the proxy's to bear. That judgement belongs to a human operator or an orchestration playbook that can actually assess cluster health and replication lag. + +Kroxylicious gives you the mechanism to flip the routing target without restarting anything. When to pull that trigger is your call. + +### Offset continuity requires out-of-band replication + +The proxy routes frames; it doesn't replicate log data. If consumers need to pick up where they left off on the secondary cluster, offsets have to line up — which means your replication pipeline ([KIP-1279](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1279%3A+Cluster+Mirroring) or equivalent) needs to be in place and caught up *before* you flip the switch, not after. Flipping the switch is not a substitute for having done that work. + +--- + +## Egress, ingress, or does it matter? + +Post 1 promised we'd call out where each pattern lands on the egress/ingress axis. From a protocol standpoint connection switching is neutral — the mechanics work either side of the wire. In practice, your use case has a strong opinion about placement. + +When talking about DR failover it's obvious once you say it out loud. If the proxy lives in the broker domain — in front of the brokers, a reverse proxy — and the primary site goes down, so does the proxy. Your switch just went boom along with the primary site... The secondary site's ingress proxy was never in the traffic path to begin with; you can't flip a routing target on a proxy your clients weren't talking to. You'd still need to redirect clients to a different address, which is exactly the coordination problem connection switching was supposed to eliminate. + +Thus for DR, the proxy needs to sit on the client side of the link as an egress deployment. The proxy isn't collocated with what failed. It's still in the traffic path. You can flip the routing target, and clients never notice. + +Connection switching also sits quite happily on the ingress side of the fence. If your goal is to give every team in the organisation a single `kafka.corp.internal:9092` and route their connections to the right tenant cluster behind the scenes, the proxy will most likely sit in front of the brokers — the platform team controls that domain and the stable endpoint is theirs to publish. You could run tenant dispatch egress-side, but now you're managing a fleet of client-side proxies, one per team or workload, and keeping them all pointed at the same routing config. That's a lot of instances to babysit for a problem the platform team already owns. + +So what happens when you need to perform a failover of a multi-tenant system? You put a proxy at both ends of the link — egress on the client side, ingress on the broker side. What makes this work cleanly is that each proxy is completely indifferent to what sits behind it: it dials what it's told, speaks Kafka, and has no idea whether the thing on the other end is a broker or another Kroxylicious instance. There's no shared state between them, no coordination protocol, no handshake. That's the pay-off from being a native layer 7 proxy. + +--- + +Connection switching keeps the operational promise simple: platform teams can change the physical backend without the application teams ever knowing it happened. That's not a small thing. + +The next post moves from one-to-one connection switching to fanning out traffic and assembling secondary streams on the fly: **Stream Branching**. + +[^1]: There is always some bit of infra that fails to honour the TTL properly or doesn't understand a CNAME or PTR record. diff --git a/_drafts/routing-post3.md b/_drafts/routing-post3.md new file mode 100644 index 00000000..81dfb85b --- /dev/null +++ b/_drafts/routing-post3.md @@ -0,0 +1,175 @@ +--- +layout: post +title: "Stream Branching: Mind the gap" +date: 2026-10-26 15:00:00 +1200 +author: "Sam Barker" +author_url: "https://github.com/sambarker" +# noinspection YAMLSchemaValidation +categories: blog kroxylicious-proxy +tags: [ "routing" ] +--- + +[Post 2](/TODO-post2-url) established the station network: the routing DAG is a network of stations, the client's stream is the express running down the mainline, and at each station the stationmaster can inspect passing trains. Stream branching is what the stationmaster does next. There are two flavours. In the *substitution* case, the train is quietly re-routed to a different platform — the original destination never sees it. In the *additive* case, the express continues to its primary destination while new trains depart down branch lines. The first is simple. The second is where the sharp edges are, and most of this post is about those edges. + +## The simplest branch: Replica routing + +If you run Kafka across multiple Availability Zones, you've probably already had this conversation with your finance team. You opened the monthly bill, choked on the cross-AZ data transfer line item, and dutifully turned on Kafka's built-in follower fetching (`client.rack`). The bill plummeted. You closed the Jira ticket, posted a triumphant graph in `#infra-cost-savings`, and moved on. + +Then six months later finance knocks on your door again. The cross-AZ tax has silently crept right back up. + +Why? Because relying on every client in the company to configure rack awareness is an unwinnable game of whack-a-mole: +- Someone copy-pastes a Helm chart that hardcodes `client.rack=eu-west-1a` into a pod deployed in `eu-west-1b`. +- A data science team spins up a fleet of Python consumers using a library where nobody realised follower fetching requires a specific flag. +- A new microservice goes live with the defaults, happily pulling multi-gigabyte streams across the AZ boundary from whatever partition leader happens to be awake. + +Centralised problems need centralised controls. But there's a subtler obstacle. Even if a filter could see that a particular `FetchRequest` belongs in `eu-west-1a`, knowing that isn't enough — the pipe is already nailed to broker 1 in `eu-west-1c`. The filter has nowhere to send it. The Routing API allows you to fix this: the backend connection is no longer fixed at connection time, it can be chosen per request. + +A `FetchRequest` payload just says "partition 3, from offset N". The broker ID is encoded by which connection the client sends the request on — that's the seam a custom router can exploit. When the `FetchRequest` arrives on the virtual broker 1 connection, the router inspects the partition's replica set, sees that broker 3 is in the same AZ and fully caught up, and tells the runtime to fulfil the request down a separate backend socket to broker 3 instead: + +``` +[ Consumer (AZ-a) ] ──Fetch──► [ Kroxylicious (AZ-a) ] ────► [ Broker 3 (AZ-a, Replica) ] + │ + (Skips AZ tax) + ▼ + [ Broker 1 (AZ-c, Leader) ] ← never sees the request +``` + +The application didn't configure `client.rack`. The request payload is completely untouched. Broker 1 never receives the order, and the cross-AZ charge never reaches the invoice. What made that possible: the proxy is holding two backend sockets, one to the leader and one to the local replica, and choosing between them per request. To the client, that's still one broker on one cluster. + +The same per-request dispatch mechanism opens up other possibilities beyond proximity — your custom Bugatti router from post 1 could consider: +- **Replica lag awareness:** Steering reads to the most caught-up follower to avoid trailing-edge staleness. +- **Broker load shedding:** Diverting fetch traffic away from CPU- or IO-bound leaders to give them headroom for critical writes. +- **Tiered storage offloading:** Routing historical backfills to brokers backed by cheap object storage while keeping hot tail-reads on fast NVMe. + +Whether those prove practical in production remains to be seen — they require telemetry the router might not easily have — but proximity routing alone solves a very real, very expensive problem right now. + +One constraint the dispatcher can't escape: **proxy placement**. If the proxy is in `eu-west-1b` and the consumer is in `eu-west-1a`, pulling from a broker in `eu-west-1a` still costs a cross-AZ hop to reach the dispatcher first. For replica routing to pay off, the proxy fleet must be co-located with the clients it fronts. + +--- + +## The express doesn't wait + +Stream branching has one rule, and the rest of this post follows from it: + +> **The mainline is the source of truth. Branch lines are best effort.** + +The express never waits at a junction for a branch-line train. The client produces a batch. The router sends it to the primary cluster and, at the same time, sends zero or more batches down branch lines to other topics, other clusters, or both. Those can be copies, redacted copies, samples, or new records derived from the payload. The client gets back the primary's `ProduceResponse` and nothing else. Branch writes never add latency to that acknowledgement, never turn a success into an error, and never turn an error into a success. + +``` + ┌──(mainline)──► [ Primary cluster ] ──ack──┐ +[ Producer ] ──Produce──► [ Router ] │ + ▲ ├──(branch)────► [ Staging cluster ] (redacted, sampled) + │ └──(branch)────► [ audit-events topic ] (derived) + └──────────────── ProduceResponse (primary's verdict only) ◄───────────────┘ +``` + +The branch writes belong to the router, not the client. The client didn't send them and never sees their responses. The router opens its own connections to the branch targets, sends its own `ProduceRequest`s, and decides what to do when they fail. + +This rule is a design choice, not a missing feature. The obvious alternative is to hold the acknowledgement until every branch has landed. That doesn't make the writes atomic. It just ties production to your least reliable cluster, as [Mind the gap](#mind-the-gap) explains. + +## What it costs to stop at the station + +The cheapest branch is a faithful copy: same records, same batches, sent on to a different place. The router can pass the record batch bytes through with only the header changed (more on which fields later). It never has to decompress anything. + +Once the stationmaster needs to look *inside* the carriages, the cost goes up a lot. Redacting PII, deriving an audit record from the payload, or filtering individual records all mean decompressing the batch, deserialising records, transforming them, then re-serialising and recompressing. On a busy topic with `zstd` batches that's real CPU, and it happens inline on the proxy for every batch that takes the branch. Size your proxy fleet for the most expensive station on the line, not for the mainline. + +Sampling sits somewhere in the middle. Dropping whole batches is close to free. Dropping individual records means rewriting batches. Which one you want depends on whether "10% of the traffic" means 10% of requests or 10% of records. For load testing it's usually the first, which is lucky. + +Keep it stateless. Every station handles each batch on its own. Joins, windows, and aggregations over time need state that survives restarts, rebalances, and late data, and Kafka Streams and Kafka Connect exist to manage exactly that. A router that starts accumulating state across batches has become a badly built stream processor. + +## Mind the gap + +Here's what you're signing up for. + +### There is no undo + +A Kafka log only ever appends. When a batch lands on one cluster and fails on the other, nothing can take the successful write back. That leaves four outcomes for any batch: + +| Primary | Branch | What the client sees | What the branch ends up with | +|---------|---------|----------------------|------------------------------| +| ✅ | ✅ | Success | A copy. The happy path. | +| ✅ | ❌ | Success | A hole | +| ❌ | ✅ | Error, so it retries | A copy now, and another one after the retry | +| ❌ | ❌ | Error, so it retries | Whatever the retry produces | + +The second row is the one people expect: the branch falls behind and nobody tells the client. The third row is the one that catches people out. The primary returned an error (or the ack timed out), but the branch write already went through. The client retries. The primary deduplicates the retry if the first attempt actually landed, because that's what idempotent producers are for. The branch, though, receives the retried batch as a fresh write from the router. Unless the router has done the extra work described below, the branch now has a duplicate the primary doesn't. + +So in normal running the branch both loses data and duplicates it, independently of the primary. If that sentence makes you uncomfortable, stream branching is the wrong tool for that particular job. That isn't a criticism. It's the reason post 1 sent atomic, lossless replication off to MirrorMaker 2 or [KIP-1279](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1279%3A+Cluster+Mirroring) cluster mirroring. + +### Waiting for the branch makes the mainline only as reliable as the branch + +The obvious way to close the gap is to hold the client's acknowledgement until both writes succeed. It doesn't work. + +If the router waits for the branch, the client's produce latency becomes the slower of the two clusters, and the staging cluster you sized on a budget sets the latency of production. If the router *fails* the client whenever the branch fails, an outage on staging becomes an outage on production. You've wired production's reliability to your least reliable cluster. Even if you accept both of those, you still haven't got atomicity: a branch write can succeed while the primary times out, and you're back in row three of the table. + +A branch must never back up onto the mainline. Which leads to the next problem. + +### Backpressure: drop or block? + +The primary is healthy and the staging cluster is struggling. Branch writes start queueing in the proxy. That queue has to have a limit, because proxy memory isn't infinite and running out of it takes the mainline down as well. When the queue is full, the router has two options: + +- **Drop.** Throw away branch writes until the branch catches up. The mainline is unaffected and the branch gets a hole. This is the right default for nearly every branching use case, and it's what "best effort" means in practice. +- **Block.** Stop reading from the client until the branch drains. The branch stays complete, and the mainline slows down to match the branch. Now you're coupled again, just more slowly. + +Whichever one you choose, make the drops visible. A branch that loses data silently is one you'll eventually trust when you shouldn't. Count what was dropped, per branch, and alert on it, so that when someone asks whether staging saw the batch that triggered the bug, you can actually answer. + +### Partitions don't have to line up + +The client's partitioner picked a partition based on the *primary* topic's partition count. The branch topic might have a different number of partitions, or might not have partitions that mean the same thing. If the branch is a faithful shadow with the same partition count, partition *n* maps to partition *n* and per-key ordering survives. If not, the router is repartitioning. It has to choose a branch partition itself, which means either re-hashing the key or making some other decision, and the ordering guarantees on the branch are whatever that decision gives you. Derived streams, such as an audit feed keyed by user rather than by order, almost always fall into the second case. That's fine as long as you meant it. + +### Duplicates on the branch + +Kafka producers use idempotent delivery by default. The broker deduplicates retried writes, so a message is committed exactly once even if the network drops the acknowledgement and the producer retries. This depends on a Producer ID (`PID`) that the broker assigns at initialisation, plus a per-partition sequence number on every batch. If the broker sees a batch it has already committed (same PID, same sequence), it quietly discards the duplicate. The client never needs to know. + +**The mainline PID belongs to the client, and it is never translated.** The client negotiated its PID with the primary cluster, and the router passes it through unchanged, exactly as in connection switching. Every write that carries the client's identity goes to the cluster that issued it. Nothing to map, nothing to invent, nothing to lose. Not needing to translate it is a large part of what separates branching from weaving. Once one client identity has to be valid on several clusters at once, the proxy is translating something far less forgiving than a node ID. That's a problem for the weaving posts. + +**Branch PIDs belong to the router.** The branch cluster's identity is something the router created. The client doesn't know the branch cluster exists, and nothing downstream will ever try to resume or share that identity. The router sends its own `InitProducerIdRequest` to the branch cluster, rewrites the producer ID and epoch in each record batch header before sending it down the branch, and owns that identity from start to finish. This is safe *because* the client doesn't know about it: an identity the client can't see is one the client can't collide with. (Those header fields are covered by the batch CRC, so rewriting them means recomputing the checksum. It's cheap compared with decompressing anything, but it isn't free.) + +That ownership settles who is talking to the branch. It doesn't settle what the branch deduplicates, which brings back row three of the table. There are two ways to handle sequence numbers: + +- **Pass the client's sequence numbers through** under the router's PID. If the branch receives exactly the same batches as the primary, in the same order, from the start of the session, then a client retry carries the same sequence number to both clusters and *both* deduplicate it. That closes row three. It only works as long as the branch is a faithful copy. Sample a batch away, drop one under backpressure, or lose one to a branch error, and the branch sees a gap in the sequence and rejects everything after it with `OUT_OF_ORDER_SEQUENCE_NUMBER`. The router then has to notice, bump its epoch on the branch, and start again, and the batches in flight during that recovery are gone. +- **Assign the router's own sequence numbers.** The router numbers branch batches contiguously however it likes, so sampling, dropping, and derived streams all just work. But the branch can no longer recognise the client's retries, because each one is a new batch from the router. Row three now produces duplicates on the branch every time. + +Branches that only sample or derive have to use the second approach. Faithful shadows can try the first, as long as you accept that every gap forces a reset, and that it only helps while each partition's traffic stays on one client connection (more on that next). In both cases the branch's deduplication history is separate from the primary's, and writes the primary would have discarded as duplicates may be accepted on the branch. + +**The branch identity belongs to a connection, not to the client.** The router's branch PID only lasts as long as the client connection it was negotiated for. When that connection closes, the proxy tears down everything attached to it, branch identity included. And the client has one PID but several connections: a producer opens one to each broker that leads a partition it writes to, and in a horizontally scaled proxy fleet those connections can land on different proxy instances. The routers serving those connections have no way to coordinate, so each one negotiates its own PID with the branch cluster. One client identity on the primary becomes several unrelated identities on the branch. + +That's harmless while each partition's traffic stays on one connection. It stops being harmless when the traffic moves, and in Kafka it moves routinely: + +- **Leadership changes.** A partition's leader moves from broker 1 to broker 2, so the client starts sending that partition's batches down a different connection, possibly through a different proxy instance. It carries on with the same PID and the next sequence number. On the primary, that's an ordinary continuation. On the branch, the batches arrive under a PID that has never written to that partition. Whatever sequence numbers the router uses, the branch has no history for that PID, so it can't recognise any batch the client retries across the move. +- **Reconnects.** A network blip, an idle timeout, a broker restart, a proxy restart: anything that drops the client's connection drops the branch identity with it. The client reconnects and resends its in-flight batches, some of which had already reached the branch under the old PID. They arrive under a new PID, and the branch accepts them all as new writes. Reconnecting to the same proxy instance doesn't help, because the old connection's state is already gone. + +Both are the same failure. The client's retries for a partition move from one branch identity to another, and the branch can only deduplicate within a single identity. + +The mainline is unaffected. Its PID belongs to the client, which holds it across every connection, every proxy instance, and every reconnect, so the primary recognises the retries and discards the ones it has already committed. That's the asymmetry working in your favour: the identity that matters is held by the one party that sees all of the client's traffic. But leadership changes, reconnects, and rolling upgrades are routine, so duplicates on the branch are a normal part of running stream branching, not an edge case. + +Use shadow data for observability, not for accounting. If someone wants to reconcile the branch against the primary to the last record, the answer is replication, not branching. + +### Transactions don't branch + +If the client is a transactional producer, the batches it produces are marked transactional, and they're only meaningful together with the `AddPartitionsToTxn` and `EndTxn` calls the client sends to the *primary's* transaction coordinator. The commit and abort markers that tell consumers which records count aren't produced by the client at all. The primary's coordinator writes them. The router never sees them in the produce stream, so it can't copy them. + +That leaves the router with two bad options for a transactional branch: + +- **Strip the transactional flag** and write the branch batches as plain idempotent writes. They're visible on the branch straight away, *including records from transactions the primary later aborts*. A `read_committed` consumer on the branch will read data that, as far as the primary is concerned, never existed. +- **Run a shadow transaction** on the branch: the router starts its own transaction against the branch cluster's coordinator, adds partitions as the client does, and commits or aborts when the client's `EndTxn` succeeds or fails on the primary. That's a lot of protocol to mirror, and the two commits are still separate. The primary can commit while the branch aborts, and nothing coordinates the two. Post 1 put it bluntly and it still holds: the Kafka protocol has no cross-cluster transaction coordinator, and routing can't create one. + +For shadowing and sampling, stripping the flag and documenting that the branch may contain aborted data is usually the honest choice. For derived streams that feed something financial, don't branch from a transactional producer at all. + +## Know when to get off the train + +Stream branching is a great tool for a specific set of problems, and it's easy to use it for the wrong ones. A rough guide: + +- **You want production-shaped traffic somewhere safe, and losing some is fine.** Branch it. Redact it on the way, sample it if staging is small, and watch the drop counters. +- **You want a stream nobody produces yet, derived statelessly from one that already exists.** Branch it. That's the case that makes the proxy worth running: no application team has to deploy anything. +- **You need the derived stream to be stateful**, such as joins, windows, sessionisation, or deduplication across batches. Use Kafka Streams or a Connect pipeline that reads from the primary. They manage state. A proxy shouldn't. +- **You need a complete, ordered, offset-consistent copy of the primary on another cluster.** Use replication: MirrorMaker 2, KIP-1279 cluster mirroring, or your vendor's equivalent. They read the primary's committed log after the fact, so they can't have the asymmetric failures described above. +- **You need writes to two clusters to be atomic.** Nothing at layer 7 can give you that. Redesign so the writes share a cluster. + +The pattern works because it's honest about having two tiers: one stream that carries the guarantee and some that don't. Keep that distinction clear in your config, your dashboards, and your conversations with whoever consumes the branch, and stream branching lets you build streams nobody had to code for. Blur it, and sooner or later someone will be surprised by a gap you could have warned them about. + +--- + +Both flow patterns are now covered. Connection switching picks the track. Stream branching adds branch lines. Next we leave the railway for the sorting office: **topic weaving**, where the router no longer just directs streams but decides which topics exist, where they live, and what they're called. Producer identity comes up again there, and this time the client *does* know about every cluster involved. + +If you're branching streams already, or want to and have hit a gap this post didn't cover, find us on [GitHub](https://github.com/kroxylicious/kroxylicious), [Slack](https://kroxylicious.slack.com), or [Bluesky](https://bsky.app/profile/kroxylicious.io). diff --git a/_drafts/stash-replica-routing.md b/_drafts/stash-replica-routing.md new file mode 100644 index 00000000..ff7ae6b0 --- /dev/null +++ b/_drafts/stash-replica-routing.md @@ -0,0 +1,60 @@ +# Stash: replica routing pattern + +Removed from `routing-post1.md` — to be picked up in post 2 (cluster aliasing) as an intra-cluster optimisation. + +## Why it's not a peer pattern + +The other three patterns (cluster aliasing, union clusters, virtual topics) all answer "which cluster/topic does this request go to". Replica routing answers a different question: "given we've already decided which cluster, which broker within it serves this fetch". It's a different level of abstraction, and it only meaningfully applies to `Fetch` requests — produce requires the leader, and most admin operations that are partition-aware could use it but it's not the primary motivation. + +## The pattern + +- Client is connected to a VKC backed by, say, the DE cluster +- Client sends a `Fetch` for a partition whose leader is broker 1 in `eu-west-1c` +- The proxy is sitting in `eu-west-1a`, where broker 3 holds a valid in-sync replica of that partition +- Rather than forwarding down the existing pipe to broker 1, the proxy routes the request down a separate connection to broker 3 +- Broker 1 never sees the request. The client never knows there was a choice. + +The key implementation detail: the proxy doesn't rewrite the request and send it to a different address on the same connection. It has a separate backend connection to broker 3 and routes the `Fetch` down that pipe instead. The request itself is unchanged — what changes is which pipe it travels down. + +## Why this needs Layer 7 + +A Layer 4 proxy sees "connection to broker 1" and can only forward or drop. A Layer 7 proxy reads the `Fetch` request, understands which partition is being requested, knows the replica set and where each replica sits, and can choose which backend connection to use. The routing decision happens at the Kafka protocol level, not the TCP level. + +## Relationship to Kafka's built-in rack awareness + +This is a complement to rack awareness, not a replacement. `client.rack` + follower fetching works — when clients set it, and set it correctly. Replica routing extends the same idea: the proxy knows its own position in the topology and can apply locality policy centrally, covering clients that haven't configured rack awareness without requiring them to change anything. + +## Naming considered + +- **Topology-aware routing** — too tied to the implementation word "topology", was the original name +- **Locality routing** — too narrow, implies proximity is the only policy +- **Broker aliasing** — implies a static mapping, but the choice is dynamic +- **Replica routing** — landed here: names the thing being chosen between (replicas), leaves the policy open (proximity, lag, load, cost), intra-cluster by definition + +## Other policies beyond proximity + +- Replica lag — prefer the most caught-up replica +- Load — prefer the least busy broker +- Cost — prefer replicas on cheaper storage tiers + +None of these are proximity, which is why the name shouldn't bake in "locality". + +## Failure mode + +Proxy placement. A client in `us-east-1a` hitting a proxy in `us-east-1b` that then routes to a broker in `us-east-1a` adds a cross-AZ hop instead of eliminating one. The proxy fleet has to be co-located with clients for locality savings to materialise. + +--- + +# Stash: why PIDs aren't node IDs (for the topic weaving post) + +Cut from the stream branching post. Branching never translates the client's PID: the main line passes it through and branch PIDs are router-owned, so this argument isn't needed there. It belongs in weaving, where one client identity has to be valid on several clusters at once. + +When you first look at PID management through the proxy, it looks like a problem we've already solved. [Post 2](/TODO-post2-url#when-is-2--2) introduced a bijection for node IDs: a deterministic, stateless mapping between downstream and upstream IDs at each router. PID translation looks like the same problem, and the similarities are convincing. Both are integers. Both are cluster-scoped. Both are assigned by the cluster, not chosen by the client. Both are opaque values the client just carries around and hands back. So the proxy can keep a downstream-to-upstream mapping for both, right? That's the obvious answer, and each problem with it looks fixable on its own until you've fixed several and the whole thing still doesn't hold together. The reason is that although node IDs and producer IDs look identical on the wire, they mean completely different things. A node ID is **spatial**: a label, a coordinate. A PID is **temporal**: not just a label but a claim to be a particular live producer. We can fake a node ID with arithmetic because it's only an address. We have to track a producer ID with state because the broker enforces it as an identity, and fences anyone who presents a stale one. + +**Node ID translation is transport-scoped.** A `MetadataResponse` tells the client where the brokers are. The proxy can lie as much as it likes here, because the broker never sees the lies, let alone challenges them. The proxy can also forget: if it throws away its mapping and rebuilds it from the next `MetadataResponse`, it tells the same lies again. Nothing notices and nothing breaks. + +**PID allocation is negotiation-scoped.** Since Kafka 3.0, producers run in idempotent mode by default. To support that, the producer sends `InitProducerIdRequest` to ask the broker for an identity: a (PID, epoch) pair. The broker tracks that identity and uses it to deduplicate retried batches during that session. It will also evict any connection that presents the same PID with a lower epoch. That's the fencing mechanism, and it stops two connections claiming the same identity at once. If you need the identity to survive across sessions, so that a restarted producer picks up where it left off, that's what `transactional.id` is for. A second `InitProducerIdRequest` with the same `transactional.id` is deliberately treated as a hostile takeover: fencing is how Kafka makes sure only one producer instance is ever active for a given transactional identity. + +Node IDs and PIDs are both cluster-scoped: issued by one cluster, honoured within it, meaningless outside it. But the node ID mapping is stateless. If the proxy loses it, the proxy rediscovers and recomputes it. A PID can't be recovered that way. Once the negotiation is lost, so is the identity. + +The upshot is that the proxy can't manage PIDs globally, and the Routing API doesn't try to. No built-in abstraction handles this for you. Whoever writes the router has to work out what their routing topology means for producer identity. diff --git a/_posts/2026-10-12-these-are-not-the-brokers-you-are-connecting-to.md b/_posts/2026-10-12-these-are-not-the-brokers-you-are-connecting-to.md new file mode 100644 index 00000000..222df16c --- /dev/null +++ b/_posts/2026-10-12-these-are-not-the-brokers-you-are-connecting-to.md @@ -0,0 +1,121 @@ +--- +layout: post +title: "These are not the brokers you are connecting to" +date: 2026-10-12 15:00:00 +1200 +author: "Sam Barker" +author_url: "https://github.com/sambarker" +# noinspection YAMLSchemaValidation +categories: blog kroxylicious-proxy +tags: [ "routing" ] +--- + +Back in May, Tom wrote about [a proof of concept for routing]({% post_url 2026-05-21-topic-routing %}) — Kafka clients producing and consuming from topics scattered across multiple clusters, with no idea that's what they're doing. The proxy handles the fan-out, the response merging, the session state. The client just sees topics. + +That POC has been quietly maturing. We're now in the process of turning it into something you can actually run in production, and in a few weeks we'll be talking about it at [Current in San Francisco](https://current.confluent.io/san-francisco/sessions#session-SESS-165). In a session called *"These are not the brokers you are connecting to"* — my employer does not condone lying, but the proxy will quite happily tell your clients whatever you need it to. + +This post is the first in a short series leading up to that talk. The goal here is to establish some vocabulary and get into the gory details I won't have time for onstage. I've been thinking about the design space routing enables and I'm convinced there are a handful of distinct patterns worth naming separately. There are no hard boundaries between the patterns, they all leverage the same underlying infrastructure, but they are distinct because they trade off different aspects of the problem space. The names are mine and while I'll happily accept better ones, we need something to start a conversation (plus I'm right 😉). + +The deeper dives on each pattern will follow in subsequent posts. + +## Pick your poison + +If you've tried to migrate a live Kafka workload between clusters, consolidate regional deployments, or shift cloud providers, you will have run into the same core problem: Kafka is not HTTP-based (stop rolling your eyes at the back). Kafka clients are genuinely indifferent to geography — they'll talk to a broker in the same pod just as happily as one in Timbuktu. What they care about, deeply and lastingly, is *which* broker they're talking to. Broker 1 is Broker 1. You can't quietly swap it out, move it between availability zones, or retire it without a lot of fuss. The load balancer doesn't get a say. Most proxies can only manage Kafka as a layer 4 protocol — and layer 4 is BOOOOORRRRING (I will not listen to arguments from the network engineer in the corner). + +So when you need to reshape the underlying infrastructure without breaking application teams, you usually end up picking which operational headache you dislike the least. + +**Application-level dual writes** look clean in architecture diagrams. In production, network blips cause asymmetric failures, message ordering drifts, and duplicate delivery becomes every application team's problem. Getting twenty teams to deploy matching code changes on the same timeline is an exercise in cat-herding that usually ends with someone's Friday afternoon becoming someone else's out of hours page. + +**Replication pipelines** (MirrorMaker 2, et al.) work great for asynchronous backup, but for live consolidation knowing whether you're in sync is a very difficult question, not to mention doubling storage and network costs, and asking consumers to deal with offset translation. You're running twice as much hardware to move bytes you already owned, fine if the migration ever actually ends. + +**Maintenance windows** are conceptually simple right up until a legacy service ignores DNS TTLs, holds onto stale sockets, and drops messages silently when the old brokers finally go dark. Sunday 2am is when the core reporting pipeline splutters to a halt. + +## What a Layer 7 proxy buys you + +Jumping up to Layer 7 changes the game. Rather than blindly punting TCP bytes around, the proxy speaks Kafka. It knows a `Metadata` response from a `Produce` request, and it can answer a client's question about where the brokers live with whatever answer you need. + +From day one, Kroxylicious gave you a Virtual Kafka Cluster (VKC) — a stable endpoint for your clients. But until recently, it was an opinionated pipe. That was a compliment: it did real work. It could encrypt records, enforce auth, rewrite headers — all without the client noticing. But it was still a fixed pipe: one client socket in, one backend socket out, wired to a broker on whatever cluster you pointed it at on boot. The filters were stateless with respect to the broker topology: an encryption filter might talk out-of-band to a KMS for keys, but it didn't care which physical cluster lived behind the socket. + +Think of it like the horse in the late 19th century. A horse gets you from A to B across familiar ground without needing a national highway project. You feed it, you mount it, and you go. Horses were great. + +[Proposal 70](https://github.com/kroxylicious/design/blob/main/proposals/070-routing-api.md), the Routing API, is our combustion engine. The filters haven't gone anywhere — encryption, auth enforcement, and header rewriting all ride along inside the car. But the platform underneath can now make routing decisions on the fly. As a project, we provide the API so *any* car is possible, but we only supply the Model T — apocryphally in any colour you like, so long as it's black. If you need a bespoke Bentley or Bugatti router, you can build it. + +The trade-off is the same one history made. A horse rider and a car driver both pick their own destination. But while the horse is happy on grass, the car only delivers that speed and range if there are roads, fuel stops, and mechanics. + +Routing is the exact same trade: more range, more power, but you depend on the world outside the proxy. Kroxylicious can't and shouldn't make your business policy decisions — it doesn't know which tenant belongs on which tier or when a cluster swap is safe. Those decisions belong to you. What the proxy gives you is the steering wheel, the throttle, and a much bigger map. + +We get it: running a proxy is operational overhead. We wish swapping a CNAME was enough as much as you do — but to a Kafka client, a CNAME flip is an unexpected divorce. That's why we're all here. + +## Name them we must + +After reviewing the [routing design proposal](https://github.com/kroxylicious/design/pull/70) and thinking through the problem space, I think there are four distinct deployment patterns worth naming: **connection switching**, **stream branching**, **topic weaving**, and **partition weaving**. As with all patterns the boundaries are fuzzy and often more than one applies at once. The thing that convinces me they're real is that they have descriptive power — each one has its own tradeoffs, failure modes, and a distinct place on the spectrum from the clear(ish) waters of connection switching to the mangrove swamp of partition weaving. Like all good abstractions, they're useful. + +You can deploy the proxy on either side of the network boundary: helping traffic egress from the client domain (forward proxy territory), or managing ingress into the broker domain (reverse proxy territory). I prefer *egress* and *ingress* because they describe traffic flows from your perspective rather than abstract networking theory. Some patterns land far more naturally on one side than the other, and we'll call that out as we go, but the underlying mechanics are the same. + +If you're reaching for the [Enterprise Integration Patterns](https://www.enterpriseintegrationpatterns.com/patterns/messaging/MessageRoutingIntro.html) book right now — well done for being as old as I am, the rest of you just Googled it. It describes what happens to individual messages, which is interesting, but I'm talking about whole streams. Those patterns are what makes all this possible under the hood; what I'm describing sits a layer or two higher. + +The patterns build on each other along a single axis: how deep into the Kafka protocol the router has to reach. The first two — **connection switching** and **stream branching** — are flow patterns. Think of them like a railway network: you're directing moving trains, either throwing points at the junction before departure or dispatching carriages down branch lines as the train passes through. The second two — **topic weaving** and **partition weaving** — are topology patterns, where you step off the track to stitch discrete streams into a single logical map. + +Connection switching operates at the connection level — which broker, which forms part of which cluster, is this connection actually being sent to? Stream branching works at the stream level — where does this request go? Topic weaving reaches into the topic catalog — which topics are visible and where do they live? Partition weaving goes deepest — how do physical partition spaces map onto a logical one? Each step requires the router(s) to understand more of the protocol than the last. But the complexity doesn't compound as routing is managed through a DAG which allows them to be composed rather than monolithic. A router handling topic visibility doesn't care whether the partitions behind it are woven from multiple clusters. A router rewriting partition numbers doesn't care whether the topic catalog above it spans two physical clusters or twenty. + + +### Connection switching + +Until recently a VKC provided a 1:1 mapping between the thing the client addressed and the thing the proxy dialled. The routing API breaks that constraint — one VKC can now connect to one or more physical backends, and the target isn't fixed. The VKC is stable; what's behind it doesn't have to be. + +Every organisation has a cage it can't close. The CFO wants it gone, but somewhere upstream is "the Thing" nobody dares touch that still talks to the Kafka cluster inside it. The cluster is full. The cage is full. You need more capacity, and you can't get it here. + +Before routing, that means updating `bootstrap.servers` in half a million places — and crucially, touching the Thing. + +With connection switching, you leave a Kroxylicious instance in the cage and point it at a new cluster wherever you actually have headroom. Every client still connects to the same address. The proxy dials the new cluster behind the scenes. When you're ready, update the route. When the cage is empty, tell the CFO. Nobody had to touch the Thing. + +Before routing, Kroxylicious was a pipe with opinions — it could inspect and rewrite the stream, but it was inherently connection-oriented. One client socket, one backend, straight through. Connection switching breaks the static part of that: the backend can now change. The routing decision is made once at connection time — the points are thrown before the train departs, and after that it's a straight run to the destination. It's still 1:1; the interesting thing is that the 1 on the right side is now chosen, not fixed. + +### Stream branching + +You're chasing an issue in the reporting pipeline that only shows up in the sixth hour of the run and nobody can figure out which record trips it up. You've been there, right? What you really want is access to the live data with a debugger. Alas, this job is stuffed full of Personally Identifiable Information — so you're out of luck. You've been there and got *that* t-shirt. What now? You build a router that shadows production traffic to a staging cluster, piping it through a redaction filter on the way so the PII becomes gibberish before it touches staging brokers. The client is still producing to one VKC. But the proxy is now opening *two* backend connections — one to the primary cluster, one to staging — and writing every record to both. The acknowledgement the client gets back is from the primary; the shadow write happens on the side. One stream in, two streams out. If you've got your EIP book off the shelf already, yes — it's a Wire Tap with a Message Translator on the diverted stream. + +That's one flavour of stream branching — the secondary stream is a copy of the primary, diverted somewhere and the client never realises. But the diverted branch doesn't have to be a copy. All staging clusters are capable of handling production load, aren't they? No? Ohh right, you have budgets. We still want to stage some real traffic, but we can't handle the full volume. A filter on the staging route drops 90% of the produce requests it sees and forwards the rest — we can do that as nothing cares about the acks. + +Every record passes through the routing DAG before it hits the broker. Something in that DAG can inspect the payload, derive something from it — an audit log your compliance team needs, record telemetry for downstream aggregation, or an anonymised event feed — and emit that derived data to a topic anywhere the routing config points — a different topic on the same cluster, or a cluster the client has never heard of. The primary stream is untouched. The additional streams just appear, derived from the original client's activity. The clients don't change. Which means anything that previously required every application to cooperate — the audit trail, the telemetry, the thing all but one of your clients never supported — you can now just invent that stream yourself. This all works for stateless transformations performed inline. If you need state, joins, or windowing, Kafka Streams or Kafka Connect have your back. + +I think of stream branching as a railway network, where the routing DAG forms the station network itself. The client's primary stream runs straight down the express mainline. At each station along the line (a router node in the DAG), the stationmaster can inspect passing trains, reassemble carriages (record batches), or dispatch brand new trains down branch lines toward audit or staging clusters — all without delaying express traffic on the mainline. The power of the network is that complexity stays completely self-contained: one station handles PII redaction, the next branches an audit feed, and neither needs to know what happens up or down the line. + +Stream branching is fundamentally a dual-write pattern, and dual writes have sharp edges. Kafka logs have no undo button: if a write succeeds on the primary cluster but fails on the secondary, you cannot roll back the primary. Returning an error to the client triggers a retry, sending duplicate batches to the primary; returning success leaves the secondary silently behind. + +That is why stream branching is best suited for auxiliary streams — PII-redacted shadows in staging, sampled load, derived telemetry — where losing or duplicating the odd record doesn't matter. If your architecture strictly requires atomic, lossless replication across two physical clusters, an L7 proxy isn't the right tool; you want dedicated replication pipelines like MirrorMaker 2 (or KIP-1279 Cluster Mirroring). The dedicated post on stream branching gets into the gory details. + +### Topic weaving + +Your New York application team wants to consume `london_orders`, `tokyo_orders`, and `newyork_orders`. Those topics live on three different clusters — one in each region, provisioned by three different teams at three different points in time. Without topic weaving, the NY team needs bootstrap addresses for all three clusters, separate client configurations for all three, and the operational joy of managing credentials and failover for all three. With topic weaving, you point them at one VKC. The routing layer stitches the backing clusters together into a single logical view. The client asks for metadata and gets back a broker list that includes all brokers from all regions as part of the same cluster. As far as the client believes, they all live in its local cluster. + +That's the common case: different topics, different clusters, one coherent view for the client. Your router is what holds the map — which topic lives where, which broker list to return, what gets dispatched where. + +There's a subtler case. What if two of your clusters both host a topic called `orders`? Think of it like a sorting office: you write *George Street* on the parcel and drop it at the counter. The sorting office decides whether you mean George Street, Edinburgh or George Street, Dunedin — a city in New Zealand that Scottish settlers named after Edinburgh, gave the same street names, and promptly scrambled the layout. (Guess why I know.) The sender doesn't need to know both exist. Neither does your client. Because the proxy controls what gets returned in a metadata response, your router decides which topics are visible and under what names — which means in principle it can arbitrate which names reach clients, or whether the duplicate surfaces at all. + +You look after a gaggle of Kafka clusters. You know how they got there. You're not proud of all of them. You can't herd them — geese are worse than cats, they honk back — but you don't have to admit to anyone else how many geese there actually are, or what they're called. + +Two things to keep in mind. Consumer group co-ordination stays pinned to physical clusters — that's almost always fine, because a group's offset state is tied to the topic-partitions it's consuming and those live on one backend, but it's worth knowing the seam is there. And transactions don't cross cluster boundaries: a transaction that writes to topics on two different backing clusters is two independent transactions, and there is no way to make it otherwise — the Kafka protocol has no cross-cluster transaction coordinator and routing can't invent one. If cross-topic atomicity matters, those topics need to share a physical cluster. + +### Partition weaving + +Topic weaving weaves (obviously 😉) whole topics together to present a unified catalog. Which invites the natural question: why stop at the topic catalog? Kafka's real unit of parallelism and storage is the partition. Can we weave at that layer instead? + +Yes — but now we have to be *really* careful with the protocol. + +With partition weaving, a single logical topic has its partitions composed from physical topics across multiple clusters — which don't even need to share a name. + +Take an illustrative scenario: your third-party risk engine in `us-east` is hardcoded to read from a topic called `orders`. It has always read from `orders`. It will continue to read from `orders` (because vendor software rarely accommodates your topology reorganisations). Meanwhile, your business is growing in Europe, and sending every EU event across the Atlantic to land on `us-east` is expensive, slow, and fragile. The EU team provisions `orders-eu` on a local cluster sized for their producer load, and EU producers write there directly. + +Rather than waiting on a vendor feature request or building intermediate replication pipelines, the router presents both physical topics as one logical `orders` topic to the risk engine. When the engine requests metadata for `orders`, it gets back 32 partitions: partitions 0–15 are backed by `orders` on `us-east`, and 16–31 by `orders-eu` on `eu-west`. When a consumer fetches from partition 20, the proxy transparently routes the `FetchRequest` to `eu-west` (rewriting it for partition 4) and translates the response back. The risk engine reads the full partition space without a single configuration change, and EU producers write locally without crossing the Atlantic. + +Perceptive readers, all of you, will be asking how consumer groups survive this fiction. You can’t simply commit an offset for partition 20 to a coordinator whose cluster only knows 0–15 — Kafka brokers validate committed topic-partitions against their local metadata and reject unmapped ones with `UNKNOWN_TOPIC_OR_PARTITION`. + +Instead, the proxy intercepts `OffsetCommit` and `OffsetFetch` requests, demuxes the partition tuples across the backing clusters, and translates virtual partition 20 to physical partition 4 on `eu-west`. Each coordinator only ever receives offset commits for partitions that genuinely exist on its cluster, and the proxy stitches the responses back together into a single seamless reply for the consumer. + +What the proxy has to do here is far more demanding than in any of the previous patterns: partition numbers must be rewritten consistently across every Kafka RPC that mentions them — and they get everywhere, as you might expect. That's the essential complexity of the pattern, not incidental overhead. It's a lot of surfaces to get right. The operational contract also shifts: both physical topics have to honour the mapping, and your router is now the source of truth for what `orders` means. Transactions don't cross cluster boundaries here either — same caveat as topic weaving, but worth repeating because with partition weaving it's easier to forget the seam is there. + +--- + +These four patterns give us a working vocabulary for the rest of the series. The posts that follow take each one in turn — what it can do, what it can't, and why. Some of the limits are ours and will close over time. Some are the Kafka protocol's: transactions have no external coordinator, and no amount of clever routing changes that. And some would require the proxy to grow a consensus layer of its own — which is a whole different animal, and not one we're planning to domesticate any time soon. + +If you're thinking about Kafka topology problems, have a use case that doesn't fit cleanly into any of these patterns, or want to tell us where the vocabulary breaks down — find us on [GitHub](https://github.com/kroxylicious/kroxylicious), [Slack](https://kroxylicious.slack.com), or [Bluesky](https://bsky.app/profile/kroxylicious.io). diff --git a/proposals/289-review-comments.json b/proposals/289-review-comments.json new file mode 100644 index 00000000..551912b0 --- /dev/null +++ b/proposals/289-review-comments.json @@ -0,0 +1,51 @@ +[ + { + "type": "review_thread_reply", + "thread": "T5", + "replyToId": 4210755206, + "body": "Good point \u2014 you're right that we don't want to focus on record-level routing here. The design space does cover it (a router reconstituting a branch request can go as fine-grained as it likes), but that's not where current implementations sit. I've tweaked the language in the axis paragraph to avoid the ambiguity, and the stream branching section now covers the routing granularity point in more detail.", + "status": "posted" + }, + { + "type": "review_thread_reply", + "thread": "T6", + "replyToId": 4211918741, + "body": "Horses for courses \u2014 Kafka Streams is absolutely the right answer for some of this. Where the proxy approach has the edge: no duplicate storage cost (you're deriving data in flight rather than materialising a new topic), and no framework overhead for what are often stateless transforms. Kafka Streams and Connect are powerful but they're not lightweight \u2014 if all you need is \"emit a record count to a metrics topic as this batch passes through\", standing up a Streams application is a lot of machinery for a stateless operation. I've added a sentence to the section to make this clearer.", + "status": "posted" + }, + { + "type": "review_thread_reply", + "thread": "T9", + "replyToId": 4212088090, + "body": "Oops that wasn't quite the example I was trying to make. I've tweaked the text to make it clearer the woven topic is only on the consumer side.", + "status": "posted" + }, + { + "type": "review_thread_reply", + "thread": "T4", + "replyToId": 4210723236, + "body": "Glad the others landed. *Stream branching* was the best I could come up with \u2014 open to better suggestions. *Branching streams* is another candidate; the difference is whether you're naming the act (branching a stream) or the result (streams that branch). I went with the former but I'm not attached to it.", + "status": "posted" + }, + { + "type": "review_thread_reply", + "thread": "T1", + "replyToId": 4210638469, + "body": "Fixed in df2f68b \u2014 updated to broker rather than cluster.", + "status": "posted" + }, + { + "type": "review_thread_reply", + "thread": "T8", + "replyToId": 4223067354, + "body": "Fair point on consumer lag \u2014 you're right that cluster-side observation is the proper place for complete lag metrics. On counts/telemetry, the proxy isn't doing the final aggregation \u2014 it's just emitting inline record telemetry for downstream systems to aggregate. I've updated the examples in this section to focus on clean, stateless inline operations (a redaction filter for PII, a sampling filter to thin traffic for smaller staging environments, and derivation filters for audit logs/telemetry).", + "status": "posted" + }, + { + "type": "review_thread_reply", + "thread": "T9", + "replyToId": 4223173007, + "body": "Yep, exactly \u2014 if you control the consumer and don't mind it managing multiple topic subscriptions/assignments, topic weaving gets you part of the way there. Partition weaving is what lets you keep that complexity entirely in the infrastructure so downstream consumers don't even need to know the second topic exists.\n\nI've reworked this section to frame it around extending weaving down to Kafka's fundamental unit of parallelism (partitions), swapped the example to a third-party risk engine consuming an `orders` topic, and explained how consumer group coordination remains pinned to a single cluster without the coordinator needing to know about the seam.", + "status": "posted" + } +]