Repository navigation
Conversation
parmesant
force-pushed
the
feat/distributed-schema-resolution
branch
from
November 13, 2026 06:08
097ca3b to
fc899cc
Compare
parmesant
force-pushed
the
feat/distributed-schema-resolution
branch
from
November 14, 2026 06:45
6f64aa9 to
9d98d0d
Compare
This comment has been minimized.
This comment has been minimized.
parmesant
force-pushed
the
feat/distributed-schema-resolution
branch
from
November 15, 2026 03:45
19f3e2d to
420f424
Compare
parmesant
force-pushed
the
feat/distributed-schema-resolution
branch
from
November 17, 2026 06:12
16eec63 to
b1f2a18
Compare
When several ingestors see the same new field with different types in the same window (e.g. `a: Int64` on one, `a: Utf8` on another), each wrote its own schema and parquet, and the strict schema merge on the querier failed, making the stream unqueryable. Ingestors now ask a single schema authority before converting arrow files whose fields they have not resolved yet: - Authority is the Prism node if one is registered, otherwise the querier (lowest live node id). New endpoint POST /internal/schema/resolve, intra-cluster only, gated by the new admin-only ResolveSchema action. - The authority keeps a write-once field registry per stream. The first type claimed wins; a conflicting type is renamed to `<name>_<type suffix>` (chained up to 3 times). Reserved columns (p_timestamp, time and custom partitions, etc.) are never renamed and come back as unresolvable. - The registry is persisted to `.stream/.schema` (with a `parseable.registry_version` metadata key) before replying, so an answer never changes, and is rebuilt from it on restart. - Ingestors persist the answers in `.field_resolution.json` in the stream's staging directory, rename columns at conversion with a zero-copy schema swap, align their in-memory schema with the authority's types, and only advertise resolved fields in their staged schema file. - If the authority is unreachable, conversion of the affected files is deferred to the next cycle. After P_SCHEMA_RESOLVE_MAX_DEFERRED_CYCLES (default 3) failed cycles, ingestion for that stream returns 503 with Retry-After until resolution succeeds. - At startup, deferred files in processing directories go back to the staging root for the periodic cycle to retry. The query side stays strict: fetch_schema still rejects conflicting types, so no read-time casts are introduced. Tolerant merges are used only on the write side. New options: P_SCHEMA_RESOLVE_MODE (off | prefer | require, default require; ignored in standalone mode), P_SCHEMA_RESOLVE_MAX_DEFERRED_CYCLES, P_SCHEMA_RESOLVE_CONCURRENCY. Also: - the schema key hash now separates field name and type, to avoid collisions - get_all_schemas skips unreadable schema files instead of panicking
- kafka: a chunk rejected because its stream is blocked by schema resolution is retried every 60s and its offsets are committed only after it succeeds. Previously the error was logged and the offsets committed, silently dropping the records. - authority: clear the cached registry once a stream deletion completes, so a stream recreated under the same name starts from an empty registry instead of inheriting the old field types. - registry: a rename target can no longer be a column name carried by the same request. A file holding both `a` and `a_utf8` previously mapped both to `a_utf8`, failing the rename and leaving the file unconverted in staging. - authority: drop the querier-side Prism check and NOT_AUTHORITY. Only the OSS querier or Prism mounts the resolve route, so the cluster can only ever have one authority. - client: clear the cached authority on a 404/405, so an ingestor that picked a node without the route (an enterprise querier chosen before Prism registered) finds the right node on its next attempt. - document that the querier/Prism must be upgraded before ingestors in `require` mode
- replace P_SCHEMA_RESOLVE_MODE (off/prefer/require) with P_SCHEMA_RESOLUTION (default true); drop the prefer fallback to local types, which reintroduced the conflict. An authority without the endpoint is a failed cycle: upgrade the querier/Prism before ingestors. - re-resolve a file whose cached renames would collide with a column it carries, instead of leaving it unconverted in staging - the authority takes the tenant only from authentication: verified intra-cluster-tenant with P_CLUSTER_SECRET, else the caller's own tenant; ingestors refuse to resolve tenant streams without the secret - address DeepSource findings
A file whose cached rename collided with one of its own columns was re-resolved, and the new answer replaced the global (name, type) rename. Files already accepted under the old mapping then failed conversion with duplicate columns and stayed in processing_* until restart, the mapping could flip back and forth between cycles, and the same field split across two columns. Rename targets are now append-only per (name, type). Each file takes the first target not used by another of its columns, so a file converts the same way under any later view and files without a clash keep the first target. The view file keeps its entry format, one entry per target in grant order.
- look up the schema authority on every request and take the first live node of the authority type (Prism if registered, else the querier). The cluster runs one such node, so ordering by node_id and the 30s discovery cache added nothing; requests only happen for fields a node has not resolved yet. - remove MAX_SUFFIX_DEPTH. Types that share a suffix (every list is `list`) chained one level per type, so a fifth list type for a field became unresolvable and blocked the stream. The chain now continues until it reaches a free or compatible name; it always ends because only finitely many names are registered or proposed. Only a conflict on a reserved column is unresolvable.
Ingest-time conflict renaming took the column suffix from the raw JSON type, while schema V1 stores the new column under its own overrides: every number as Float64, and date strings in time-named fields as Timestamp. So V1 integers landed in `x_int64` columns typed Float64, and date strings in `x_utf8` columns typed Timestamp. In distributed mode the schema authority names the same data after the stored type (`x_float64`, `x_timestamp_ms`), so one field's values split across two columns depending on which path renamed them. Both rename paths (batch-level and per-record) now take the suffix from the type the new column is stored as: the same inference and V1 overrides run on the values being moved. V0 streams keep `_int64`, since V0 stores integers as Int64. Streams that already have `x_int64` or `x_utf8` siblings keep that data. New conflicts go to the correctly named column.
A record carrying a conflicting `f` and a literal `f_utf8` lost one of the two values: the ingest-time rename moved `f` onto `f_utf8` inside the same JSON map, so one value overwrote the other before any file was written. Both rename passes now build each record through one helper. Kept keys claim their names first; a moved value whose target is already a non-null key of the record, or a stream column the value does not fit, chains one more suffix (`f_utf8` -> `f_utf8_utf8`) until the name is free. This is the same chain the schema authority uses, so ingestors and the authority agree on the name. A null at the target is overwritten, since null and absent mean the same. Renaming onto an existing sibling column of another type used to fail the event; it now moves to the next free name.
- kafka: bound chunks in flight per partition. A chunk blocked by schema resolution keeps its records until it succeeds; with unbounded concurrency the partition kept being read into ever more held chunks for the length of the block. Now blocked chunks stop the reads and the consumer waits. - authority: fence resolutions against stream deletion. Deletion takes the stream's authority lock before removing its files and holds it until they are gone, and resolve re-checks the deleting flag under that lock, so a resolution in flight can no longer write `.schema` back for a deleted stream or over its replacement. - ingestor: publish the resolved view before moving the in-memory schema to the resolved types, and read the view under the schema lock in commit_schema. An event built against the old type no longer sees the new type without the rename that allows it and is not rejected. The per-node schema file now also advertises the newly resolved fields right away.
A chunk whose stream rejects writes until its schema is reconciled is held and retried, with its offsets uncommitted. Capping chunks in flight to bound those held records stopped draining the partition's queue; once full, the consumer loop shared by every partition waited on it, stalling all topics for as long as the block lasted. Now the first blocked chunk pauses its partition and the last one to go through resumes it. Chunk processing is unbounded again, so blocked chunks drain the queue instead of filling it, and the pause bounds what a blocked partition holds. The processor reports a block as `SchemaBlocked` carrying the records back, and the retry moves to the partition worker, which owns the consumer.
…Kafka commits A stream switches a field to the authority's type whenever it learns it: from a rename, from the registry snapshot sent to a lagging node, or at restart. commit_schema tolerated events still carrying the old type only when a rename existed, so a type learned from the snapshot rejected valid events built just before the switch. It now tolerates the old type whenever the stream's type is the authority's; the event's arrow file is resolved at conversion like any other unresolved type. Kafka chunks of a partition finish out of order: one can wait out a schema block while later ones go through. Each committed its own offset, so a later chunk could commit past records still waiting, and a restart or rebalance would skip them. Offsets are now committed only through the last consecutive completed chunk, under a lock so the committed offset never moves back. A partition revoked while paused is resumed when its worker is dropped.
A revoked partition's worker is dropped without the revoke waiting for it, so its Drop could resume the partition after a new worker for the same partition had already paused it. The new worker's later blocked chunks did not pause again, and the resumed partition kept filling with sleeping chunks. The count of blocked chunks now belongs to the partition and is shared by all its workers; the partition is paused exactly while the count is above zero. A dropped worker gives back only its own chunks. Each worker also pauses on its own first blocked chunk, so it does not depend on an older worker's pause having survived the reassignment.
With P_CLUSTER_SECRET set, an ingestor resolving a schema for a non-default tenant sent the root username as intra-cluster-userid. That user only exists in the default tenant, so IntraClusterRequest created no session and the route's ResolveSchema check rejected the call. New fields stayed unconverted and ingestion fell back to 503s. The resolve route now authorises in the handler. A verified cluster-secret caller is a node acting for the tenant it names in intra-cluster-tenant; no user of that tenant is needed. Without the secret the caller must still hold ResolveSchema and acts for its own tenant. The body still cannot name another tenant. Also share the intra-cluster-tenant header name as a constant.
Name list renames after their element type. get_datatype_suffix gave every list type the same `list` suffix, so ingest-time renaming sent a second list type of a field into the first one's column and rejected the batch. Across ingestors this happened all the time: after a resolution an ingestor holds another ingestor's list type under `f_list`, and its own lists were rejected. Suffixes now carry the element type (`list_utf8`, `list_list_int64`), and ingestion chains past a column of another nested type like the registry does, so both sides pick the same column for the same type. Never grant one rename target to two proposals of a request. A file carrying `f` and a literal `f_utf8` of the same type, next to an existing `f_utf8_utf8`, got that target for both. The file could never be renamed, and the stream blocked after the deferral cycles with the authority up. Seed a new registry only when the saved one does not exist. Any failed read or undecodable file was treated as missing, so the authority rebuilt the registry from per-node schemas in listing order and could overwrite types it had already granted. It now returns an error, and ingestors retry next cycle.
parmesant
force-pushed
the
feat/distributed-schema-resolution
branch
from
November 17, 2026 14:04
b1f2a18 to
69baaf6
Compare
Comment on lines
+153
to
+165
| let fields: Vec<Field> = unresolved | ||
| .iter() | ||
| .map(|(name, data_type)| Field::new(name, data_type.clone(), true)) | ||
| .collect(); | ||
| let request_id = self.request_id(&fields); | ||
| let request = ResolveRequest { | ||
| protocol_version: PROTOCOL_VERSION, | ||
| request_id, | ||
| node_id: ctx.node_id.clone(), | ||
| tenant_id: ctx.tenant_id.clone(), | ||
| stream: ctx.stream.to_owned(), | ||
| known_version: view.registry_version, | ||
| fields, |
There was a problem hiding this comment.
Wide streams stop accepting events
ensure_resolved sends every unresolved field from all selected files in one request, but the authority rejects bodies larger than 1 MiB. A wide stream, or several accepted batches that introduce enough fields before conversion, can exceed that limit even though ingestion allows 10 MiB per request.
Every later cycle resends the same oversized field set. Its files never convert, and ingestion starts returning 503 after the configured failed cycles. Split the fields into requests that fit the authority's limit.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/schema_registry/ingestor.rs
Line: 153-165
Comment:
**Wide streams stop accepting events**
`ensure_resolved` sends every unresolved field from all selected files in one request, but the authority rejects bodies larger than 1 MiB. A wide stream, or several accepted batches that introduce enough fields before conversion, can exceed that limit even though ingestion allows 10 MiB per request.
Every later cycle resends the same oversized field set. Its files never convert, and ingestion starts returning 503 after the configured failed cycles. Split the fields into requests that fit the authority's limit.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When several ingestors see the same new field with different types in the
same window (e.g.
a: Int64on one,a: Utf8on another), each wrote itsown schema and parquet, and the strict schema merge on the querier failed,
making the stream unqueryable.
Ingestors now ask a single schema authority before converting arrow files
whose fields they have not resolved yet
New options: P_SCHEMA_RESOLVE_MODE (off | prefer | require, default
require; ignored in standalone mode), P_SCHEMA_RESOLVE_MAX_DEFERRED_CYCLES,
P_SCHEMA_RESOLVE_CONCURRENCY.
Description
This PR has:
Do not merge until schema requests are split to fit the authority's body limit.
Fix with agent prompt
Summary
Adds a shared schema authority so distributed ingestors agree on field types before writing Parquet.
parmesantdismissed the competing-authorities finding because only one query node can run. Greptile accepted that topology constraint.Diagram
%%{init: {'theme': 'neutral'}}%% flowchart LR A[Accepted events] --> B[Arrow files] B --> C[Collect unresolved fields] C --> D[Schema authority] D --> E[Save field decisions] E --> F[Rename columns] F --> G[Write Parquet] D -->|Request too large| H[Defer files] H --> C H --> I[Reject new writes after failed cycles]Reviews (14) · Last reviewed commit: "fix: keep rename targets distinct and ne..." · Reviewed by Greptile