Skip to content

Distributed schema resolution - #1856

Open
parmesant wants to merge 13 commits into
parseablehq:mainfrom
parmesant:feat/distributed-schema-resolution
Open

parmesant wants to merge 13 commits into
parseablehq:mainfrom
parmesant:feat/distributed-schema-resolution

Conversation

@parmesant

@parmesant parmesant commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

When several ingestors see the same new field with different types in the
same window (e.g. a: Int64 on one, a: Utf8 on another), each wrote its
own schema and parquet, and the strict schema merge on the querier failed,
making the stream unqueryable.

Ingestors now ask a single schema authority before converting arrow files
whose fields they have not resolved yet

New options: P_SCHEMA_RESOLVE_MODE (off | prefer | require, default
require; ignored in standalone mode), P_SCHEMA_RESOLVE_MAX_DEFERRED_CYCLES,
P_SCHEMA_RESOLVE_CONCURRENCY.

Description


This PR has:

  • been tested to ensure log ingestion and log query works.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added documentation for new or modified features or behaviors.

RetriggerView in GreptileConfidence Score: 4/5

Do not merge until schema requests are split to fit the authority's body limit.

Fix All in CodexFindings

  1. P1 Wide streams stop accepting events ▶
Fix with agent prompt
### Issue 1
src/schema_registry/ingestor.rs:153-165
`ensure_resolved` sends every unresolved field from all selected files in one request, but the authority rejects bodies larger than 1 MiB. A wide stream, or several accepted batches that introduce enough fields before conversion, can exceed that limit even though ingestion allows 10 MiB per request.

Every later cycle resends the same oversized field set. Its files never convert, and ingestion starts returning 503 after the configured failed cycles. Split the fields into requests that fit the authority's limit.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds a shared schema authority so distributed ingestors agree on field types before writing Parquet.

  • One schema authority assigns field types across distributed ingestors.
  • Ingestors resolve staged fields before writing parquet.
  • Kafka partitions hold blocked batches until the stream can accept them.
  • Conflicting JSON fields get column names that match their stored types.
  • License header spacing changes without changing behavior.

parmesant dismissed the competing-authorities finding because only one query node can run. Greptile accepted that topology constraint.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Accepted events] --> B[Arrow files]
  B --> C[Collect unresolved fields]
  C --> D[Schema authority]
  D --> E[Save field decisions]
  E --> F[Rename columns]
  F --> G[Write Parquet]
  D -->|Request too large| H[Defer files]
  H --> C
  H --> I[Reject new writes after failed cycles]
Loading

Reviews (14) · Last reviewed commit: "fix: keep rename targets distinct and ne..." · Reviewed by Greptile

Comment thread src/parseable/streams.rs Resolved
Comment thread src/schema_registry/client.rs Resolved
Comment thread src/schema_registry/ingestor.rs Resolved
Comment thread src/schema_registry/authority.rs Resolved
Comment thread src/schema_registry/ingestor.rs Resolved
greptile-apps[bot]
greptile-apps Bot previously approved these changes Nov 12, 2026
greptile-apps[bot]
greptile-apps Bot previously approved these changes Nov 12, 2026
Comment thread src/event/format/mod.rs Resolved
@parmesant
parmesant force-pushed the feat/distributed-schema-resolution branch from 097ca3b to fc899cc Compare November 13, 2026 06:08
Comment thread src/connectors/kafka/processor.rs Resolved
Comment thread src/schema_registry/authority.rs Resolved
Comment thread src/schema_registry/ingestor.rs Resolved
Comment thread src/connectors/kafka/processor.rs Resolved
@parmesant
parmesant force-pushed the feat/distributed-schema-resolution branch from 6f64aa9 to 9d98d0d Compare November 14, 2026 06:45
Comment thread src/schema_registry/ingestor.rs Resolved
@greptile-apps

This comment has been minimized.

Comment thread src/connectors/kafka/processor.rs Resolved
greptile-apps[bot]
greptile-apps Bot previously approved these changes Nov 15, 2026
Comment thread src/schema_registry/client.rs Resolved
@parmesant
parmesant force-pushed the feat/distributed-schema-resolution branch from 16eec63 to b1f2a18 Compare November 17, 2026 06:12
Comment thread src/schema_registry/authority.rs Resolved
When several ingestors see the same new field with different types in the
same window (e.g. `a: Int64` on one, `a: Utf8` on another), each wrote its
own schema and parquet, and the strict schema merge on the querier failed,
making the stream unqueryable.

Ingestors now ask a single schema authority before converting arrow files
whose fields they have not resolved yet:

- Authority is the Prism node if one is registered, otherwise the querier
  (lowest live node id). New endpoint POST /internal/schema/resolve,
  intra-cluster only, gated by the new admin-only ResolveSchema action.
- The authority keeps a write-once field registry per stream. The first type
  claimed wins; a conflicting type is renamed to `<name>_<type suffix>`
  (chained up to 3 times). Reserved columns (p_timestamp, time and custom
  partitions, etc.) are never renamed and come back as unresolvable.
- The registry is persisted to `.stream/.schema` (with a
  `parseable.registry_version` metadata key) before replying, so an answer
  never changes, and is rebuilt from it on restart.
- Ingestors persist the answers in `.field_resolution.json` in the stream's
  staging directory, rename columns at conversion with a zero-copy schema
  swap, align their in-memory schema with the authority's types, and only
  advertise resolved fields in their staged schema file.
- If the authority is unreachable, conversion of the affected files is
  deferred to the next cycle. After P_SCHEMA_RESOLVE_MAX_DEFERRED_CYCLES
  (default 3) failed cycles, ingestion for that stream returns 503 with
  Retry-After until resolution succeeds.
- At startup, deferred files in processing directories go back to the
  staging root for the periodic cycle to retry.

The query side stays strict: fetch_schema still rejects conflicting types,
so no read-time casts are introduced. Tolerant merges are used only on the
write side.

New options: P_SCHEMA_RESOLVE_MODE (off | prefer | require, default
require; ignored in standalone mode), P_SCHEMA_RESOLVE_MAX_DEFERRED_CYCLES,
P_SCHEMA_RESOLVE_CONCURRENCY.

Also:
- the schema key hash now separates field name and type, to avoid
  collisions
- get_all_schemas skips unreadable schema files instead of panicking
- kafka: a chunk rejected because its stream is blocked by schema
  resolution is retried every 60s and its offsets are committed only
  after it succeeds. Previously the error was logged and the offsets
  committed, silently dropping the records.
- authority: clear the cached registry once a stream deletion completes,
  so a stream recreated under the same name starts from an empty
  registry instead of inheriting the old field types.
- registry: a rename target can no longer be a column name carried by
  the same request. A file holding both `a` and `a_utf8` previously
  mapped both to `a_utf8`, failing the rename and leaving the file
  unconverted in staging.
- authority: drop the querier-side Prism check and NOT_AUTHORITY. Only
  the OSS querier or Prism mounts the resolve route, so the cluster can
  only ever have one authority.
- client: clear the cached authority on a 404/405, so an ingestor that
  picked a node without the route (an enterprise querier chosen before
  Prism registered) finds the right node on its next attempt.
- document that the querier/Prism must be upgraded before ingestors in
  `require` mode
- replace P_SCHEMA_RESOLVE_MODE (off/prefer/require) with
  P_SCHEMA_RESOLUTION (default true); drop the prefer fallback to local
  types, which reintroduced the conflict. An authority without the
  endpoint is a failed cycle: upgrade the querier/Prism before ingestors.
- re-resolve a file whose cached renames would collide with a column it
  carries, instead of leaving it unconverted in staging
- the authority takes the tenant only from authentication: verified
  intra-cluster-tenant with P_CLUSTER_SECRET, else the caller's own
  tenant; ingestors refuse to resolve tenant streams without the secret
- address DeepSource findings
A file whose cached rename collided with one of its own columns was
re-resolved, and the new answer replaced the global (name, type) rename.
Files already accepted under the old mapping then failed conversion with
duplicate columns and stayed in processing_* until restart, the mapping
could flip back and forth between cycles, and the same field split across
two columns.

Rename targets are now append-only per (name, type). Each file takes the
first target not used by another of its columns, so a file converts the
same way under any later view and files without a clash keep the first
target. The view file keeps its entry format, one entry per target in
grant order.
- look up the schema authority on every request and take the first
  live node of the authority type (Prism if registered, else the
  querier). The cluster runs one such node, so ordering by node_id and
  the 30s discovery cache added nothing; requests only happen for
  fields a node has not resolved yet.
- remove MAX_SUFFIX_DEPTH. Types that share a suffix (every list is
  `list`) chained one level per type, so a fifth list type for a field
  became unresolvable and blocked the stream. The chain now continues
  until it reaches a free or compatible name; it always ends because
  only finitely many names are registered or proposed. Only a conflict
  on a reserved column is unresolvable.
Ingest-time conflict renaming took the column suffix from the raw JSON
type, while schema V1 stores the new column under its own overrides:
every number as Float64, and date strings in time-named fields as
Timestamp. So V1 integers landed in `x_int64` columns typed Float64,
and date strings in `x_utf8` columns typed Timestamp.

In distributed mode the schema authority names the same data after the
stored type (`x_float64`, `x_timestamp_ms`), so one field's values
split across two columns depending on which path renamed them.

Both rename paths (batch-level and per-record) now take the suffix from
the type the new column is stored as: the same inference and V1
overrides run on the values being moved. V0 streams keep `_int64`,
since V0 stores integers as Int64. Streams that already have
`x_int64` or `x_utf8` siblings keep that data. New conflicts go to the
correctly named column.
A record carrying a conflicting `f` and a literal `f_utf8` lost one of
the two values: the ingest-time rename moved `f` onto `f_utf8` inside the
same JSON map, so one value overwrote the other before any file was
written.

Both rename passes now build each record through one helper. Kept keys
claim their names first; a moved value whose target is already a
non-null key of the record, or a stream column the value does not fit,
chains one more suffix (`f_utf8` -> `f_utf8_utf8`) until the name is
free. This is the same chain the schema authority uses, so ingestors and
the authority agree on the name. A null at the target is overwritten,
since null and absent mean the same.

Renaming onto an existing sibling column of another type used to fail
the event; it now moves to the next free name.
- kafka: bound chunks in flight per partition. A chunk blocked by schema
  resolution keeps its records until it succeeds; with unbounded
  concurrency the partition kept being read into ever more held chunks
  for the length of the block. Now blocked chunks stop the reads and the
  consumer waits.
- authority: fence resolutions against stream deletion. Deletion takes
  the stream's authority lock before removing its files and holds it
  until they are gone, and resolve re-checks the deleting flag under that
  lock, so a resolution in flight can no longer write `.schema` back for
  a deleted stream or over its replacement.
- ingestor: publish the resolved view before moving the in-memory schema
  to the resolved types, and read the view under the schema lock in
  commit_schema. An event built against the old type no longer sees the
  new type without the rename that allows it and is not rejected. The
  per-node schema file now also advertises the newly resolved fields
  right away.
A chunk whose stream rejects writes until its schema is reconciled is
held and retried, with its offsets uncommitted. Capping chunks in
flight to bound those held records stopped draining the partition's
queue; once full, the consumer loop shared by every partition waited
on it, stalling all topics for as long as the block lasted.

Now the first blocked chunk pauses its partition and the last one to
go through resumes it. Chunk processing is unbounded again, so blocked
chunks drain the queue instead of filling it, and the pause bounds what
a blocked partition holds. The processor reports a block as
`SchemaBlocked` carrying the records back, and the retry moves to the
partition worker, which owns the consumer.
…Kafka commits

A stream switches a field to the authority's type whenever it learns it:
from a rename, from the registry snapshot sent to a lagging node, or at
restart. commit_schema tolerated events still carrying the old type only
when a rename existed, so a type learned from the snapshot rejected
valid events built just before the switch. It now tolerates the old type
whenever the stream's type is the authority's; the event's arrow file is
resolved at conversion like any other unresolved type.

Kafka chunks of a partition finish out of order: one can wait out a
schema block while later ones go through. Each committed its own offset,
so a later chunk could commit past records still waiting, and a restart
or rebalance would skip them. Offsets are now committed only through
the last consecutive completed chunk, under a lock so the committed
offset never moves back. A partition revoked while paused is resumed
when its worker is dropped.
A revoked partition's worker is dropped without the revoke waiting for it,
so its Drop could resume the partition after a new worker for the same
partition had already paused it. The new worker's later blocked chunks did
not pause again, and the resumed partition kept filling with sleeping
chunks.

The count of blocked chunks now belongs to the partition and is shared by
all its workers; the partition is paused exactly while the count is above
zero. A dropped worker gives back only its own chunks. Each worker also
pauses on its own first blocked chunk, so it does not depend on an older
worker's pause having survived the reassignment.
With P_CLUSTER_SECRET set, an ingestor resolving a schema for a
non-default tenant sent the root username as intra-cluster-userid.
That user only exists in the default tenant, so IntraClusterRequest
created no session and the route's ResolveSchema check rejected the
call. New fields stayed unconverted and ingestion fell back to 503s.

The resolve route now authorises in the handler. A verified
cluster-secret caller is a node acting for the tenant it names in
intra-cluster-tenant; no user of that tenant is needed. Without the
secret the caller must still hold ResolveSchema and acts for its own
tenant. The body still cannot name another tenant.

Also share the intra-cluster-tenant header name as a constant.
Name list renames after their element type. get_datatype_suffix gave
every list type the same `list` suffix, so ingest-time renaming sent a
second list type of a field into the first one's column and rejected
the batch. Across ingestors this happened all the time: after a
resolution an ingestor holds another ingestor's list type under
`f_list`, and its own lists were rejected. Suffixes now carry the
element type (`list_utf8`, `list_list_int64`), and ingestion chains past
a column of another nested type like the registry does, so both sides
pick the same column for the same type.

Never grant one rename target to two proposals of a request. A file
carrying `f` and a literal `f_utf8` of the same type, next to an
existing `f_utf8_utf8`, got that target for both. The file could never
be renamed, and the stream blocked after the deferral cycles with the
authority up.

Seed a new registry only when the saved one does not exist. Any failed
read or undecodable file was treated as missing, so the authority
rebuilt the registry from per-node schemas in listing order and could
overwrite types it had already granted. It now returns an error, and
ingestors retry next cycle.
@parmesant
parmesant force-pushed the feat/distributed-schema-resolution branch from b1f2a18 to 69baaf6 Compare November 17, 2026 14:04
Comment on lines +153 to +165
let fields: Vec<Field> = unresolved
.iter()
.map(|(name, data_type)| Field::new(name, data_type.clone(), true))
.collect();
let request_id = self.request_id(&fields);
let request = ResolveRequest {
protocol_version: PROTOCOL_VERSION,
request_id,
node_id: ctx.node_id.clone(),
tenant_id: ctx.tenant_id.clone(),
stream: ctx.stream.to_owned(),
known_version: view.registry_version,
fields,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Wide streams stop accepting events

ensure_resolved sends every unresolved field from all selected files in one request, but the authority rejects bodies larger than 1 MiB. A wide stream, or several accepted batches that introduce enough fields before conversion, can exceed that limit even though ingestion allows 10 MiB per request.

Every later cycle resends the same oversized field set. Its files never convert, and ingestion starts returning 503 after the configured failed cycles. Split the fields into requests that fit the authority's limit.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/schema_registry/ingestor.rs
Line: 153-165

Comment:
**Wide streams stop accepting events**

`ensure_resolved` sends every unresolved field from all selected files in one request, but the authority rejects bodies larger than 1 MiB. A wide stream, or several accepted batches that introduce enough fields before conversion, can exceed that limit even though ingestion allows 10 MiB per request.

Every later cycle resends the same oversized field set. Its files never convert, and ingestion starts returning 503 after the configured failed cycles. Split the fields into requests that fit the authority's limit.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant