Skip to content

Recognise all recoverable schema incompatibilities in vstream plan failures - #96

Open
orware wants to merge 2 commits into
mainfrom
fix/schema-incompat-matcher-dropped-column
Open

Recognise all recoverable schema incompatibilities in vstream plan failures#96
orware wants to merge 2 commits into
mainfrom
fix/schema-incompat-matcher-dropped-column

Conversation

@orware

@orware orware commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Problem

IsVStreamSchemaIncompatibilityError (added in #89) requires Code: FAILED_PRECONDITION in the error text.

vstreamer wraps every plan failure with failed to build table replication plan for table <name>, but the inner cause varies with the shape of the DDL — and only two are raised through vterrors with a code attached. The rest are plain fmt.Errorf, so they arrive code-less and the matcher rejects them on its very first check:

Inner cause Raised when Has a code? Matched before?
column <c> not found in table <t> column appended, stream now requests it, event predates the ALTER yes yes
cannot use column names in vstream filter ... columns inserted mid-table (ADD COLUMN ... AFTER) or reordered yes yes
cannot determine table columns for <t> column dropped, schema narrower than the event no no
failed to build ENUM and SET column integer to string mappings ENUM/SET column dropped, value list unrecoverable no no

Two of the four were unmatched, and both have been seen in production. Each falls through to the generic error return, so the operator gets an opaque Internal error rather than the FailedPrecondition and the explicit "run a historical re-sync" guidance #89 added — precisely the situations that guidance exists for.

All four recover the same way: drop the cursor, run a historical sync.

Change

Gate on the failed to build table replication plan wrapper — common to all variants — and enumerate the four recoverable inner causes. The FAILED_PRECONDITION check is removed rather than relaxed: it excluded real failure shapes while adding no selectivity the wrapper does not already provide.

What is deliberately excluded

The wrapper is not sufficient on its own. Two other errors share it and are not recoverable by re-syncing. Both have negative tests:

  • unsupported type: <n>, position: <i> — a historical sync would hit the same unsupported column type.

  • unknown table <t> in schema — this one is worth spelling out, because it looks like a stale-cursor condition and isn't. historian.GetTableForPos returns (nil, nil) on a miss, and Engine.GetTableForPos then falls back to the live schema rather than erroring:

    if mt != nil { return mt, nil }
    // We got nothing from the historian, which typically means that it's not enabled.
    // ... falls back to the live schema

    So the historian fails open, and this error instead means an undecodable GTID or a table genuinely absent from the tablet's schema. The latter can be transient during an online DDL rename swap, where resetting the cursor would force an unnecessary historical sync.

Tests

Eight table-driven cases — four positive, four negative. The two newly matched positives were each verified to fail against the previous matcher.

go test ./lib passes. (./cmd/... needs make proto first, as CI does via test-ci: proto.)

Relationship to #97

Independent, and they compose. #97 adds opt-in automatic recovery once a schema incompatibility is detected; this PR determines what counts as one. Either can merge first — but detection without #97 only improves the error message, and #97 without this one only auto-recovers from two of the four causes.

🤖 Generated with Claude Code

IsVStreamSchemaIncompatibilityError required "Code: FAILED_PRECONDITION"
in the error text. Two of the three ways vstreamer reports a schema
mismatch carry that code, but the DROP COLUMN variant does not: it is
raised as a plain fmt.Errorf inside buildTableColumns and reaches the
connector with no gRPC code attached.

The result was that a deploy dropping a column from a replicated table
fell through to the generic error return, so the operator got an opaque
Internal error rather than the FailedPrecondition and the explicit
"run a historical re-sync" guidance added in #89.

Gate on the "failed to build table replication plan" wrapper, which all
three variants carry, and add "cannot determine table columns" to the set
of recognised causes. The FAILED_PRECONDITION check is dropped rather
than relaxed because it excluded a real failure shape while adding no
selectivity the wrapper does not already provide.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@orware orware changed the title Match dropped-column schema incompatibility in vstream errors Recognise all recoverable schema incompatibilities in vstream plan failures Aug 3, 2026
"failed to build ENUM and SET column integer to string mappings" shares
the "failed to build table replication plan" wrapper and is recovered the
same way, but was falling through to the generic error return. It is
raised when an ENUM or SET column is dropped and its value list can no
longer be recovered to decode the integers in the replayed event. Like
the drop-column variant it carries no gRPC code.

Two other errors share the wrapper and are deliberately excluded, each
with a negative test:

  - "unsupported type: <n>, position: <i>", since a historical sync would
    hit the same unsupported column type.

  - "unknown table <t> in schema", which is not a stale-cursor condition.
    A historian miss returns (nil, nil) and the schema engine falls back
    to the live schema rather than erroring, so this indicates an
    undecodable GTID or a table genuinely absent from the tablet schema.
    The latter can be transient during an online DDL rename swap, where
    resetting the cursor would force an unnecessary historical sync.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@orware
orware force-pushed the fix/schema-incompat-matcher-dropped-column branch from 80ca485 to 408e6f4 Compare August 3, 2026 21:16
@orware

orware commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

Reproduced all four matched causes end-to-end against live PlanetScale branches, driving the connector's real Read() path — capture a cursor, write rows, apply the DDL, write more rows, resume from the stale cursor. Both via direct ALTER and via deploy request (online DDL), with and without track-schema-versions. 16 cells; every deploy-request result was identical to its direct-ALTER equivalent.

Three of the four matched causes reproduce exactly and are caught by this matcher:

DDL error matched
trailing ADD COLUMN column X not found in table Y yes
mid-table ADD ... AFTER cannot use column names in vstream filter yes
DROP COLUMN cannot determine table columns yes

One caveat worth recording so it isn't mistaken for verified: I could not reproduce the ENUM variant (failed to build ENUM and SET column integer to string mappings). Dropping an ENUM column hits the column-count check first and surfaces as cannot determine table columns; with schema tracking enabled it decodes cleanly. The production sighting that motivated that clause was on a mysql80-2026-04-08 build, so the upstream issue looks to have been fixed since.

I've left the clause in — it matches a real error that was observed in production, and it's inert if the error no longer occurs — but happy to drop it if you'd rather the matcher only cover shapes reproducible on current builds.

orware added a commit that referenced this pull request Aug 5, 2026
Fivetran tells us per table whether new columns should be synced
(TableSelection.include_new_columns), and the SDK guide makes acting on it
the connector's job. We never read the field: includedColumns() looks only at
TableSelection.columns, so a column added after a connection was set up is
never named in the VStream projection and its values never arrive. No error
is raised, syncs report success, and only a historical re-sync recovers.

Widening the projection is not something we can simply do up front, though.
Naming a column that did not exist at the replay position is exactly what
produces "column X not found in table Y" against a pre-DDL TABLE_MAP. So the
stream starts on Fivetran's selection as-is and rebuilds the projection only
once it observes the DDL event for the table.

Resuming there is safe because the cursor is already past the schema change:
vstreamer emits a GTID event carrying EncodePosition(vs.pos) immediately
before the DDL event, vs.pos already includes the DDL's own GTID, and vtgate
converts that GTID to the VGTID we consume in place. Only post-DDL row events
are replayed, so their TABLE_MAP carries the new column and the plan resolves
without help from vttablet's schema historian -- this needs no
--track-schema-versions.

The serializer needed the same treatment. columnSelection and columnWriters
are both built from TableSelection.columns, so a widened projection would
have been discarded on the way out, and a missing writer is a hard error
rather than a silent skip.

Both opt-ins must be set: the new propagate_new_columns source setting and
Fivetran's per-table include_new_columns. A column Fivetran explicitly
deselected is present-and-false in the map and stays excluded; only columns
it never named at all are adopted. The setting defaults to false and is
labelled experimental.

Scope, verified against a live branch with tracking off rather than inferred:

  - Fixes new columns never arriving. The pre-DDL row is delivered on the
    narrow projection and the post-DDL row carries the new column's value.
  - Does not change hard stalls. When Fivetran's selection already names the
    new column, filterExistingColumns keeps it (it exists live), so the first
    window fails before reaching the DDL and the gate never fires. The error
    is identical with the setting on or off and is still matched by
    IsVStreamSchemaIncompatibilityError, so recovery stays with #96/#97.
  - Does not change dropped columns on a lagging cursor. The width check
    fires on the pre-DDL row before the DDL is reached, so that case still
    depends on the historian. rebuildProjection drops vanished columns for
    coherence, which matters only for a drop landing mid-Read.

Note for anyone tempted to loosen the projection instead: falling back to
SELECT * silently corrupts data across a mid-table ADD. analyzeExprs copies
Table.Fields positionally for a StarExpr and never calls findColumn, so
nothing catches the skew from the st.Fields[:len(tm.Types)] truncation. On a
lagging cursor with ADD COLUMN newcol AFTER a over (id, a, b, c), * delivered
b's value as newcol and c's value as b with no error at all. The explicit
projection is a guardrail, not the defect.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant