Recognise all recoverable schema incompatibilities in vstream plan failures - #96
Recognise all recoverable schema incompatibilities in vstream plan failures#96orware wants to merge 2 commits into
Conversation
IsVStreamSchemaIncompatibilityError required "Code: FAILED_PRECONDITION" in the error text. Two of the three ways vstreamer reports a schema mismatch carry that code, but the DROP COLUMN variant does not: it is raised as a plain fmt.Errorf inside buildTableColumns and reaches the connector with no gRPC code attached. The result was that a deploy dropping a column from a replicated table fell through to the generic error return, so the operator got an opaque Internal error rather than the FailedPrecondition and the explicit "run a historical re-sync" guidance added in #89. Gate on the "failed to build table replication plan" wrapper, which all three variants carry, and add "cannot determine table columns" to the set of recognised causes. The FAILED_PRECONDITION check is dropped rather than relaxed because it excluded a real failure shape while adding no selectivity the wrapper does not already provide. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"failed to build ENUM and SET column integer to string mappings" shares
the "failed to build table replication plan" wrapper and is recovered the
same way, but was falling through to the generic error return. It is
raised when an ENUM or SET column is dropped and its value list can no
longer be recovered to decode the integers in the replayed event. Like
the drop-column variant it carries no gRPC code.
Two other errors share the wrapper and are deliberately excluded, each
with a negative test:
- "unsupported type: <n>, position: <i>", since a historical sync would
hit the same unsupported column type.
- "unknown table <t> in schema", which is not a stale-cursor condition.
A historian miss returns (nil, nil) and the schema engine falls back
to the live schema rather than erroring, so this indicates an
undecodable GTID or a table genuinely absent from the tablet schema.
The latter can be transient during an online DDL rename swap, where
resetting the cursor would force an unnecessary historical sync.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
80ca485 to
408e6f4
Compare
|
Reproduced all four matched causes end-to-end against live PlanetScale branches, driving the connector's real Three of the four matched causes reproduce exactly and are caught by this matcher:
One caveat worth recording so it isn't mistaken for verified: I could not reproduce the ENUM variant ( I've left the clause in — it matches a real error that was observed in production, and it's inert if the error no longer occurs — but happy to drop it if you'd rather the matcher only cover shapes reproducible on current builds. |
Fivetran tells us per table whether new columns should be synced
(TableSelection.include_new_columns), and the SDK guide makes acting on it
the connector's job. We never read the field: includedColumns() looks only at
TableSelection.columns, so a column added after a connection was set up is
never named in the VStream projection and its values never arrive. No error
is raised, syncs report success, and only a historical re-sync recovers.
Widening the projection is not something we can simply do up front, though.
Naming a column that did not exist at the replay position is exactly what
produces "column X not found in table Y" against a pre-DDL TABLE_MAP. So the
stream starts on Fivetran's selection as-is and rebuilds the projection only
once it observes the DDL event for the table.
Resuming there is safe because the cursor is already past the schema change:
vstreamer emits a GTID event carrying EncodePosition(vs.pos) immediately
before the DDL event, vs.pos already includes the DDL's own GTID, and vtgate
converts that GTID to the VGTID we consume in place. Only post-DDL row events
are replayed, so their TABLE_MAP carries the new column and the plan resolves
without help from vttablet's schema historian -- this needs no
--track-schema-versions.
The serializer needed the same treatment. columnSelection and columnWriters
are both built from TableSelection.columns, so a widened projection would
have been discarded on the way out, and a missing writer is a hard error
rather than a silent skip.
Both opt-ins must be set: the new propagate_new_columns source setting and
Fivetran's per-table include_new_columns. A column Fivetran explicitly
deselected is present-and-false in the map and stays excluded; only columns
it never named at all are adopted. The setting defaults to false and is
labelled experimental.
Scope, verified against a live branch with tracking off rather than inferred:
- Fixes new columns never arriving. The pre-DDL row is delivered on the
narrow projection and the post-DDL row carries the new column's value.
- Does not change hard stalls. When Fivetran's selection already names the
new column, filterExistingColumns keeps it (it exists live), so the first
window fails before reaching the DDL and the gate never fires. The error
is identical with the setting on or off and is still matched by
IsVStreamSchemaIncompatibilityError, so recovery stays with #96/#97.
- Does not change dropped columns on a lagging cursor. The width check
fires on the pre-DDL row before the DDL is reached, so that case still
depends on the historian. rebuildProjection drops vanished columns for
coherence, which matters only for a drop landing mid-Read.
Note for anyone tempted to loosen the projection instead: falling back to
SELECT * silently corrupts data across a mid-table ADD. analyzeExprs copies
Table.Fields positionally for a StarExpr and never calls findColumn, so
nothing catches the skew from the st.Fields[:len(tm.Types)] truncation. On a
lagging cursor with ADD COLUMN newcol AFTER a over (id, a, b, c), * delivered
b's value as newcol and c's value as b with no error at all. The explicit
projection is a guardrail, not the defect.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Problem
IsVStreamSchemaIncompatibilityError(added in #89) requiresCode: FAILED_PRECONDITIONin the error text.vstreamer wraps every plan failure with
failed to build table replication plan for table <name>, but the inner cause varies with the shape of the DDL — and only two are raised throughvterrorswith a code attached. The rest are plainfmt.Errorf, so they arrive code-less and the matcher rejects them on its very first check:column <c> not found in table <t>cannot use column names in vstream filter ...ADD COLUMN ... AFTER) or reorderedcannot determine table columns for <t>failed to build ENUM and SET column integer to string mappingsTwo of the four were unmatched, and both have been seen in production. Each falls through to the generic error return, so the operator gets an opaque
Internalerror rather than theFailedPreconditionand the explicit "run a historical re-sync" guidance #89 added — precisely the situations that guidance exists for.All four recover the same way: drop the cursor, run a historical sync.
Change
Gate on the
failed to build table replication planwrapper — common to all variants — and enumerate the four recoverable inner causes. TheFAILED_PRECONDITIONcheck is removed rather than relaxed: it excluded real failure shapes while adding no selectivity the wrapper does not already provide.What is deliberately excluded
The wrapper is not sufficient on its own. Two other errors share it and are not recoverable by re-syncing. Both have negative tests:
unsupported type: <n>, position: <i>— a historical sync would hit the same unsupported column type.unknown table <t> in schema— this one is worth spelling out, because it looks like a stale-cursor condition and isn't.historian.GetTableForPosreturns(nil, nil)on a miss, andEngine.GetTableForPosthen falls back to the live schema rather than erroring:So the historian fails open, and this error instead means an undecodable GTID or a table genuinely absent from the tablet's schema. The latter can be transient during an online DDL rename swap, where resetting the cursor would force an unnecessary historical sync.
Tests
Eight table-driven cases — four positive, four negative. The two newly matched positives were each verified to fail against the previous matcher.
go test ./libpasses. (./cmd/...needsmake protofirst, as CI does viatest-ci: proto.)Relationship to #97
Independent, and they compose. #97 adds opt-in automatic recovery once a schema incompatibility is detected; this PR determines what counts as one. Either can merge first — but detection without #97 only improves the error message, and #97 without this one only auto-recovers from two of the four causes.
🤖 Generated with Claude Code