Evaluate volatile column DEFAULTs per row instead of freezing them into cached plans - #316
Merged
Merged
Conversation
…gines Add coverage for nextval/currval/setval, SERIAL column allocation, and DEFAULT expressions (nextval, currval, scalar functions, UUID_V7) across strict, schemaless, KV, and columnar engines. Also cover the unknown-sequence and unevaluable-DEFAULT error paths.
Add wire-level integration tests asserting UUID_V7(), NOW(), and nextval() DEFAULT expressions produce fresh values per execution rather than replaying a cached plan's frozen result, including across repeated byte-identical INSERT statements.
The schemaless-document arm dropped every column's DEFAULT clause, the primary key's included, when building catalog ColumnInfo. A declared default never reached evaluation, so the column was written absent and read back as NULL. Recover each field's default from its declared type in stored.fields, the way the columnar arm already does.
Introduce nextval, currval, and setval as volatile scalar functions routed at plan time to SqlCatalog sequence methods, with per-session last-value tracking for currval. Add a plan-level volatility scan and a Volatile variant so a plan containing a volatile call (sequence functions, UUID/ULID/nanoid generators, DEFAULT expressions) is excluded from the plan cache and re-evaluated on every execution instead of freezing a value. Wrap lowered physical plans in a LoweredPlan that carries this cache verdict through the gateway, and thread the new SQLSTATEs and error constructors needed to reject invalid sequence access.
nextval/currval DEFAULTs now resolve through the catalog instead of being silently dropped. evaluate_default_expr and materialize_row_defaults require a SqlCatalog handle, and ConvertContext carries an optional Arc<dyn SqlCatalog> so INSERT and UPSERT conversion can reach it after planning returns. Consolidate the four drifted copies of the SqlError -> Control-Plane Error mapping (only one mapped RetryableSchemaChanged) into a single plan_error_map module shared by query planning and DEFAULT materialization. Extract engine routing for row-shaped INSERT/UPSERT converters into a dedicated route module, and DML helper parameters into a params module.
…AULTs
SERIAL and BIGSERIAL columns now expand to their backing type plus a
DEFAULT nextval('{collection}_{field}_seq') naming the sequence the
caller creates, instead of only the bare type. parse_fields_clause and
parse_fields_clause_from_pairs take the collection name to build that
sequence name.
Add a DDL-time gate that classifies and parses every declared column
DEFAULT through the same paths evaluate_default_expr uses, so a call
to an unregistered function is refused at CREATE with SQLSTATE 42883
instead of surfacing at the first INSERT. The gate parses a setval or
sequence-accessor DEFAULT without evaluating it, so declaring a column
never advances a sequence.
Prepare and describe round trips for an INSERT with a nextval DEFAULT must plan the statement without allocating from the sequence; only executing the statement advances it.
A column declared with a numeric type keeps its OID, comparison semantics, and exact integer round trip when followed by a DEFAULT clause or a NOT NULL modifier, across the document (strict and schemaless) and columnar engines.
A vector-primary insert splits each row into the vector field and a payload map before the shared DEFAULT pass runs. Cover a nextval and a UUID_V7 key DEFAULT and a non-key payload DEFAULT, each on a column the insert omits.
…serts Extract the KV engine's DEFAULT-materialization helper into a shared declared_defaults module and reuse it for vector-primary collection inserts, which previously bypassed EngineRules::plan_insert and never expanded a declared DEFAULT. Materialization runs before the row's declared-type coercion and range checks, so a default is validated exactly like a supplied literal. Track whether any materialized default came from a volatile expression (e.g. nextval) on VectorPrimaryInsert, matching the KV insert plan's existing gate, so a plan containing one is excluded from the plan cache instead of replaying a frozen value.
A write plan's OutputSchema previously carried no columns, so the simple-query protocol fell back to deriving a RowDescription from the row payload rather than the catalog. Thread the parsed RETURNING spec through every planning entry point down to build_output_schema, which now announces the target collection's declared columns for a named clause and defers to the row-derived shape only for RETURNING *. Propagate the same announced schema through the Calvin dispatch and response paths and the neutral DDL DML path, so a RETURNING write renders identically regardless of which route it took. Split output_schema.rs into a directory of build/columns/returning modules along the way, and fix parse_type_str to strip DEFAULT/NOT NULL modifiers off a catalog column's raw declared-type text before resolving its wire type. Propagate a vector-primary insert's msgpack serialization error instead of discarding it into an empty payload.
The memtable stores every timeseries instant in epoch milliseconds, but a client reads a declared TIMESTAMP/TIMESTAMPTZ cell as epoch microseconds. Raw scans and RETURNING rows from ingest handed back the raw millisecond count unscaled, understating the stored instant by three orders of magnitude on read. Add CoreLoop::ts_instant_columns to list a collection's declared instant columns and scale_instant_cells to rescale those columns' values from milliseconds to microseconds once rows leave a scan or ingest. A BIGINT TIME_KEY shares the same storage column but is not a declared instant, so it keeps the integer that was inserted.
Drop the separate result_fields_for_returning path and its private SqlDataType-to-pg mapping. Describe now infers RETURNING columns through the same build_output_schema call used for SELECT projections, so the extended-query and simple-query paths can never disagree on a column's type. Update the wire test to read numeric RETURNING columns by their announced OID (int2/int4/int8, float4/float8) instead of assuming int8/float8, since OutputSchema reports the narrower declared width.
… scans An output schema for a JOIN previously defaulted every projected column to Text. It now resolves each column against the catalog of the side it came from, keyed on the qualified name the join executor emits, falling back to Text only for a bare name two sides declare with different types. A grouped, non-bucketed timeseries scan now announces its GROUP BY keys with their catalog types instead of leaving the whole schema untyped, matching the shape its aggregate encoder emits.
A join reads a timeseries collection through scan_collection, which hands back its stored epoch-millisecond value, while a client reads a TIMESTAMP cell as epoch microseconds. Hash, nested-loop, and sort-merge joins now rescale the declared-instant cells they read from their own local scans immediately before emission, after every predicate has run so the join itself still compares milliseconds against milliseconds. A side supplied by a sub-plan is excluded: its rows were already rescaled by the handler that produced them, so rescaling again would double the correction.
A grouped timeseries scan reduces every key to a string, and the aggregate encoder rendered every one of them back as text regardless of the column's real type. It now resolves each GROUP BY column's storage kind — declared instant, integer, float, or text — from the collection's declared schema or its resident memtable schema, and renders the key with that type. A declared TIMESTAMP key converts its stored milliseconds to the microseconds a client expects, matching how a direct SELECT of the same column renders it.
…UP BY A stored TIMESTAMP time key can be read directly, projected through a JOIN, or used as a GROUP BY key, each through its own scan and encoder. Cover that all three render one stored instant identically, anchored against the instant the INSERT actually supplied.
Each engine's EngineRules now picks the WriteRoute (Document or ColumnarFamily) when it builds an Insert or Upsert plan, carrying it on the SqlPlan variant instead of re-deriving it later from the engine type. The conversion layer reads the route directly, removing the separate route-resolution module.
A projection that renames a joined declared-instant cell (an alias or a computed expression) hid it from the millisecond-to-microsecond rescale, which matched on the emitted output name. Rescale now runs on the merged row against the keys the join itself wrote, before any projection or computed column touches it, so a rename or computation downstream cannot bypass the scale.
ColumnType::from_declared_type resolves a catalog fields entry's raw DDL text (e.g. "INT DEFAULT 5") to its ColumnType through one shared classifier, and ColumnType::is_instant answers whether a column carries an instant. The Control Plane's parse_type_str and the Data Plane's declared_type_is_instant / memtable_column_type now both defer to these instead of keeping their own copies of the integer/float keyword lists and timestamp-variant matches, so the two planes cannot drift on which columns are instants or which spellings resolve to which type. DECLARED_INT_KEYWORDS and DECLARED_FLOAT_KEYWORDS name the accepted PostgreSQL wire-width spellings once, shared with the corresponding IntWidth/FloatWidth classifiers.
Split planner::defaults into a module with a ColumnDefaults / CompiledDefault pair: declaration text is classified and parsed once into a DefaultKind (generator, literal, or parsed expression), and each row then evaluates the compiled form without re-parsing. The KV insert path, declared-defaults materialization, and the columnar/ document row-expansion path (expand_row_defaults) all build one ColumnDefaults outside their row loops instead of re-invoking evaluate_default_expr per row per column. Drop Volatility::Stable, which named a per-statement reuse boundary nothing in the planner ever used; Volatility now distinguishes only Immutable (foldable at plan time) from Volatile (fresh per call).
Route the TYPEGUARD type-expression parser through the same ColumnType resolver CONVERT and CREATE use, so one declared spelling (VARCHAR(n), TIMESTAMP WITH/WITHOUT TIME ZONE, SYSTEM_TIMESTAMP, DECIMAL(p, s), and every registered numeric alias) resolves to the same type everywhere, and a trailing word after a keyword is rejected instead of silently dropped. Generalize the CREATE-time column DEFAULT gate into a shared column_default module and reuse it from CONVERT's column list and from a new typeguard DEFAULT/VALUE gate, so a declaration naming an unregistered or non-deterministic function is refused at DDL time across all three surfaces instead of failing at first write. Split the CONVERT COLLECTION handler out of its single file into a directory of driver, column-defs, type-map, and typeguard-columns modules to hold the added validation without growing an already large file.
farhan-syah
added
the
run-ci
nextval/currval/setval only evaluate through SqlCatalog's sequence state at plan time, in a FROM-less SELECT or a column DEFAULT. A call over a FROM relation instead reached the row evaluator, which holds no sequence state and silently returned NULL for every row. Name the three accessor names once in a shared sequence_accessor module and use it from both the constant folder and the new resolver gate. TableScope::is_row_scope and ColumnScope::is_row_scope report whether an expression sits behind a relation; convert_function_depth refuses a sequence accessor there with the new SqlError::SequencePerRowUnsupported. Wire the refusal through to SQLSTATE 0A000 (feature_not_supported) on both the plan-error and pgwire error-mapping paths via a new crate::Error::FeatureNotSupported, and through error_classify for the native surface.
This was referenced Sep 10, 2026
farhan-syah
deleted the
fix/sequence-accessors-and-volatile-defaults
branch
September 10, 2026 11:07
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Non-deterministic expression evaluation ran at plan-construction time, and plans are cached by SQL text. Sequences were the loudest instance, not a special case.
DEFAULT UUID_V7()on a primary key raisedSQLSTATE 23505on the second identical INSERT. The cached plan replayed the first execution's frozen UUID. That defect was live onmainand unfiled.Reported defects fixed
nextval/currval/setvalraised42883SqlCatalogagainst the sequence registrycurrvalread the node-wide counterevaluate_default_exprreturnedOption, andNonedropped the column silentlyResult. Silent omission is not expressibledefault: None, dropping every DEFAULTCREATE COLLECTIONagainst the sameFunctionRegistrythe resolver usesSERIALcreated its sequence, and nothing linked the column to itDEFAULT nextval('<collection>_<field>_seq')Model adopted
Function volatility is a first-class axis, orthogonal to
FunctionCategory. A volatile function never folds into a reusable plan. This follows PostgreSQL.Defects found while fixing, also closed
These were not in the source report.
TIMESTAMPstored in milliseconds and read as microsecondsRETURNINGannounced an emptyOutputSchema, disagreeing with an equivalentSELECTon the same rowGROUP BYCONVERT COLLECTION c TO document_strict (name WIDGET)silently produced aStringcolumnDEFAULTclause in aCONVERTcolumn definition was parsed and discardedDEFAULT no_such_function()wrote NULL on every row and reported nothing, becausenodedb_query::functions::eval_functionreturnsOk(Value::Null)for an unknown nameConsolidation
ColumnType::from_declared_typeinnodedb-types.CREATE COLLECTION,CONVERTandCREATE TYPEGUARD.nodedbintoEngineRules, removing amatch enginefrom outsidenodedb-sql.Testing
16100 workspace tests pass. 367 cluster tests pass.
cargo clippy --workspace --all-targets -- -D warningsis clean.Coverage added in
sql_sequences.rs,sql_default_expressions.rs,sql_default_volatility.rs,sql_declared_column_types.rs,sql_default_vector_primary.rs,timeseries_join_time_rendering.rs,sql_convert_column_defs.rs,sql_typeguard_default_gate.rs.Known limits
TimeseriesScanarrives in microseconds while a local side is millisecondsONcompares mismatched unitstime_bucketover a join has no coverageVALUE LOWER(status)passes the column-DEFAULT gatenodedb-litereadsSqlPlan::Insert.engineand matchesengine_typeinlower_insertroute: _at the nextnodedb-sqlversion bump0A000SELECT nextval('s') FROM tis refused. Tracked by #314Filed as #317 (the first two limits), #318 (the typeguard guard) and #319 (a
SERIALDDL-surface mismatch found while fixing).Registering the accessors makes them pass the plan-time existence gate. A per-row
SELECT-list has no evaluation for them, so the resolver refuses that scope rather
than letting
nodedb_query::functions::eval_functionreturnOk(Value::Null)perrow. Per-row evaluation needs a post-Data-Plane stamping stage that does not exist
today.
nodedb/src/control/gateway/fuser.rsconcatenates payloads in arrivalorder, not output-row order.
Breaking changes
RowDescriptionOIDs narrow to the declared width.INTEGERannouncesINT4, notINT8.DECIMAL/NUMERICandVECTOR/GEOMETRYannounceTEXT.RETURNINGon the simple-query path renders through the column's real type, so a whole float renders1rather than1.0.Issues closed
Closes #294 — the source report.
Closes #298 — a timeseries
TIME_KEYdeclaredTIMESTAMPannounced OID 25 carrying raw epoch milliseconds. It announces OID 1114 and renders a timestamp.Closes #313 — an unknown sequence in a DEFAULT raised
42601. It raises42704.#314 stays open. This branch preserves its
0A000refusal and adds coverage pinning it.