Milestones 3–4: AST, transformers, and the tree-shape gate - #3
Merged
Conversation
Add the typed AST (package ast, DuckDB-native node shapes), the public parser API (Parse/ParseStatement/ParseExpr), and transformers for the full expression layer and SELECT tree: query nodes, set operations (n-ary UNION flattening), CTEs (incl. recursive conversion and USING KEY), table refs, joins, PIVOT/UNPIVOT, samples, windows (named-window resolution), GROUP BY grouping sets, result modifiers, and SHOW/DESCRIBE desugaring. Every upstream transform-time rewrite the pinned build performs is replicated: operator calls as functions, IN/ANY/ALL and BETWEEN desugaring, list comprehensions, ARRAY(subquery), interval literals, star modifiers, prepared-parameter numbering, and constant typing (INTEGER through UHUGEINT, DECIMAL width/scale). internal/serialize renders the AST as json_serialize_sql-compatible JSON; cmd/serialize-diff diffs it statement-by-statement against the pinned DuckDB CLI. The full corpus SELECT subset (25,414 statements) now matches the oracle exactly: 25,301 match, 0 mismatch, 113 skipped (statements the oracle itself refuses to serialize). A deterministic sample of 2,108 statements is vendored as goldens for parser/serialize_test.go; comparator normalizations for upstream's unordered containers are documented on serialize.Equal. The lexer learns upstream's Unicode-whitespace handling, the matcher exposes an Expression entry point for ParseExpr, and debug-parse gains an -ast mode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B918Vu4wvBaZcRUX4nkQdr
DML: INSERT (column lists, BY NAME/POSITION, DEFAULT VALUES, OR
REPLACE/IGNORE and ON CONFLICT, RETURNING), UPDATE (aliased targets,
FROM, tuple SET), DELETE (USING), TRUNCATE and MERGE INTO, with
prepared-parameter numbering end-to-end. DML statements inside a CTE
body become INSERT/UPDATE/DELETE query nodes mirroring upstream's
serialization (TRUNCATE lands in its delete-node form).
DDL: CREATE TABLE (columns, constraints, generated columns, CTAS),
VIEW, SCHEMA, INDEX, SEQUENCE and TYPE; the ALTER families; DROP
families. MACRO/SECRET/TRIGGER stay ErrUnsupported for milestone 5.
The corpus accept/reject harness now classifies with the full Parse
pipeline, so transformer-raised Parser Errors count alongside the
matcher's syntax errors; the todo entries this resolves are removed.
Chasing the remaining disagreements fixed a batch of transformer
behaviors against the pinned oracle:
- regex ANY/ALL (~, ~*, !~, !~* over lists) desugars to upstream's
CASE over list_transform/list_contains/list_filter
- OPERATOR(schema.op) keeps its schema and always builds a function
(comparison spellings included); the prefix form folds qualifiers
into the name; three-part qualifiers raise upstream's error
- other-operator chains ('a' || 'b' || 'c') carry a location only on
the outermost fold, like the binary ladder levels
- FROM-clause UNPIVOT target lists parse (shape bug), multiple IN
groups raise upstream's single-pivot-element error, and the pivot
ref location covers the clause body
- statement PIVOT ... IN (SELECT ...) enum-discovery targets parse
- bare SHOW expands like SHOW ALL TABLES
- number literals beyond the double range become ±Inf DOUBLE
constants, serialized as upstream's bare Infinity tokens
- UPDATE targets span through SET; DELETE/TRUNCATE targets carry no
location; qualified UPDATE SET targets raise upstream's error
- ORDER BY ALL inside ARRAY(SELECT ...) clones the star under the
__array_internal_idx alias
The full-corpus serialize sweep against the pinned binary is clean
(25,974 SELECT-subset statements: 25,846 match, 0 mismatch, 128
skipped where the oracle refuses to serialize), and the vendored
goldens are regenerated from it. New hand-written shape tests cover
the DML and DDL AST surface.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B918Vu4wvBaZcRUX4nkQdr
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements milestones 3 and 4 of PLAN.md: the public AST and Parse API, hand-written transformers for the full SELECT tree plus DML and DDL, and the
json_serialize_sqltree-shape gate — all pinned against the milestone-2 oracle.Milestone 3 — AST + transformer core
ast/: the public AST — statements, query nodes, table refs, expressions, values, and DDL infos, porting upstream's parser node classes.parser/:Parse/ParseStatement/ParseExpron top of the milestone-2 engine, withErrorcarrying message, offset, line, and column; statement spans tile the input.parser/transform_*.go): the port of upstream'stransformer/, replicating its effective behavior — operator desugarings (LIKE family, IN, ANY/ALL, list comprehensions,ARRAY(SELECT), interval literals,IS TRUE/FALSE, EXTRACT), prepared-parameter numbering, named windows, GROUP BY dedup/CUBE/ROLLUP set orders, SHOW/DESCRIBE/SUMMARIZE expansion, PIVOT/UNPIVOT, and upstream's location-tracking quirks (point spans for additive operators, outermost-only spans on chains, location-free synthesized nodes).internal/serialize/: renders the AST injson_serialize_sql-compatible JSON;serialize.Equalcompares structurally with the few documented normalizations for upstream's unordered hash containers.cmd/serialize-diff: sweeps ad-hoc statements or the corpus SELECT subset against a live oracle binary and writes the vendored goldens.cmd/debug-parse -ast: dumps the transformed AST as JSON.Milestone 4 — DML + DDL transformers
ErrUnsupportedfor milestone 5.Conformance
TestCorpus) now classifies with the full Parse pipeline, so transformer-raised Parser Errors count alongside the matcher's syntax errors; the 29 todo entries this resolves are removed. Getting the gate green fixed a batch of oracle-verified behaviors: the regex~/~*/!~/!~*ANY/ALL CASE desugar,OPERATOR(schema.op)semantics, FROM-clause UNPIVOT shapes,PIVOT ... IN (SELECT ...), bareSHOW, out-of-range literals as ±Inf DOUBLEs (serialized as upstream's bareInfinitytokens), JOIN BY type aliases, and DML target-location conventions.TestSerializeGoldens): 2,153 vendored goldens sampled from the corpus SELECT subset.parser/dml_test.go,parser/ddl_test.go).go build && go vet && gofmt -l && go test -race ./...all pass.🤖 Generated with Claude Code
https://claude.ai/code/session_01B918Vu4wvBaZcRUX4nkQdr
Generated by Claude Code