Codegen Go and TypeScript SyntaxKinds, ASTs, factories, visitors, type guards, encoders, decoders

[andrewbranch]

Thanks to @virtulis for some additional tests; supersedes #3195.

I will try to list the more interesting changes here, but I recommend just looking at the changed files that are not thousands of lines of generated code diff to see where usage actually changed; it's fairly small.

Actual AST changes made:

• Renamed SyntaxKind.JSDocTag to SyntaxKind.JSDocUnknownTag
• Deleted SyntaxKind.CommaListExpression (unused in Corsa)
• In Corsa, reordered VariableDeclarationList factory params to align with Strada
• Some scalar properties were previously left out of factory update functions for unclear reasons; they are universally included now.

Cosmetic AST changes made to Corsa:

• As casts always match their struct name, e.g. AsTypeParameter() → AsTypeParameterDeclaration()
• Embedding hierarchy expanded to match branded interfaces in Strada, e.g. PrefixUnaryExpression goes from embedding ExpressionBase directly to a chain of UpdateExpressionBase → UnaryExpressionBase → ExpressionBase, because these give the corresponding TypeScript interface _updateExpressionBrand, _unaryExpressionBrand, and so on.
• Every node automatically gets a named type alias for Node and those aliases are consistently used in struct field types and factory/visitor parameters.

Making changes

• edit _scripts/ast.json
• run hereby generate:ast

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR moves AST infrastructure toward a schema-driven codegen pipeline, generating Go/TypeScript AST types and protocol helpers while removing older per-package generators.

Changes:

• Introduces schema + generators to emit Go AST/kind definitions and TS AST scaffolding.
• Switches @typescript/ast consumers from nodes.ts/hand generators to ast.ts + generated factory/visitor artifacts.
• Refactors encoder/RemoteNode infrastructure to rely on generated protocol tables and node accessors; updates CI and tests accordingly.

Reviewed changes

Copilot reviewed 57 out of 134 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
_scripts/generate-go-ast.ts	New generator for Go AST structs, factories, visitors, kind guards, and kind enum output.
_scripts/ast.schema.json	Adds JSON schema for AST definition inputs used by generators.
_packages/ast/src/scanner.ts	Updates scanner typings/imports and defines a JSDoc token-kind union tied to SyntaxKind.
_packages/ast/src/index.ts	Re-exports new ast.ts surface and drops nodes.ts export.
_packages/ast/src/enums/tokenFlags.ts	Updates generated-file header to new Hereby generator format.
_packages/ast/src/enums/tokenFlags.enum.ts	Updates generated-file header to new Hereby generator format.
_packages/ast/src/enums/syntaxKind.ts	Updates generated SyntaxKind output and header to reference kind_generated.go.
_packages/ast/src/enums/syntaxKind.enum.ts	Updates generated SyntaxKind enum output and header to reference kind_generated.go.
_packages/ast/src/enums/outerExpressionKinds.ts	New generated JS enum output for OuterExpressionKinds.
_packages/ast/src/enums/outerExpressionKinds.enum.ts	New generated TS enum output for OuterExpressionKinds.
_packages/ast/src/enums/nodeFlags.ts	Updates generated-file header to new Hereby generator format.
_packages/ast/src/enums/nodeFlags.enum.ts	Updates generated-file header to new Hereby generator format.
_packages/ast/src/enums/modifierFlags.ts	Updates generated-file header to new Hereby generator format.
_packages/ast/src/enums/modifierFlags.enum.ts	Updates generated-file header to new Hereby generator format.
_packages/ast/src/clone.ts	Switches to generated factory imports and updates literal cloning to carry tokenFlags.
_packages/ast/src/astnav.ts	Switches to ast.ts + generated factory types; fixes JSDoc kind guard naming and token factory typing.
_packages/ast/src/ast.ts	New hand-written AST facade layering generated AST types + core shared interfaces.
_packages/ast/scripts/generateVisitor.ts	Removes legacy visitor generator script (now superseded by schema-driven generation).
_packages/ast/scripts/generateFactory.ts	Removes legacy factory generator script (now superseded by schema-driven generation).
_packages/ast/package.json	Updates exports/scripts to use generated factory and new generate:ast script.
_packages/api/test/sync/ast.test.ts	Updates tests for new AST type names and updated factory signatures/flags.
_packages/api/test/sync/api.test.ts	Adds roundtrip/clone coverage and factors out tsserver path helper.
_packages/api/test/encoder.test.ts	Adjusts encoder tests to updated factory signatures and RemoteNode typing.
_packages/api/test/async/api.test.ts	Mirrors sync test additions/updates for async API and clone/roundtrip checks.
_packages/api/src/node/protocol.ts	Removes hand-maintained childProperties table and re-exports generated protocol tables.
_packages/api/src/node/protocol.generated.ts	Adds generated childProperties + singleChildNodePropertyNames tables.
_packages/api/src/node/node.ts	Splits heavy RemoteNode code into generated + infrastructure modules and re-exports public surface.
_packages/api/src/node/node.infrastructure.ts	New shared infrastructure helpers/constants for RemoteNode decoding.
_packages/api/src/node/node.generated.ts	Adds generated RemoteNode/RemoteNodeList implementation and generated property getters.
_packages/api/src/node/encoder.ts	Refactors encoder to use generated node-data helpers and new child-presence semantics.
_packages/api/src/node/encoder.generated.ts	Adds generated helpers for node-data typing + common-data bit packing.
_packages/api/src/enums/typePredicateKind.ts	Updates generated-file header to new Hereby generator format.
_packages/api/src/enums/typePredicateKind.enum.ts	Updates generated-file header to new Hereby generator format.
_packages/api/src/enums/typeFlags.ts	Updates generated-file header to new Hereby generator format.
_packages/api/src/enums/typeFlags.enum.ts	Updates generated-file header to new Hereby generator format.
_packages/api/src/enums/symbolFlags.ts	Updates generated-file header to new Hereby generator format.
_packages/api/src/enums/symbolFlags.enum.ts	Updates generated-file header to new Hereby generator format.
_packages/api/src/enums/signatureKind.ts	Updates generated-file header to new Hereby generator format.
_packages/api/src/enums/signatureKind.enum.ts	Updates generated-file header to new Hereby generator format.
_packages/api/src/enums/signatureFlags.ts	Updates generated-file header to new Hereby generator format.
_packages/api/src/enums/signatureFlags.enum.ts	Updates generated-file header to new Hereby generator format.
_packages/api/src/enums/objectFlags.ts	Updates generated-file header to new Hereby generator format.
_packages/api/src/enums/objectFlags.enum.ts	Updates generated-file header to new Hereby generator format.
_packages/api/src/enums/elementFlags.ts	Updates generated-file header to new Hereby generator format.
_packages/api/src/enums/elementFlags.enum.ts	Updates generated-file header to new Hereby generator format.
_packages/api/src/enums/diagnosticCategory.ts	Updates generated-file header to new Hereby generator format.
_packages/api/src/enums/diagnosticCategory.enum.ts	Updates generated-file header to new Hereby generator format.
_packages/api/package.json	Adds encoder generation step and updates generator scripts to new pipeline.
Herebyfile.mjs	Adds generate:ast task, updates SyntaxKind enum source, and standardizes TS enum headers.
.github/workflows/ci.yml	Updates CI generation steps to run generate:ast instead of older package scripts.

[jakebailey]

Curious if this could be generated, given it seems 1:1

[andrewbranch]

Surely. ModifierFlags are currently generated from the Go source the way other enums (besides SyntaxKind, now) are, so there would be a bit more setup to do this correctly, but it would be nice.

[jakebailey]

In a way I wish we'd use a more compact language like KDL, or a JS file with a little DSL, but, JSON is ubiquitous, so

[andrewbranch]

I was also tempted to use KDL, but I figured anything besides JSON would be controversial, having schema support is nice, importing with no processing is nice, and any other choice would be controversial, but it would be free to switch down the road if we want.

[jakebailey]

Yeah, the JSON schema thing is helpful.

Evil voice: it would work for yaml too

[jakebailey]

were these things from the spec? or is this rename okay?

[andrewbranch]

There was a replace-all that got much more than intended (same thing that caused the accidental checker.Type method change) which I tried to undo everything undesriable, but I thought these were talking about node kinds, so I left them. Now that you mention it, I think it was copied from the spec, but I don't know how relevant that is now. I'm ambivalent about this; do you want me to undo it?

[jakebailey]

Probably doesn't matter that much, outside them matching Strada's comments

[jakebailey]

LGTM, but if you want to do the other additions I won't complain 😄

Will probably conflict with whatever @ahejlsberg is doing with reparsing, but hopefully not too hard to fix up.

[andrewbranch]

There are things that I'd like to improve and additional things I'd like to generate, but need to get some more time-sensitive stuff out of the way first.