Make REPL.Syntax a compact index with named structural lookups #1178

Story

As a Sidekick using xmd repl, I want to discover available syntax through a compact index and request complete references by name, so learning what exists does not fill the conversation with every symbol's schemas and usage details.

The same named-reference mechanism explains ordinary components and structural constructs such as If and All. Structural usage guidance remains reachable when the discovery response becomes smaller.

Example

<REPL.Syntax />

This returns exact names grouped by category, each with a short purpose and instructions for requesting details.

<REPL.Syntax names={["If", "File"]} />

This returns the complete selected references. Structural lookup also works through the existing shared surfaces:

xmd syntax If

<Syntax names={["If"]} />

Current gap and measured evidence

Bare REPL.Syntax calls the full symbol Markdown renderer. Although it omits long-form package documentation, it still renders prop tables, JSON schemas, return schemas, origins and detailed descriptions for the entire vocabulary.

In an inspected live session, the bare lookup returned 79,569 characters for 79 symbols. A simultaneous named lookup added 9,953 characters; the combined feedback packet was 89,556 characters. These are character counts, not token counts. The resulting reference stays in the agent conversation.

Structural constructs already appear in bare listings, but packages/core/src/syntax-reference.ts explicitly excludes structural entries from named selection. A listed name such as If therefore fails as an unknown component. The language specification currently documents that refusal.

Pagination alone limits each response's size without separating discovery from detailed reference. This story makes that separation explicit.

Contract

Compact discovery

• Bare <REPL.Syntax /> returns an index of the current entry vocabulary: exact names, existing categories and short purposes, followed by the named-lookup invocation.
• The index omits prop tables, JSON schemas, return schemas, detailed usage and origin boilerplate. It remains an index as the vocabulary grows rather than appending a full reference per symbol.
• Build the index from the same structured symbols as detailed lookup. Do not maintain a separate REPL catalog or parse the detailed Markdown back into an index.
• Preserve current profile, include and Plugin selection, ordinary component precedence, reference-only testing guidance and truthful availability. Current session component changes appear on subsequent reads; saved discovery remains historical.
• Named ordinary-component lookup continues to supply complete metadata and documentation. No silent truncation replaces a complete named reference.
• The required compact default is REPL.Syntax. Preserve bare canonical <Syntax />, xmd syntax and versioned JSON behavior unless a deliberate shared-format change is specified and its consumers updated. Reusing the structured catalog does not require changing every presentation.

Named structural references

• Named xmd syntax, <Syntax names={…} /> and <REPL.Syntax names={…} /> select engine-owned and installed structural constructs from the reference available to that surface.
• Each selected construct supplies its authored spelling and forms, purpose, applicable props, structural constraints and documentation sufficient to use it. Installed parent/child relationships remain explicit; no ordinary-component return or binding contract is invented.
• Metadata and documentation come from the existing engine or installation owner. No pretend component registration supplies structural reference material.
• Mixed structural/component requests return each selected reference once in established symbol order. Preserve unique-name validation, ordinary as capture and all-or-nothing refusal when any requested name is genuinely unknown.
• Preserve the distinction between enclosing authoring references and effective execution availability, including narrowed evaluations and installed construct identity. Documentation grants no execution authority and invokes no expansion handler.
• Preserve package-owned documentation selection and custom-component behavior. Ship structural lookup before or together with the compact index so removing detailed structural guidance from discovery does not make it inaccessible.

Acceptance and verification

1. A representative broad vocabulary returns a complete compact index with all expected names and short purposes, no schema/table dump, and an actionable named lookup. Record before/after character sizes on the same fixture; a renamed heading over unchanged detailed output fails acceptance.
2. Selecting ordinary components still returns their full contracts. Selecting an engine construct, an installed parent/child pair and a mixed structural/component list returns the correct usage reference.
3. An unknown name in a mixed request refuses without partial output. Narrowed execution reports truthful availability without granting capability or executing a structural handler.
4. Profile/include changes and ordinary same-named component precedence remain correct. Source and distributed-binary named lookups have the documentation they require.
5. Update specs/repl-spec.md, specs/executable-mdx-spec.md §5.3.1, relevant CLI/component documentation and Sidekick guides together. Bootstrap guidance explains index-first discovery and named detail reads.

Focused evidence belongs in packages/core/tests/syntax-component.test.ts, syntax-api.test.ts, syntax-catalog.test.ts, packages/cli/tests/syntax-cli.test.ts and the affected Sidekick controls and installed-structure tests. Use deno task test --changed plus explicit compiled/subprocess tests that the import graph cannot select.

Relationships and scope

Follow-up to completed #758 and #806. #1161 owns token-consumption diagnostics; this story reduces the observed discovery payload. #1177 owns repeated Claude model switching.

Pagination for exceptionally large individual references may be evaluated separately from the required index/detail split. This story adds no structural execution capability and does not change runtime dispatch, permissions or component resolution.

Reserved higher-value pilot assignment

The maintainer asked for more valuable work than the earlier #1207/#1210 pair. This issue is reserved for one fresh Codex Implementor pilot, coordinated by #1199's existing owner (chat 01a1bcee-8df7-77b3-86db-21dcde6a4b3c), after all accepted workflow changes and independent review-summary repair pass review/checks. Use the same exact frozen workflow revision as paired issue #1183. The earlier pair was withdrawn before any pilot dispatch.

Bounded scope: Implement the compact REPL.Syntax index and complete named structural references already specified here, preserving canonical bare Syntax/CLI/JSON behavior. Use the existing catalog, keep complete named details and current availability/precedence. No new discovery framework, language changes, whole-spec reading requirement or token accounting work (#1201 owns that; legacy #1161 is closed). Keep #1197 version drift separate.

The user authorized unattended planning, implementation, applicable/independent reviews, focused PR publication and CI without additional routine approval. Preserve finite budgets, original failures and native identity; no silent model substitution or reset. Recheck current ownership and whether the bug persists before dispatch. Genuine new contract decisions remain explicit blockers; continue the other task if one blocks. Keep PRs reviewable for the user's decision; no merge authorization is granted here or for #1203.

Record actual workflow/source pins, first-result validity, guide-read timing, correction/additional-call counts, summary-overflow handling and manual interventions. Collect one short debrief from the original Implementor once useful work is complete. Use deterministic regression evidence rather than fabricating an overflow or adding another paid review run.