RPCN - Support for optional fields

[SpicyPete]

RPCN: adopt restored schema flags and fix drifted templates

Background

cloudv2 #27724 (merged 2026-07-14) bumped benthos/v4 4.53 → 4.73 in redpanda-connect-api, which re-serializes the Connect config schema before serving it. Benthos only began emitting per-field is_optional / is_advanced / is_secret / is_deprecated at v4.59, so until that bump the console never received them and the frontend compensated with a defensive heuristic ladder (checkRequired) that guessed required-ness from proto flags and name patterns.

The two RPCs still differ in what they carry (verified against cloudv2 source and the embedded RPCN 4.100 schema):

Signal	ListComponents (proto FieldSpec)	GetPipelineServiceConfigSchema (raw JSON)
optional / advanced / deprecated	✅	✅
is_secret	❌ no field in the proto at all	✅
defaults	⚠️ string defaults only — int/bool/collection defaults arrive as ''	✅ exact required arrays computed by the backend with full default knowledge

So required/secret truth must come from the raw JSON path; the proto alone cannot distinguish "required" from "default lost in serialization".

What's changed

Schema-signal enrichment

• New enrichComponentsWithConfigSchema (utils/schema-enrichment.ts) stamps requiredBySchema and secret onto proto-derived specs by walking the raw config-schema JSON (object/array/map wrappers unwrapped so JSON-Schema nodes line up with proto children).
• Degrades gracefully: a schema that is missing, unparsable, or pre-4.59 (no is_optional anywhere) leaves specs unstamped.
• New useEnrichedComponents hook consolidates the recipe for all three YAML-generating consumers (pipeline editor, onboarding wizard, template form panel) and keeps components empty until the schema query settles, so YAML is never generated from un-enriched specs. Display-only consumers (gallery grid, tiles, command palette) intentionally stay on plain parseSchema.

Required-field detection

• checkRequired is now: deprecated guard → trust requiredBySchema stamp → documented degraded proto ladder (for pre-4.59 dataplanes only). Dead heuristic branches deleted.
• generateDefaultValue no longer zero-fills fields whose default was dropped by the proto — they're omitted so engine defaults apply (previously e.g. auto_replay_nacks defaulting to true was rendered as false).

Visual editor forms

• Deprecated fields are excluded from form fields and generated configs; existing values surface in the raw-YAML section.
• Secret masking is now the schema's is_secret flag unioned with the old name regex (the flag misses e.g. AWS session tokens; the regex is all we have on older dataplanes).
• Handles object nodes benthos serializes with an empty kind (batching policies).

Template fixes (drift caught by this work)

Four templates referenced components/fields the engine never shipped or has since renamed — invisible because the test snapshot was frozen at RPCN 4.65 with the components allowlisted as known-missing:

• dynamodb-cdc: aws_dynamodb_stream → aws_dynamodb_cdc (tables list, start_from/snapshot_mode)
• oracle-cdc: oracle_cdc → oracledb_cdc (connection_string, include)
• sqlserver-cdc: sql_server_cdc → microsoft_sql_server_cdc (connection_string, include)
• iceberg: catalog_uri/warehouse → catalog.url / optional catalog.warehouse — this one was user-visible broken: the component exists, so the template showed in the gallery and its YAML failed dataplane lint on every deploy
• Refreshing assets/rp-connect-schema-full.json 4.65 → RPCN 4.100 (the cloud schema cloudv2 actually serves) also caught redpanda_migrator having dropped its batching field; removed from the mirror template.
• KNOWN_MISSING_COMPONENTS is now empty — every template is fully validated by the regression tests (schemaField paths resolve, stitched YAML contains only known fields, required fields always emitted).

Behavior changes to be aware of

• The redpanda input's topics and seed_brokers are no longer marked required — they genuinely are .Optional() in the engine (regexp_topics_include is the alternative). The wizard still auto-populates them.
• Optional fields whose defaults don't survive the proto are omitted from starter YAML instead of being zero-filled.
• Hardcoded template slot required values were audited against the served schema: templates stricter than the engine (e.g. S3 region/credentials) are deliberate curation; the one looser case (postgres-cdc slot_name) is safe because defaultWhenBlank always supplies a value.

Deferred (needs cloudv2 changes)

• is_secret can never ride the proto path — the field doesn't exist in FieldSpec; a protogen regen won't help.
• cloudv2's convertFieldSpec still drops non-string defaults. Serializing them would let the degraded checkRequired ladder be deleted entirely.

[github-actions]

✅ Clean — no registry drift, off-token colours, or ad-hoc classes

App: frontend · Scope: diff vs origin/master · Files: 31

	Count
⚠️ Outdated registry components	0
🛠 Locally-modified components	0
❓ Unknown to registry	0
🎨 Off-token palette colours	0
🔢 Ad-hoc utility classes	0

Generated by lookout audit-changes.

[SpicyPete]

@claude review

[github-actions]

Claude finished @SpicyPete's task in 3m 24s —— View job

---

Code Review — RPCN optional field support

• Read core logic (schema-enrichment.ts, schema.ts, use-enriched-components.ts)
• Review visual editor form changes
• Review template fixes & deploy logic
• Review tests
• Post consolidated review

---

Reviewed the full diff against origin/master (skimming the 42k-line rp-connect-schema-full.json regen, reading the logic files closely). This is a careful, well-documented change — the layering (proto specs → JSON-schema enrichment → checkRequired) is clean, degradation paths are explicit, and the test coverage (schema-enrichment.test.ts, template-form-schema.test.tsx, ground-truth fixtures) is genuinely strong: it pins the exact proto-loss cases the design hinges on (checkpoint_limit int default, auto_replay_nacks bool, empty-string consumer_group, is_secret on root_cas). Nice work.

Findings below are all minor / non-blocking.

1. Template-form default reset now depends on the config-schema query settling

template-form-panel.tsx:174-184 — the one-shot form.reset(defaultValues) is now gated on isComponentsLoading (which is isComponentListLoading || isSchemaLoading). Previously it fired as soon as componentListResponse arrived. useEnrichedComponents already returns a non-empty components array the moment ListComponents resolves (enrichment degrades gracefully when configSchema is undefined), so if the config-schema query stalls indefinitely without erroring (React Query keeps isLoading: true), the schema-driven default reset never fires. On error isLoading flips to false, so only an indefinite hang is affected, and the form remains usable with its first-render defaults — impact is low. Worth a comment noting the intentional trade-off (waiting for schema-exact required-ness at the cost of resilience to a hung schema query), or a fallback that resets on componentList alone after a timeout.

2. List-entry scalars are force-set to Scalar.PLAIN

template-deploy.ts:116-118 — expanded list entries get scalar.type = Scalar.PLAIN. For the current call sites (table identifiers, validated by entryPattern to ^[a-zA-Z0-9_$]+$) this is safe, but a plain scalar bypasses the quoting the YAML serializer would otherwise apply, so a future list slot without a strict pattern could emit an entry like 123 or true that the engine parses as int/bool instead of string. Consider dropping the explicit PLAIN and letting the serializer decide, or documenting that list slots must be pattern-constrained to bare strings.

3. Module-level enrichment cache is never invalidated

use-enriched-components.ts:24-37 — lastEnrichment is a module singleton holding references to the (multi-MB) component list, config-schema string, and cloned spec tree for the lifetime of the session. It's a single slot, so a new componentList/configSchema identity replaces it, and the memo rationale is sound — just flagging that these large objects stay resident even after all consumers unmount. Fine to leave; noting for awareness.

4. PR-description wording is stale vs. the code (doc only)

The description says useEnrichedComponents "keeps components empty until the schema query settles, so YAML is never generated from un-enriched specs." The actual code (and the hook's own doc comment) does the opposite — components is populated as soon as ListComponents resolves and is re-emitted enriched once the schema settles, with checkRequired falling back to proto flags in between. The code behavior is the better one; just update the PR body so reviewers aren't misled about the invariant.

Nits

• template-schema.ts:51 — checkRequired(field as unknown as RawFieldSpec): the double cast is only needed because resolveFieldByPath is typed to return the proto FieldSpec while the runtime value is the stamped RawFieldSpec. Consider having resolveFieldByPath return RawFieldSpec for enriched trees to drop the as unknown.
• template-deploy.ts:84-92 — the Pair visitor calls substituteScalar purely to detect a null (drop) result, then the Scalar visitor recomputes it to apply the substitution. Harmless on small template docs, just slightly redundant.

Not verified

I did not run bun run type:check && bun run lint && bun run test in this review (would require a full dependency install). Please make sure those pass — in particular the removed useListComponentsQuery/parseSchema imports in onboarding-wizard.tsx and pipeline/index.tsx don't leave an unused useMemo or import behind for Biome.
· branch rpcn/optional-field-support