Skip to content

Extract indexing field metadata behind a format-neutral value object - #1351

Open
jwils wants to merge 1 commit into
joshuaw/indexer-ingestion-adaptersfrom
joshuaw/indexing-field-metadata
Open

Extract indexing field metadata behind a format-neutral value object#1351
jwils wants to merge 1 commit into
joshuaw/indexer-ingestion-adaptersfrom
joshuaw/indexing-field-metadata

Conversation

@jwils

@jwils jwils commented Aug 15, 2026

Copy link
Copy Markdown
Collaborator

Why

Indexer::RecordPreparer read its per-field indexing metadata straight out of the JSON schema artifacts. It received raw hashes shaped like {"type" => "ID!", "nameInIndex" => "id"} and called eg_meta.fetch("nameInIndex") directly.

That made the JSON schema representation part of the indexer's record-preparation contract. elasticgraph-indexer is meant to stay ingestion-format-neutral, but an ingestion format with no JSON schemas (such as the planned elasticgraph-proto_ingestion runtime) could only supply metadata by synthesizing JSON-schema-shaped hashes with camelCase keys.

What

  • Add RecordPreparer::FieldMetadata, a value object with type and name_in_index.
  • RecordPreparer::TypeMetadata now exposes fields_by_name (a Hash<String, FieldMetadata>) instead of eg_meta_by_field_name.
  • RecordPreparer uses field.name_in_index and field.type instead of fetching camelCase JSON schema keys.
  • JSONIngestion::RecordPreparerFactory converts the JSON schema ElasticGraph metadata into FieldMetadata, so the JSON schema key names stay inside elasticgraph-json_ingestion.
  • Add a spec that builds a RecordPreparer directly from TypeMetadata and FieldMetadata, with no JSON schema involved. This is the path a non-JSON ingestion format will use.

Behavior is unchanged. This is a pure refactor of the contract between the indexer and the ingestion format gems.

Follow-up

This removes the shape coupling. A later PR still needs to give non-JSON formats a source for this metadata. Today name_in_index and the indexing type are recorded only in the versioned JSON schema artifacts.

Runtime metadata is not a sufficient substitute. Its graphql_fields_by_name entries record name_in_index, but they carry no field type, which RecordPreparer needs to select a scalar indexing preparer and to recurse into nested objects. The entries are also unversioned and describe the GraphQL field set, not the ingestion field set.

Verification

  • elasticgraph-indexer, elasticgraph-json_ingestion, elasticgraph-indexer_lambda, elasticgraph-warehouse_lambda, elasticgraph-lambda_support, elasticgraph-warehouse, elasticgraph-local and elasticgraph-apollo unit suites: 673 examples, 0 failures
  • script/lint, script/spellcheck, script/type_check: all green
  • bundle exec rake schema_artifacts:check: artifacts up to date

Stack

Current PR is marked with ->.

`Indexer::RecordPreparer` read its per-field metadata straight out of the
JSON schema artifacts: it received raw `{"type" => ..., "nameInIndex" =>
...}` hashes and called `eg_meta.fetch("nameInIndex")`. That made the
JSON schema representation part of the indexer's record-preparation
contract, so an ingestion format that has no JSON schemas could only
supply metadata by synthesizing JSON-schema-shaped hashes.

The metadata is now a `RecordPreparer::FieldMetadata` value object with
`type` and `name_in_index`, and `RecordPreparer::TypeMetadata` exposes
`fields_by_name` instead of `eg_meta_by_field_name`. The JSON schema key
names stay inside `elasticgraph-json_ingestion`, which converts them when
it builds its type metadata. Behavior is unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant