Skip to content

Generalize indexing schema versions - #1284

Draft
jwils wants to merge 1 commit into
joshuaw/indexing-field-metadatafrom
joshuaw/generalize-indexing-schema-version
Draft

Generalize indexing schema versions#1284
jwils wants to merge 1 commit into
joshuaw/indexing-field-metadatafrom
joshuaw/generalize-indexing-schema-version

Conversation

@jwils

@jwils jwils commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator

Why

The indexing pipeline spoke in json_schema_version. That key is specific to one ingestion format. A proto ingestion format has no JSON schemas, so it cannot supply that value.

This PR makes the shared pipeline speak in a format-neutral schema_version. Each ingestion format maps its own versioning concept onto that key.

What

  • Add SCHEMA_VERSION_KEY = "schema_version" and use it in elasticgraph-indexer, elasticgraph-warehouse_lambda, and the shared test support.
  • JSONIngestion::IngestionAdapter restores the json_schema_version key before it validates an event against the JSON schemas, so the artifacts stay unchanged. That patch is private to the JSON gem, so elasticgraph-schema_artifacts gains no new public API and keeps no knowledge of the generic key.
  • JSONIngestion::IndexingEventDecoder maps the publisher's json_schema_version onto schema_version while it decodes JSON Lines.
  • JSONIngestion::IngestionAdapter dispatches on schema_version and selects the closest available JSON schema version.

Publishers keep sending json_schema_version. The JSON schema artifacts keep the json_schema_version const. Only the decoded in-memory event uses the new key.

Deviation from the original PR

The original version of this PR also defaulted a missing schema version to the latest available version. I did not carry that over.

Ingestion adapter dispatch now uses handles_event?, which tests for the presence of schema_version. A default would make the JSON adapter silently accept any event without a version, so malformed events would fail late instead of early. A missing schema_version still produces a validation failure.

Verification

  • elasticgraph-indexer, elasticgraph-json_ingestion, elasticgraph-schema_artifacts, and elasticgraph-warehouse_lambda unit suites: 566 examples, 0 failures
  • script/lint, script/spellcheck, script/type_check: all green
  • bundle exec rake schema_artifacts:check: artifacts up to date

Stack

Current PR is marked with ->.

@jwils
jwils force-pushed the joshuaw/generalize-indexing-schema-version branch from 690e775 to ec0cb6b Compare July 1, 2026 01:25
@jwils jwils changed the title Generalize indexing schema version Allow omitted JSON schema versions Jul 1, 2026
@jwils
jwils force-pushed the joshuaw/generalize-indexing-schema-version branch from ec0cb6b to ab31afa Compare July 1, 2026 01:51
@jwils jwils changed the title Allow omitted JSON schema versions Generalize indexing schema versions Jul 1, 2026
@jwils
jwils force-pushed the joshuaw/generalize-indexing-schema-version branch from ab31afa to 609617b Compare July 1, 2026 18:17
@jwils
jwils force-pushed the joshuaw/generalize-indexing-schema-version branch from 609617b to 22bce59 Compare August 15, 2026 23:36
@jwils
jwils changed the base branch from joshuaw/indexer-ingestion-adapters to joshuaw/indexing-field-metadata August 15, 2026 23:36
@jwils
jwils force-pushed the joshuaw/generalize-indexing-schema-version branch from 22bce59 to 09425ae Compare August 16, 2026 00:03
@jwils
jwils force-pushed the joshuaw/generalize-indexing-schema-version branch from 09425ae to f90f1aa Compare August 16, 2026 00:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant