Generalize indexing schema versions - #1284
Draft
jwils wants to merge 1 commit into
Draft
Conversation
jwils
force-pushed
the
joshuaw/generalize-indexing-schema-version
branch
from
July 1, 2026 01:25
690e775 to
ec0cb6b
Compare
jwils
force-pushed
the
joshuaw/generalize-indexing-schema-version
branch
from
July 1, 2026 01:51
ec0cb6b to
ab31afa
Compare
jwils
force-pushed
the
joshuaw/generalize-indexing-schema-version
branch
from
July 1, 2026 18:17
ab31afa to
609617b
Compare
jwils
force-pushed
the
joshuaw/generalize-indexing-schema-version
branch
from
August 15, 2026 23:36
609617b to
22bce59
Compare
jwils
changed the base branch from
joshuaw/indexer-ingestion-adapters
to
joshuaw/indexing-field-metadata
August 15, 2026 23:36
This was referenced Aug 15, 2026
jwils
force-pushed
the
joshuaw/generalize-indexing-schema-version
branch
from
August 16, 2026 00:03
22bce59 to
09425ae
Compare
jwils
force-pushed
the
joshuaw/generalize-indexing-schema-version
branch
from
August 16, 2026 00:10
09425ae to
f90f1aa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The indexing pipeline spoke in
json_schema_version. That key is specific to one ingestion format. A proto ingestion format has no JSON schemas, so it cannot supply that value.This PR makes the shared pipeline speak in a format-neutral
schema_version. Each ingestion format maps its own versioning concept onto that key.What
SCHEMA_VERSION_KEY = "schema_version"and use it inelasticgraph-indexer,elasticgraph-warehouse_lambda, and the shared test support.JSONIngestion::IngestionAdapterrestores thejson_schema_versionkey before it validates an event against the JSON schemas, so the artifacts stay unchanged. That patch is private to the JSON gem, soelasticgraph-schema_artifactsgains no new public API and keeps no knowledge of the generic key.JSONIngestion::IndexingEventDecodermaps the publisher'sjson_schema_versionontoschema_versionwhile it decodes JSON Lines.JSONIngestion::IngestionAdapterdispatches onschema_versionand selects the closest available JSON schema version.Publishers keep sending
json_schema_version. The JSON schema artifacts keep thejson_schema_versionconst. Only the decoded in-memory event uses the new key.Deviation from the original PR
The original version of this PR also defaulted a missing schema version to the latest available version. I did not carry that over.
Ingestion adapter dispatch now uses
handles_event?, which tests for the presence ofschema_version. A default would make the JSON adapter silently accept any event without a version, so malformed events would fail late instead of early. A missingschema_versionstill produces a validation failure.Verification
elasticgraph-indexer,elasticgraph-json_ingestion,elasticgraph-schema_artifacts, andelasticgraph-warehouse_lambdaunit suites: 566 examples, 0 failuresscript/lint,script/spellcheck,script/type_check: all greenbundle exec rake schema_artifacts:check: artifacts up to dateStack
Current PR is marked with
->.