Add skip_record_validation_for indexer config for sampled backfill validation

[vermatron]

Add skip_record_validation_for indexer config for sampled backfill validation

Why

During large backfills of already-validated data, per-record JSON schema validation is wasted work. Every record walks the full schema (regex, enum, min/max, format, abstract-type discriminators) even though the source has already been validated upstream. Today there's no way to trade that cost for throughput.

This adds a config option that skips per-record validation for a chosen fraction of records, per GraphQL type, while keeping a sampled slice validated as a canary so schema drift still surfaces. It's a sibling of the existing skip_derived_indexing_type_updates backfill knob and follows the same shape.

Design notes:

• skip_record_validation_for maps a type name to a fraction in [0.0, 1.0]. 0.0 (or an absent key) validates everything, 1.0 skips everything, and values in between sample. The value is the fraction skipped, so 0.9 skips 90% and validates 10%.

• The skip decision is a Zlib.crc32 of the event id (type:id@vversion) bucketed into [0.0, 1.0) and compared against the rate. Same event id, same decision, so a retry never flips a record between validated and skipped, even across pods. String#hash won't do here: its seed is per-process, so two pods would disagree.

• The event envelope is always validated. Only the per-record schema walk gets sampled.

• Skipping isn't silent. Processor counts skipped records per batch and logs one RecordValidationSkipped line with per-type counts, the same way it logs ElasticGraphIndexingLatencies. Logging per record would drown a 1.0 backfill in log lines, so the count is aggregated per batch.

• Safety net for skipped abstract-type records: once validation is off, a record with a missing or unknown __typename can reach RecordPreparer. It now raises a typed RecordPreparer::UnknownTypeError, and Factory#build rescues it into a FailedEventError, so that one failure mode degrades a single event instead of the batch. The rescue is narrow (a typed error, not a bare rescue KeyError) so it can't hide schema-artifact bugs, and its message drops the offending value to avoid leaking record data. A pre-walk check was considered and dropped: it would re-implement prepare_for_index's recursion over nested fields and drift out of sync with it.

• That safety net is the only per-record isolation guarantee, and the config documentation says so explicitly rather than implying a broader one. With validation off, other malformed data that the schema walk would have caught can still raise an unhandled error and fail the whole batch: a value IndexingPreparers::Integer cannot coerce, a missing id_source path, or - after build has already returned success - the rollover index suffix and routing value computed lazily in Update#metadata during router.bulk. Since such a batch produces no partial-failure response, the queue redelivers all of its events and the malformed record fails them again on each retry until it drains to the DLQ. These paths are not new code, but this config is what makes them reachable, so the caveat ships with it.

The field defaults to {}, so nothing changes unless you set it. Additive and minor-release-safe.

What

Config:

indexer:
  skip_record_validation_for:
    Widget: 0.9      # skip validation for 90% of Widget records, validate 10%
    Component: 1.0   # skip validation for all Component records

• config.rb: new skip_record_validation_for JSON schema property (object, per-type number in [0, 1], additionalProperties: false, default {}); convert_values coerces rates to Float. The description: states which failure mode is isolated and which ones can still fail a batch.
• operation/factory.rb: new skip_validation?(type, event) helper; build skips record validation when sampled, records the skipped type on the result, and rescues UnknownTypeError. BuildResult gains validation_skipped_for.
• processor.rb: aggregate RecordValidationSkipped log per batch when any record was skipped.
• record_preparer.rb: new RecordPreparer::UnknownTypeError; the abstract-type fetch raises it with a value-free message.
• indexer.rb: wire config.skip_record_validation_for into the factory.
• RBS signatures updated for all of the above.
• elasticgraph-local config_schema.yaml: the new property added to the shared config validation schema.

13 files changed, 393 insertions(+), 18 deletions(-).

Verification

• script/run_specs (COVERAGE=1, real Elasticsearch): 5222 examples, 0 failures. Coverage holds at the project's expected level (the one non-100% file, gem_spec.rb, is pre-existing and untouched here).
• script/type_check (Steep): no type errors.
• script/lint (Standard Ruby): 890 files, no offenses.
• script/spellcheck (codespell): clean.
• bundle exec rake schema_artifacts:check: up to date (runtime config only, no artifact changes).
• bundle exec rake site:validate: 148 runs, 0 failures.

New tests:

• config_spec.rb: integer YAML rates coerce to Float (1 to 1.0), and out-of-range rates (1.5, -0.1) are rejected at config load.
• operation/factory_spec.rb: a skipped type builds operations without record validation; non-skipped types still fail on bad records; envelope validation still runs for skipped types; fractional sampling (stubbed Zlib.crc32 for both branches); retry stability; the derived-index path under skip; and the unknown-__typename case returning a FailedEventError instead of raising.
• processor_spec.rb: a batch with skips logs one RecordValidationSkipped with the right count/counts_by_type; a batch with no skips logs none.

[CLAassistant]

All committers have signed the CLA.

[marcdaniels-toast]

@vermatron I'm digging into this PR. Meanwhile can you merge the latest origin/main into it?

[marcdaniels-toast]

The description warns that skipped records won't get rejection for constraints like regex/enum/format, but it doesn't mention that a malformed record can also raise an unhandled exception that fails the whole batch, not just that one record (e.g. IndexingPreparers::Integer#prepare_for_indexing raises Errors::IndexOperationError on a non-coercible value, and that isn't rescued in Factory#build).

Given your statement that "one bad record fails on its own instead of crashing the batch" as a property of this feature, I think it's worth being explicit in the docs that this guarantee only covers the missing/unknown __typename case. Other malformed-data failures can still crash the batch. Something like:

Note: only a missing or unknown __typename on an abstract-type field is guaranteed to fail in isolation (as a structured event failure) when validation is skipped. Other malformed data that per-record validation would normally catch (e.g. a value that can't be coerced to its expected type) may raise an unhandled error that fails the entire batch, not just the offending record.

[ashitverma-toast]

On actually isolating these rather than documenting them: it needs a seam that covers both layers, plus typed errors so that a genuine schema-artifact bug (eg_meta.fetch("nameInIndex") raises a bare KeyError too) doesn't get quietly demoted to a per-event data failure. I'm happy to build that here if you want it in scope. It's also defensible to treat "skipping validation can cost you a batch" as an accepted, documented consequence of an opt-in backfill knob, which is what the description now says plainly. Which would you prefer?