Skip to content

feat(verdict): verdict policies, metric catalogs and the evidence verdict command (0.2.0) - #19

Merged
siddhant573 merged 17 commits into
LambdaTest:mainfrom
lt-zeeshan:feat/verdict-policy
Oct 26, 2026
Merged

feat(verdict): verdict policies, metric catalogs and the evidence verdict command (0.2.0)#19
siddhant573 merged 17 commits into
LambdaTest:mainfrom
lt-zeeshan:feat/verdict-policy

Conversation

@lt-zeeshan

@lt-zeeshan lt-zeeshan commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

Problem

A pack records what happened in a run, but there is no standard way to judge it against a team's quality gates (pass rate, failed steps, a producer's own metrics) and get a machine-readable verdict.

Change (0.2.0, contract stays 0.1, additive only)

  • Decisions 0047 (verdict command and policy format), 0048 (metric catalog snapshotted in the pack), 0049 (CEL for expressions), 0050 (exit codes and the verdict file next to the pack; proposed).
  • Schemas: policy, metrics-catalog and verdict tool schemas.
  • Policy: YAML gates over metrics (min, max, eq, …), with optional where filters and expr gates in CEL; sum / mean / percentile registered, with a cost limit.
  • Metric catalogs: a built-in evidence-core catalog (totals, pass rate, steps, run duration and status, failure records) that every pack supports, recomputed from the pack's own files; a producer catalog is snapshotted at meta/metrics-catalog.yaml and its metrics are referenced as metrics.<name>. validate and merge handle the snapshot (merge keeps one copy only when every source's is byte-identical).
  • lintPolicy: every policy problem at once, against a catalog.
  • verdict(pack, {policies}) / evidence verdict: one verdict document per policy, never written into the pack; writeVerdicts(docs, dir) writes <policy-id>.verdict.json. Exit codes 0 pass, 1 fail, 3 errored, 2 usage.
  • Exports: verdict, lintPolicy, judge, loadPackFacts, loadPolicy, loadPolicies, loadCatalog, lintCatalog, writeVerdicts, verdictFileName, combinedExitCode, PolicyUsageError, CORE_CATALOG, CATALOG_SNAPSHOT_PATH, EVALUATOR.
  • New dependency @marcbachmann/cel-js 8.0.0, so engines.node is now >=20.19.

Tests

  • npm test 448/448 (main: 194), tsc --noEmit clean, dist built.
  • Conformance corpus for verdicts: pass, fail, errored, unfinalized, merged, CEL.
  • The first three commits fail the design-links test on their own (the decision records link files added later); the branch as a whole passes.

Notes

  • Not published. kane-cli consumes it through verdict, lintPolicy, loadCatalog, writeVerdicts and CORE_CATALOG.
  • No checksums.txt: a pack's identity in a verdict is the sha256 of the .evidence zip (null for a directory pack).
  • Metric-to-metric references are left for later.

Part of LambdatestIncPrivate/v16#2261

🤖 Generated with Claude Code

lt-zeeshan and others added 15 commits October 21, 2026 02:21
@lt-zeeshan @claude
docs(decisions): 0047-0050 verdict policy, metric catalog, CEL, exit …
0eb44ef 
…codes

Records the verdict command and policy format (0047), the metric catalog
format with the evidence-core built-ins and the in-pack snapshot (0048), CEL
as the expression language with @marcbachmann/cel-js (0049), and the 0/1/3
exit codes with the verdict as a sidecar (0050, proposed). Amends 0012 so the
in-pack verdict stays status/totals while policy verdicts live beside the pack.

Part of LambdatestIncPrivate/v16#2261
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lt-zeeshan @claude
feat(schemas): policy, metrics-catalog and verdict tool schemas
6e73240 
Registers the three tool schemas beside the versioned tree, adds
loadToolSchema to compile them once, and adds the CEL dependency the verdict
engine uses (engines node >=20.19, see decision 0049).

Part of LambdatestIncPrivate/v16#2261
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lt-zeeshan @claude
feat(verdict): CEL expression wrapper with sum/mean/percentile and a …
9e634fe 
…cost limit

compileExpr type-checks an expression for the pack or test context and
returns its result type or a policy.expr_syntax / policy.expr_type
diagnostic; evalExpr runs it against facts whose lists count element reads,
so one evaluation is bounded. Decision 0049.

Part of LambdatestIncPrivate/v16#2261
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lt-zeeshan @claude
feat(verdict): policy and catalog loaders, built-in evidence-core cat…
25add29 
…alog

loadPolicy checks a policy's structure (YAML, unknown keys with the closest
key, schema, gate shape, duplicate ids) and keeps line numbers; loadCatalog
and lintCatalog check a catalog against its schema and rules. The built-in
catalog ships as JSON so bundlers inline it. Diagnostic gains optional hint,
line and column. Decisions 0047, 0048.

Part of LambdatestIncPrivate/v16#2261
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lt-zeeshan @claude
feat(verdict): lintPolicy - every policy problem at once
46c1cad 
checkPolicy checks gates against evidence-core and the producer catalog
(unknown or prefixed names, operator per type, threshold type and range,
where only on per-test metrics, expressions that are bools, requires);
lintPolicy runs it with loadPolicy and sorts by line. Decision 0047.

Part of LambdatestIncPrivate/v16#2261
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lt-zeeshan @claude
feat(validate,merge): metric catalog snapshot in the pack
bf965e9 
run.yaml gains the optional metrics_catalog block {id, version, sha256,
source}; validate checks meta/metrics-catalog.yaml exists, passes the catalog
lint, hashes to the recorded sha256 and carries the recorded id and version.
merge keeps the snapshot when every source carries the identical one. No
checksums.txt: a verdict identifies a pack by the sha256 of its sealed file.
Decision 0048.

Part of LambdatestIncPrivate/v16#2261
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lt-zeeshan @claude
feat(verdict): pack facts and the metric resolver
9464ddc 
loadPackFacts reads a pack once (run.yaml, top-level tests, catalog snapshot,
the files the catalogs read, the sealed file's sha256, an L0 report).
resolveMetric recomputes built-ins, follows a producer metric's verify,
checks type, unit and range, cross-checks a recompute against a stored value,
and applies the merge rule on merged packs. Decision 0048.

Part of LambdatestIncPrivate/v16#2261
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lt-zeeshan @claude
feat(verdict): judge - one policy against one pack
e866c6d 
judge checks the pack is finalized and L0-valid and the policy is clean
against the pack's catalog, then resolves and compares every gate and works
out the result, the gate lists and the exit code (0 pass, 1 fail, 3
errored). The document matches verdict.schema.json. Decisions 0047, 0050.

Part of LambdatestIncPrivate/v16#2261
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lt-zeeshan @claude
feat(verdict): verdict() library export and the evidence verdict command
1d5e713 
verdict(pack, {policies}) judges a pack against several policies and
returns one VerdictDoc each; writeVerdicts writes <policy-id>.verdict.json.
The CLI writes them next to the pack (or -o), never into it, and exits 0
pass, 1 fail, 3 errored, 2 usage; --lint checks policies with no pack,
against --catalog when given. The CLI reports the package version.
Decisions 0047, 0050.

Part of LambdatestIncPrivate/v16#2261
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lt-zeeshan @claude
test(conformance): verdict corpus - pass, fail, errored, unfinalized,…
8e34ffc 
… merged, CEL

Seven cases under fixtures/0.1/verdict, each a pack built by the real
finalize/merge, a policy and an expectation; the harness judges the
directory and a sealed zip of it and requires the same document.

Part of LambdatestIncPrivate/v16#2261
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lt-zeeshan @claude
docs: verdict contract, README and changelog; version 0.2.0
9fbb9b4 
The commands contract gains the verdict section (policy format, built-ins,
catalogs, CEL, lint codes, verdict file, exit codes, library). README and
CHANGELOG describe the command and the Node 20.19 floor. Not published.

Part of LambdatestIncPrivate/v16#2261
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lt-zeeshan @claude
docs(fixtures): describe the pass case's operators accurately
c56e6b8 
Part of LambdatestIncPrivate/v16#2261
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lt-zeeshan @claude
fix(verdict): a where that is not a bool errors instead of dropping t…
44e4778 
…ests

An aggregate where that gave a non-bool (e.g. `test.status`) silently left
every test out, so a failure-count gate could pass with zero. A catalog
where whose type is known to be non-bool is now catalog.expr at load, and a
non-bool result at evaluation errors the gate with verdict.expr_error.

Part of LambdatestIncPrivate/v16#2261
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lt-zeeshan @claude
fix(verdict): check every merged source's stored metric before folding
f916ee9 
On a merged pack, sum/min/max/mean folded each source's <label>/<name>
value after only a number check, so a source with the wrong unit or type
could be summed into a passing value. Each source is now checked like a
single stored value: type and unit (verdict.stored_type), then value type
and range.

Part of LambdatestIncPrivate/v16#2261
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lt-zeeshan @claude
fix(verdict): look metric names and pack keys up as own properties only
49b8fab 
A policy metric named constructor, toString, hasOwnProperty or __proto__
was found on Object.prototype, taken for a built-in, and crashed policy
lint. Metric, run.yaml metrics and pack-file lookups now read own keys
only, so such a name is policy.unknown_metric, and a catalog metric that
really is named constructor still resolves. getKey walks own keys only
(a verify path or merge rule key never reads an inherited name), and
the expression facts copy a __proto__ key as a plain key.

Part of LambdatestIncPrivate/v16#2261
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lt-zeeshan
lt-zeeshan marked this pull request as ready for review October 22, 2026 04:29
lt-zeeshan and others added 2 commits October 22, 2026 12:34
@lt-zeeshan @claude
feat(merge): report the seal time as output.endedAt
9127fdc 
The merge report now carries endedAt, the latest ended (else started) among
the eligible packs, with or without --finalize. A caller that merges live and
seals the pack itself stamps the same time --finalize would, without reading
the inputs again; a pack the rules skipped never counts.

Part of LambdatestIncPrivate/v16#2261
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KWVpJasfRTmthme49f5vTv
@lt-zeeshan @claude
fix(verdict): review findings - merged folds, step limit, missing val…
3b38b14 
…ues, min_producer, usage

- A merge: sum/min/max/mean metric now folds every merged_from source and
  errors (verdict.metric_missing) when one stored no value, instead of
  folding the rest. A source that was itself a merged pack contributes its
  own sources' <label>/<inner label>/<name> values, which were skipped, so a
  merge of merged packs judged one source's number (6.5 instead of 21 in the
  new test).
- Expressions get a second limit: 5,000,000 evaluation steps, counted on the
  CEL evaluator's run method. The list-read meter only saw lists from the
  facts, so nested loops over a list literal or a map result ran unbounded.
- An aggregate whose kept test has no value for its path errors the gate
  instead of leaving the test out, so a mistyped catalog path is not read as
  0. Catalogs narrow with where: has(test.<path>), as kane-cli's does.
- A metric's min_producer is enforced: a gate errors
  (verdict.producer_version) when a test was written by an older producer or
  the pack records no producer version.
- A policy that does not check clean against the pack's catalog is a usage
  error (exit 2) listing every problem of every policy; nothing is judged. A
  pack lacking the catalog a policy requires still errors that verdict.

Contract, decisions 0048-0050, README, changelog and the conformance corpus
(errored-unknown-metric becomes usage-unknown-metric) follow.

Part of LambdatestIncPrivate/v16#2261
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KWVpJasfRTmthme49f5vTv
@siddhant573
siddhant573 merged commit 4700685 into LambdaTest:main Oct 26, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@lt-zeeshan @siddhant573